Graphics and the GPU

22.0 What this chapter gives you#

1. You will be able to explain why a graphics chip is built differently from a processor, and what “throughput” buys that “speed” does not.
2. You will be able to draw a line, fill a shape and blend two transparent images by hand, using the same arithmetic the machine uses.
3. You will be able to describe how a three-dimensional shape is stored, and trace one point of it through every coordinate space to a pixel on screen.
4. You will be able to multiply a point by a 4x4 matrix and say exactly why the fourth row and column are there.
5. You will be able to name every stage of the rasterisation pipeline, in order, and say which stages you can write code for.
6. You will be able to read a short shader program and say what runs once per corner and what runs once per pixel.
7. You will be able to explain mipmaps, depth testing, shadow maps and physical material parameters without reaching for magic words.
8. You will be able to say how ray tracing differs from rasterisation, why it was impossible in real time for forty years, and what changed in 2018.
9. You will be able to read a real graphics card specification sheet line by line and predict which number will limit your program.
10. You will be able to explain why the same chip that draws a game also trains a neural network, which is the bridge into Chapters 46 to 51.

22.1 Why a separate chip#

PLAIN22.1.1 in simple words#

1. A screen at 1920 by 1080 has 2,073,600 little coloured squares called pixels.
2. At 60 pictures per second, that is about 124 million pixel values to work out every second. At 4K and 120 pictures per second it is nearly a billion.
3. Every one of those pixels is worked out almost independently of its neighbours. Nobody has to wait for anybody else.
4. A main processor is built to do one job very quickly, one step after another, with clever tricks to guess what comes next.
5. That cleverness is expensive in chip area and power, and it is wasted when the same simple sum has to run two million times.
6. So we build a second chip full of very simple workers instead of a few very clever ones.
7. A graphics chip in 2026 has thousands of these simple workers. A desktop processor has eight to twenty-four clever ones.
8. Each simple worker is slower than a clever one. Together they finish the pile of work far sooner.

PLAIN22.1.2 a picture in your head#

1. Imagine a post office that has to stamp one million envelopes today.
2. Option one: hire four brilliant clerks. Each can read handwriting, fix wrong addresses, phone customers and think around problems.
3. Each brilliant clerk stamps four envelopes a second. Four clerks give sixteen a second. The million envelopes take seventeen hours.
4. Option two: hire two thousand school students. Each can only do one thing: press the stamp where the arrow points.
5. Each student stamps one envelope a second. Two thousand students give two thousand a second. The million envelopes take about eight minutes.
6. If a student has to wait for a new box of envelopes, the supervisor simply turns to another student who already has a box. Nobody stands idle.
7. This is the whole idea. The brilliant clerks are a processor. The students are a graphics chip.

Where this comparison breaks: the students are not independent. They are lined up in rows of thirty-two, and every student in a row must press the stamp at the same instant on the same command. If half the row needs a different action, the other half must stand still and wait its turn. A processor core has no such restriction, and this is the single biggest reason some tasks are hopeless on a graphics chip even though they contain a lot of work.

PLAIN22.1.3 a worked example#

1. Suppose we need to add 1 to every number in a list of 16,384 numbers.
2. A processor core running one instruction per cycle at 5 GHz, with no vector tricks, needs 16,384 cycles, about 3.3 microseconds.
3. With vector instructions it handles 16 numbers per instruction, so 1,024 instructions, about 0.2 microseconds.
4. A graphics chip with 16,384 lanes gives every number its own lane. One pass. At 2.4 GHz that pass is under a microsecond including overheads.
5. But the work has to get to the chip first. Sending 16,384 four-byte numbers over the connection to the card takes about 65 kilobytes of transfer.
6. Over a PCIe 5.0 x16 link at roughly 55 gigabytes per second of real throughput, 65 kilobytes takes about 1.2 microseconds each way.
7. So for this tiny job, the copying costs more than the computing. The graphics chip loses.
8. Change the list to 100 million numbers and the picture inverts completely. The chip wins by a factor of tens.
9. This is the rule: a graphics chip pays a fixed toll and then charges very little per item. It only wins when there are a great many items.

PLAIN22.1.4 what is really happening inside#

1. Inside a processor core, most of the silicon is not doing arithmetic.
2. It is guessing which way a branch will go, reordering instructions, keeping large caches full, and undoing work that turned out to be wrong.
3. All of that exists to make one chain of instructions finish sooner. We call this design latency-optimised: it minimizes the waiting time of one task.
4. A graphics chip does almost none of that. It is throughput-optimised: it maximizes how many tasks finish per second, and does not care that any single one is slow.
5. When a processor core asks memory for a value and memory takes 300 cycles to answer, the core tries to find other work in the same instruction stream. Often it cannot, and it stalls.
6. When a graphics chip asks memory for a value, it simply parks that group of threads and runs a different group that already has its data.
7. Because every group’s working values sit in a huge on-chip register file, the switch costs nothing. There is no saving and restoring.
8. So a processor hides waiting with caches and guessing. A graphics chip hides waiting with sheer numbers of ready threads.
9. The threads are not free-running. They are grouped, and each group shares one instruction pointer. Thirty-two threads, one instruction, thirty-two different pieces of data.
10. When threads in one group need to take different branches, the hardware runs both paths and switches off the lanes that should not be active. This is called divergence and it wastes time.

TECHNICAL22.1.5 the engineer’s version#

1. SIMD means Single Instruction, Multiple Data. One instruction operates on a fixed-width vector register. Michael Flynn defined the category in 1966.
2. On x86, AVX-512 gives 512-bit registers, that is 16 lanes of 32-bit float per instruction. The width is visible in the instruction set.
3. SIMT means Single Instruction, Multiple Threads. NVIDIA coined the term for the G80 architecture in 2006. It is a programming model over SIMD hardware.
4. In SIMT you write scalar code for one thread. The hardware bundles threads into fixed groups and issues one instruction for the whole group.
5. NVIDIA calls a group a warp, and it has been 32 threads on every NVIDIA architecture since G80. That is an implementation detail, not a standard, but it has not changed in twenty years.
6. AMD calls a group a wavefront. GCN used 64 threads. RDNA, from 2019, defaults to 32 (wave32) and can also run wave64. Intel Xe issues SIMD8, SIMD16 or SIMD32 depending on the compiled kernel.
7. Divergence is handled by an execution mask. Both sides of a branch execute in sequence with inactive lanes predicated off. Worst case cost is the sum of both paths.
8. Since Volta in 2017, NVIDIA hardware keeps a per-thread program counter, called independent thread scheduling. Threads in a warp can reconverge more flexibly, but they still issue as a warp.
9. Latency hiding is quantified by occupancy: resident warps divided by the maximum the hardware supports. Registers and shared memory per thread limit it. NVIDIA ships an occupancy calculator with the CUDA toolkit.
10. Approximate access latencies on recent NVIDIA parts, measured by microbench marks rather than published by the vendor, and therefore approximate:

11. The register file on an NVIDIA H100 streaming multiprocessor is 256 KB. Across 132 of them that is 33 MB of registers, larger than the L2 cache of most processors. That inversion is the signature of a throughput design.
12. Tools to observe this: nvidia-smi for utilization and clocks, Nsight Compute for per-kernel occupancy and stall reasons, rocm-smi and rocprof on AMD, Xcode’s Metal debugger on Apple silicon.

WORDS22.1.6 remember these#

1. Latency — how long one job takes — the delay from issue to completion of a single operation or memory request.
2. Throughput — how much work finishes per second — sustained operations or bytes per unit time, independent of any one job’s delay.
3. SIMD — one order, many hands — one instruction applied across a fixed-width vector register file.
4. SIMT — scalar code, grouped execution — NVIDIA’s model where per-thread programs are executed in lockstep warps on SIMD hardware.
5. Warp / wavefront — a row of workers — the hardware scheduling group: 32 threads on NVIDIA, 32 or 64 on AMD.
6. Divergence — the row splits — threads in one group taking different branches, forcing serialized execution with lane masking.
7. Occupancy — how many rows are waiting to work — resident warps per streaming multiprocessor as a fraction of the architectural maximum.

22.2 Two-dimensional graphics first#

PLAIN22.2.1 in simple words#

1. Before any three-dimensional trickery there is a very simple idea: a big block of memory, one entry per pixel.
2. That block is called the framebuffer. Write a number into a slot and that pixel changes colour. That is all drawing ever is.
3. For a 1920 by 1080 screen with four bytes per pixel, the block is 1920 times 1080 times 4, which is 8,294,400 bytes, about 7.9 mebibytes.
4. Drawing a line means deciding which slots to write. There is no such thing as a diagonal line in a grid, only a staircase of squares that looks like one.
5. Filling a shape means finding, for each row, where the shape starts and stops, and writing every slot between.
6. Copying a rectangle of pixels from one place to another is called a blit. It is the workhorse of all older graphics.
7. A small movable picture, drawn over the background without disturbing it, is called a sprite.
8. Making one image show through another is blending, and it uses a fourth number per pixel telling how solid that pixel is.
9. Refusing to draw outside a rectangle is clipping, and it is what stops one window scribbling over another.

PLAIN22.2.2 a picture in your head#

1. Think of a very large sheet of squared paper, 1920 squares wide and 1080 squares tall.
2. You have coloured pencils and one rule: you may only fill in whole squares. You may never draw between them.
3. To draw a line from one corner to another you hold a ruler over the paper and fill the square nearest the ruler in each column.
4. To draw a triangle you find, in each row, the leftmost and rightmost squares inside it, and fill everything between.
5. To move a picture you rub out the squares it was on and fill in a new set. The paper does not remember anything, so you must repaint the background.
6. To make something look like coloured glass you mix the new colour with what was already in the square instead of replacing it.

Where this comparison breaks: real drawing never rubs anything out. The machine paints the whole sheet again from scratch, sixty or more times a second, into a fresh sheet while you are looking at the previous one. There are always at least two sheets, and they swap. Nothing is ever edited in place while it is visible, because you would see the edit happening.

PLAIN22.2.3 a worked example#

1. Let us draw a line from pixel (0,0) to pixel (6,4) using Bresenham’s algorithm, published by Jack Bresenham of IBM in 1965.
2. The idea: step one column at a time, keep a running error, and step down a row only when the error says we have drifted too far.
3. Set dx = 6, dy = 4. Set the decision value D = 2 times dy minus dx, so D = 8 - 6 = 2. Start at y = 0.
4. Rule at each column: plot the pixel, then if D is greater than 0, increase y by 1 and add (2dy - 2dx) to D; otherwise add 2dy to D.
5. Here is every step, with the true mathematical line for comparison:

6. Every chosen pixel is the one nearest the true line. No division, no floating point, no multiplication inside the loop. Only integer addition.
7. Here is the whole thing in C, for the case where the line goes right and down and is wider than it is tall:

void line(int x0, int y0, int x1, int y1) {
    int dx = x1 - x0, dy = y1 - y0;
    int D = 2 * dy - dx;
    int y = y0;
    for (int x = x0; x <= x1; x++) {
        plot(x, y);
        if (D > 0) { y++; D += 2 * dy - 2 * dx; }
        else       { D += 2 * dy; }
    }
}

8. Now blending. Suppose a red marker at 25 percent opacity is drawn over a pale grey background.
9. Source colour is (200, 30, 30) with alpha 0.25. Destination is (240,240,240).
10. The standard source-over formula is: result = source times alpha, plus destination times (1 minus alpha).
11. Red: 200 x 0.25 + 240 x 0.75 = 50 + 180 = 230.
12. Green: 30 x 0.25 + 240 x 0.75 = 7.5 + 180 = 187.5, which rounds to 188.
13. Blue is the same as green: 188. The final pixel is (230, 188, 188), a pale pink. Exactly what a thin red wash over grey looks like.

PLAIN22.2.4 what is really happening inside#

1. The framebuffer is ordinary memory. On a modern machine it lives in the graphics card’s own memory, not in main memory.
2. Pixels are stored row by row. The distance in bytes from one row to the next is called the pitch or stride, and it is often larger than width times bytes per pixel, because rows are padded for alignment.
3. A blit is a rectangle copy that respects both pitches. Dan Ingalls wrote the first widely copied version, BitBLT, as a microcoded routine on the Xerox Alto in 1975. The Commodore Amiga shipped a famous dedicated engine for it in
4. A sprite, on very old hardware, was not drawn into the framebuffer at all. A separate circuit overlaid it as the picture was being sent to the screen.
5. Today a sprite is simply a small textured rectangle drawn by the normal pipeline, and the word survives only as a habit.
6. Alpha is a fourth channel next to red, green and blue. 0 means fully see through, 1 (or 255) means fully solid.
7. Clipping is done by comparing each pixel’s coordinates with a rectangle before writing. Modern hardware does it with a scissor test that costs nothing.
8. Your window manager does not let applications write to the screen at all. Each window draws into its own private buffer.
9. A separate program, the compositor, then draws all those buffers onto the real screen in the right order, with the right transparency, at the right moment.
10. This is why a frozen application leaves a stale image rather than a hole, and why shadows and rounded corners on windows are cheap.

TECHNICAL22.2.5 the engineer’s version#

1. The first true framebuffer is usually credited to Richard Shoup’s SuperPaint at Xerox PARC, working in 1973, holding 640 by 486 pixels at 8 bits each in 307 kilobytes of shift-register memory that cost a fortune.
2. Bresenham’s algorithm appeared as “Algorithm for computer control of a digital plotter”, IBM Systems Journal volume 4 number 1, 1965. He wrote it in 1962.
3. The compositing algebra everyone uses comes from Thomas Porter and Tom Duff, “Compositing Digital Images”, SIGGRAPH 1984. It defines twelve operators; the one you meet daily is over.
4. Straight alpha stores colour and alpha separately. Premultiplied alpha stores colour already multiplied by alpha, which makes over cheaper and makes filtering correct:

straight:      out = src.rgb * src.a + dst.rgb * (1 - src.a)
premultiplied: out = src.rgb          + dst.rgb * (1 - src.a)
out alpha:     out.a = src.a + dst.a * (1 - src.a)

5. Filtering straight-alpha images produces dark or bright halos, because the colour of fully transparent texels leaks in. Premultiplied alpha does not have this fault. This is a convention question that bites everybody once.
6. Blending must happen in linear light, not in the sRGB values you store. Blending sRGB numbers directly is wrong and produces dark edges. This is a standard requirement, defined in IEC 61966-2-1 for sRGB, and routinely ignored.
7. The hardware blend stage is configurable but not programmable. In OpenGL you set it with glBlendFuncSeparate and glBlendEquation; in Vulkan through VkPipelineColorBlendAttachmentState.
8. Real framebuffer sizes, with a 32-bit colour buffer and a 32-bit depth buffer:

9. Compositors in the wild: Quartz Compositor on macOS since 2001, the Desktop Window Manager on Windows since Vista in 2006, Mutter and KWin on Wayland, SurfaceFlinger on Android.
10. Tools: xwd and import capture X11 windows; weston-info reports Wayland compositor capabilities; RenderDoc captures the actual draw calls of any frame on Windows, Linux and Android.

WORDS22.2.6 remember these#

1. Framebuffer — the memory that is the picture — a linear array of pixel values with a defined format and pitch, scanned out to the display.
2. Pitch — bytes per row — the stride between the start of consecutive rows, including alignment padding.
3. Blit — copy a rectangle — a two-dimensional block transfer between memory regions, historically by a dedicated engine.
4. Alpha — how solid a pixel is — a per-pixel coverage or opacity channel used by the Porter-Duff compositing operators.
5. Premultiplied alpha — colour already dimmed by alpha — storage where RGB is pre-scaled by A, required for correct filtering.
6. Clipping — do not draw outside here — restricting rasterisation to a region, implemented as a scissor rectangle test.
7. Compositor — the program that assembles the desktop — the system process that blends per-window buffers into the scanout surface.

22.3 The three-dimensional problem#

PLAIN22.3.1 in simple words#

1. A screen is flat. A world is not. So we need a way to write down a solid shape as numbers, and a recipe for flattening it.
2. A shape is stored as a list of points in space. Each point is three numbers: how far right, how far up, how far forward.
3. A point on its own is invisible. Points are joined into flat patches, and almost every patch in every game is a triangle.
4. Three points always lie on one flat surface. Four points may not. That single fact is why the whole industry uses triangles.
5. A whole object is a mesh: a list of points plus a list of which three points make each triangle.
6. Each point also carries extra facts, most importantly a normal: an arrow saying which way the surface faces at that point.
7. Without normals, lighting is impossible, because light depends entirely on the angle between the surface and the lamp.
8. To get from a stored shape to a picture we move the numbers through several different ways of measuring, one after another.
9. First relative to the object itself, then to the world, then to the camera, then to the flat screen, then to actual pixel positions.

PLAIN22.3.2 a picture in your head#

1. Think of a paper model of a house that you built on your desk.
2. When you measured the pieces you measured them from a corner of the house. The chimney is 3 centimetres up from the roof. That is the house’s own ruler.
3. Then you put the house on a model railway board. Now the chimney is 40 centimetres from the left edge of the board. That is the board’s ruler.
4. Then you kneel down and look at the board through a cardboard tube. Now what matters is how far the chimney is in front of your eye and how far off to the side. That is your ruler.
5. Then you trace what you see onto the flat end of the tube. Things far away trace smaller. That is the flattening.
6. Finally you photocopy the tracing onto a page of a particular size. That is turning it into pixels.
7. Five rulers, one after the other. The house never moved. Only the way we measured it changed.

Where this comparison breaks: the tracing step is not a passive act of looking. It is a specific piece of arithmetic that also throws away everything behind you, everything too close, and everything too far, and it records how far away each traced point was so that nearer things can hide farther ones later. A cardboard tube does none of that bookkeeping.

PLAIN22.3.3 a worked example#

1. Here is the smallest useful mesh: a square made of two triangles.
2. Four corners, listed once each. Position, then the normal arrow, then a pair of numbers for where to sample a picture (we meet those in section 22.7).

index  position (x,y,z)   normal        uv
  0    (-1, -1, 0)        (0, 0, 1)     (0, 0)
  1    ( 1, -1, 0)        (0, 0, 1)     (1, 0)
  2    ( 1,  1, 0)        (0, 0, 1)     (1, 1)
  3    (-1,  1, 0)        (0, 0, 1)     (0, 1)

triangles: (0, 1, 2) and (0, 2, 3)

3. Notice corners 0 and 2 are used by both triangles. That is the point of the index list: store each corner once, refer to it many times.
4. A naive format would store 6 corners for 2 triangles. This stores 4. On a real mesh the saving is close to a factor of three, because in a closed surface each vertex is shared by about six triangles.
5. Storage cost here: 4 vertices times (3 + 3 + 2) floats times 4 bytes = 128 bytes, plus 6 indices times 2 bytes = 12 bytes. Total 140 bytes.
6. A detailed game character is roughly 50,000 to 150,000 triangles. At about 32 bytes per vertex and roughly as many vertices as half the triangle count, a 100,000-triangle character costs on the order of 2 to 3 megabytes before textures.
7. The order of the three indices matters. (0,1,2) walked round the triangle is anticlockwise when seen from the front. That is how the hardware later works out which side you are looking at.

PLAIN22.3.4 what is really happening inside#

1. A model file on disk holds, at minimum: vertex positions, indices, and one or more sets of the extra per-vertex facts.
2. Common extras: normal, tangent (needed for surface-detail textures), one or two sets of texture coordinates, vertex colour, and for animated models a list of which bones move this vertex and by how much.
3. It also holds material references: which texture files, which shader, which numeric settings.
4. Now the five coordinate spaces, each reached by multiplying by a matrix.
5. Model space, also called object space. Coordinates measured from the object’s own origin. This is what the artist made and what the file stores.
6. World space. Multiply by the model matrix. This places, rotates and scales the object into the shared scene. Every object in the level now shares one ruler.
7. View space, also called eye or camera space. Multiply by the view matrix. This re-measures everything relative to the camera, which now sits at the origin looking down one axis.
8. Clip space. Multiply by the projection matrix. This applies perspective and defines a box: anything outside it is not visible and is thrown away.
9. Normalized device coordinates. Divide every component by the fourth component, w. This is the step that actually makes distant things smaller. The visible region becomes a tidy cube from -1 to 1.
10. Screen space. Apply the viewport transform, which stretches that tidy range onto the real pixel grid, for example 0 to 1919 across.
11. Every one of these steps except the divide is a matrix multiply, and matrices can be multiplied together beforehand, so in practice three matrices are combined into one and applied once per vertex.

TECHNICAL22.3.5 the engineer’s version#

1. Triangles win for four concrete reasons: three points are always coplanar; three points are always convex; barycentric interpolation across a triangle is well defined and cheap; and any polygon can be triangulated.
2. Quads and higher polygons are used in modelling tools for editing convenience and subdivision. They are triangulated before reaching the hardware. This is a convention, not a limit of the mathematics.
3. Index buffers are usually 16-bit (up to 65,535 vertices) or 32-bit. The post-transform vertex cache, an implementation detail typically holding 16 to 32 entries, rewards index orders that reuse recent vertices. Tools such as Tom Forsyth’s linear-speed vertex cache optimizer and meshoptimizer reorder indices for this.
4. Winding order defines facing. OpenGL defaults to counter-clockwise front faces (GL_CCW); Direct3D historically defaults to clockwise. Both are configurable. Getting it wrong makes objects vanish.
5. Vertex normals are usually the area-weighted or angle-weighted average of the adjacent face normals. Hard edges are made by splitting the vertex so the two sides carry different normals, which is why a cube needs 24 vertices, not 8.
6. Common interchange formats:

7. glTF 2.0 is deliberately close to what a graphics API wants: buffers, buffer views, accessors, and a physically based material model. It is called “the JPEG of 3D” for that reason.
8. The matrix chain, written the way most maths texts and OpenGL write it, with column vectors on the right:

 clip = P * V * M * v_model
 ndc  = clip.xyz / clip.w        (the perspective divide)
 win  = viewport(ndc)

9. Handedness and depth range are the two places every engine disagrees. OpenGL traditionally uses a right-handed view space and clip depth from -1 to 1. Direct3D, Vulkan and Metal use depth 0 to 1. Vulkan also flips the Y axis of clip space relative to OpenGL. These are specification differences, not opinions, and porting code without fixing them produces upside-down or inside-out images.
10. Reversed-Z, storing near objects at depth 1.0 and far at 0.0 with a floating-point depth buffer, drastically improves precision. It is now standard practice in serious engines and requires a 0-to-1 depth range.

WORDS22.3.6 remember these#

1. Vertex — a corner point — a record holding a position plus arbitrary attributes, consumed by the vertex shader.
2. Mesh — a shape as a list of corners and triangles — indexed geometry: a vertex buffer plus an index buffer plus a topology.
3. Normal — which way the surface faces — a unit vector perpendicular to the surface, used in all lighting terms.
4. Winding order — the direction you walk round a triangle — the vertex order that defines front and back faces for culling.
5. Model space — measured from the object — object-local coordinates as authored.
6. View space — measured from the camera — coordinates after the view matrix, with the eye at the origin.
7. Clip space — the pre-divide box — homogeneous coordinates where visibility is tested against -w <= x,y <= w.
8. Normalized device coordinates — the tidy cube — post-divide coordinates in [-1,1] for x and y, resolution independent.

22.4 The mathematics, taught gently#

PLAIN22.4.1 in simple words#

1. Three ideas carry all of three-dimensional graphics: the vector, the dot product, and the matrix. Everything else is decoration.
2. A vector is just a list of numbers. In graphics it is usually three or four. It can mean a position, or a direction, or a colour.
3. The dot product of two vectors is: multiply matching entries, add up the results. One number comes out.
4. That one number tells you how much two directions agree. If they point the same way it is large. At right angles it is exactly zero. Opposite, negative.
5. The cross product of two vectors gives a third vector at right angles to both. It is how you find which way a surface faces.
6. A matrix is a grid of numbers that describes a transformation: a rotation, a scaling, a shear, a move.
7. Multiplying a point by a matrix gives the moved point. Multiplying two matrices gives one matrix that does both jobs.
8. For three-dimensional work we use a 4x4 grid, not 3x3, and the reason is surprisingly simple: a 3x3 matrix cannot move something.
9. The same multiply, done on much bigger grids, is what a neural network does. The chip does not know or care which one you meant.

PLAIN22.4.2 a picture in your head#

1. Think of a matrix as a machine that eats an arrow and spits out a different arrow, always in a consistent way.
2. Feed it the arrow pointing one step east. It gives you back some arrow. Write that down as the first column.
3. Feed it the arrow pointing one step up. Write that answer down as the second column. Same for one step forward, third column.
4. Now you know everything the machine does, because any arrow is just so many easts plus so many ups plus so many forwards.
5. That is literally what a matrix is: the columns are where the basic direction arrows land.
6. And here is the problem. Feed such a machine the arrow of length zero, the origin. Multiply zero by anything and you get zero. The origin can never move.
7. So a 3x3 machine can turn, stretch and squash, but never shift. To allow shifting we add a fourth number to every arrow and a fourth row and column to every machine.

Where this comparison breaks: the fourth number is not a fake. It is a real coordinate in a real four-dimensional space, and the division by it at the end is the whole of perspective. Treating it as a bookkeeping trick works right up until you meet the projection matrix, at which point you need the honest story, which is in the technical block below.

PLAIN22.4.3 a worked example#

1. First the dot product. Take a surface facing straight up, normal N = (0,1,0). Take a direction to the lamp, L = (0.6, 0.8, 0).
2. Check L is length 1: 0.6 squared is 0.36, 0.8 squared is 0.64, they sum to 1.00. Good.
3. Dot product: (0 x 0.6) + (1 x 0.8) + (0 x 0) = 0.8.
4. So this surface receives 80 percent of the light it would receive if the lamp were straight overhead. That is Lambert’s cosine law in one multiplication.
5. If the lamp were below the surface, the dot product would come out negative, and we clamp it to zero. Surfaces do not receive negative light.
6. Now the full journey of one point. Take a vertex at (1, 1, 0) in model space.
7. The model matrix moves the object 2 units right and 5 units away from the camera. So world position is (3, 1, -5).
8. The camera sits at the origin looking down the negative z axis, so the view matrix is the identity and view position is also (3, 1, -5).
9. The projection matrix: 90 degree vertical field of view, 16:9 aspect, near plane 0.1, far plane 100. Its non-zero entries are

 P[0][0] = 1 / (tan(45 deg) * (16/9)) = 0.5625
 P[1][1] = 1 / tan(45 deg)            = 1.0
 P[2][2] = (far+near)/(near-far)      = -1.002002
 P[2][3] = (2*far*near)/(near-far)    = -0.2002002
 P[3][2] = -1

10. Apply it to the homogeneous point (3, 1, -5, 1): x = 0.5625 x 3 = 1.6875. y = 1.0 x 1 = 1.0. z = (-1.002002 x -5) + (-0.2002002 x 1) = 5.01001 - 0.2002 = 4.80981. w = -1 x -5 = 5.
11. Clip space point is (1.6875, 1.0, 4.80981, 5). Note w is now 5, not 1. The projection matrix stashed the distance into w.
12. Divide everything by w. Normalized device coordinates are (0.3375, 0.2, 0.961962). All three are inside -1 to 1, so the point is visible.
13. Viewport transform onto a 1920 by 1080 window, with y counted downwards: screen x = (0.3375 + 1) / 2 x 1920 = 0.66875 x 1920 = 1284. screen y = (1 - 0.2) / 2 x 1080 = 0.4 x 1080 = 432.
14. The vertex lands on pixel (1284, 432), and its stored depth is 0.981. Done.
15. Now move the same object twice as far away, to world z = -10. Redo it: w becomes 10, x stays 1.6875, so ndc x = 0.16875 and screen x = 1082.
16. The point moved from 1284 to 1082, that is 202 pixels closer to the centre at
    960. Half the distance from centre, for twice the depth. That is perspective, and it fell out of one division.

PLAIN22.4.4 what is really happening inside#

1. A 4x4 matrix has a clean structure. The top-left 3x3 block does rotation, scale and shear. The rightmost column does translation. The bottom row is usually (0,0,0,1) and does nothing.
2. When the bottom row is not (0,0,0,1), you get perspective. The projection matrix puts -1 in the bottom row so that w ends up holding the view-space distance.
3. A point is written with w = 1. A direction is written with w = 0.
4. That single difference makes translation apply to points and not to directions, which is exactly right: moving a house does not change which way north is.
5. Matrix multiplication is not commutative. Rotate then move is a different result from move then rotate. Ninety percent of beginner bugs are this.
6. Because matrices combine, an engine computes one combined matrix on the processor per object per frame, and the graphics chip applies it once per vertex.
7. For a 100,000-vertex model that is one 4x4 by 4x4 multiply on the processor and 100,000 4x4 by 4x1 multiplies on the graphics chip.
8. Each of those is 16 multiplies and 12 adds. That is 28 arithmetic operations with no branching, no dependence on neighbours, and perfectly predictable memory access.
9. That shape of work is exactly what thousands of simple lanes are for. It is also, and this is the point of the chapter, exactly the shape of the work in a neural network.

TECHNICAL22.4.5 the engineer’s version#

1. Homogeneous coordinates were introduced by August Ferdinand Möbius in 1827, long before computers, in his work on projective geometry.
2. Larry Roberts applied them to computer graphics in his 1963 MIT doctoral thesis on machine perception of solids, which is where the projection matrix as we use it comes from.
3. The geometric meaning of the dot product: a dot b equals |a||b| cos(theta). For unit vectors it is simply cos(theta).
4. The geometric meaning of the cross product: a cross b is perpendicular to both, with magnitude |a||b| sin(theta), which equals the area of the parallelogram they span. Direction follows the right-hand rule in a right-handed system.
5. Triangle face normal from vertices A, B, C is normalize(cross(B - A, C - A)). Half the magnitude of that cross product is the triangle’s area, which is why the same computation drives area-weighted normal averaging.
6. Column-major versus row-major is a storage question, not a mathematical one. OpenGL and GLSL store matrices column-major; Direct3D’s older maths libraries are row-major and write vectors on the left. The same transform then appears transposed. This is a convention clash and a permanent source of bugs.
7. The general perspective projection matrix, OpenGL convention, depth mapped to [-1, 1]:

 [ f/aspect  0   0                     0                  ]
 [ 0         f   0                     0                  ]
 [ 0         0   (far+near)/(near-far) 2*far*near/(near-far) ]
 [ 0         0  -1                     0                  ]
 where f = 1 / tan(fovy / 2)

8. Vulkan and Direct3D map depth to [0, 1], which changes the third row. Reversed -Z swaps near and far in that row and pairs with a GREATER depth test.
9. Depth precision: with a 24-bit fixed-point depth buffer, near = 0.1 and far = 1000, 90 percent of the stored range is used up within the first metre. Raising the near plane helps far more than raising the buffer’s bit depth. Reversed-Z with a 32-bit float buffer largely solves it.
10. The connection to machine learning, stated plainly. A vertex transform is a 4x4 matrix times a 4x1 vector. A fully connected neural network layer is an N x M weight matrix times an M x 1 activation vector, usually batched into an N x M times M x B matrix multiply.
11. Both are the same primitive: multiply-accumulate over contiguous data with no control flow. The hardware unit that does one does the other. Tensor cores, introduced on NVIDIA Volta in 2017, are simply matrix-multiply units with a fixed small tile size, and they are used by both graphics and machine learning code. Chapter 46 picks this up in detail.
12. Libraries: GLM for C++ (header-only, mirrors GLSL), numpy for experimenting, cglm for C. To check a matrix by hand, print it and verify that the fourth column is your translation.

WORDS22.4.6 remember these#

1. Vector — a list of numbers with a direction meaning — an element of a real vector space, here R3 or R4.
2. Dot product — how much two directions agree — the scalar product, which equals |a||b|cos(theta) and is zero for perpendicular vectors.
3. Cross product — a direction at right angles to two others — the vector product, magnitude |a||b|sin(theta), used for face normals.
4. Matrix — a grid that transforms — a linear map expressed in a chosen basis; its columns are the images of the basis vectors.
5. Homogeneous coordinates — the extra fourth number — projective coordinates where (x,y,z,w) and (kx,ky,kz,kw) name the same point, enabling translation and perspective as linear operations.
6. Perspective divide — dividing by w — the non-linear step that converts clip space to normalized device coordinates and makes distance shrink things.
7. Model-view-projection matrix — the three rulers combined — the product P * V * M, uploaded as a uniform and applied per vertex.

22.5 The rasterisation pipeline, stage by stage#

PLAIN22.5.1 in simple words#

1. Rasterisation is the answer to one question, asked once per triangle: which pixels does this triangle cover?
2. The whole machine is a production line. Corners go in one end, coloured pixels come out the other.
3. First the machine reads the corners out of memory.
4. Then it runs your program on each corner, which is where the matrices from the last section get applied.
5. Then, optionally, it can create extra corners to add detail, or make whole new shapes.
6. Then it throws away anything outside the visible box, and cuts triangles that are half in and half out.
7. Then it does the perspective divide and maps everything onto real pixel coordinates.
8. Then it works out, for each triangle, exactly which pixel centres fall inside it.
9. For each of those it can check the depth early and skip the expensive part.
10. Then it runs your second program once per surviving pixel, which decides the colour.
11. Then it blends that colour with what is already there and writes it into memory.
12. Some of those stages you write yourself. The rest are fixed circuits you can only configure.

PLAIN22.5.2 a picture in your head#

1. Think of a car factory line.
2. At the start, a robot picks parts out of bins. That is vertex fetch.
3. The first station is staffed by a worker who can do anything you train them to do. That is the vertex shader.
4. The next stations can add extra parts or duplicate the chassis. Those are tessellation and the geometry shader, and most cars skip them.
5. Then a quality gate throws out anything that will not fit in the showroom. That is clipping.
6. Then a stamping press converts the shape into a fixed grid pattern. That is rasterisation, and it is a fixed machine, not a person.
7. Then another trainable worker paints each square. That is the fragment shader.
8. Then the final fixed machine decides whether the new paint covers the old and files the result. That is depth test and blending.

Where this comparison breaks: a factory line handles one car at a time per station. This line has thousands of cars at every station simultaneously, and the stations are not physically separate. On real hardware the same shader cores run the vertex program and the fragment program, switching between them as work arrives. The line is a logical order, not a physical layout.

PLAIN22.5.3 a worked example#

1. Draw one triangle whose screen positions after the divide and viewport transform are A = (100, 100), B = (300, 100), C = (100, 300).
2. The bounding box is x from 100 to 300 and y from 100 to 300. That is 201 by 201, so 40,401 candidate pixels.
3. The triangle is half of that box, so about 20,200 pixels are actually inside.
4. Each candidate is tested with three edge functions. For edge AB the function is E(x,y) = (B.y - A.y)(x - A.x) - (B.x - A.x)(y - A.y).
5. Substituting A and B: E(x,y) = 0 x (x - 100) - 200 x (y - 100), which is -200(y - 100). It is negative for y above 100 and zero on the line.
6. A pixel is inside when all three edge functions have the same sign. Test the pixel centre at (150.5, 150.5) and it passes all three.
7. Once inside, the hardware computes barycentric weights. For the pixel (150.5, 150.5): u = 0.2525 toward B, v = 0.2525 toward C, and 0.495 toward A.
8. Those weights interpolate every attribute. If A, B and C had depths 0.5, 0.7 and 0.9, this pixel’s depth is 0.495 x 0.5 + 0.2525 x 0.7 + 0.2525 x 0.9 = 0.6515.
9. That depth is compared against what is already in the depth buffer. If the stored value is nearer, this pixel is discarded before the fragment shader ever runs.
10. Fragments are always processed in 2x2 groups, called quads, even at the triangle edge where one or two of the four are outside. The wasted ones are called helper lanes. On thin triangles this can waste half the work, which is why very high triangle counts stop paying off.

PLAIN22.5.4 what is really happening inside#

1. Here is the full order, with which parts you can program:

   [ vertex + index buffers in VRAM ]
                |
                v
        vertex fetch                    fixed function
                |
                v
        VERTEX SHADER                   you write this
                |
                v
     tessellation (optional)            2 of 3 stages yours
                |
                v
     GEOMETRY SHADER (optional)         you write this
                |
                v
        clipping                        fixed function
                |
                v
     perspective divide                 fixed function
                |
                v
     viewport transform                 fixed function
                |
                v
     triangle setup                     fixed function
                |
                v
     rasterisation                      fixed function
                |
                v
     early depth test                   fixed, configurable
                |
                v
       FRAGMENT SHADER                  you write this
                |
                v
     depth / stencil test               fixed, configurable
                |
                v
        blending                        fixed, configurable
                |
                v
   [ framebuffer in VRAM ]

2. Vertex fetch reads the index buffer, looks up each index in the vertex buffer, and assembles a struct for each vertex.
3. Vertex shader runs once per vertex. Its only required output is the clip space position. It may output anything else you want passed along.
4. Tessellation is three stages: a control shader you write, a fixed tessellator that subdivides a patch by the factors you gave, and an evaluation shader you write that positions each new vertex.
5. Geometry shader can emit zero, one or many primitives per input primitive. It is flexible and, on most hardware, slow. Modern engines avoid it.
6. Clipping removes primitives fully outside the view volume and cuts those that straddle it, producing new vertices with interpolated attributes.
7. Perspective divide divides x, y and z by w.
8. Viewport transform maps the -1 to 1 range onto the pixel rectangle and maps depth into the depth buffer’s range.
9. Triangle setup computes the edge functions and the per-attribute interpolation gradients once per triangle, so the per-pixel work is cheap.
10. Rasterisation walks tiles of the bounding box and tests pixel centres against the edge functions, emitting fragments.
11. Early depth test compares depth before the fragment shader. It is only safe when the shader does not change depth and does not discard pixels.
12. Fragment shader runs once per fragment and outputs colour, and optionally depth.
13. Blending combines the shader’s output with the existing pixel, then the result is written to the framebuffer.

TECHNICAL22.5.5 the engineer’s version#

1. The programmable stages, in Direct3D 11 and OpenGL 4 terms, are: vertex, hull (tessellation control), domain (tessellation evaluation), geometry, and pixel (fragment). Compute exists outside the graphics pipeline.
2. Everything else is fixed function. Fixed does not mean unconfigurable. The depth test comparison, blend factors, cull mode, scissor rectangle, viewport and stencil operations are all set through pipeline state.
3. Modern GPUs are tile-based or use binning even on desktop. AMD RDNA’s Draw Stream Binning Rasterizer and NVIDIA’s tiled caching, first observed on Maxwell in 2014, sort triangles into screen tiles to improve cache locality. These are implementation details and are not exposed in the APIs.
4. Rasterisation rules are specified precisely so that adjacent triangles neither double-draw nor leave gaps. Direct3D and Vulkan both define a top-left fill rule and fixed-point vertex snapping, typically 8 subpixel bits.
5. Early depth test is called Early-Z or Hi-Z. Writing to gl_FragDepth, calling discard, or using certain blend modes disables it. A single stray discard in a shader can cost a large fraction of frame time.
6. Attribute interpolation must be perspective correct: the hardware interpolates attribute/w and 1/w linearly in screen space, then divides. Doing it without the divide produces the warped textures of 1990s consoles. The original PlayStation of 1994 famously lacked perspective correction.
7. The mesh shader pipeline, introduced with NVIDIA Turing in 2018 and standardized in Direct3D 12 Ultimate in 2020 and Vulkan’s VK_EXT_mesh_shader in 2022, replaces vertex, tessellation and geometry with two programmable stages: task and mesh. It is a genuine architectural change, not a rename.
8. Fragment quads: derivatives such as dFdx and dFdy, used for texture mipmap selection, only exist because fragments are shaded in 2x2 groups. This is why the quad is a hardware invariant and not an optimization.
9. Observation tools: RenderDoc and NVIDIA Nsight Graphics let you step through a captured frame stage by stage, inspect the post-vertex-shader output, and see per-draw pixel counts. GL_ARB_pipeline_statistics_query and Direct3D’s pipeline statistics queries report primitives in, primitives clipped, and fragment shader invocations.

WORDS22.5.6 remember these#

1. Rasterisation — deciding which pixels a shape covers — converting primitives into fragments by testing sample positions against edge functions.
2. Fragment — a candidate pixel — a potential contribution to a pixel, carrying interpolated attributes, before depth and blend decide its fate.
3. Barycentric coordinates — how far toward each corner — the three weights that express a point inside a triangle as a combination of its vertices.
4. Early-Z — check depth before painting — a depth test executed before the fragment shader, disabled by depth writes or discard in the shader.
5. Fixed function — a circuit you configure, not program — a hardware stage with settable state but no user code.
6. Quad — the 2x2 shading group — the minimum granularity of fragment shading, required for screen-space derivatives.
7. Mesh shader — the new front end — a compute-like stage that emits meshlets directly, replacing vertex, tessellation and geometry stages.

22.6 Shaders#

PLAIN22.6.1 in simple words#

1. A shader is a small program that runs on the graphics chip instead of the processor.
2. The name is misleading. A shader does not only handle shading. It is just the name for any program that runs at one of the programmable stages.
3. A vertex shader runs once for each corner. Its job is to say where that corner ends up.
4. A fragment shader runs once for each candidate pixel. Its job is to say what colour that pixel should be.
5. Both are written in a small C-like language. The most common are GLSL for OpenGL and Vulkan, HLSL for Direct3D, and MSL for Apple’s Metal.
6. Values that are the same for every invocation, such as the camera matrix or the time, are called uniforms. You set them from the processor.
7. Values that the vertex shader computes and that get smoothly blended across the triangle for the fragment shader are the varying outputs.
8. Pictures the shader can look things up in are textures, and the object describing how to look them up is a sampler.
9. Thousands of copies of the same shader run at the same instant on different data. No copy can see another copy’s variables.
10. A compute shader is the same machinery without any drawing at all. You just say how many invocations you want and they run. That is what opened the chip to work that has nothing to do with pictures.

PLAIN22.6.2 a picture in your head#

1. Imagine a huge examination hall with twenty thousand desks.
2. Every candidate has the exact same question paper. That is the shader program.
3. On the front board are some facts everyone needs: today’s date, the exchange rate. Those are uniforms. Nobody may change them.
4. In each candidate’s envelope is their own individual data: a corner position, or a pixel coordinate. That is the per-invocation input.
5. There is a reference library at the side of the hall that anyone may consult. That is texture memory.
6. Candidates may not talk to each other. Each writes one answer and leaves.
7. The invigilator does not wait for candidate 1 before starting candidate 2. Everyone works at once.

Where this comparison breaks: candidates in the same row of thirty-two must turn the page at the same moment. If one candidate needs to read a different section of the paper, the whole row goes there and the others sit idle. And in a compute shader, small groups of candidates do get a shared scratch table they can all write on, which is the one exception to the no-talking rule and the reason compute shaders can do things fragment shaders cannot.

PLAIN22.6.3 a worked example#

1. Here is a complete, working vertex shader in GLSL. It transforms a position and passes a texture coordinate and a normal onward.

#version 330 core
layout(location = 0) in vec3 aPos;
layout(location = 1) in vec3 aNormal;
layout(location = 2) in vec2 aUV;

uniform mat4 uMVP;
uniform mat3 uNormalMatrix;

out vec3 vNormal;
out vec2 vUV;

void main() {
    vNormal     = normalize(uNormalMatrix * aNormal);
    vUV         = aUV;
    gl_Position = uMVP * vec4(aPos, 1.0);
}

2. Line by line: the three in variables are read from the vertex buffer. The two uniform values were set once by the processor.
3. gl_Position is the one required output: the clip space position from section 22.4.
4. vNormal and vUV are outputs that the hardware will interpolate across the triangle.
5. Here is the matching fragment shader. It looks up a texture and applies one diffuse light.

#version 330 core
in vec3 vNormal;
in vec2 vUV;

uniform sampler2D uAlbedo;
uniform vec3 uLightDir;

out vec4 fragColour;

void main() {
    vec3  base = texture(uAlbedo, vUV).rgb;
    float ndotl = max(dot(normalize(vNormal), uLightDir), 0.0);
    fragColour = vec4(base * (0.1 + 0.9 * ndotl), 1.0);
}

6. vNormal and vUV arrive already interpolated. They are not the same values the vertex shader wrote; they are a weighted blend of three vertices’ values.
7. sampler2D is the combined texture and sampler. texture() does the lookup, including filtering and mipmap selection, in one hardware instruction.
8. dot(N, L) is the Lambert term from section 22.4. The 0.1 is a crude ambient floor so that unlit sides are not pure black.
9. Now count the work. At 1920 by 1080 with every pixel covered once, this fragment shader runs 2,073,600 times per frame. At 60 frames per second that is 124 million invocations per second, of a program with one texture fetch and about ten arithmetic operations.
10. A compute shader, for comparison, invents its own grid. This one inverts an image:

#version 430
layout(local_size_x = 16, local_size_y = 16) in;
layout(rgba8, binding = 0) uniform image2D uImage;

void main() {
    ivec2 p = ivec2(gl_GlobalInvocationID.xy);
    vec4  c = imageLoad(uImage, p);
    imageStore(uImage, p, vec4(1.0 - c.rgb, c.a));
}

11. There is no triangle, no vertex, no framebuffer. You dispatch a grid of workgroups and each of the 256 invocations per group handles one pixel.

PLAIN22.6.4 what is really happening inside#

1. Your shader text is not what the chip runs. It goes through two compilations.
2. First, the source is compiled to a portable intermediate form. For Vulkan that is SPIR-V, a binary format standardized by Khronos in 2015. For Direct3D it is DXIL, based on LLVM bitcode.
3. Second, the graphics driver compiles that intermediate form into the actual machine code of the specific chip installed in your machine.
4. That second step is why the same game runs on chips whose instruction sets are completely different and undocumented.
5. Uniforms are not registers. They live in a small buffer in memory that every invocation reads. Because every invocation reads the same address, the read is broadcast, and it is close to free.
6. Varying outputs are written into on-chip storage by the vertex shader, then the triangle setup stage turns them into interpolation gradients, and the fragment shader reads interpolated values.
7. A texture fetch is not a normal memory read. Dedicated texture units handle address wrapping, format decoding, filtering between neighbouring texels, and mipmap level choice, all in hardware.
8. Because a texture fetch may take hundreds of cycles, the scheduler issues it and immediately switches to another warp. This is exactly the latency hiding from section 22.1.
9. The number of registers your shader needs directly limits how many warps can be resident. A shader with too many live variables reduces occupancy and can run slower despite doing less arithmetic.

TECHNICAL22.6.5 the engineer’s version#

1. History, briefly. Fixed-function transform and lighting arrived on the NVIDIA GeForce 256 in August 1999. Programmable vertex and pixel shaders were specified in Direct3D 8 in November 2000 and first shipped on the GeForce 3 in February 2001, written in assembly with instruction-count limits in the low tens.
2. High level languages followed: Cg and HLSL in 2002, GLSL in OpenGL 2.0 in
   2004. Unified shader hardware, where one pool of cores runs all stages, arrived with the ATI Xenos in the Xbox 360 in 2005 and NVIDIA G80 in November
3. Compute shaders were standardized in Direct3D 11 in 2009 and OpenGL 4.3 in
   2012. Metal and Vulkan had them from the start.
4. Shader stage inputs and outputs are declared with explicit locations, and the interface between stages must match or linking fails. In Vulkan this is validated at pipeline creation, not at draw time.
5. Uniform buffer objects have a guaranteed minimum size of 16 KiB in OpenGL and Vulkan; shader storage buffers are much larger and writable. Push constants in Vulkan give a small, fast path, with a guaranteed minimum of 128 bytes.
6. Interpolation qualifiers change behaviour: smooth is perspective correct and is the default, noperspective is linear in screen space, flat takes the value from the provoking vertex with no interpolation.
7. Workgroup limits are guaranteed minimums, not typical values. OpenGL 4.3 guarantees at least 1024 invocations per workgroup and 32 KiB of shared memory; Vulkan guarantees only 128 invocations and 16 KiB. Real desktop hardware offers 1024 and 32 KiB or more, but relying on that is not portable.
8. Subgroup or wave intrinsics let invocations in one warp exchange values directly through registers. subgroupAdd, subgroupBallot and friends were added in Vulkan 1.1 in 2018. They are the fastest reduction primitive available and are heavily used in machine learning kernels.
9. Tools: glslangValidator and glslc compile GLSL to SPIR-V; spirv-cross converts between shader languages; dxc compiles HLSL to DXIL and SPIR-V; NVIDIA’s nvdisasm and AMD’s Radeon GPU Analyzer show the real machine code and register counts.

WORDS22.6.6 remember these#

1. Shader — a program that runs on the graphics chip — user code executed at a programmable pipeline stage in SIMT fashion.
2. Uniform — a value that is the same everywhere — read-only per-draw constant data, uploaded from the host into a constant or uniform buffer.
3. Varying — a value blended across the triangle — a vertex shader output interpolated per fragment, usually perspective correct.
4. Sampler — the rules for reading a picture — the object holding filter mode, wrap mode, mip bias and comparison state used by a texture fetch.
5. SPIR-V — the portable half-compiled shader — the Khronos binary intermediate representation consumed by Vulkan and OpenCL, standardized in 2015.
6. Compute shader — a shader with no drawing — a general kernel dispatched over an explicit grid of workgroups with shared local memory.
7. Workgroup — a small team with a shared table — a block of invocations that share on-chip memory and can synchronize with a barrier.

22.7 Textures and surface detail#

PLAIN22.7.1 in simple words#

1. A triangle mesh gives shape. It does not give pattern, colour or fine detail.
2. Those come from textures: ordinary images wrapped onto the surface.
3. To wrap an image you need to say, for each corner of each triangle, which point of the image it corresponds to.
4. That pair of numbers is called a UV coordinate, running from 0 to 1 across the image in each direction.
5. Between corners, the machine blends UV values, so every pixel of the triangle knows where to look in the image.
6. The image pixel you land on is called a texel, to keep it separate from a screen pixel. They are almost never the same size.
7. If a texel is bigger than a pixel the image looks blocky. If it is much smaller, distant surfaces sparkle and crawl horribly.
8. The fix for the second problem is to store the same image several times, each half the size of the last. That stack is a mipmap.
9. Textures do more than colour. One can store which way each tiny bump of the surface faces, giving fine detail with no extra triangles.
10. Textures are the single biggest consumer of graphics memory in any real game.

PLAIN22.7.2 a picture in your head#

1. Think of wrapping a present with patterned paper.
2. The box is the mesh. The paper is the texture. Deciding which part of the paper touches which face of the box is UV mapping.
3. To do it properly you first cut the paper into a flat pattern that unfolds to fit the box. Artists really do this, and call it unwrapping.
4. Now step back across the room. The pattern is too fine to see and appears to shimmer. So instead you use a version of the paper printed with a coarser pattern for viewing from far away.
5. You keep a whole set: full detail, half detail, quarter detail, down to a single average-coloured square. You choose by distance.

Where this comparison breaks: paper has no thickness and no lighting behaviour. A modern surface uses four or five separate images at once for the same patch: one for colour, one for the direction of tiny bumps, one for how metallic it is, one for how rough it is, and often one for how much light gets trapped in crevices. They are all sampled with the same UV coordinates.

PLAIN22.7.3 a worked example#

1. Take a 4096 by 4096 texture. That is 16,777,216 texels.
2. Stored as four bytes per texel, uncompressed, it needs 64 mebibytes.
3. Now add the mipmap chain: 2048 squared, then 1024, then 512, and so on to 1x1. Each level is a quarter of the previous, so the whole chain adds one third. Total 85.3 mebibytes.
4. Now compress it. Here are the real costs for the same image:

5. A game character might use five such textures. Uncompressed that is 426 mebibytes for one character. In BC7 it is 107 mebibytes. Compression is not optional.
6. Now filtering. Suppose a pixel lands at UV (0.30012, 0.50008) on a 1024 by 1024 texture. That is texel position (307.32, 512.08).
7. Nearest filtering picks texel (307, 512) and ignores the fractions.
8. Bilinear filtering blends four texels: (307,512), (308,512), (307,513), (308,513), with weights (1-0.32)(1-0.08) = 0.6256, 0.32 x 0.92 = 0.2944, 0.68 x 0.08 = 0.0544, and 0.32 x 0.08 = 0.0256. Those four sum to 1.
9. Trilinear filtering does that twice, on the two nearest mipmap levels, then blends between them by the fractional level. Eight texels, one result.
10. Anisotropic filtering takes several trilinear samples along the direction the surface is stretched. At 16x it may take up to sixteen of them. It is the single best-value image quality setting in any game.

PLAIN22.7.4 what is really happening inside#

1. Mipmaps exist because of sampling theory, not because of laziness. If the texture has more detail than the pixel grid can represent, that detail folds back as false patterns. That is aliasing.
2. The hardware picks a mipmap level automatically. It compares the UV of the pixel to the right and the pixel below, using the 2x2 quad from section 22.5, and takes the base-2 logarithm of the larger change.
3. A normal map stores, in the red, green and blue channels, the x, y and z of a tiny surface normal, in a coordinate frame attached to the surface.
4. Because it is only a normal, not real geometry, the silhouette of the object never changes. Look along a bumpy wall and the edge is still perfectly flat.
5. A bump map or height map stores only height, and the normal is derived from the slope. It is older, smaller, and less exact.
6. A cube map is six square textures forming a box around a point. Look up with a direction vector instead of a UV pair. This is how skies and reflections are stored.
7. Block compression works on 4x4 blocks. Each block stores two endpoint colours and a small index per texel that says how far between them the texel lies.
8. That fixed block size is the point: any texel’s address can be computed directly, so the texture unit can decompress a single block on demand without touching the rest of the image.

TECHNICAL22.7.5 the engineer’s version#

1. Texture mapping was introduced in Edwin Catmull’s 1974 University of Utah doctoral thesis, alongside the depth buffer. Mipmapping came from Lance Williams, “Pyramidal Parametrics”, SIGGRAPH 1983. The name is from the Latin “multum in parvo”, much in little.
2. Bump mapping is James Blinn, “Simulation of Wrinkled Surfaces”, SIGGRAPH 1978. Environment mapping is Blinn and Newell, 1976.
3. Compression families by platform, all standards:

4. BC6H stores high dynamic range data and is the correct choice for environment and light probe maps. BC5 stores two channels and is the standard choice for normal maps, with z reconstructed in the shader as sqrt(1 - x^2 - y^2).
5. Anisotropic filtering degree is capped by hardware, commonly 16x on desktop. The cost is extra texture fetches, so it is bandwidth-bound, not arithmetic-bound. Measured cost in modern games is typically under 3 percent of frame time at 16x.
6. Real memory budgets, observed rather than specified. In 2026, a demanding PC title at 4K with ray tracing and the highest texture setting commonly reports 10 to 14 gigabytes of video memory allocated, of which the large majority is textures. At 1080p with medium textures the same title fits in 6 to 8 gigabytes.
7. This is why 8 gigabyte cards became controversial from 2023 onward. The argument is not about raw speed; it is about what happens when the working set does not fit and textures must stream over PCIe every frame, producing stutter and visibly blurred surfaces.
8. Virtual texturing, sparse textures and the Direct3D 12 Sampler Feedback feature, added in 2020, let an engine load only the mip levels and tiles actually sampled. Unreal Engine 5’s Nanite and Virtual Shadow Maps rely on this class of technique.
9. Tools: texconv and compressonator for offline compression; RenderDoc shows every bound texture with its exact format and mip chain; nvidia-smi and radeontop report video memory in use, though allocated is not the same as actively needed.

WORDS22.7.6 remember these#

1. Texel — a pixel of a texture — an addressable element of a texture image, distinct from a screen pixel.
2. UV coordinate — where on the picture this corner sits — normalized texture coordinates in [0,1], interpolated per fragment.
3. Mipmap — the stack of shrinking copies — a prefiltered image pyramid, each level half the linear size, selected by screen-space derivatives.
4. Bilinear filtering — blend the four nearest dots — weighted average of the 2x2 texel neighbourhood within one mip level.
5. Trilinear filtering — blend between two shrink levels too — bilinear in each of two adjacent mip levels, then linearly interpolated between them.
6. Anisotropic filtering — extra samples along the stretched direction — multiple trilinear taps along the axis of greatest UV derivative, up to the hardware cap.
7. Normal map — a picture of which way each speck faces — a texture encoding tangent-space normals, giving lighting detail without geometry.
8. Block compression — fixed-size squeezed tiles — lossy fixed-rate texture formats decoded in the texture unit, allowing random access.

22.8 Making it look real#

PLAIN22.8.1 in simple words#

1. Once you can put a coloured triangle on screen, everything else is about deciding what colour, and which triangle wins.
2. Which triangle wins is settled by the depth buffer: a second image that stores, for every pixel, how far away the nearest thing drawn so far is.
3. Draw order stops mattering. Each pixel simply keeps the nearest.
4. When two surfaces are at almost the same distance, rounding error makes the winner flicker between them. That is z-fighting.
5. You can also skip any triangle whose back is facing you, since you cannot see the inside of a solid object. That is back-face culling, and it removes about half of all triangles for free.
6. Colour comes from a lighting model: a formula combining a base ambient level, a part that depends on the angle to the lamp, and a shiny highlight.
7. Shadows are handled by rendering the scene once from the lamp’s position, storing only depth, then checking each pixel against that.
8. Modern materials are described not by made-up numbers but by physical ones: base colour, whether it is metal, and how rough it is.
9. Light that has bounced off other surfaces is called global illumination and is the hardest part of the whole subject.
10. Finally, effects applied to the finished image, such as blur and colour grading, are called post-processing.

PLAIN22.8.2 a picture in your head#

1. Think of painting a stage set with an assistant holding a tape measure.
2. Every time you are about to paint a square, the assistant tells you how far away the last thing painted on that square was.
3. If your new object is nearer, you paint and the assistant writes down the new distance. If it is further, you skip it.
4. That is the depth buffer, and it means you can paint the scenery in any order at all.
5. Now for shadows: you walk to where the lamp is, look at the set from there, and write down how far the nearest surface is in every direction.
6. Back at your easel, for each square you ask: is this point further from the lamp than the nearest thing the lamp could see in that direction? If yes, it is in shadow.

Where this comparison breaks: the tape measure has limited precision. Two surfaces a millimetre apart at a hundred metres get the same reading, and the painter flickers between them. That is z-fighting, and it is not a bug in the idea, it is a limit of storing distance in a finite number of bits.

PLAIN22.8.3 a worked example#

1. Take a surface with normal N = (0,1,0), light direction L = (0.6, 0.8, 0), and view direction V = (0, 1, 0). Shininess exponent 32.
2. Diffuse term: N dot L = 0.8. Lambert’s law, straight from section 22.4.
3. Phong specular uses the mirror-reflected light direction: R = 2(N dot L)N - L = (0, 1.6, 0) - (0.6, 0.8, 0) = (-0.6, 0.8, 0).
4. R dot V = (-0.6 x 0) + (0.8 x 1) = 0.8. Raise to the power 32: 0.8^32 = 0.00079. Almost no highlight.
5. Blinn-Phong instead uses the halfway vector between light and view: H = normalize(L + V) = normalize(0.6, 1.8, 0) = (0.3162, 0.9487, 0).
6. N dot H = 0.9487. Raise to the power 32: 0.9487^32 = 0.185. A clearly visible highlight.
7. Those two answers differ by a factor of 230. The models are not interchangeable at the same exponent.
8. To match Blinn-Phong to Phong you need roughly four times the exponent. Check: 0.9487^128 = 0.00118, which is close to Phong’s 0.00079. The rule of thumb holds.
9. Full Blinn-Phong for this pixel, with ambient 0.05, diffuse strength 0.8, specular strength 0.5: result = 0.05 + 0.8 x 0.8 + 0.5 x 0.185 = 0.05 + 0.64 + 0.0925 = 0.7825.

PLAIN22.8.4 what is really happening inside#

1. The formulas, written out plainly. Lambert diffuse: kd * max(dot(N, L), 0). Phong specular: ks * pow(max(dot(R, V), 0), shininess) with R = reflect(-L, N). Blinn-Phong specular: ks * pow(max(dot(N, H), 0), shininess) with H = normalize(L + V).
2. Blinn-Phong is cheaper, because computing H is fewer operations than reflecting a vector, and it behaves better at grazing angles where Phong produces an unnatural hard cut-off.
3. Shadow mapping in exact steps: render the scene from the light, store depth only; then when shading, transform the pixel’s world position into the light’s space, look up the stored depth, and compare.
4. Two errors follow immediately. If the comparison is exact, surfaces shadow themselves in a stripey pattern called shadow acne, fixed with a small bias. Too much bias and shadows detach from their objects, called peter-panning.
5. Physically based rendering replaces made-up shininess with measurable parameters: albedo (the base colour a surface reflects diffusely), metalness (0 for non-metal, 1 for metal, nothing in between in the standard model), and roughness (0 mirror, 1 completely scattered).
6. The reason it works is energy conservation: a surface may not reflect more light than it receives. Older models happily broke that and artists corrected it by eye, which is why old games look different under different lighting.
7. Global illumination is light that has bounced at least once. Direct lighting is one lamp to one surface to the eye. Indirect is lamp to wall to floor to eye, and it is what makes a room look like a room.
8. Ambient occlusion is a cheap stand-in: darken creases and contact points, because less bounced light reaches them.

TECHNICAL22.8.5 the engineer’s version#

1. The depth buffer, or z-buffer, is from Edwin Catmull’s 1974 thesis, with Wolfgang Strasser describing the same idea independently in 1974.
2. Z-fighting is a precision problem. With a 24-bit fixed-point buffer, near 0.1 and far 1000, the gap between representable depths is about 6 millimetres at 100 metres and about 60 centimetres at 1000 metres. Two surfaces closer together than that gap cannot be told apart.
3. Fixes, in order of preference: move the near plane out; use reversed-Z with a 32-bit float depth buffer; use a depth bias with glPolygonOffset; last resort, physically separate the surfaces.
4. Back-face culling uses the sign of the signed screen-space area of the triangle after projection. It removes about half the triangles of a closed mesh and costs nothing, so it is on by default in every engine.
5. Lighting model history: Henri Gouraud published per-vertex interpolated shading in 1971. Bui Tuong Phong published his reflection model and per-pixel normal interpolation in Communications of the ACM in June 1975, dying of leukaemia in the same year at 32. James Blinn published the halfway-vector variant at SIGGRAPH in 1977.
6. Cook and Torrance published the first physically motivated microfacet model in
   1982. The parameterization the industry actually uses comes from Brent Burley’s “Physically Based Shading at Disney”, SIGGRAPH 2012, usually called the Disney principled BRDF.
7. The standard real-time specular term today is the Cook-Torrance form D * F * G / (4 (N dot L)(N dot V)), with GGX for the normal distribution D (Walter and others, 2007), the Schlick approximation for Fresnel F (1994), and a Smith height-correlated term for G.
8. Shadow map filtering: percentage closer filtering (Reeves, Salesin and Cook,
   1987. samples several nearby depths and averages the comparison results, not the depths. Cascaded shadow maps split the view frustum into 3 or 4 ranges, each with its own map, and are the standard for outdoor sunlight.
9. Screen space ambient occlusion was introduced by Crytek in Crysis in 2007. Ground truth ambient occlusion (GTAO, 2016) is the common modern choice.
10. Post-processing effects and their real costs at 4K, order of magnitude, on a mid-range 2025 card: tone mapping under 0.1 ms, bloom 0.3 to 0.8 ms, motion blur 0.5 to 1.5 ms, depth of field 0.5 to 2 ms, screen space reflections 1 to 3 ms. In a 16.7 ms frame budget these add up quickly.

WORDS22.8.6 remember these#

1. Depth buffer — a per-pixel record of nearest distance — the z-buffer, tested and optionally written per fragment, making draw order irrelevant.
2. Z-fighting — flickering between two surfaces — depth quantization error when two surfaces fall within one representable depth step.
3. Back-face culling — skip triangles facing away — discarding primitives by the sign of their projected signed area.
4. Diffuse — brightness by angle to the lamp — the Lambertian term, proportional to max(N dot L, 0).
5. Specular — the shiny highlight — the mirror-like lobe, modelled by Phong, Blinn-Phong or a microfacet distribution such as GGX.
6. Shadow map — a depth picture taken from the lamp — a depth buffer rendered from the light, compared against per fragment to test occlusion.
7. Albedo, metalness, roughness — colour, is-it-metal, how scattered — the three core parameters of the metallic-roughness physically based material model.
8. Ambient occlusion — darkening in the creases — an estimate of how much of the surrounding hemisphere is blocked at each point.

22.9 Ray tracing#

PLAIN22.9.1 in simple words#

1. Rasterisation asks: for each triangle, which pixels does it cover?
2. Ray tracing asks the opposite: for each pixel, which triangle does it see?
3. To answer that, you shoot an imaginary straight line from the eye through that pixel and find the first thing it hits.
4. Once you have hit something, you can shoot more lines: toward the lamp to test for shadow, or in the mirror direction to find a reflection.
5. That is why ray tracing gets reflections, refraction and correct shadows almost for free, while rasterisation needs a separate trick for each.
6. The cost is brutal. Testing one line against every triangle in a scene of ten million triangles is ten million tests, for every one of two million pixels.
7. The fix is to sort the triangles into a tree of nested boxes, so that one test against a big box can eliminate a million triangles at once.
8. Even with the tree, this was far too slow for real time for about forty years.
9. In 2018 graphics chips gained circuits dedicated to walking that tree and testing lines against triangles, and real-time ray tracing became possible.
10. Games do not use it for everything. They rasterise most of the image and use rays only for shadows, reflections or bounced light. That is hybrid rendering.

PLAIN22.9.2 a picture in your head#

1. Imagine finding which shop in a city a laser pointer is aimed at.
2. The stupid method is to visit all forty thousand shops and check each one.
3. The sensible method is a nested set of boxes. Is the beam inside the city boundary? Yes. Which district? North. Which street in that district? Which building on that street? Which shop in the building?
4. Five or six questions instead of forty thousand. Each question halves or better the remaining candidates.
5. That nested set of boxes is called a bounding volume hierarchy.

Where this comparison breaks: the city does not move. A game scene does, every frame, and the hierarchy has to be rebuilt or repaired for anything that moved, by the processor or by the chip itself. That rebuild cost is a real and significant part of the frame time and is invisible in screenshots.

PLAIN22.9.3 a worked example#

1. A ray is written as P(t) = O + tD, where O is the origin, D is a unit direction, and t is distance travelled.
2. A sphere of radius r centred at C is every point where the distance to C equals r.
3. Substituting one into the other gives a quadratic in t: t^2 + 2t(D dot (O - C)) + |O - C|^2 - r^2 = 0.
4. Real numbers. Eye at the origin O = (0,0,0). Direction D = (0,0,-1). Sphere at C = (0,0,-10) with radius 2.
5. O - C = (0,0,10). D dot (O - C) = -10. So b = -20 and c = 100 - 4 = 96.
6. Discriminant = b^2 - 4c = 400 - 384 = 16. Its square root is 4.
7. t = (20 plus or minus 4) / 2, so t = 8 or t = 12.
8. The nearer hit is t = 8, at the point (0, 0, -8). Correct: the front of the sphere is at z = -10 + 2 = -8.
9. A negative discriminant means the ray misses. A discriminant of exactly zero means it grazes the surface.
10. For triangles the standard method is Moller-Trumbore, published in 1997. It solves for the barycentric coordinates and the distance in one go, using two cross products and needing no precomputed plane equation.

PLAIN22.9.4 what is really happening inside#

1. A bounding volume hierarchy is a tree. Each node holds a box that contains everything below it. Leaves hold a handful of triangles.
2. Traversal keeps a small stack. Test the ray against the two child boxes; descend into the nearer one first; if you find a hit closer than the far box, you never open the far box at all.
3. Ray-box tests are cheap: six comparisons using the slab method. Ray-triangle tests are more expensive, so the tree exists to minimize how many you do.
4. In a well-built tree, a ray touches roughly 20 to 60 nodes and 2 to 10 triangles in a scene of millions.
5. The 2018 hardware does two specific things. One unit walks the tree and does box tests. Another does triangle intersection. Both run while the shader cores do something else.
6. Path tracing is ray tracing taken to its logical end: at every hit, pick a random new direction, keep going, and average over many random paths. It converges to the true answer given enough samples.
7. Real time can afford about one to two paths per pixel. That gives a violently noisy image.
8. So the last stage is a denoiser: an algorithm, today usually a small neural network, that turns a noisy one-sample image plus depth and normal information into a clean one.
9. Without the denoiser, real-time path tracing would be useless. It is not a polish step, it is a load-bearing part of the system.

TECHNICAL22.9.5 the engineer’s version#

1. Arthur Appel described ray casting for shading solids in 1968. Turner Whitted added recursive reflection and refraction in “An Improved Illumination Model for Shaded Display”, Communications of the ACM, June 1980. His famous test image took 74 minutes on a VAX 11/780.
2. James Kajiya formalized the rendering equation and path tracing in “The Rendering Equation”, SIGGRAPH 1986. Everything since is a method of approximating that one integral.
3. Bounding volume hierarchies for ray tracing go back to Rubin and Whitted in 1980 and Kay and Kajiya in 1986. The surface area heuristic, the standard build quality metric, is from Goldsmith and Salmon, 1987.
4. Hardware acceleration arrived with NVIDIA Turing, announced 20 August 2018 and shipping 20 September 2018, in the RTX 2080 and 2080 Ti. NVIDIA quoted 10 billion rays per second for the 2080 Ti.
5. AMD added ray accelerators with RDNA 2 in November 2020; RDNA 4, in March 2025, doubled ray-triangle throughput per compute unit and added oriented bounding boxes. Intel shipped ray tracing units with Arc Alchemist in 2022.
6. API support is a standard: DirectX Raytracing (DXR) announced March 2018, Vulkan Ray Tracing extensions ratified November 2020, and Metal ray tracing from 2020. All expose a two-level acceleration structure: a bottom level per mesh, a top level of instances.
7. The shader types DXR defines are ray generation, intersection, any-hit, closest-hit and miss. Traversal itself is fixed function and deliberately not programmable, which is what allows it to be a dedicated circuit.
8. Denoising: the practical breakthrough was Chaitanya and others, “Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder”, SIGGRAPH 2017. NVIDIA’s Ray Reconstruction, shipped in DLSS 3.5 in September 2023, replaces the hand-written denoiser with a trained network.
9. Honest performance picture, as of 2026. Full path tracing at 4K on the fastest consumer card of the day still typically renders internally at 1080p or lower and relies on both denoising and upscaling to reach the output resolution. Native-resolution real-time path tracing is not a solved problem. Vendors showing “4K path traced” numbers are almost always showing an upscaled image, and independent testers routinely point this out.

WORDS22.9.6 remember these#

1. Ray casting — shoot a line, find what it hits — solving for the nearest surface intersection along a parameterized ray.
2. Bounding volume hierarchy — nested boxes for fast searching — a tree of axis-aligned boxes over primitives, traversed to prune intersection tests.
3. Path tracing — follow random bounces and average — Monte Carlo integration of the rendering equation.
4. Hybrid rendering — rasterise most, trace some — using rasterisation for primary visibility and rays only for selected effects.
5. RT core — the circuit that walks the tree — fixed-function hardware for ray-box and ray-triangle intersection, introduced in 2018.
6. Denoiser — clean up a noisy estimate — a spatial and temporal filter, now usually a neural network, reconstructing an image from few samples.

22.10 Upscaling and frame generation#

PLAIN22.10.1 in simple words#

1. Rendering costs roughly scale with pixel count. Going from 1080p to 4K is four times the pixels and roughly four times the cost.
2. So there is an obvious cheat: render at a lower resolution and enlarge the result cleverly.
3. Naive enlargement looks soft and blurry, so for a long time nobody did it.
4. The clever version uses information from previous frames. Slightly shift the camera by a fraction of a pixel each frame, and over several frames you have sampled the scene far more finely than one frame allows.
5. That trick, on its own, is called temporal anti-aliasing. It reuses history to remove jagged edges.
6. Upscalers use the same history but also target a higher output resolution, and the modern ones use a trained neural network to decide how to combine old and new samples.
7. There is a second, more controversial idea: instead of rendering more frames, invent them. Look at two real frames and manufacture the picture in between.
8. That makes the counter say a bigger number and makes motion look smoother, but it cannot make the game respond faster. It usually makes it respond slower.
9. Neither trick is free. Both introduce specific, visible artefacts.

PLAIN22.10.2 a picture in your head#

1. Think of photographing a page of small print in poor light.
2. One photo is grainy and unreadable. But take eight photos, each shifted by a fraction of a millimetre, and stack them, and the text becomes sharp.
3. That is temporal accumulation. You bought detail with time instead of with sensor resolution.
4. Now suppose the page is being pulled away while you shoot. To stack the photos you must first work out how far the page moved between each one, and slide each photo back into place.
5. That correction is called reprojection, and the arrows telling you how far things moved are motion vectors.
6. When a corner of the page was hidden behind your thumb in the earlier photos, there is no history for that region, and the stack has nothing to add.

Where this comparison breaks: a page moves rigidly. A game scene has objects moving in front of each other, transparent surfaces with no motion vectors at all, shadows that move differently from the objects casting them, and particles. Each of these breaks the correction in its own way, which is why upscaler artefacts cluster around fast movement, thin objects, and things seen through glass.

PLAIN22.10.3 a worked example#

1. Output resolution 3840 by 2160, which is 8,294,400 pixels.
2. The standard quality presets and their internal render sizes:

3. Performance mode renders 2,073,600 pixels instead of 8,294,400. That is 25 percent of the work for the parts of the frame that scale with resolution.
4. In practice the measured speed-up is smaller, typically 1.6x to 2.2x rather than 4x, because vertex work, shadow map rendering and the upscaler itself do not shrink.
5. Now frame generation, and the latency arithmetic that people get wrong.
6. Suppose the game natively runs at 60 frames per second, one frame every 16.7 milliseconds.
7. To insert a frame between real frames N and N+1, the system must already have rendered N+1. So it holds N+1 back, shows the invented frame first, then shows N+1.
8. The counter now reads 120 frames per second. But the newest real image reaches your eye about one frame later than it would have, plus the time to generate the fake one.
9. Measured end-to-end click-to-photon latency typically rises by 8 to 15 milliseconds with single frame generation, before any latency-reduction feature claws some back.
10. So the honest statement is: frame generation improves smoothness, not responsiveness, and slightly harms responsiveness. It is at its best when the base frame rate is already comfortable, and at its worst below about 45 frames per second where it is most tempting.

PLAIN22.10.4 what is really happening inside#

1. The renderer jitters the projection matrix by a sub-pixel offset each frame, following a low-discrepancy sequence such as Halton.
2. It also outputs a motion vector for every pixel: where that surface point was in the previous frame, in screen coordinates.
3. The upscaler takes this frame’s low-resolution colour, its depth, its motion vectors, and the previous output frame.
4. It warps the previous output using the motion vectors, then decides per pixel how much of that warped history to trust.
5. The old approach used hand-written rules: colour clamping to a neighbourhood box, rejecting history that looks too different. It causes ghosting when it trusts too much and flicker when it trusts too little.
6. The new approach feeds all of that into a small neural network trained on matched pairs of low-resolution input and high-quality reference images.
7. Frame generation additionally computes an optical flow field between two real frames, warps both toward the midpoint, and fills disocclusions with a network.
8. The generated frame is never used as history for the next real frame. It is displayed and discarded.

TECHNICAL22.10.5 the engineer’s version#

1. Timeline, all verifiable release dates. DLSS 1.0, February 2019, per-game trained, widely judged poor. DLSS 2.0, March 2020, generic temporal network, the version that made the idea work. DLSS 3, October 2022, adds Frame Generation, RTX 40 only. DLSS 3.5, September 2023, adds Ray Reconstruction. DLSS 4, January 2025, replaces the convolutional network with a transformer model and adds Multi Frame Generation on RTX 50.
2. As of August 2026 NVIDIA’s current release is DLSS 4.5, with a second generation transformer model and a dynamic multi-frame multiplier advertised up to 6x, meaning five generated frames per rendered frame.
3. AMD FidelityFX Super Resolution: FSR 1.0, June 2021, spatial only. FSR 2.0, May 2022, temporal, open source, runs on competitors’ hardware. FSR 3, September 2023, adds frame generation. FSR 4, March 2025, machine learning based and limited to RDNA 4.
4. Intel XeSS launched August 2022 with two code paths: XMX matrix instructions on Intel Arc, and a DP4a integer path elsewhere that is measurably weaker. XeSS 2, December 2024, added frame generation and a low-latency mode.
5. Latency reduction features are separate technologies: NVIDIA Reflex (2020), AMD Anti-Lag, Intel Xe Low Latency. They shorten the render queue. They do not remove the frame-holding cost of frame generation.
6. Honest quality comparison, as of 2026. Independent testing outlets including Digital Foundry and Hardware Unboxed consistently report the transformer-based DLSS as the most stable and detailed at a given preset, FSR 4 as close behind on supported hardware and a large jump over FSR 3, and XeSS as good on Intel hardware and weaker on its fallback path. This is expert consensus from controlled comparison, not a measured standard, and reasonable people disagree about specific titles.
7. The known artefact list, which applies to all of them in some degree: ghosting trails behind fast movement; smearing where an object uncovers background; shimmer on thin geometry such as fences and wires; loss of fine texture detail at aggressive presets; and, for frame generation, warped heads-up-display elements and text in the generated frames.
8. The cost of running the upscaler itself is real: roughly 1 to 2 milliseconds per frame at 4K output on recent hardware. Below about 40 frames per second this overhead eats a noticeable share of the gain.
9. There is a genuine industry disagreement worth stating. One camp argues these techniques are legitimate reconstruction, no different in kind from mipmaps or texture compression. The other argues they let publishers ship games that cannot run acceptably at native resolution, and that quoting frame rates with generated frames included is misleading. Both positions are held by serious people.

WORDS22.10.6 remember these#

1. Temporal anti-aliasing — reuse earlier frames to smooth edges — accumulation of sub-pixel jittered samples across frames with motion-vector reprojection.
2. Motion vector — where this pixel was last frame — a per-pixel screen-space displacement produced by the renderer for reprojection.
3. Reprojection — slide the old frame into place — warping history buffers by motion vectors before reuse.
4. Ghosting — trails behind moving things — an artefact of trusting stale history whose motion vectors were wrong or missing.
5. Disocclusion — newly revealed area with no history — a region uncovered this frame, which temporal methods must invent.
6. Frame generation — invent a picture between two real ones — interpolating an intermediate frame from optical flow, raising frame counters and latency together.

22.11 Inside a real graphics chip#

PLAIN22.11.1 in simple words#

1. A graphics chip is not one big pool of lanes. It is many small identical blocks, each a small computer in its own right.
2. NVIDIA calls a block a streaming multiprocessor. AMD calls it a compute unit. Intel calls it an Xe core. They are the same idea.
3. Each block has its own instruction scheduler, its own arithmetic lanes, its own registers, its own small fast memory, and its own texture unit.
4. The marketing number, “21,760 CUDA cores”, is the total count of arithmetic lanes, not the number of independent computers.
5. The real count of independent computers on that chip is 170.
6. Alongside the general lanes sit two kinds of special circuit: matrix units for machine learning, and ray tracing units for the tree walking of section 22.9.
7. All the blocks share a large cache and a memory controller that talks to the card’s own memory chips.
8. That memory is the real bottleneck of almost every workload, and its speed is the number you should look at first.

PLAIN22.11.2 a picture in your head#

1. Think of a large open-plan office split into 170 identical pods.
2. Each pod has 128 desks, one supervisor per 32 desks, a shared whiteboard, and a shelf of reference books.
3. The whiteboard is shared memory: fast, tiny, and only visible inside the pod.
4. Down the corridor is the building library, shared by all pods. That is the L2 cache.
5. Outside the building is the city archive. That is the card’s memory. Fetching from there takes a long walk, so you send someone and get on with other work.

Where this comparison breaks: the desks in a pod are not independent workers. Groups of 32 share one supervisor who reads out one instruction at a time. The pod is more like a rowing eight than an office.

PLAIN22.11.3 a worked example#

1. Here is a real specification sheet, the NVIDIA GeForce RTX 5090, launched 30 January 2025 at 1,999 US dollars, decoded line by line.

2. Now the arithmetic that the sheet does not show you.
3. 21,760 divided by 170 gives 128 FP32 lanes per block. 680 divided by 170 gives 4 matrix units per block.
4. Peak FP32 rate = lanes x 2 (a multiply-add counts as two) x clock = 21,760 x 2 x 2.41 GHz = 104.8 TFLOPS. This matches the published figure exactly, which tells you it is a paper number from a formula, not a measurement.
5. Bandwidth = bus width x per-pin data rate / 8 = 512 x 28 Gbps / 8 = 1,792 GB/s. The 28 gigabits per second per pin is the GDDR7 speed grade used.
6. A 512-bit bus with 32-bit memory chips means 16 chips. 32 GB across 16 chips is 2 GB each. The board layout follows directly from the bus width.
7. Now the ratio that matters most: 104.8 TFLOPS against 1,792 GB/s is about 58 floating-point operations available for every byte you can read.
8. So any calculation doing fewer than 58 operations per byte of data is limited by memory, not by arithmetic. Most real ones are. This ratio is the whole of the roofline model in one line.
9. Power: 575 watts over 750 square millimetres is about 0.77 watts per square millimetre, which is why these cards carry three-slot coolers and vapour chambers.

PLAIN22.11.4 what is really happening inside#

1. Inside one block, the lanes are split into four partitions of 32. Each partition has one warp scheduler that issues one instruction per cycle to its own group.
2. The register file is enormous by processor standards: 256 KB per block on recent NVIDIA parts, so more register storage than L1 cache.
3. That is deliberate. Registers are what let dozens of thread groups stay resident so the scheduler always has someone ready to run.
4. Shared memory is a scratchpad the programmer controls, not a cache. On recent NVIDIA consumer parts it shares a 128 KB pool with L1, split as you choose.
5. Tensor cores do not do one multiply. They do a whole small matrix multiply per instruction, for example a 16x8 by 8x16 tile, accumulating into a result.
6. RT cores do box and triangle intersection while the general lanes are free to do other work.
7. The memory controller is split into many channels. Each GDDR chip has its own path, and requests from many blocks are coalesced and reordered to keep every channel busy.
8. The PCIe link is not for the drawing. It is for uploading geometry and textures, sending commands, and reading results back.

TECHNICAL22.11.5 the engineer’s version#

1. Memory technologies and real bandwidth, all from published specifications:

2. JEDEC published the GDDR7 standard, JESD239, in March 2024. It is the first JEDEC DRAM standard to use PAM3 signalling, three voltage levels carrying 1.5 bits per symbol, and it specifies up to 32 Gb/s per pin.
3. JEDEC published HBM4, JESD270-4, in April 2025, with revision 4A in December
   2025. It doubles the per-stack interface to 2048 bits, runs up to 8 Gb/s per pin, reaches up to 2 TB/s per stack, and allows up to 64 GB per stack with 16-high 32 Gb dies.
4. HBM is stacked vertically on an interposer beside the die, which is why it is confined to expensive datacentre parts. GDDR is separate packages on the board, which is why it is on everything else.
5. Naming honesty. A “CUDA core” is one FP32 lane inside a SIMD datapath with no independent program counter. AMD’s “stream processor” is the same thing. Neither is a core in the sense used for a processor. This is marketing vocabulary that stuck.
6. TFLOPS honesty. AMD publishes 48.7 TFLOPS FP32 for the RX 9070 XT, which has 4,096 stream processors at up to 2.97 GHz. The plain formula gives 24.3. The published figure counts dual-issue FP32, which only applies when the compiler can pair instructions. NVIDIA’s 104.8 TFLOPS for the RTX 5090 uses the plain formula. Comparing the two numbers directly is not valid.
7. PCIe 5.0 x16 has a raw signalling rate of 32 GT/s per lane, giving a theoretical 63 GB/s each way after 128b/130b encoding, and roughly 50 to 55 GB/s achievable in practice.
8. Power delivery on high-end 2025 cards uses the 12V-2x6 connector, the revised form of 12VHPWR, rated at 600 watts. The original 12VHPWR connector had well-documented melting failures on RTX 4090 cards in 2022, traced to incomplete insertion and uneven current sharing between pins.
9. Tools: nvidia-smi -q dumps clocks, power, temperature and memory; nvtop and radeontop give live views; Nsight Compute reports achieved occupancy, memory throughput as a percentage of peak, and the exact stall reasons per kernel.

WORDS22.11.6 remember these#

1. Streaming multiprocessor — one pod of lanes — NVIDIA’s independent scheduling and execution block; AMD’s compute unit, Intel’s Xe core.
2. CUDA core — one arithmetic lane — an FP32 lane within a SIMD datapath, not an independently scheduled core.
3. Tensor core — the matrix multiplier — a fixed-tile matrix multiply-accumulate unit, first shipped on NVIDIA Volta in 2017.
4. Shared memory — the pod’s whiteboard — programmer-managed on-chip scratchpad shared by a workgroup.
5. GDDR — fast memory chips on the board — graphics DDR DRAM; GDDR7 uses PAM3 and reaches 32 Gb/s per pin under JESD239.
6. HBM — memory stacked next to the die — high bandwidth memory on an interposer; HBM4 under JESD270-4 reaches 2 TB/s per stack.
7. Roofline — the arithmetic-per-byte break-even — the ratio of peak compute to peak bandwidth that decides whether a kernel is compute or memory bound.

22.12 The software stack#

PLAIN22.12.1 in simple words#

1. Your program never talks to the graphics chip directly. It talks to a library with a defined set of functions, called a graphics API.
2. Behind that library sits the driver, written by the chip maker, which turns those calls into the chip’s own commands.
3. Old APIs let you say “draw this” one thing at a time, and the driver did an enormous amount of guessing and bookkeeping.
4. New APIs make you state everything up front, in explicit objects, and then record long lists of commands yourself.
5. That is more work for the programmer and much less guessing for the driver, which means less overhead and more predictable timing.
6. Commands are not executed as you write them. They are recorded into a buffer, submitted in a batch, and executed later by the chip.
7. Because the chip runs behind the processor, at any moment one of them is waiting for the other. Which one is waiting decides how you make the program faster.
8. Shaders must be turned into the chip’s real machine code at some point. If that happens the first time you enter a new area, the game stutters.

PLAIN22.12.2 a picture in your head#

1. Think of ordering from a kitchen through a waiter.
2. The old API is a waiter who takes one instruction at a time, remembers the whole table’s preferences, and silently fixes contradictions.
3. That waiter is convenient and unpredictable. You never know when they will disappear to sort something out.
4. The new API makes you write the entire order on a card, in full, and hand it over. The waiter only carries cards.
5. It is more effort. But nothing surprising happens, and several people can write cards at the same time.

Where this comparison breaks: the kitchen is always about one full order behind. By the time your card is being cooked you are already writing the next one. If you ever ask “is my food ready yet” and wait for the answer, both of you stop. That is what a synchronization stall costs, and it is the single most common performance mistake in graphics programming.

PLAIN22.12.3 a worked example#

1. A frame is CPU-bound or GPU-bound. Here is how to tell, with numbers.
2. Measure two things: how long the processor takes to prepare the frame, and how long the chip takes to draw it.

 case A   CPU 12.0 ms  |============|
          GPU  6.0 ms  |======|              -> CPU-bound, 83 fps
 case B   CPU  4.0 ms  |====|
          GPU 15.0 ms  |===============|     -> GPU-bound, 67 fps

3. In case A, buying a faster graphics card changes nothing. The chip already finishes early and waits.
4. In case A, lowering the resolution also changes nothing, because resolution costs the chip, not the processor.
5. What helps in case A: fewer draw calls, fewer objects, better batching, a lower-overhead API.
6. In case B, resolution and shader quality are exactly the right dials.
7. A quick field test: drop the resolution by half. If the frame rate barely moves, you are CPU-bound.

PLAIN22.12.4 what is really happening inside#

1. The API validates your calls, tracks state, and hands work to the driver.
2. The driver contains a shader compiler that turns the portable intermediate form into the specific chip’s instructions, and a command translator that builds the hardware command packets.
3. Those packets go into a ring buffer in memory. The chip reads the ring buffer independently through direct memory access.
4. Work is submitted to queues. A modern chip has separate queues for graphics, for compute, and for copying, and they can run at the same time.
5. Because they run at the same time, you need explicit fences and barriers to say “this must finish before that starts”. Getting those wrong causes corruption that only appears on some hardware.
6. Shader compilation is the classic stutter cause. A game may contain tens of thousands of shader variants. Compiling one takes a few milliseconds to a few hundred. Doing it at the moment an effect first appears drops a frame.
7. The fix is to compile everything ahead of time and cache it, which is why many games now show a “compiling shaders” screen on first run or after a driver update.
8. A driver update invalidates the cache, because the generated machine code may change. That is why the stutter returns after updating a driver.

TECHNICAL22.12.5 the engineer’s version#

1. API history, with dates. OpenGL 1.0 from Silicon Graphics in 1992. Direct3D in
   1996. OpenGL 2.0 with GLSL in 2004. Direct3D 11 in 2009. AMD Mantle in 2013, which was donated and became the basis of Vulkan. Apple Metal in June 2014. Direct3D 12 with Windows 10 in July 2015. Vulkan 1.0 on 16 February 2016. Vulkan 1.4 in December 2024. WebGPU shipped in Chrome 113 in May 2023 and its W3C specification was still at Candidate Recommendation stage in January 2026.
2. What changed with the low-level APIs, precisely: pipeline state is baked into immutable objects instead of being set piecemeal; resource lifetime and residency become the application’s job; memory is allocated and sub-allocated by the application; synchronization is explicit; and command buffers can be recorded from many threads at once.
3. The measured payoff is draw call overhead. A Direct3D 11 draw call costs roughly 1 to 10 microseconds of processor time depending on how much state changed with it; Direct3D 12 and Vulkan reduce this to well under 1 microsecond and let the cost be spread across threads.
4. The cost is code volume. A minimal triangle in OpenGL is about 100 lines. In Vulkan it is 800 to 1,500. This is a real and widely acknowledged trade, and it is why most people use an engine rather than the API directly.
5. Pipeline state objects are the mechanism behind first-run stutter. Direct3D 12 added Pipeline State Object libraries and Vulkan added pipeline caches; both Vulkan’s VK_EXT_graphics_pipeline_library (2022) and Direct3D 12’s state object work aim to reduce combinatorial explosion. Valve’s Steam Deck distributes precompiled pipeline caches to users precisely for this reason.
6. Queue families in Vulkan are advertised by the device. A typical discrete card exposes one graphics-and-compute queue family, one or more compute-only, and one transfer-only queue using the copy engines.
7. Diagnosis tools: RenderDoc for frame capture and per-draw timing; PIX on Windows; Nsight Systems and Radeon GPU Profiler for timeline views showing exactly where the processor and the chip wait for each other; VK_LAYER_KHRONOS_validation for correctness.

WORDS22.12.6 remember these#

1. Graphics API — the agreed set of drawing functions — the specified interface between application and driver, such as Vulkan or Direct3D 12.
2. Driver — the translator to real hardware — vendor software containing the shader compiler, memory manager and command packet builder.
3. Command buffer — a written list of orders — a recorded sequence of GPU commands, submitted to a queue for later execution.
4. Queue — a lane of work into the chip — an independent submission channel; graphics, compute and transfer queues can run concurrently.
5. Pipeline state object — the whole draw configuration frozen — an immutable object bundling shaders, blend, depth and raster state, compiled once.
6. CPU-bound — the processor is the limit — frame time dominated by host-side preparation, unaffected by resolution.
7. GPU-bound — the chip is the limit — frame time dominated by device execution, responsive to resolution and shader cost.

22.13 Graphics chips for everything else#

PLAIN22.13.1 in simple words#

1. For thirty years these chips did one thing. Then people noticed the shape of the work, not the subject of the work, was what suited them.
2. Anything made of millions of independent, identical, arithmetic-heavy steps fits. Weather models, molecular simulation, oil exploration, cryptography and, above all, neural networks.
3. Before 2007 you had to disguise your problem as a drawing problem. People really did pack scientific data into textures and read the answer out of a picture.
4. In 2007 NVIDIA released CUDA, which let you write ordinary C-like code for the chip with no pretence of drawing. That is the moment the field opened.
5. Speed is quoted in FLOPS, floating-point operations per second. A TFLOP is a trillion of them per second.
6. But the number depends entirely on how precise the numbers are. Halve the precision and the same silicon roughly doubles the rate.
7. And the FLOPS number is usually not what limits you. Memory bandwidth is. Most real work spends its life waiting for data.
8. The chips sold for datacentres and the chips sold for games are the same family with very different memory, very different precision support, and very different prices.

PLAIN22.13.2 a picture in your head#

1. Think of a factory that can assemble 100,000 items per hour.
2. The loading bay can only bring in enough parts for 20,000 items per hour.
3. The factory’s rated capacity is a fiction. You will build 20,000 items per hour, and buying faster assembly robots will change nothing.
4. The loading bay is memory bandwidth. The robots are FLOPS.
5. The only real fixes are a bigger loading bay, or redesigning the product so each part gets used in more items before it is thrown away.

Where this comparison breaks: you can genuinely redesign the work. Reusing data that is already on the chip, by tiling a matrix multiply so each loaded block feeds many operations, is the single most important optimization in the field, and it is why a well-written matrix multiply reaches near peak while a naive one reaches a few percent.

PLAIN22.13.3 a worked example#

1. Real throughput of one NVIDIA H100 SXM, at each precision, dense (that is, not counting the sparsity doubling that vendors like to quote):

2. From FP64 to FP8 is a factor of about 30 on the same chip. Nothing changed except how many bits each number uses.
3. This is why the AI field moved to lower precision so aggressively. It is not a small saving.
4. Now the bandwidth check. The H100 SXM has 3,350 GB/s. At FP16 it can do 989 trillion operations per second.
5. That is 989e12 / 3350e9 = about 295 operations for every byte read. Any calculation with a lower ratio than that is bandwidth-limited.
6. Generating one token from a 70-billion-parameter model in 16-bit precision requires reading roughly 140 gigabytes of weights, and does about two operations per weight. That is 1 operation per byte. Hopelessly bandwidth-limited.
7. At 3,350 GB/s, reading 140 GB takes about 42 milliseconds. That is the physical floor for one token on one such chip, no matter how fast its arithmetic is.
8. A gaming card against a datacentre accelerator, same generation family:

PLAIN22.13.4 what is really happening inside#

1. A general-purpose program for a graphics chip is called a kernel. You launch a grid of thread blocks; each block runs on one streaming multiprocessor.
2. The programming model is deliberately the same as a compute shader. Same hardware, different front door.
3. Data must be copied to the card’s memory first, unless the system has a shared memory design. That copy crosses PCIe and is often the slowest part.
4. Matrix multiply is the workhorse. The efficient version splits the matrices into tiles that fit in shared memory, loads each tile once, and reuses it across many multiply-accumulate operations.
5. Tensor cores make this even more extreme: one instruction consumes a whole tile and produces a whole tile of results.
6. Lower precision helps twice: the arithmetic units go faster, and the same bytes of bandwidth carry twice as many numbers.
7. That second effect is often the larger one, which is exactly the point about bandwidth being the true limit.

TECHNICAL22.13.5 the engineer’s version#

1. Timeline. NVIDIA announced CUDA with the G80 architecture in November 2006; the public toolkit shipped in 2007. Khronos released OpenCL 1.0 in December 2008, initiated by Apple. Both are still maintained; CUDA has by far the larger ecosystem, which is a business fact as much as a technical one.
2. The scientific computing shift came first. GPU-accelerated entries appeared in the TOP500 supercomputer list from 2008 onward, and Tianhe-1A took the number one spot in November 2010 using NVIDIA Tesla parts.
3. The machine learning shift is usually dated to AlexNet, Krizhevsky, Sutskever and Hinton, which won the ImageNet competition in 2012 trained on two GeForce GTX 580 cards with 3 GB each. That result is established fact and is the hinge of the modern field.
4. Precision formats in use, all standards or de facto standards: IEEE 754 binary 64 and binary 32; IEEE 754 binary16; bfloat16, from Google Brain, which keeps FP32’s exponent range with 8 fewer mantissa bits; TF32, NVIDIA’s 19-bit internal tensor format from 2020; FP8 in the E4M3 and E5M2 variants, standardized by an Arm, Intel and NVIDIA joint proposal in 2022; and FP4 from Blackwell in 2024.
5. Marketing claim, flagged as such: every headline TOPS figure from every vendor in this space is quoted at the lowest supported precision, with structured sparsity assumed. NVIDIA’s 3,352 AI TOPS for the RTX 5090 is FP4 with sparsity. Dividing by four gives a fairer comparison against an older FP16 figure.
6. Active research, flagged as such: whether FP4 and lower training is generally viable rather than viable for specific layers is not settled as of 2026. Inference at 4 bits with careful quantization is established practice; 4-bit training is not.
7. The roofline model, from Williams, Waterman and Patterson in 2009, is the standard tool: plot achievable performance against arithmetic intensity, with a slanted bandwidth roof and a flat compute roof. Nsight Compute draws it for you per kernel.
8. Chapter 46 takes this forward into what a neural network actually is and why these numbers govern what can be trained and served.

WORDS22.13.6 remember these#

1. FLOPS — sums per second — floating-point operations per second; TFLOPS is 10^12 and PFLOPS is 10^15.
2. Kernel — a program launched over a grid — a device function executed by many threads organized into blocks.
3. Arithmetic intensity — sums per byte fetched — the ratio of operations to bytes moved, which determines the binding limit under the roofline model.
4. Sparsity — assume half the weights are zero — NVIDIA’s 2:4 structured sparsity, which doubles quoted tensor throughput when applicable.
5. bfloat16 — 16 bits with FP32’s range — a truncated FP32 with 8 exponent bits and 7 mantissa bits, from Google Brain.
6. CUDA — the first non-graphics door into the chip — NVIDIA’s proprietary parallel computing platform and C-like language, public from 2007.

22.14 Integrated, discrete and mobile chips#

PLAIN22.14.1 in simple words#

1. A discrete graphics card is a separate board with its own memory chips and its own power supply, plugged into the processor over a link.
2. An integrated graphics chip is built into the same package as the processor and has no memory of its own. It borrows the system’s memory.
3. That borrowing is the whole story of integrated graphics. System memory is several times slower than a graphics card’s memory.
4. Apple’s approach is different: one large pool of fast memory that both the processor and the graphics chip address directly, with no copying.
5. Phone and tablet chips use a further trick. Instead of drawing the whole screen at once, they cut it into small tiles and finish each tile completely in tiny on-chip memory.
6. That saves enormous amounts of memory traffic, which saves power, which is why a phone can render a detailed scene on a battery.
7. A phone chip can do almost everything a desktop chip can do, including ray tracing on recent models. What it cannot do is sustain it, because of heat.

PLAIN22.14.2 a picture in your head#

1. Imagine painting a large mural.
2. A desktop chip paints the whole wall at once, walking back to the paint store for every colour.
3. A phone chip masks off one square metre, keeps that square’s paints on a small tray at hand, finishes it completely, then moves the tray to the next square.
4. Far fewer trips to the store. The store is memory, and every trip costs battery.

Where this comparison breaks: the tile has to know which parts of the mural touch it before it starts, so all the shapes must be sorted into tiles first. That sorting step is real work and real memory traffic, and it is why tile-based designs do best on scenes with moderate geometry and lose their advantage as triangle counts explode.

PLAIN22.14.3 a worked example#

1. Memory bandwidth, the number that decides everything here:

2. An integrated chip on a normal desktop shares that 96 GB/s with the processor, which is also using it. In practice it sees perhaps 60 to 70 GB/s.
3. That is about 4 percent of an RTX 5090’s bandwidth. No amount of arithmetic capability rescues that.
4. Apple’s M5 Max at 614 GB/s is in the same range as a mid-range discrete card, and unlike a discrete card it can address up to 128 GB of it.
5. That combination is why Apple laptops became popular for running large models locally: not raw speed, but a large pool of reasonably fast memory with no copying.

PLAIN22.14.4 what is really happening inside#

1. Integrated graphics carve a region out of system memory. Some is reserved at boot, more is allocated on demand. The processor and graphics chip may share a last-level cache, which helps.
2. Unified memory on Apple silicon means one physical pool and one address space. A buffer written by the processor is visible to the graphics chip with no copy and no PCIe transfer.
3. Tile-based deferred rendering, the mobile approach, works in two passes. The first pass runs all the vertex work and records which triangles fall in which tile, writing that list to memory.
4. The second pass takes one tile at a time, typically 16x16 or 32x32 pixels, loads nothing, rasterises every triangle in that tile’s list into on-chip memory, and writes only the final colours out.
5. Because the depth buffer for a tile lives on-chip, depth traffic to main memory is almost eliminated. So is blending traffic.
6. “Deferred” here refers to deferring fragment shading until visibility within the tile is resolved, so hidden surfaces are never shaded at all.
7. What a phone cannot do is sustain power. A phone chip may draw 10 watts for a few seconds and then must fall back to 3 to 5 watts or the case becomes too hot to hold. Frame rates fall accordingly.

TECHNICAL22.14.5 the engineer’s version#

1. Tile-based rendering came from Imagination Technologies’ PowerVR, whose first parts shipped in 1996 and which powered the Sega Dreamcast in 1998 and every iPhone until Apple’s own designs from the A11 in 2017.
2. Arm Mali and Qualcomm Adreno both use tiling; Adreno can also switch to immediate mode per render pass, which Qualcomm calls FlexRender.
3. Vulkan and Metal expose tiling directly through render pass load and store operations. Declaring LOAD_OP_DONT_CARE and STORE_OP_DONT_CARE correctly is the single biggest mobile optimization, because it tells the driver a tile never needs to be read from or written to main memory.
4. Apple silicon current figures, from Apple’s own specifications: M5, launched 15 October 2025, has an 8 or 10 core GPU with a neural accelerator in each core and 153.6 GB/s of unified bandwidth. M5 Pro and M5 Max followed on 3 March 2026, with up to 20 and up to 40 GPU cores, and 307 GB/s and 614 GB/s respectively.
5. AMD’s integrated designs in handheld consoles, and Apple’s designs, blur the old boundary: both are integrated in the strict sense yet perform like entry-level to mid-range discrete parts.
6. What a phone GPU genuinely cannot do, as of 2026: hold large working sets, because bandwidth and thermal budget both punish it; sustain high clocks for more than a few minutes; and match the fixed-function ray tracing throughput of a 300-watt desktop part, though recent Arm, Qualcomm, Imagination and Apple designs all have hardware ray tracing.
7. Tools: Arm Performance Studio and Qualcomm Snapdragon Profiler report tile counts, overdraw and bandwidth per render pass; Xcode’s Metal debugger shows per-tile memory traffic on Apple silicon.

WORDS22.14.6 remember these#

1. Discrete GPU — a separate card with its own memory — a device with dedicated VRAM connected over PCIe.
2. Integrated GPU — built into the processor package — a device sharing system DRAM and its bandwidth with the host.
3. Unified memory — one pool, no copying — a single physical memory and address space shared coherently by CPU and GPU, as on Apple silicon.
4. Tile-based deferred rendering — finish one small square at a time — binning primitives into screen tiles and shading each tile entirely in on-chip memory.
5. Binning pass — sorting shapes into squares — the first pass of a tiled renderer, producing per-tile primitive lists.
6. Thermal throttling — slowing down to stay cool — clock reduction driven by sustained power and temperature limits, dominant on mobile devices.

22.98 Common wrong ideas#

1. Wrong: a GPU is just a faster CPU. Right: it is a slower processor repeated thousands of times. Single-thread performance on a GPU lane is several times worse than on a modern CPU core; the win is entirely in parallel count.
2. Wrong: 21,760 CUDA cores means 21,760 independent processors. Right: it means 21,760 arithmetic lanes across 170 independently scheduled blocks, with groups of 32 lanes forced to execute the same instruction.
3. Wrong: more TFLOPS means a faster card. Right: TFLOPS is a formula, not a measurement, vendors count it differently, and most real workloads are limited by memory bandwidth long before they run out of arithmetic.
4. Wrong: the fourth number in a 4x4 matrix is a mathematical trick with no meaning. Right: it is a real projective coordinate, and the division by it is literally what produces perspective.
5. Wrong: a normal map adds geometric detail. Right: it only changes the direction used in lighting. Silhouettes stay perfectly flat, and the illusion collapses at grazing angles.
6. Wrong: ray tracing replaced rasterisation. Right: every shipping game in 2026 rasterises primary visibility and uses rays for selected effects. Full path tracing runs at reduced internal resolution and depends on denoising.
7. Wrong: frame generation makes a game more responsive. Right: it raises the frame counter and smoothness while increasing click-to-photon latency by roughly 8 to 15 milliseconds, because a real frame must be held back.
8. Wrong: upscaling from a lower resolution always looks worse than native. Right: because native rendering has no anti-aliasing history, a good temporal upscaler often resolves fine detail and edges better than native with simple anti-aliasing, while being worse in fast motion.
9. Wrong: the depth buffer stores distance in metres. Right: it stores a non-linear function of distance, which is why precision is concentrated near the camera and why reversed-Z with a float buffer is a large improvement.
10. Wrong: shader stutter means the game is badly optimized in general. Right: it usually means shader machine code is being generated on first use, which precompilation and a persistent cache fix without changing anything else.

22.99 Chapter summary in 20 lines#

1. A screen is millions of independent small calculations per frame, which is a different shape of problem from a single fast instruction stream.
2. A processor is latency-optimised: a few clever cores with caches, branch prediction and reordering, built to finish one task quickly.
3. A graphics chip is throughput-optimised: thousands of simple lanes that hide memory waiting by switching to another ready group of threads.
4. Threads run in fixed groups, warps of 32 on NVIDIA and wavefronts of 32 or 64 on AMD, sharing one instruction at a time; divergence costs real time.
5. All drawing ends in a framebuffer. Lines come from integer stepping algorithms such as Bresenham’s from 1965, and transparency from the Porter-Duff over operator of 1984.
6. Three-dimensional shapes are meshes of triangles with per-vertex normals, because three points are always flat, always convex, and always interpolatable.
7. A vertex travels through model, world, view, clip, normalized device and screen space, one matrix multiply per step plus one division by w.
8. Four-by-four matrices with homogeneous coordinates, an idea from Möbius in 1827, are used because a 3x3 matrix cannot express translation or perspective.
9. That same matrix multiply, on larger matrices, is exactly what a neural network layer computes, which is why one chip serves both fields.
10. The rasterisation pipeline has programmable stages, vertex, tessellation, geometry and fragment, separated by fixed-function clipping, divide, viewport transform, triangle setup, rasterisation, depth test and blending.
11. Shaders are compiled twice: once to a portable form such as SPIR-V, once by the driver into the installed chip’s own machine code.
12. Textures supply detail, and mipmaps exist to prevent aliasing, not to save memory; block compression is what makes real texture budgets possible.
13. The depth buffer, from Catmull in 1974, makes draw order irrelevant, and its finite precision is the sole cause of z-fighting.
14. Lighting moved from Phong in 1975 and Blinn in 1977 to physically based models parameterized by albedo, metalness and roughness, following Disney’s 2012 formulation.
15. Ray tracing answers the opposite question from rasterisation, needs a bounding volume hierarchy to be tractable, and became real time only with dedicated hardware from 2018.
16. Path tracing at real-time rates produces a noisy image that a neural denoiser reconstructs; the denoiser is load-bearing, not cosmetic.
17. Upscaling and frame generation trade image quality and latency for pixel throughput; the trades are real, measurable and worth stating honestly.
18. Inside a chip: independent blocks with their own schedulers, registers, shared memory and texture units, sharing an L2 cache and a wide memory controller feeding GDDR7 or HBM.
19. The ratio of peak arithmetic to peak bandwidth, about 58 operations per byte on an RTX 5090, decides whether any given program is compute or memory bound.
20. Integrated chips are limited by shared system bandwidth, Apple’s unified memory sidesteps copying entirely, and mobile chips save power by finishing the picture one small on-chip tile at a time.