Lecture 11 Notes for Class on 10/2/08 about the GPU architecture
The class started with some introduction to the different stages in the graphics pipeline. Then, we had a quick overview of the different filtering algorithms namely Point sampling, Bilinear filtering, Trilinear filtering and Anisotropic filtering. About the Texture cache, it mainly exploits the spatial and temporal locality due to the overlap between the fragments in the textures.
Pixel Shading:
When it comes to pixel shading, it was in 1999, DirectX 7 was capable of not just rendering the pixel color based on one fragment, but rather one can overlay one fragment over another and compute the color of a pixel based on the combination of these fragments to render more complex images. DirectX 9 allowed a programmer to write a program that can read the input from interpolated values, textures, lighting and a result of other math operations and compute the color for each pixel.
ROP (Raster Operations):
While computing the color of a pixel, there may be more than one fragment that maps to the same pixel. A fragment may be transparent or semitransparent and we may need to add up the colors to find the color of this particular pixel. In this process, something called Antialiasing is done, which tries to remove sharp edges in the image. An example is, you can see the edge between the red and the green portions of the following image.
What antialiasing does is that the sharpness of the edges is removed by giving an intermediate color for the pixel that is near the edge based on the colors of its neighboring pixels. One can see that the border between the red and the green segments is now given an intermediate color between red and green in the image below after antialiasing.
Typically all these operations are read/modify/write to the frame buffer. The color of the current pixel is read from the buffer, it is modified based on the fragment that we are working on and is updated back into the frame buffer.
Alpha Blending and Z Buffers:
Alpha blending is used to render translucent objects. The alpha component of the pixel contains its opacity and based on this alpha component the resulting color of this pixel is calculated by the formula,
Result = alpha * Src + (1-alpha) * Dst
While different fragments are combined to render a particular pixel, the fragments that are closer to the viewing point block the ones behind them and the ones closest to the viewing point determines the color of the pixel. Thus we need some kind of ordering for these fragments based on the distance from the viewing point. While storing the X and the Y coordinates, the Z values are also stored, but in a special buffer called the Z-buffer which is nothing but a 2D array of these Z values with the same X, Y dimensions as the color frame buffer. And before writing the color, the Z-buffer test given below is performed.
- Read the Z value from memory
- Compare the candidate Z to the Z from memory; if the candidate Z is NOT in front of the previous Z, discard the pixel
- Otherwise, write the new Z value to the Z buffer and write (or blend) the new color to the color frame buffer
Frame Buffer Interface:
All these read modify write operations that we discussed above are done to the Frame buffer using the Frame Buffer Interface (FBI), which is a fairly complicated piece of circuit, which does compression and decompression on the fly. The bandwidth that is required here is very high (there are also multiple fragments to every pixel) and is the most critical component of the performance of the GPU. The FBI of the GeForce is a crossbar and has independent memory controllers for 4+ independent memory banks for more efficient access to frame buffer.
Thus in summary, we see that we need to do some programming for the vertex part and some programming for the fragment part of the pipeline. Thus the GPU has separate processors for both parts of the pipeline.
The GPU architects need to decide how much of hardware that should be given to each of them based on the requirement of how many vertices we need to process per second and how many fragments we need to process per second in each of the processors. This load balancing in hardware is difficult because sometimes, we may need more processing power for the vertices, and other times we may need more processing power for the fragments based on the problem.
But, if the same load balancing when done in software, it is easier. Thus if they have both units for vertices and fragment processing, one could dynamically load balance the vertex and fragment processing based on the type of the problem. This unification happened 3–4 years ago, when one processor capable of doing both vertex and fragment processing was designed.
Now, since it is becoming more programmable, the compute core was made the focus of the architecture and everything is built around it so that we can do a dynamic load balancing.
Recalling the GPU pipeline, each vertex is a compute task. This Vertex thread issue unit creates all the units of execution and dynamically assigns them to different cores. Similarly, once it is done processing the vertex, the next step is rasterization and so on, for which the same compute structure is used. By having special drivers, we can use this GPU for general purpose computing too. The following figure gives another view of the pipeline,
Vertex control has a collection of triangles/geometry that is a stream of vertices. The transform takes in this stream of vertices, one triangle at a time and gives out the transformed (rotated, scaled etc) stream of vertices. This is fed into the Geometry, where the programmable part of the vertex processing takes place. Then the Lighting, and Setup are also similar, in the sense, one operation is performed on a stream of data, which is independent of each other. Raster is a little different in the sense that it takes in the stream of vertices and produces a stream of fragments. These fragments further go through the Fragment Shading stages. In summary, most of these operations process streams of data that are independent of one another. These operations can be visualized as built in kernels operating on streams of data. The Unit of Execution is processing of one stream element in one kernel (NVIDIA calls it a thread).
These streams are very long and the elements are independent of each other. They are broken down into chunks, called strips or blocks. A thread block is the unit of execution, processing one block of data by one kernel.
“From Shader Code to a Teraflop: How Shader Cores Work”
by Kayvon Fatahalian, Stanford University
A GPU has a bunch of Shader cores, a Tex to perform the texture operations with other special units like Rasterizer, Output blend etc. This is a heterogeneous chip multiprocessor that is highly tuned for graphics.
Here is a sample Shader Code,
It has a shader sampling function that will be called for every element in the stream and this illustrates the data parallelism. It takes in one fragment record (with 3 elements) as an input and performs the operations and gives out another fragment as the output, which is similar to a regular processor with a Fetch/Decode, ALU and an Execution context for the temporary variables.
Since this is very similar to a CPU core, why should we not use a normal CPU core to do it for us? A CPU core has the usual out-of-order control logic, a branch predictor, prefetcher etc as given in the following figure,
The out-of-order control logic tries to hide the latency in the instructions by identifying independent instructions, and thus exploiting the ILP present in the code. The branch predictor helps in aiding the out of order execution by predicting the outcomes of branches. But, the question is that do we really need these out-of-order, branch predictor and prefetching logic?
Actually, the Graphic Processing has enough amount of DLP and thus we can afford to get rid of these logic that goes after exploiting ILP in the code. The memory prefetcher can be removed because, we already have special ways of accessing the memory through texture cache, Frame Buffer Interface etc. Thus the CPU core can be slimmed down into a very simple core that would just help a single instruction stream execute fast. And since we have independent data, it would be advantageous to have multiple cores working on them independently.
Amongst the different cores, most of the time the instructions that are executed are the same and the only difference is the data on which these instructions act on. Thus the next step is to reduce the complexity due to fetching and decoding by doing it only once for a set of cores. Now, each of these instructions is a vector instruction that needs to be generated by the compiler.
Thus if we have 16 such cores, which will be 128 ALUs, we can process 16 simultaneous instruction streams. Here, there are different units processing the primitives, vertices and fragments etc independently.
But, what happens when there is a branch instruction in the code?
The hardware computes the condition for each of the 8 cores and individually. If the condition evaluates to false, that particular core idles, but rather the cores that evaluated the condition to be true proceed with execution. When the cores reach the set of instructions that has to be executed for the false case, the cores that had the condition evaluated to false start executing and the ones that evaluated it to true start idling. It is very clear that there is a lot of loss in performance when there is a branch in the code. The worst case happens when there is a nested branch and only very few lanes get to execute and others idle most of the time.
In an architecture like this, the SIMD processing does not really imply SIMD instructions. One can write his code with either explicit vector instruction or write it with scalar instructions with implicit data parallelism and let the hardware convert it into SIMD instructions. The later is the methodology used by ATI Graphics and Nvidia and the former is used by Intel/AMD x86 SSE and Intel Larrabee. NVIDIA calls this as Single Instruction Multiple Threads or warps.
What happens when there are stalls?
Stalls occur when a core cannot run the next instruction because of a dependency on a previous operation. Now, since we do not have any caches as in a normal CPU, we need to come up with a new mechanism to hide the latencies due to stalls. The advantage that we have is again the DLP. Since we have a lot of independent fragments, we can interleave processing of many fragments on a single core by context switching between them and hide the latency caused by the stalls. Thus when one of the threads has a stall, we can switch to another thread and keep the processor busy all the time. But storing these contexts is the problem associated with this mechanism. There are different ways of doing it. We can have a lot of small contexts stores or lesser number of medium sized ones or have very few large context stores.
