Fall 2011, Lecture 16: NVIDIA GPUs (I)
(modified from notes of Fall08 lecture 11 and Fall07 lecture 13)
GPU Architecture
Of the steps in the graphics pipeline, vertex processing and shading are completely programmable. Rasterization is hard to do in software and very easy to do with fixed function hardware, so there is little benefit to programmability. Original GPUs had separate logical elements for vertex and fragment processing, as shown below
Since some images have large amounts of vertices and others have detailed fragment/shader processing, hardware load balancing can be difficult.
To try and solve this problem, modern GPUs use a unified shader architecture for efficiency. One compute node is capable of processing both vertices and fragments.
Since the GPU was becoming more programmable, the compute core was made the focus of the architecture. This has led to the wave of applications that are using the GPU as a general processing node.
Recalling the GPU pipeline, each vertex is a compute task. There are about 10,000 to 100,000 vertices per frame. A stream of vertices is sent to the GPU from the host, each vertex has some state associated with it and they are sent in phases of matching state. Once the vertices arrive at the GPU, every vertex gets its own thread. These vertices are transformed, which amounts to a matrix multiplication operation. The vertices then go through a programmable geometry manipulation. The output of this stage is a stream of primitives that are rasterized into pixels. In the case of transparency, the Z-buffer blends the fragments together based on whether or not the fragment in front is opaque or translucence. Opacity overrules translucence so the order of operations matters, pixels need to be processed in the order they originated.
We can generalize the graphics pipeline that was described above into streams arriving into processing kernels. The stream can be split up into blocks and the GPU operates on a block at a time. This leads to a thread block, which is the base processing granularity of the GPU.
“From Shader Code to a Teraflop: How Shader Cores Work”
by Kayvon Fatahalian, Stanford University
The GPU architecture includes many shader cores, texture units and fixed function units such as rasterization hardware. GPUs are heterogeneous multi-core processors that are tuned to stream problems.
The GPU needs to distribute work to all of its compute units. This is done in fixed function hardware in current GPUs. Intel’s Larrabee chip has a programmable scheduler and rasterization unit.
A shader is shown below:
A kernel like this is applied to all fragments in the stream that have a given set of parameters. This program takes in one fragment record (with 3 elements) as an input and modulates it with some light before it returns the fragment.
Modern CPUs have a complex fetch unit to exploit ILP keep pipeline full and a large execution context when its caches are included. The GPU removes elements that help a single unit run quickly. Throughput is important. Large numbers of simple fragment processing cores are put in parallel.
GPUs exploit extreme parallelism in graphics applications. Since every pixel is completely independent from every other pixel, compute nodes become the focus of the chip. So, instead of using caches to try and hide memory latency, threads are used instead to expose more parallelism.
Amongst the different cores, most of the time the instructions that are executed are the same and the only difference is the data on which these instructions act on. Thus the next step is to reduce the complexity due to fetching and decoding an instruction by doing it only once for a set of cores. Each of these instructions is now a vector instruction.
Thus if we have 16 such cores, which will be 128 ALUs, we can process 16 simultaneous instruction streams. Here, there are different units processing the primitives, vertices and fragments independently.
But, what happens when there is a branch instruction in the code?
The hardware computes the condition (or predicate) for each of the 8 cores. If the condition evaluates to false, that particular core knows not to commit its result, but the cores that evaluate the condition to be true proceed with execution. A particularly bad case for this model is if there is a nested branch and the amount of control divergence (the threads don’t all follow the same path of execution) is high. Only a few lanes would be computing relevant results at a time.
In an architecture like this, the SIMD processing does not really imply SIMD instructions. One can write his code with either explicit vector instruction or write it with scalar instructions with implicit data parallelism and let the hardware convert it into SIMD instructions. The later is the methodology used by AMD (ATI) and Nvidia and the former is used by Intel/AMD x86 SSE and Intel Larrabee. NVIDIA calls this as Single Instruction Multiple Threads.
What happens when there are stalls?
Stalls occur when a core cannot run the next instruction because of a dependency on a previous operation or a long latency memory operation. Since we do not have any caches as in a normal CPU, we need to come up with a new mechanism to hide the latencies due to stalls. Since we have a lot of independent fragments, we can interleave the processing of many fragments on a single core by context switching between them and hide the latency caused by the stalls. Thus when one of the threads has a stall, we can switch to another thread and keep the processor busy all the time. But storing these contexts takes space. There are many different ways of implementing this mechanism. We can have a lot of small contexts stores or lesser number of medium sized ones or have very few large context stores.
GeForce−8 Series HW overview
The G80 is composed of several streaming multiprocessors (SM) as pictured below.
Each of these streaming multiprocessors has eight streaming processors (SP) and two super-function units (SFU). The SPs contain very simple ALUs for the most basic/common operations used in graphics processing while the SFUs are used when more complex operations are required (these should be avoided when possible). The SM performs multi-threaded instruction dispatch in vectors of 32. These vectors are referred to as warps in Nvidia terminology. There may be as many as 16 warps per a thread block for a total of 512 threads per a thread block. The large number of threads is used to hide memory access latency.
