Lecture 22, Nov. 9th, 2011

Introduction to CUDA

CUDA is NVIDIA’s parallel computing architecture. It enables dramatic increases in computing performance by harnessing the power of the GPU. GPU is originally designed for graphics application so that it adopts the streaming processing model because there are millions objects like vertices and pixels in the graphics application. CUDA is a programing model to utilize the computing resources for general purpose computing.

CUDA is similar to C language. It matches the architecture closer than C. Because GPU utilizes the SIMT architecture, users need to write the program for a thread in a general way and specify the number of threads in this kernel. By this way, user kicks off batches of threads on GPU. The programming model of GPU also implies GPU is a dedicated super-threaded, massively data parallel co-processor.

CUDA Platform

Currently, NVIDIA high-end GPUs not integrated into CPU. GPU is connected to the CPU through the PCI Express bus. Every time CPU wants to communicate with GPU, it needs to go through drivers in the OS. The driver would load the computation programs to GPU through the PCI-E bus. In order to get good performance speedup over CPU, the computation program needs to have very hight arithmetic intensity relative to PCI-E bus. Otherwise, this interface between CPU and GPU would become bottleneck for the computation.
CUDA API is an extension to the ANSI C programming language. So users could pick it up very quickly. GPU is viewed as a compute device that is a coprocessor to the CPU (also called host) and has its own DRAM. The data-parallel portions of an application are executed on the device as kernels which run in parallel on many threads. Thread in GPU is extremely lightweight compared to CPU thread. So GPU can have many threads running simultaneously and it is designed to utilize this big number of threads to hide latency.

CUDA compilation

Any source file containing CUDA language extensions must be compiled with NVIDIA’s own compiler driver nvcc. NVCC can output either C code for CPU or PTX object code directly. PTX is not NVIDIA’s GPU real instruction set but it’s intermediate representation. So the PTX program running in the GPU needs to be compiled to the machine dependent instruction. This compilation is done by a Just-in-Time compiler (JIT) in the GPU. So we can treat PTX as both a Virtual Machine and ISA. An executable compiled in device emulation mode (nvcc -deviceemu) runs completely on the host using the CUDA runtime. There is no need of any device and CUDA driver for that. When running in device emulation mode, one can use host native debug support lke breakpoints, inspection, etc. Recently, NVIDIA released CUDA GDB, which is a real debugger for both CPU and GPU CUDA parts. There are other useful tools like CUDA MEMCHECK for invalid memory access detection and CUDA Visual Profiler which we used for the lab in class.

CUDA program optimization

Before going on to the optimization of CUDA codes, there are some prerequisites of algorithm implemented in CUDA. This does not mean algorithms without these characteristics cannot be implemented in CUDA, but they are necessary for performance perspective. These characteristics are

1. Enough parallelism for GPU to hide latency since GPU is designed for streaming processing.
2. High Arithmetic Intensity relative to the PCI-E bus, and quite high to device memory. Otherwise, the GPU’s computing capability cannot be fully utilized because it would be bound by the memory.
3. Few dependency. GPU’s ability to deal with dependency is quite low because its original graphics application doesn’t contain dependency and it’s designed for throughput.

After satisfying these prerequisites, we can think about the optimization of CUDA programs. The suggested priority of that is

1. Memory coalescing is first priority.
2. Take advantage of shared memory because shared memory have very high bandwidth and low latency compared to DRAM.
3. Use parallelism efficiently to keep the GPU busy at all times. And we might need many threads and thread blocks to guarantee that.
4. Leave bank conflicts and divergence for last. Because smaller conflicts are not usually worth avoiding if doing that would cost more instructions.

For the data transfer between GPU and CPU, the interface between them (PCI-E bus) is has much lower bandwidth than the device memory to device memory bandwidth. (4GB/s for PCI-E vs 80GB/s for Quadro FX 5600). So one important thing the programmer needs to consider is to minimize data transfers between CPU and GPU. One large transfer is much better than many small ones since the data transfer needs to go through driver every time. And memory allocation in GPU needs to go through CPU OS too, so the dynamic memory allocation for GPU is very slow and expensive in terms of performance.

Communication and Synchronization in CUDA

In the graphics application, Data parallel streams that represent independent vertices, triangles, fragments, and pixels in the graphics world and they never communicate. But in the compute mode, some communications are allowed. For example, threads in a thread block can communicate with each other via shared memory and kernels can communicate thorugh global device memory. But thread blocks can’t communicate inside a kernel.

For the synchronization inside a kernel, only threads inside a thread block are allowed to synchronize using the barrier. But blocks cannot synchronize. In CUDA, kernels and DMAs are asynchronous by default. So in order to support the inter-kernel synchronization, CUDA provides the synchronization management with streams and events. Stream is an in-order sequence of bulk operations and streams can be arbitrarily interleaved or executed concurrently. In other side, programmers can explicitly record or wait for a event.