NVidia Pascal's GPU architecture most exciting feature

Few months ago NVidia announced the Pascal GPU architecture and more specifically the GP100 GPU. This is a monstrous GPU with more than 15 billion transistors built using a 16nm FinFET fabrication. Though, the alleged performance numbers are arguably impressive (10.6 TFlops SP, 5.3 TFlops DP) I personally think that this is not the most impressive feature of this GPU.

The most impressive feature I found on as advertised is the unified memory support. In CUDA 6 and CC3.0 & CC3.5 devices (Kepler architecture) this term had been first introduced. But it didn't actually provide any real benefits at the time other than programming laziness. In particular, the run-time took care of moving the whole data to/from the GPU memory whenever it was used on either the host or GPU. The GP100 memory unification seems far more complete as according to specifications it seems to take memory unification to the next level. It supports data migration at the granularity of memory page! This means that programmer is able to "see" the whole system memory and the run-time takes care of which memory page should be moved at the time it is actually needed. This is a great feature! It allows porting CPU programs to CUDA without caring which data will actually be accessed.

For instance, imagine having a huge tree or graph structure and and you have a GPU kernel that needs to access just a few nodes on it without knowing which beforehand. Using the Kepler memory unification feature would require copying the whole structure from the host to GPU memory which could potentially cannibalize performance. The Pascal memory unification would actually copy only the memory pages residing on the accessed nodes, instead. This releases programmer from a great pain and that's why I think this is the most exciting feature.

I really hope this feature will be eventually supported on consumer GPU variants and stays not just an HPC feature for in Tesla products. I also hope that AMD will also support such a feature in its emerging ROCm platform.

Resources:

NVidia Tesla P100 whitepaper

About SPIR-V and OpenCL 2.1

Less than a couple of months ago the provisional release of OpenCL 2.1 and SPIR-V byte code was announced. SPIR-V is now defined completely from the ground-up and is not a patched LLVM derivative anymore. The real good news is that Khronos will push extended language features (C++) to be supported through an offline compiler.

Let me explain. I believe that the source compiler should never have been a part of the device driver. Using an offline compiler and feeding the OpenCL runtime with kernels in a bytecode format would allow the runtime to be more lightweight, which is very important especially for mobile devices. It would also prove to be less error prone and more portable. My sense is that OpenCL consumption by the runtime will be obsolete in future OpenCL releases. I always found weird the way by supplying text source code to the library during runtime. This is a change for good and it will allow vendors releasing their implementations faster.

I also want to note that NVidia, after all these years of stagnation, silently released drivers supporting OpenCL 1.2 which allows us dreaming a future driver supporting SPIR-V. They do not support Fermi through (only Kepler & Maxwell).

Maxwell for the masses (GM206)

As you probably already know the mainstream version of Maxwell GPU has already been released in the form of GM206. The graphics card bearing the chip is the GTX-960. The card seems to be pretty efficient and a significant improvement over Kepler especially in compute applications which is the one aspect that I'm particularly interested in. There has been some controversy of course due to its short memory bus (128bit) which entails a peak memory bandwidth of 112GB/sec. However, the larger cache memory should help alleviating this bottleneck.

The Zotac GTX-960 AMP! edition

In order to give you a taste about the compute capabilities of Maxwell I provide the results of experimenting with the OpenCL NBody example (16384 bodies) from the NVidia SDK 4.2 (the last one with OpenCL support). The GTX-960 yields a well above of 1TeraFlop performance which is impressive. I also performed executions with 3 more GPUs. All results are depicted in the chart that follows.

The red bars represent measured performance in GFLOPs and the green ones the efficiency as the ratio measured/peak GFLOPs performance.

The Maxwell architecture seems to address many issues with compute efficiency of its predecessor. However, there are two drawbacks. First, the low memory bandwidth as mentioned above and second, the quite low compute performance in double precision operations which is set now at 1/32 ratio with regard to single precision operations.

One last observation is the quite good performance of the AMD GPU although the example application had been developed by NVidia and it's reasonable to think that it is optimized for its own GPUs. This could be one of the main reasons that they stopped supporting the OpenCL paradigm.

Least required GPU parallelism for kernel executions

GPUs require a vast number of threads per kernel invocation in order to utilize all execution units. As a first thought one should spawn at least the same number of threads as the number of available shader units (or CUDA cores or Processor Elements). However, this is not enough. The type of scheduling should be taken into account. Scheduling in Compute Units is done by multiple schedulers which in effect restricts the group of shader units in which a thread can execute. For instance the Fermi SMs consist of 32 shader units but at least 64 threads are required because 2 schedulers are evident in which the first can schedule threads only on the first group of 16 shader units and the other on the rest group. Thus a greater number of threads is required. What about the rest GPUs? What is the minimum threading required in order to enable all shader units? The answer lies on schedulers of compute units for each GPU architecture.

NVidia Fermi GPUs

Each SM (Compute Unit) consists of 2 schedulers. Each scheduler handles 32 threads (WARP size), thus 2x32=64 threads are the minimum required per SM. For instance a GTX480 with 15 CUs requires at least 960 active threads.

NVidia Kepler GPUs

Each SM (Compute Unit) consists of 4 schedulers. Each scheduler handles 32 threads (WARP size), thus 4x32=128 threads are the minimum requirement per SM. A GTX660 with 8 CUs requires at least 1024 active threads.

In addition, more independent instructions are required in the instruction stream (instruction level parallelism) in order to utilize the extra 64 shaders of each CU (192 in total).

NVidia Maxwell GPUs

Same as Kepler. A GTX660 with 8 CUs requires at least 1024 active threads. A GTX980 with 16 CUs requires 2048 active threads.

The requirement for instruction independency does not apply here (only 128 threads per CU).

AMD GCN GPUs

Regarding the AMD GCN units the requirement is more evident. This is because each scheduler handles threads in four groups, one for each SIMD unit. This is like having 4 schedulers per CU. Furthermore the unit of thread execution is done per 64 threads instead of 32. Therefore each CU requires the least of 4x64=256 threads. For instance a R9-280X with 32 CUs require a vast amount of 8192 threads! This fact justifies the reason for which in many research papers the AMD GPUs fail to stand against NVidia GPUs for small problem sizes where the amount of active threads is not enough.

Maxwell further lowers double precision performance for GeForce GPUs

Now this double precision mockery seems to have no end. For top end Fermi based GPUs the ratio was 1/8 which was just acceptable. For the rest Fermi GPUs the ratio became 1/12. Thereafter, Kepler further reduced it to 1/24. And today we learn that the first Maxwell GPUs further cut it to 1/32!

As long as NVidia wants to sell as more Teslas as it gets we will never be able to achieve acceptable performance in double precision arithmetic from consumer cards. Actually, using a consumer GPU (excluding GTX Titan) for a compute intensive problem does not worth considering the CPU improvements with 256bit AVX2 plus the addition of FMA instructions. And certainly not everyones has 1000$ to waste for a GTX Titan. I would expect a decent double precision performance from a mid-range card of, lets say 300$, but unfortunately that's not the case.

I hope the next architecture dubbed Volta will not emply a 1/128 ratio though it doesn't actually make much difference if it is 1/32, 1/64 or 1/128. These ratios turn double precision compute on consumer cards meaningless.

Source: http://www.tomshardware.com/reviews/geforce-gtx-750-ti-review,3750.html#xtor=RSS-182

My new teraflopper accelerator (?)

Althought the title above would seem to be eye catching some years ago, modern GPUs offer multiples of teraflops of compute power. Thus, my "slight" upgrading from GTX465 (Fermi based) to GTX660 (Kepler based) GPU appear as not so remarkable news.

Now, let me elaborate on what I mean by the word "slight". I mainly use the GPU as a development device for the CUDA and OpenCL environments (actually, I almost never use it for playing games). Therefore, I'm particularly interested on the compute performance of these cards. As it has been shown in various reviews the compute performance of Fermi based cards have proven more efficient than Kepler based cards. However, Kepler GPUs tend to be more power efficient.

The older GTX465

GTX660

Nevertheless, the GTX660 provides multiple benefits over the GTX465. First, the available memory on device is doubled, 2GB instead of 1GB. This is important as I can run compute kernels for bigger workloads. Additionally, the memory bandwidth is significantly higher, 144GB/sec instead of 102GB/sec. This proves usefull for memory bound kernels. Regarding the compute throughput of both cards the GTX465 features 855GFlops in single precision operations whilst the GTX660 features a decend 1880GFlops. All these in just 140W TDP instead of 200W TDP. This is why the GTX660 requires just one 6-pin power connector instead of 2.

Now, the drawback. The GTX660 exhibits significantly less peak performance in double precision operations as NVidia tends to criple double precision potential of desktop products on the Kepler GPUs by a factor of 1/24 rather than the 1/8 it did in the first Fermi generation. This yields a 78GFlops for the GTX660 and 106GFlops for the GTX465.

I definitely wanted GPU with CUDA support as most of my work is based on it so I couldn't consider getting a Radeon GPU. However, in the future I intend to focus on open standards i.e. OpenCL.

OpenCL performance of GTX 660

As I upgraded my GPU card I wanted to run some OpenCL benchmarks. I have developed a small tool that performs some benchmarks by identifying the best configuration in terms of workgroup size, vector size and workitem granularity on each kernel. I also included results from a Radeon HD7750 which is an even less power hungry card as it does not require a power connector at all.

AMD HD7750

Benchmark results:

	Reduction GB/sec	Leibniz formula for PI computation (SP) GFLOPS	Leibniz formula for PI computation (DP) GFLOPS
GTX465	89.56	304.26	26.56
GTX660	108.12	656.08	21.77
HD7750	63.15	214.45	13.81

Result chart:

It doesn't seems so bad. Memory bandwidth is improved and the single precision computations are more than doubled! As predicted the double precision computation exhibits a noticable performance drop but it can be tolerated.

Peak throughput numbers were used from the respective wikipedia pages:
http://en.wikipedia.org/wiki/GeForce_400_Series
http://en.wikipedia.org/wiki/GeForce_600_Series