HPC Accelerators with 3 D Memory
After a decade evolving in the High Performance Computing arena, GPU equipped supercomputers have conquered the top500 and green500 lists, providing us unprecedented levels of computational power and memory bandwidth. This year, major vendors have introduced new accelerators based on 3D memory, like Xeon Phi Knights Landing by Intel and Pascal architecture by Nvidia. This paper reviews hardware features of those new HPC accelerators and unveils potential performance for scientific applications, with an emphasis on Hybrid Memory Cube (HMC) and High Bandwidth Memory (HBM) used by commercial products according to roadmaps already announced. I. BACKGROUND: PROCESSORS The icon for processors over the last 50 years has been Moore's law, but doubling the number of transistors every 18 months was often confused with doubling performance. Mainly because it was not hard to convert the number of functional units into GFLOPS, particularly on those early years when pipelining in the 80's and superscalar designs in the 90's were driving commercial models. Last decade, the pillar was multicore, but once again, the idea turned out not to be scalable for a CPU design. In the meantime, we realize that scalability is possible on GPUs, and that is, in essence, its contribution along last decade, and the primary reason for the rapid transition to many core GPUs that we are witnessing lately. Nowadays, disruptive technologies such as heterogeneous multi cores and GPUs offer excellent performance/cost ratios for scientific applications, and an increasing number of developers have learnt to program them to take full advantage of that emerging power. GPUs have moved closer to CPUs in terms of functionality and programmability, and CPUs have also acquired features that are GPU alike. Two good exponents of this stronger CPU GPU coupling are the Fusion project led by AMD to integrate a CPU and GPU on a single chip, and the Larrabee project led by Intel to develop a many core hybrid platform using x86 CPUs. The Intel movement continued with Knights Ferry, Knights Corner and Knights Landing to establish the MIC (Many Integrated Core) Architecture [4] and finally release the Xeon Phi family of accelerators [5] In 2016, the last generation of Xeon Phi was released to include memory controllers for 3D DRAM, allowing programmers to use the x86 instruction set architecture and choose where they allocate DRAM memory, either using typical DDR modules or novel 3D cubes [6] In parallel with the CPU evolution, the GPU started its own way towards high performance computing fifteen years ago. Graphics programming experienced a revolution with the advent of shaders, methods to program vertex and pixel processors to leverage creativity in visual effects. First, HLSL (High Level Shading Language, 2001) led by Microsoft for its Direct3D pipeline, and right after GLSL (OpenGL Shading Language, 2002) the OpenGL counterpart, became popular at that time, followed by Cg (C for Graphics, 2003) developed by Nvidia with Microsoft as partner. In 2005, Nvidia unified vertex and pixel shaders leading to a more versatile core design called streaming processor. Soon multi core vertex and pixel processors at Nvidia (2001) which were ancestors for the subsequent multi core CPUs at Intel (2005) turned many core with general purpose capabilities, and in November, 2006, CUDA (Compute Unified Device Architecture) [8] was announced as the hardware and software paradigm to design and program GPUs for HPC. Now reaching its tenth anniversary, the evolution of CUDA has been impressive. Table I summarizes major achievements according to Nvidia and the NSF. We may fairly say that for the first time in HPC history supercomputing was democratized, combining three features never gathered before: Price, power and ubiquity. The free availability of CUDA tools to interact with other software communities (open source compilers, wrappers, back ends) plus the generosity of CUDA programmers, with thousands of source codes and libraries, created a friendly ecosystem for developing applications. And the last milestone attained by GPUs is energy efficiency, with all top 40 supercomputers within the green500 list [3] composed of accelerators, 31 of them being Nvidia GPUs. In terms of raw computational power, GPUs are ahead of CPUs roughly an order of magnitude. But peak processing power is harder to reach on the GPU, so this difference shortens in practice. Typical rates are sensitive to application's nature, but in general, a GPU programmer is happy squeezing 40% of peak performance, whereas a CPU programmer is often disappointed in the 60% range. Another remarkable difference lies in the programming model. Multi core CPUs use vector processing when enabling multimedia extensions (MMX, SSE, AVX) instruction level parallelism when enabling HyperThreading, and coarse grain at thread level parallelism, either relying on the scheduler of the operating system or programming explicitly via POSIX threads. Many core GPUs are programmed using the SIMT (Single Instruction Multiple Thread) model to enable data parallelism in a more scalable manner, particularly in the big data era. The idea is to run the same program in all cores, but each thread instantiates on a different data subset for each core to work effectively in parallel. That way, the more cores we have available, the smaller the working region becomes for each core, thus making the execution time to be reduced 2016 IEEE International Conference on Computational Science and Engineering, IEEE International Conference on Embedded and Ubiquitous Computing, and International Symposium on Distributed Computing and Applications to Business, Engineering and Science 978 1 5090 3593 9/16 $31.00 (c) 2016 IEEE DOI 10.1109/.64 320 CSE EUC DCABES.2016.203 TABLE I. THE IMPRESSIVE EVOLUTION OF CUDA OVER THE LAST DECADE. Year 2008 2016 Multiplier Number of GPUs accepting CUDA 100.000.000 600.000.000 6x CUDA downloads monthly 10.000 300.000 30x CUDA enabled supercomputers within Top500 list 1 10 Fermi 53 Kepler 104x Aggregate performance for those supercomputers 77 TFLOPS 80000 TFLOPS 1039x University courses teaching CUDA 60 800 13x Scientific papers published using CUDA 4.000 60.000 15x proportionally without even having to recompile the code. The key for a successful massive parallelism on a SIMT deployment is data partitioning. Straightforward decompositions are usually derived from 1D, 2D and 3D matrices as input data sets, overall on regular access patterns, but the goal turns more difficult in the presence of irregular accesses or dynamic data structures. II. BACKGROUND: MEMORY The evolution of memory technology is a whole different story, but we can see similarities with the processor. DRAM memory has its own Moore's law: It doubles its size (Gbytes) every 18 months. That is like the transistor count for the CPU, but unlike its partner, it does not translate that easy into speed rates. Even worse, the larger a memory circuit becomes, the slower reacts. And the more density holds, the tinier its atomic cell, that is, the capacitor. It takes a while to read typical loads of 25 30 fF, so it is faster to amplify those loads before starting to read. In essence, amplifiers increase latency, but benefit bandwidth. And since latency is unavoidable, we have tried to amortize its cost with higher bandwidths. That has been the fuel propelling SDRAM memories over the past 20 years, with an overall increase of 48x, from the first SDRAM pumped at 66 MHz to provide 533 MB/s, to the last DDR4 at 2x1600 MHz to deliver 25.6 GB/s. In the meantime, CPUs contribute with larger caches and more cache levels to minimize DRAM memory accesses, and also with bigger cache lines to promote bandwidth over latency. On the other hand, GPUs have introduced wider paths (384 bits) for a GPU using GDDR5 video memory to deliver 336 GB/s bandwidth (see Table VII using Titan X as example) In contrast, a typical CPU operating on a 4 channel motherboard hardly reaches 64 GB/s (see also Table VII for the column of the Intel Broadwell CPU) Another similarity between memory and processor is the multi core evolution. DDR (Double Data Rate) chips split memory cells into twin hemispheres sharing latency but doubling bandwidth. The DDR saga exploits this idea with DDR2, DDR3, DDR4 and GDDR5 designs the same way we have seen with dual cores, quad cores, octo cores, until we face the truth: It is not an scalable idea. The clones of a single design require more and more infrastructure for the memory chip to be able to coordinate the efforts into a single response, and therefore, latency increases proportionally to the bandwidth increment. Figure 1 summarizes the evolution of memory latency and bandwidth for the DDR family, where we see that latency now represents more than 90% of the time required for a memory read, that 20 years ago was barely 20% Life for transistor within the processor was easy thanks to the silicon miracle: Every couple of years, manufacturing nodes were shrinking gates around 30% and therefore, the atom of a processor became smaller, faster, cheaper and more power efficient on 32, 22, 14 nm. But for the memory, capacitors did not help much, because is the delay for amplifying loads what really matters. Interfaces came to rescue with smart ideas for exploiting interleaving, essentially with (1) multi banks enabled from multiple RAS (Row Access Strobe) signals, and (2) multi channel deployment for memory modules along the motherboard. Unfortunately, none of these ideas turned out to be scalable either. Ultimately, the energy cost has killed further developments for a SDRAM standard which has been pushed a long way. Additional layers of circuitry are required to compose and merge a single memory access (typically, the service of a cache line as seen in Figure 1) from all banks, chips, modules and channels involved, and that way, memory cells are place