I've spent the last few days working with tensorflow for the first time as part of a natural language processing assignment for my degree. It's been interesting (fun isn't the right word) trying to get it to run on a GPU but it got me thinking.The recent advances in deep learning have come about as GPGPU technologies have matured and the frameworks have arrived to make doing massive amounts of linear algebra on your computer much quicker and easier. Nvidia now sell chips that are designed specifically for this task and from what I understand, papers like the one featuring AlexNet would not have been possible without GPU acceleration. This point is nicely articulated by the authors: All of our experiments suggest that our results can be improved  simply by waiting for faster GPUs and bigger datasets to become  available.Given this, my question is then why haven't we seen more adoption of GPUs for traditional HPC tasks (simulation, rendering etc.)? These workloads have been around for years yet it seems like only recently that GPGPU has taken off as an approach. It appears to me that the requirements are pretty similar, namely 'read a load of data, do a load of floating point transforms on the data, save some data, repeat' but a look at the TOP500 reveals that many of the systems on there are still using CPUs (although an increasing number using 'manycore' processors like the Intel Phi which seem to straddle the CPU/GPU divide). Are there actually less similarities between traditional HPC and large-scale ML workloads than I imagine? Maybe it's that GPUs are less efficient in terms of flops/w which is what really matters when it comes to running a huge compute cluster? Is it just down to a very effective marketing effort by Nvidia in the machine learning space?
Large Scale Machine Learning vs Traditional HPC Hardware
hardware;machine learning;high performance;gpu
It appears to me that the requirements are pretty similar, namely  'read a load of data, do a load of floating point transforms on the  data, save some data, repeatI think that covers a lot of traditional HPC, but by no means all. I have only limited experience with CUDA, but my understanding is that data dependent branching kills GPU performance. All the threads within a warp have to execute the same instruction at the same time. If different threads within a warp take different branches, they have to be executed serially. That's a pretty serious constraint. Sometimes problems can be recast to work within that constraint, but not always.