Now I have been believing that FEM/CFD is supposed to be faster on a GPU unit -  here I am using CUDA as solid example.  However, I have not been able to find a convincing paper where the benchmark actually appear to me that 'Yes, this is true!'. Can I please be pointed to one? Or if not, is there any reasons why GPU unit can suck when compared to CPU for CFD/FEM? Could it have anything to do with sparse matrices structure? In terms of performance index like speed/degree of parallelism etc.
The real myth of GPU (specifically CUDA) really speed up FEM/CFD
finite element;fluid dynamics;performance;cuda
Here's the deal with GPUs. On a GPU, every single core is slow. Really slow. However, you have thousands of cores. If you can effectively use the thousands of cores at a time, then your algorithm will run better on the GPU. If you cannot, then it will run much slower on the GPU. Linear algebra is one domain where parallelism is really well established. Thus the best way to write for a GPU is to essentially have the GPU do all of the linear algebra: it essentially becomes a card to compute Ax=b and A*B much faster than the CPU (this fact is pretty easy to check, look at the numerous benchmarks or even just open up MATLAB and type in A*B for both matrices and GPU matrices). But there's a caveat: data transfer to GPUs is really slow. Also, memory allocation on GPUs is really slow. So while the linear algebra is fast, you have to deal with the fact that:Serial performance is awful.Allocating memory dynamically on the GPU destroys performance.Transferring back and forth between the CPU and GPU is slow.This puts constraints on your algorithm: you need to try to leave as much on the GPU as possible, transferring back and forth the minimum amount, while trying to avoid serial parts from running on the GPU. Likewise, the GPU can easily do the linear algebra 1000x faster than the CPU (which is usually the performance bottleneck) so in many cases you can effectively manage this dilemma and end up with large performance gains.One interesting alternative are the Xeon Phi. These cards have much faster data transfer, much better serial performance, and can allocate memory much better. However, the tradeoff is it's less specialized to be a dumb linear algebra solver and you thus have to pay a heftier price, and in return its linear algebra performance is about half that of a GPU. However, this can be much easier to develop code for (OpenMP parallel codes will use it automatically, and you can use a Xeon Phi card as another node via MPI, so if you've already parallelized a code you can use the same code with the Phi) and, by allowing you to more effectively keep data on the acclerator or using the increased data transfer speeds, can be much faster than a GPU in real-world (S)PDE solving. Of course, it depends greatly on the implementation.