Computational fluid dynamics (CFD) requires a vast amount of compute cycles on contemporary large-scale parallel computers. Hence, performance optimization is a pivotal activity in this field of computational science. Not only does it reduce the time to solution, but it also allows to minimize the energy consumption. In this work we study performance optimizations for an MPI-parallel lattice Boltzmann-based flow solver that uses a sparse lattice representation with indirect addressing. First we describe how this indirect addressing can be minimized in order to increase the single-core and chip-level performance. Second, the communication overhead is reduced via appropriate partitioning, but maintaining the single core performance improvements. Both optimizations allow to run the solver at an operating point with minimal energy consumption.
In case of lattice Boltzmann methods (LBM) the decisive quantity is called ( @math ) and is measured in bytes per fluid lattice site update ( @math ) @cite_17 . There are several propagation step variants for LBM that achieve lowest data traffic per FLUP: the two-grid one-step algorithm with non-temporal stores @cite_13 , Bailey et. al's AA-pattern @cite_9 , and Geier's Eso-Twist @cite_18 @cite_12 . The latter two both work with a single grid only and arrange the processing order in a clever way to work around data dependencies. The first two variants were successfully implemented in the fluid flow solver framework @cite_14 , which relies on a sparse lattice structure of the simulation domain @cite_0 . In contrast to a full array approach, this introduces indirect data accesses, but delivers, if done correctly, excellent performance not only for flow in simple geometries but also in porous media like fixed-bed reactors or foams.
@cite_18 @cite_14 @cite_9 @cite_0 @cite_13 @cite_12 @cite_17
2791769657 2950444855 2113754927 1980037648 1971876223 2073875590
Traditionell werden numerische Stromungssimulationen in einer zyklischen Sequenz autonomer Teilschritte durchgefuhrt. Seitens Wissenschaftlern existiert jedoch schon lange der Wunsch nach mehr Interaktion mit laufenden Simulationen. Seit dem masgeblichen Report der National Science Foundation im Jahre 1987 wurden daher neue Formen der wissenschaftlichen Visualisierung entwickelt, die sich grundlegend von den traditionellen Verfahren unterscheiden. Insbesondere hat der sogenannte Computational Steering-Ansatz reges Interesse bewirkt. Damals wie heute ist die Anwendung des Verfahrens jedoch eher die Ausnahme denn die Regel. Ursachlich dafur sind zu grosen Teilen Komplexitat und Restriktionen traditioneller Hochleistungssysteme. Im Rahmen dieser Arbeit wird daher als Alternative zu dem traditionellen Vorgehen die immense Leistungsfahigkeit moderner Grafikkartengenerationen fur die Berechnungen herangezogen. Das sogenannte GPGPU-Computing eignet sich insbesondere fur die Anwendung der Lattice-Boltzmann-Methode im Bereich numerischer Stromungssimulationen. Auf Grundlage des LBM-Verfahrens wird im Rahmen dieser Arbeit prototypisch eine interaktive Simulationsumgebung basierend auf dem Computational Steering-Paradigma entwickelt, das alle Prozesse zur Losung von Stromungsproblemen innerhalb einer einzelnen Anwendung integriert. Durch die Konvergenz der hohen massiv parallelen Rechenleistung der GPUs und der Interaktionsfahigkeiten in einer einzelnen Anwendung kann eine erhebliche Steigerung der Anwendungsqualitat erzielt werden. Dabei ist es durch Einsatz mehrerer GPUs moglich, dreidimensionale Stromungsprobleme mit praxisrelevanter Problemgrose zu berechnen und gleichzeitig eine interaktive Manipulation und Exploration des Stromungsgebiets zur Laufzeit zu ermoglichen. Dabei ist der erforderliche finanzielle Aufwand verglichen mit traditionellen massiv parallelen Verfahren verhaltnismasig gering. Memory-bound algorithms show complex performance and energy consumption behavior on multicore processors. We choose the lattice-Boltzmann method (LBM) on an Intel Sandy Bridge cluster as a prototype scenario to investigate if and how single-chip performance and power characteristics can be generalized to the highly parallel case. First we perform an analysis of a sparse-lattice LBM implementation for complex geometries. Using a single-core performance model, we predict the intra-chip saturation characteristics and the optimal operating point in terms of energy to solution as a function of implementation details, clock frequency, vectorization, and number of active cores per chip. We show that high single-core performance and a correct choice of the number of active cores per chip are the essential optimizations for lowest energy to solution at minimal performance degradation. Then we extrapolate to the MPI-parallel level and quantify the energy-saving potential of various optimizations and execution modes, where we find these guidelines to be even more important, especially when communication overhead is non-negligible. In our setup we could achieve energy savings of 35 in this case, compared to a naive approach. We also demonstrate that a simple non-reflective reduction of the clock speed leaves most of the energy saving potential unused. Lattice Boltzmann Methods (LBM) are used for the computational simulation of Newtonian fluid dynamics. LBM-based simulations are readily parallelizable; they have been implemented on general-purpose processors, field-programmable gate arrays (FPGAs), and graphics processing units (GPUs). Of the three methods, the GPU implementations achieved the highest simulation performance per chip. With memory bandwidth of up to 141 GB s and a theoretical maximum floating point performance of over 600 GFLOPS, CUDA-ready GPUs from NVIDIA provide an attractive platform for a wide range of scientific simulations, including LBM. This paper improves upon prior single-precision GPU LBM results for the D3Q19 model by increasing GPU multiprocessor occupancy, resulting in an increase in maximum performance by 20 , and by introducing a space-efficient storage method which reduces GPU RAM requirements by 50 at a slight detriment to performance. Both GPU implementations are over 28 times faster than a single-precision quad-core CPU version utilizing OpenMP. Classic vector systems have all but vanished from recent TOP500 lists. Looking at the recently introduced NEC SX-9 series, we benchmark its memory subsystem using the low level vector triad and employ the kernel of an advanced lattice Boltzmann flow solver to demonstrate that classic vectors still combine excellent performance with a well-established optimization approach. To investigate the multi-node performance, the flow field in a real porous medium is simulated using the hybrid MPI OpenMP parallel ILBDC lattice Boltzmann application code. Results for a commodity Intel Nehalem-based cluster are provided for comparison. Clusters can keep up with the vector systems, however, require massive parallelism and thus much more effort to provide a good domain decomposition. Abstract This report presents a comprehensive survey of the effect of different data layouts on the single processor performance characteristics for the lattice Boltzmann method both for commodity “off-the-shelf” (COTS) architectures and tailored HPC systems, such as vector computers. We cover modern 64-bit processors ranging from IA32 compatible (Intel Xeon Nocona, AMD Opteron), superscalar RISC (IBM Power4), IA64 (Intel Itanium 2) to classical vector (NEC SX6+) and novel vector (Cray X1) architectures. Combining different data layouts with architecture dependent optimization strategies we demonstrate that the optimal implementation strongly depends on the architecture used. In particular, the correct choice of the data layout could supersede complex cache-blocking techniques in our kernels. Furthermore our results demonstrate that vector systems can outperform COTS architectures by one order of magnitude. Several possibilities exist to implement the propagation step of lattice Boltzmann methods. This paper describes common implementations and compares the number of memory transfer operations they require per lattice node update. A performance model based on the memory bandwidth is then used to obtain an estimation of the maximum achievable performance on different machines. A subset of the discussed implementations of the propagation step are benchmarked on different Intel- and AMD-based compute nodes using the framework of an existing flow solver that is specially adapted to simulate flow in porous media, and the model is validated against the measurements. Advanced approaches for the propagation step like ''A-A pattern'' or ''Esoteric Twist'' require more programming effort but often sustain significantly better performance than non-naive but straightforward implementations.