<jupyter_start><jupyter_text>Learning objectivesIn this lab we will learn about:* Multi-node Multi-GPU programming and importance of inter-process communication frameworks.* Introduction to MPI specification and APIs.* Execution of Hello World MPI binary on single as well as multiple nodes. Multi-Node Multi-GPU ProgrammingAs we move from a single node to multiple nodes, the basic multi-GPU programming concepts like domain decomposition and application-specific concepts like halo exchange remain the same. However, the communication becomes complex.A single process can spawn threads that can be spread within a node (potentially on multiple sockets) but it cannot cross the node boundary. Thus, scalable multi-node programming requires the use of multiple processes.Inter-process communication is usually done by libraries like OpenMPI. They expose communication APIs, synchronization constructs, etc. to the user. Let us now learn about programming in MPI. MPIMPI is a specification for the developers and users of message passing libraries. By itself, it is not a library - but rather the specification of what such a library should be. An example of MPI-compliant library is OpenMPI.It primarily addresses the message-passing parallel programming model: data is moved from the address space of one process to that of another process through cooperative operations on each process.MPI is widely used in practice for HPC applications, in academia, government agencies, and industry alike. In this lab, while we will introduce its APIs, a working understanding of MPI is highly desirable. A Hello World ExampleA C-based Hello World program is shown below:```cinclude include int main(int argc, char** argv) {    // Initialize the MPI environment    MPI_Init(NULL, NULL);    // Get the number of processes    int size;    MPI_Comm_size(MPI_COMM_WORLD, &size);    // Get the rank of the process    int rank;    MPI_Comm_rank(MPI_COMM_WORLD, &rank);    // Get the name of the processor    char processor_name[MPI_MAX_PROCESSOR_NAME];    int name_len;    MPI_Get_processor_name(processor_name, &name_len);    // Print a hello world message    printf("Hello world from processor %s, rank %d out of %d processors\n",           processor_name, rank, size);    // Finalize the MPI environment.    MPI_Finalize();}```To access the program, open the [hello_world.c](../../source_code/mpi/hello_world.c) file. Alternatively, you can navigate to `CFD/English/C/source_code/mpi/` directory in Jupyter's file browser in the left pane. Then, click to open the `hello_world.c` file.The MPI environment is initialized with `MPI_Init` through which all of MPI’s global and internal variables are constructed. A "communicator" is created between all processes that are spawned, and unique ranks are assigned to each process. `MPI_Comm_size` returns the size of a communicator, that is, the number of processes within that communicator. In our example, this call will return the number of processes requested for the job.`MPI_Comm_rank` returns the rank of a process in a communicator. Each process inside of a communicator is assigned an incremental rank starting from zero. The ranks of the processes are primarily used for identification purposes when sending and receiving messages.`MPI_Get_processor_name` obtains the name of the processor on which the process is executing and `MPI_Finalize` is used to clean up the MPI environment. No more MPI calls can be made after this call. Running MPI with or without containers**We will run MPI directly on compute nodes without using containers.** The subsequent sections assume that atleast 2 compute nodes with multiple GPUs in each node are available to the user. All our codes have been tested with CUDA-aware OpenMPI v4.1.1 with supporting libraries HPCX v2.8.1 (for UCX and HCOLL) and CUDA v11.3.0.0 on DGX-1 8 Tesla V100 compute nodes. CUDA-awareness as a concept in MPI will be explained in subsequent labs.Usually, a cluster workload manager like Slurm or PBS is present and integrated with MPI installation to launch multi-node jobs. We use `mpirun` command to run MPI assuming that the user is logged into an interactive shell with multiple nodes allocated. The other common way is to use workload manager commands like `srun` (for Slurm) directly to run MPI jobs as they are integrated with MPI internally. **Note:** We do outline the method to build and run containerized MPI using Singularity in tandem with host MPI implementation in our supplemental notebook: [MPI in a containerized environment](./containers_and_mpi.ipynb).  CompilationThe `mpicc` and `mpic++` (or `mpicxx`) compilers are used to compile and link programs with MPI. We can compile the Hello World program with the command:```bashmpicc -o hello_world hello_world.c```Ensure that MPI is installed (for exmaple, if it is built from source) and available (for example, if loaded as a module) using the folllowing command:<jupyter_code>! mpirun --version<jupyter_output><empty_output><jupyter_text>Now, let us compile the program:<jupyter_code>! cd ../../source_code/mpi && make clean && make hello_world<jupyter_output><empty_output><jupyter_text>ExecutionWe run the program using the `mpirun` command as follows:```bashmpirun -np  -npersocket  -hostfile  ./hello_world```The `-np` option specifies the total number of processes spawned by MPI runtime and `-npersocket` option specifies the number of processes to be spawned on each socket. The `-hostfile` option allows us to specify which hosts (compute nodes) to start MPI processes on. The file is a newline-separated list of hostnames which must be accessible to each other so that MPI processes can communicate.Note that DGX-1V is a dual-socket system and `` should be less than or equal to number of cores in that socket. Clearly, ``$\div$(`procs_per_socket`$\times$``) is the number of nodes used. There are several other options available to specify `` that will be discussed in subsequent labs. As we are using an OpenMPI implementation in a workload manager-based environment, the `` will be provided by Slurm and we don't need to specify this option.There are numerous other configuration options that one can overview using the `mpirun --help` command. You can check the number of sockets and cores per socket in your machine (the whole node) with the command `lscpu | grep -E 'Socket|Core'`.  Single NodeRun the program binary on a single node:<jupyter_code>! cd ../../source_code/mpi && mpirun -np 2 -npersocket 1 ./hello_world<jupyter_output><empty_output><jupyter_text>You may see some warnings. As long as the output is printed, you can ignore the warnings. In the output, you should see 2 unique ranks ranging (0 and 1) and the node's name like below:```bashHello world from processor , rank 0 out of 2 processorsHello world from processor , rank 1 out of 2 processors``` Multiple NodesLet us now run the Hello World program on 2 nodes with the following command:<jupyter_code>! cd ../../source_code/mpi && mpirun -np 4 -npersocket 1 ./hello_world<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.058923512747875356, 'fraction_numerical': 0.00424929178470255, 'mean_word_length': 4.768790849673203, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '455083', 'license': 'Apache-2.0', 'n_tokens_mistral': 1850, 'n_tokens_neox': 1732, 'n_words': 988, 'path': 'hpc/multi_gpu_nways/labs/CFD/English/C/jupyter_notebook/mpi/multi_node_intro.ipynb', 'repo_name': 'AditiRM/gpubootcamp', 'size': '7060'}