<jupyter_start><jupyter_text>Message Passing InterfaceHow do we realize practically this parallelism?Let us focus on what we have discussed until now:* We have ``machines'' with multiple processors and whose main memory is partitioned into fragmented components,* We have algorithms that can divide a problem of size $N$ among these processors so that they can run (almost) independently,* With a certain degree of approximation, we know how to compute what is the *best improvement* we can expect from a parallel program with $M$ processors on a problem of size $N$.What we need to discuss now is then: "*How can we actually implement these algorithms on real machines?*"* We need a way to define a parallel environment in which every processor is accounted for,* We need to have data formats that are aware of the fact that we have a *distributed* memory,* We need to exchange data between the various memory fragments.>"MPI (Message Passing Interface) is a **specification for a standard library** for message passing that was defined by the MPI Forum, a broadly based group of parallel computer vendors, library writers, and applications specialists." -- <NAME>, <NAME>, <NAME>, <NAME>,'' -- A high-performance, portable implementation of the MPI message passing interface standard, Parallel Computing, 22 (6), 1996.* MPI implementations consist of a specific set of routines directly callable from C, C++, Fortran, Python;* MPI uses Language Independent Specifications for calls and language bindings;* The MPI interface provides an essential *virtual* topology, synchronization, and communication functionality inside a set of processes.* There exist many implementations of the MPI specification, e.g., MPICH, Open MPI, *etc.*  Our First MPI ProgramIn all the course we are going to use the MPI inside Python programs.Let us start from the classical helloworld program:<jupyter_code>%%file ccode/helloworld.c
#include"mpi.h"
#include<stdio.h>

int main(int argc,char **argv){
 MPI_Init( &argc, &argv);
 printf("Hello, world!\n");
 MPI_Finalize();
 return 0;
}<jupyter_output>Overwriting ccode/helloworld.c<jupyter_text>We can compile it by doing``` bashmpicc helloworld.c -o helloworld```- `mpicc` is a wrapper for a C compiler provided by the Open MPI implementation of MPI.- the option `-o` sets the name of the compiled (executable) file.Let us see what is happening behind the curtains- you can first try to discover what compiler are you using by executing ```bashmpicc --version```that will give you something like```gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0Copyright (C) 2017 Free Software Foundation, Inc.```- or discover what are the library inclusion and linking options by asking for `mpicc --showme:compile` and `mpicc --showme:link`, respectively.- In general, looking at the output of the `man mpicc` command is always a good idea.> ``If you find yourself saying, "But I don't want to use wrapper compilers!", please humor us and try them. See if they work for you. Be sure to let us know if they do not work for you. '' - [https://www.open-mpi.org/faq/?category=mpi-apps](https://www.open-mpi.org/faq/?category=mpi-apps)```{note}A piece of advice: if your program is anything more realistic than a classroom exercise use `make`[^1], and save yourself from writing painfully long compiling commands, and dealing with complex dependencies more than once.```> "Make gets its knowledge of how to build your program from a file called the makefile, which lists each of the non-source files and how to compute it from other files."A very simple `Makefile` for our first test would be``` makeMPICC = mpicc The wrapper for the compilerCFLAGS += -g  Useful for debug symbolsall: helloworldhelloworld: helloworld.c  $(MPICC) $(CFLAGS) $(LDFLAGS) $? $(LDLIBS) -o $@clean:  rm -f helloworld```[^1]:[https://www.gnu.org/software/make/](https://www.gnu.org/software/make/)Let us run our first parallel program by doing:```bashmpirun [ -np X ] [ --hostfile  ]  python helloworld.py```or by using its synonym```bashmpiexec [ -np X ] [ --hostfile  ] python helloworld.py```* `mpiexec` will  run  `X` copies of `helloworld` in your current run-time environment, scheduling (by default) in a round-robin fashion by CPU slot.* if running under a supported resource manager, Open MPI's `mpirun` will usually automatically use the corresponding resource manager process starter, as opposed to, for example, rsh or ssh, which require the use of a hostfile, or will default  to  running all `X` copies on the localhost * as always, look at the manual, by doing `man mpirun`.<jupyter_code>!(cd ccode && make helloworld)
!mpiexec -np 4 ./ccode/helloworld<jupyter_output>/home/cirdan/anaconda3/envs/parallel/bin/x86_64-conda_cos6-linux-gnu-cc -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/cirdan/anaconda3/envs/parallel/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/cirdan/anaconda3/envs/parallel/include -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/cirdan/anaconda3/envs/parallel/lib -Wl,-rpath-link,/home/cirdan/anaconda3/envs/parallel/lib -L/home/cirdan/anaconda3/envs/parallel/lib  helloworld.c   -o helloworld
/home/cirdan/anaconda3/envs/parallel/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: /tmp/cc0XAkgc.o: in function `main':
helloworld.c:(.text.startup.main+0x16): undefined reference to `MPI_Init'
/home/cirdan/anaconda3/envs/parallel/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: he[...]<jupyter_text>Every process executes the line that it is a **local** routine!> A procedure is **local** if completionof the procedure depends only on the local executing process.>A procedure is **non-local** if completion of the operation may requirethe execution of some MPI procedure on another process. Such anoperation *may require communication* occurring with another userprocess.  The MPI parallel environmentThe MPI parallel environment Let us modify our `helloworld` toinvestigate the MPI parallel environment. Specifically, we want toanswer, from within the program, to the questions:1.  How many processes are there?2.  Who am I?<jupyter_code>%%file ccode/hamlet.c
#include "mpi.h"
#include <stdio.h>
int main( int argc, char **argv ){
 int rank, size;
 MPI_Init( &argc, &argv );
 MPI_Comm_rank( MPI_COMM_WORLD, &rank );
 MPI_Comm_size( MPI_COMM_WORLD, &size );
 printf( "Hello world! I'm process %d of %d\n",rank, size );
 MPI_Finalize();
 return 0;
}<jupyter_output>Overwriting ccode/hamlet.c<jupyter_text>-   How many is answered by a call to `MPI_Comm_size` as an    `int` value,-   Who am I? Is answered by a call to `MPI_Comm_rank` as an    `int` value that is conventionally called `rank` and is a    number between `0` and `size-1`.The MPI parallel environment The last keyword we need to describe is the`MPI_COMM_WORLD`, this is the standard Communicator object.> **Communicator:** A Communicator object connects a group of processes in oneMPI session. There can be more than one communicator in an MPI session,each of them gives each contained process an independent identifier andarranges its contained processes in an ordered topology.This provides-   a safe communication space, that guarantees that the code can    communicate as they need to, without conflicting with communication    extraneous to the present code, e.g., if other parallel libraries    are in use,-   a unified object for conveniently denoting communication context,    the group of communicating processes and to house abstract process    naming.The MPI parallel environment If we have saved our inquiring MPI programin the file `hamlet.c`, we can then modify our `Makefile`by modifying/adding the lines``` Makefileall: helloworld hamlethamlet: hamlet.c $(MPICC) $(CFLAGS) $(LDFLAGS) $? $(LDLIBS) -o $@clean: rm -f helloworld hamlet```Then, we compile everything by doing `make hamlet` (or, simply,`make`).<jupyter_code>!(cd ccode && make hamlet)
!mpiexec -np 6 ./ccode/hamlet<jupyter_output>mpicc			 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/cirdan/anaconda3/envs/parallel/include -g			 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/cirdan/anaconda3/envs/parallel/lib -Wl,-rpath-link,/home/cirdan/anaconda3/envs/parallel/lib -L/home/cirdan/anaconda3/envs/parallel/lib hamlet.c  -o hamlet
Hello world! I'm process 0 of 6
Hello world! I'm process 1 of 6
Hello world! I'm process 2 of 6
Hello world! I'm process 5 of 6
Hello world! I'm process 3 of 6
Hello world! I'm process 4 of 6<jupyter_text>We can rewrite the same code in Python as<jupyter_code>%%file hamlet.py
"""
Hello (parallel) world!
"""
from mpi4py import MPI

comm = MPI.COMM_WORLD 
rank = comm.Get_rank() 
size = comm.Get_size() 

print("Hello world! I'm process ",rank," of ",size)<jupyter_output>Overwriting hamlet.py<jupyter_text>What have we done here:* The instruction ```pythonfrom mpi4py import MPI```provides basic MPI definitions and types, if this was a `C` code, this would have been a *preprocessor* directive of the form `include "mpi.h"`* start MPI by creating a communicator ```pythoncomm = MPI.COMM_WORLD```For the Python code* How many is answered by a call to `comm.Get_size()` as an `int` value,* Who am I? Is answered by a call to `comm.Get_rank()` as an `int` value that is conventionally called **rank** and is a number between `0` and `size-1`.<jupyter_code>!mpiexec -n 4 python hamlet.py<jupyter_output>Hello world! I'm process  0  of  4
Hello world! I'm process  1  of  4
Hello world! I'm process  2  of  4
Hello world! I'm process  3  of  4<jupyter_text>* Every processor answers the call,* But it answers it as soon as he has done doing the computation! There is **no synchronization**.  Point-to-point communicationSending and Receiving Messages We have seen that each process within a*communicator* is identified by its *rank*, how can we [exchangedata]{.alert} between two processes?We need to posses several information to have a meaningful message-   Who is sending the data?-   To whom the data is sent?-   What type of data are we sending?-   How does the receiver can identify it? The blocking send and receive```cint MPI_Send(void *message, int count,     MPI_Datatype datatype, int dest, int tag,     MPI_Comm comm)```- `void *message`:   points to the message content itself, it can be a simple scalar or a    group of data,- `int count`:   specifies the number of data elements of which the message is    composed,- `MPI_Datatype datatype`:   indicates the [data type]{.alert} of the elements that make up the    message,- `int dest`:   the rank of the destination process,- `int tag`:   the user-defined tag field,- `MPI_Comm comm`:   the communicator in which the source and destination processes    reside and for which their respective ranks are defined.```cint MPI_Recv (void *message, int count,     MPI_Datatype datatype, int source, int tag,    MPI_Comm comm, MPI_Status *status)```- `void *message`:   points to the message content itself, it can be a simple scalar or a    group of data,- `int count`:   specifies the number of data elements of which the message is    composed,- `MPI_Datatype datatype`:   indicates the [data type]{.alert} of the elements that make up the    message,- `int dest`:   the rank of the source process,- `int tag`:   the user-defined tag field,- `MPI_Comm comm`:   the communicator in which the source and destination processes    reside,- `MPI_Status *status`:   is a structure that contains three fields named `MPI_SOURCE` ,    `MPI_TAG`, and `MPI_ERROR`.Basic MPI Data Types Of the inputs in the previous slides the only onethat is specific to MPI is the `MPI_Datatype`, these corresponds toa C data typeMPI Data Types | C Type---------------------------|-------------------------`MPI_CHAR`             |`signed char``MPI_SHORT`            |`signed short int``MPI_INT`              |`signed int``MPI_LONG`             |`signed long int``MPI_FLOAT`            |`float``MPI_DOUBLE`           |`double``MPI_LONG_DOUBLE`      |`long double``MPI_UNSIGNED_CHAR`    |`unsigned char``MPI_UNSIGNED_SHORT`   |`unsigned short int``MPI_UNSIGNED`         |`unsigned int``MPI_UNSIGNED_LONG`    |`unsigned long int`**Note:** we will see in the following how to`send`/`receive` user--defined data structures.Why "blocking" send and receive? For the `MPI_Send` to be**blocking** means that it does not return until the message dataand envelope have been safely stored away so that the sender is free tomodify the send buffer: it is a *non local* operation.**Note:** The message might be copied directly into thematching receive buffer (as in the first figure), or it might be copiedinto a temporary system buffer.  A simple send/receive exampleIf we want to test these two instructions we can write the following simple C program.<jupyter_code>%%file ccode/easysendrecv.c
#include "mpi.h"
#include <string.h>
#include <stdio.h>
int main( int argc, char **argv){
 char message[20];
 int myrank;
 MPI_Status status;
 MPI_Init( &argc, &argv );
 MPI_Comm_rank( MPI_COMM_WORLD, &myrank );
 if (myrank == 0){  /* code for process zero */
  strcpy(message,"Hello, there");
  MPI_Send(message, strlen(message)+1, MPI_CHAR, 1, 99, MPI_COMM_WORLD);
 }
 else if (myrank == 1){ /* code for process one */
  MPI_Recv(message, 20, MPI_CHAR, 0, 99, MPI_COMM_WORLD, &status);
  printf("received :%s:\n", message);
 }
 MPI_Finalize();
 return 0;
}<jupyter_output>Overwriting ccode/easysendrecv.c<jupyter_text>That could be recasted in Python by doing<jupyter_code>%%file easysendrecv.py
"""
A simple send/receive example
"""
from mpi4py import MPI

comm = MPI.COMM_WORLD 
rank = comm.Get_rank() 
size = comm.Get_size() 

if rank == 0:
    data = "Hello, there"
    comm.send(data, dest=1, tag=99)
elif rank == 1:
    data = comm.recv(source=0, tag=99)
    print('received :',data)<jupyter_output>Overwriting easysendrecv.py<jupyter_text>That we can run as the simpler program by doing:<jupyter_code>!mpiexec -np 2 python easysendrecv.py<jupyter_output>received : Hello, there<jupyter_text>for the Python version, or the following for the C version<jupyter_code>!(cd ccode && make easysendrecv)
!mpiexec -np 2 ./ccode/easysendrecv<jupyter_output>mpicc			 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/cirdan/anaconda3/envs/parallel/include -g			 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/cirdan/anaconda3/envs/parallel/lib -Wl,-rpath-link,/home/cirdan/anaconda3/envs/parallel/lib -L/home/cirdan/anaconda3/envs/parallel/lib easysendrecv.c  -o easysendrecv
received :Hello, there:<jupyter_text>So, what have we done? Process $0$ sends the content of the `char`array `message[20]`, whose size is `strlen(message)+1` size of`char` (`MPI_CHAR`) to processor `1` with tag `99` onthe communicator `MPI_COMM_WORLD`. on the other side process $1$,receives into the buffer `message[20]` an array with size `20`size of `MPI_CHAR`, from process `0` with tag `99` on thesame communicator `MPI_COMM_WORLD`.Observe that in the Python case we did not declare the size or the type of the object we were passing. The all-lowercase methods (of the `Comm` class), like `send()`, `recv()`, work by passing an object to be sent as a paramenter to the communication call, and the received object is simply the return value. These variants can communicate general Python objects.In MPI for Python, the `MPI.Comm.Send()`, `MPI.Comm.Recv()` and methods of communicator objects provide support for blocking point-to-point communications and can be used to communicate memory buffers, as we do in the C variant. Consider the following example sending a `numpy` array between two processes.<jupyter_code>%%file easysendrecv2.py
"""
A (slightly less) simple send/receive example
In which we :
- pass MPI datatypes explicitly
- use the MPI datatype discovery
"""
from mpi4py import MPI
import numpy

comm = MPI.COMM_WORLD
rank = comm.Get_rank()

if rank == 0:
    data = numpy.arange(1000, dtype='i')
    comm.Send([data, MPI.INT], dest=1, tag=77)
elif rank == 1:
    data = numpy.empty(1000, dtype='i')
    comm.Recv([data, MPI.INT], source=0, tag=77)

if rank == 0:
    data = numpy.arange(100, dtype=numpy.float64)
    comm.Send(data, dest=1, tag=13)
elif rank == 1:
    data = numpy.empty(100, dtype=numpy.float64)
    comm.Recv(data, source=0, tag=13)<jupyter_output>Overwriting easysendrecv2.py<jupyter_text>In general, buffer arguments to these calls must be explicitly specified by using a 2/3-list/tuple like `[data, MPI.DOUBLE]`, or `[data, count, MPI.DOUBLE]` (the former one uses the byte-size of data and the extent of the MPI datatype to define count).<jupyter_code>!mpiexec -np 2 python easysendrecv2.py<jupyter_output><empty_output><jupyter_text>The perform the datatype discovery, Python uses the `pickle` module. This module implements binary protocols for serializing and de-serializing a Python object structure. "Pickling" is the process converting a Python object hierarchy into a byte stream, and "unpickling" is the inverse operation, converting a byte stream (from a binary file or bytes-like object) into an object hierarchy.   A simple send/receive example : programmer smash!It is a good exerciseto try and mess things up, so let us see some damaging suggestions (test them with the previous C code):-   What happens if we have a mismatch in the tags?-   **A:** The process stays there hanging waiting for a message with a tag    that will never come...-   What happens if we have a mismatch in the ranks of the sending and    receiving processes?-    **A:** The process stays there hanging trying to match messages that will    never come...-   What happens if we use the wrong message size?-    **A:** If the size of the arriving message is longer than the expected we    get an error of `MPI_ERR_TRUNCATE: message truncated`, note    that there are combinations of wrong sizes for which things still    works-   What happens if we have a mismatch in the type?-   **A:** There are combinations of instances in which things seems to work,    **but** the code is erroneous, and the behavior is not    deterministic.  DeadlockWe have now two processes that needs to exchange some data.-   Solution 1:``` cMPI_Comm_rank(comm, &myrank);if (myrank == 0){ MPI_Send(sendbuf, count, MPI_DOUBLE, 1, tag, comm); MPI_Recv(recvbuf, count, MPI_DOUBLE, 1, tag, comm, status);}else if(myrank == 1){ MPI_Send(sendbuf, count, MPI_DOUBLE, 0, tag, comm);  MPI_Recv(recvbuf, count, MPI_DOUBLE, 0, tag, comm, status);}```-   Solution 2:``` cMPI_Comm_rank(comm, &myrank);if (myrank == 0){ MPI_Recv(recvbuf, count, MPI_DOUBLE, 1, tag, comm, status); MPI_Send(sendbuf, count, MPI_DOUBLE, 1, tag, comm);}else if(myrank == 1){  MPI_Recv(recvbuf, count, MPI_DOUBLE, 0, tag, comm, status); MPI_Send(sendbuf, count, MPI_DOUBLE, 0, tag, comm);}```-   Solution 3:``` cMPI_Comm_rank(comm, &myrank);if (myrank == 0){ MPI_Send(sendbuf, count, MPI_DOUBLE, 1, tag, comm); MPI_Recv(recvbuf, count, MPI_DOUBLE, 1, tag, comm, status);}else if(myrank == 1){ MPI_Recv(recvbuf, count, MPI_DOUBLE, 0, tag, comm, status); MPI_Send(sendbuf, count, MPI_DOUBLE, 0, tag, comm); }```In the case of Solution 1:``` cMPI_Comm_rank(comm, &myrank);if (myrank == 0){ MPI_Send(...); MPI_Recv(...);}else if(myrank == 1){ MPI_Send(...);  MPI_Recv(...);}```-   The call `MPI_Send` is blocking, therefore the message sent by    each process has to be copied out before the send operation returns    and the receive operation starts.-   For the call to complete successfully, it is then necessary that **at    least one of the two messages sent be buffered**, otherwise    ...-   a deadlock situation occurs: both processes are blocked since there    is no buffer space available!In the case of Solution 2:``` cMPI_Comm_rank(comm, &myrank);if (myrank == 0){ MPI_Recv(...); MPI_Send(...);}else if(myrank == 1){ MPI_Recv(...); MPI_Send(...); }```-   The receive operation of process $0$ must complete before its send.    It can complete **only if** the matching send of processor $1$    is executed.-   The receive operation of process $1$ must complete before its send.    It can complete **only if** the matching send of processor $0$    is executed.-   This program will always deadlock.In the case of Solution 3:``` cMPI_Comm_rank(comm, &myrank);if (myrank == 0){ MPI_Send(...); MPI_Recv(...);}else if(myrank == 1){ MPI_Recv(...); MPI_Send(...); }```-   This program will succeed even if no buffer space for data is    available.     Deadlock IssuesWe can try to salvage what the situation in the case of Solution 1 byallocating the buffer space for the send calls``` cif (myrank == 0){ MPI_Send(...); MPI_Recv(...);}else if(myrank == 1){ MPI_Send(...);  MPI_Recv(...);}```We can substitute the `MPI_Send` operation with a Send inbuffered mode``` cint MPI_Bsend(const void* buf, int count,   MPI_Datatype datatype, int dest,  int tag, MPI_Comm comm)```-   A buffered mode send operation can be started whether or not a    matching receive has been posted;-   It may complete before a matching receive is posted;-   This operation is *local*!Allocating buffer space To actually use the `MPI_Bsend` we need alsoto allocate the space for the buffer, therefore we need to use the twofunctions``` cdefine BUFFSIZE 10000int size; char *buff;// Buffer of 10000 bytes for MPI_BsendMPI_Buffer_attach( malloc(BUFFSIZE), BUFFSIZE);// Buffer size reduced to zero MPI_Buffer_detach( &buff, &size);// Buffer of 10000 bytes available again MPI_Buffer_attach( buff, size); `````` {Warning}a pointer to the buffer is passed to`MPI_Buffer_attach` while the address of the pointer is passed to`MPI_Buffer_detach` and these are both `void *`.``` Nonblocking communicationsNonblocking communications As we have seen the use of blockingcommunications ensures that-   the send and receive buffers used in the `MPI_Send` and    `MPI_Recv` arguments are safe to use or reuse after the function    call,-   but it also means that unless there is a simultaneously matching    send for each receive, the code will deadlock.There exists a version of the point-to-point communication that **returnsimmediately** from the function call before confirming that thesend or the receive has completed, these are the nonblocking send andreceive functions.-   To verify that the data has been copied out of the send buffer a    separate call is needed,-   To verify that the data has been received into the receive buffer a    separate call is needed,-   The sender should not modify any part of the send buffer after a    nonblocking send operation is called, until the send completes.-   The receiver should not access any part of the receive buffer after    a nonblocking receive operation is called, until the receive    completes.Nonblocking communications: `MPI_Isend` and `MPI_Irecv` The twononblocking point-to-point communication call are then``` cint MPI_Isend(void *message, int count,    MPI_Datatype datatype, int dest, int tag,   MPI_Comm comm, MPI_Request *send_request);int MPI_Irecv(void *message, int count,    MPI_Datatype datatype, int source, int tag,   MPI_Comm comm, MPI_Request *recv_request);```-   The `MPI_Request` variables substitute the `MPI_Status` and    store information about the status of the pending communication    operation.-   The way of saying when this communications **must** be    completed is by using the when is called, the nonblocking request    originating from `MPI_Isend` or `MPI_Irecv` is provided as    an argument.Nonblocking communications: an example<jupyter_code>%%file ccode/nonblockingsendrecv.c
#include <stdio.h>
#include <mpi.h>

int main(int argc, char **argv) {
 int a, b, size, rank, tag = 0; 
 MPI_Status status;
 MPI_Request send_request, recv_request;
 MPI_Init(&argc, &argv);
 MPI_Comm_size(MPI_COMM_WORLD, &size);
 MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (rank == 0) {
 a = 314159; 
 MPI_Isend(&a, 1, MPI_INT, 1, tag, MPI_COMM_WORLD, &send_request);
 MPI_Irecv (&b, 1, MPI_INT, 1, tag, MPI_COMM_WORLD, &recv_request);
 MPI_Wait(&send_request, &status);
 MPI_Wait(&recv_request, &status);
 printf ("Process %d received value %d\n", rank, b);
} else {
 a = 667;
 MPI_Isend (&a, 1, MPI_INT, 0, tag, MPI_COMM_WORLD, &send_request);
 MPI_Irecv (&b, 1, MPI_INT, 0, tag, MPI_COMM_WORLD, &recv_request);
 MPI_Wait(&send_request, &status);
 MPI_Wait(&recv_request, &status);
 printf ("Process %d received value %d\n", rank, b);
}
 MPI_Finalize();
 return 0;
}<jupyter_output>Writing ccode/nonblockingsendrecv.c<jupyter_text>A simple send/receive example We can compile our code by simply addingto our `Makefile```` Makefilenonblockingsendrecv: nonblockingsendrecv.c  $(MPICC) $(CFLAGS) $(LDFLAGS) $? $(LDLIBS) -o $@```then, we type `make nonblockingsendrecv`, and we run our program with getting asanswer<jupyter_code>!(cd ccode && make nonblockingsendrecv)
!mpiexec -np 2 ./ccode/nonblockingsendrecv<jupyter_output>mpicc			 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/cirdan/anaconda3/envs/parallel/include -g			 -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/cirdan/anaconda3/envs/parallel/lib -Wl,-rpath-link,/home/cirdan/anaconda3/envs/parallel/lib -L/home/cirdan/anaconda3/envs/parallel/lib nonblockingsendrecv.c  -o nonblockingsendrecv
Process 0 received value 667
Process 1 received value 314159
{'fraction_non_alphanumeric': 0.10687613667143907, 'fraction_numerical': 0.011182912200595906, 'mean_word_length': 4.548303993130099, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 60, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '381729', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 8344, 'n_tokens_neox': 7808, 'n_words': 3185, 'path': 'intro.ipynb', 'repo_name': 'Cirdans-Home/intrompi', 'size': '25852'}