<commit_before>Implementation notes
--------------------

status_error
------------
Though Status has `status_error' field, it could be used with care:
"The error field in status is not needed for calls that return only
one status, such as MPI_WAIT, since that would only duplicate the
information returned by the function itself. The current design avoids
the additional overhead of setting it, in such cases. The field is
needed for calls that return multiple statuses, since each request may
have had a different failure." 
(quote from http://mpi-forum.org/docs/mpi22-report/node47.htm#Node47)

This means that, for example, during the call to MPI_Wait
implementation is free to leave this field filled with whatever
garbage got there during memory allocation. Haskell bindings are not
blanking out freshly allocated status for now either. So beware!

Test.Runner vs TestFramework
----------------------------
In order to be able to debug MPI bindings testsuite on a single-node
MPI installation one has to be able to separate output from processes
of different rank.

OpenMPI allows to do so via the --output-filename switch of mpirun,
but MPICH2 does not have similar feature. And since most of the output
in the testsuite is done from inside test harness library, there is
very little control over output.

Obvious solution would be to redirect stdout of the process to some
other file handle via dup2(2). However, there are several downsides:
1. Binding for dup2 (hDuplicateTo) is a GHC-only solutions
2. TestFramework does not play well with this solution, shutting
   output completely when stdout is redirected (probably "ncurses" is
   disappointed to find that output is not a terminal anymore)

Nevertheless, I decided to stick to hDuplicateTo and ditch
TestFramework in favor of TestRunner, since it allows for consistent
experience across MPI implementations.

Code coverage analysis
----------------------
It's very nice to have code coverage report for testsuite to make sure
that no major piece of code is left untested. However, current
profiling mechanism does not play well with MPI: when mpirun starts
two processes (on the single node), they both try to run to the same
.tix file at once. Mayhem ensues.

In order to fix this, Testsuite.hs has been made to depend on hpc
package, and after all tests has been run, HPC API is instructed to
write tix data to files rank<n>.tix.

Command line tool "hpc" could then be used to combine those into
single .tix file, which could be used to produce code coverage report.
Simple script "bin/coverage.sh" does all this automatically. Note:
script should be run from the toplevel project dir (where
bindings-mpi.cabal is residing).

Collective operations are split
-------------------------------
Collective operations in MPI usually take a large set of arguments
that include pointers to both input and output buffers. This fits
nicely in the C programming style:
* Pointers to send and receive buffers are declared
* if (my_rank == root) then (send buffer is allocated and filled)
* both pointers are passed to collective function, which ignores
  unallocated send buffer for all non-root processes.

In Haskell there is no simple way to declare pointers beforehand and
allocate memory for them to point to later. Plus, this would make for
a lousy programming style. Plus, we wanted to encourage users to
partially apply API calls both for convenience (see below) and for
simplifying reuse of already-allocated arrays (also see below).

Therefore it was decided to split most asymmetric collective calls in
two parts - sending and receiving. Thus "gather" become "sendGather"
and "recvGather", etc.

Order of arguments and terseness of programs
--------------------------------------------
It seems that very ofter programmer finds himself in the situation
where he wants to send/receive messages between the same processess
over and over again. This is true for both point-to-point modes of
communication and for collective operations.

Which is why we've chosen to change the order of arguments in most API
calls. We rearranged arguments in a way that would make partial
application of API calls natural. For example, if your rank 0 process
often sends single message msg1 to rank 1 and various messages to rank
2, you could define the following shortcuts:

sendMsg1 = send comm rank1 tag1 msg1
sendTo2 = send comm rank2

In-place receive vs new array allocation for Storable Array
-----------------------------------------------------------
When using StorableArray API in tight numeric loops, it is best to
reuse existing arrays and avoid penalties incurred by
allocation/deallocation of memory. Which is why destinations/receive
buffers in StorableArray API are specified exclusively as
(StorableArray i e).

If you'd rather allocate new array for a particular operation, you
could use withNewArray/withNewArray_:

Instead of (recv comm rank tag arr) you would write 
(arr <- withNewArray bounds $ recv comm rank tag), and new array would
be allocated, supplied as the target of the (recv) operation and
returned to you.

You could easily write your own convenience wrappers similar to
withNewArray. For example, you could create wrapper that would take an
array size as a simple number instead of range.

Rank checking in collective functions (Storable Array)
------------------------------------------------------
Collective operations that are split into separate send/recv parts
(see above) take "root rank" as an argument. Right now no safeguards
are in place to ensure that rank supplied to the send function is
corresponding to the rank of that process. We believe that it does not
worsen the general go-on-and-shoot-yourself-in-the-foot attitide of
the MPI API, but this could be changed in the future.

Planned API coverage
--------------------
Status:
+ - supported in both APIs
A - only in StorableArray API
S - only in Serializable API

First tier:
+    MPI_Send,
+    MPI_Recv
+    MPI_Bcast,
A    MPI_Allgather
+    MPI_Barrier
A    MPI_Scatter,
A    MPI_Gather
A    MPI_Alltoall
+    MPI_Isend,
+    MPI_Wait,
     MPI_Reduce_scatter
+    MPI_Test,
+    MPI_Cancel
     MPI_Wtime,
     MPI_Wtick
     MPI_Iprobe
+    MPI_Ssend
     MPI_Probe

Second tier or under consideration:
     MPI_Op_create
     MPI_Pcontrol
     MPI_Type_commit
     MPI_Pack_external
     MPI_Register_datarep
     MPI_Op_free
     MPI_Query_thread
     MPI_Win_create,
     MPI_Put,
     MPI_Get,
     MPI_Accumulate

How to set up OpenMPI on 2 (3,4,..) nodes?
------------------------------------------
Quick intro for the impatient:
1)Set up OpenMPI on each node<commit_msg>Updated MPI API coverage status<commit_after>Implementation notes
--------------------

status_error
------------
Though Status has `status_error' field, it could be used with care:
"The error field in status is not needed for calls that return only
one status, such as MPI_WAIT, since that would only duplicate the
information returned by the function itself. The current design avoids
the additional overhead of setting it, in such cases. The field is
needed for calls that return multiple statuses, since each request may
have had a different failure." 
(quote from http://mpi-forum.org/docs/mpi22-report/node47.htm#Node47)

This means that, for example, during the call to MPI_Wait
implementation is free to leave this field filled with whatever
garbage got there during memory allocation. Haskell bindings are not
blanking out freshly allocated status for now either. So beware!

Test.Runner vs TestFramework
----------------------------
In order to be able to debug MPI bindings testsuite on a single-node
MPI installation one has to be able to separate output from processes
of different rank.

OpenMPI allows to do so via the --output-filename switch of mpirun,
but MPICH2 does not have similar feature. And since most of the output
in the testsuite is done from inside test harness library, there is
very little control over output.

Obvious solution would be to redirect stdout of the process to some
other file handle via dup2(2). However, there are several downsides:
1. Binding for dup2 (hDuplicateTo) is a GHC-only solutions
2. TestFramework does not play well with this solution, shutting
   output completely when stdout is redirected (probably "ncurses" is
   disappointed to find that output is not a terminal anymore)

Nevertheless, I decided to stick to hDuplicateTo and ditch
TestFramework in favor of TestRunner, since it allows for consistent
experience across MPI implementations.

Code coverage analysis
----------------------
It's very nice to have code coverage report for testsuite to make sure
that no major piece of code is left untested. However, current
profiling mechanism does not play well with MPI: when mpirun starts
two processes (on the single node), they both try to run to the same
.tix file at once. Mayhem ensues.

In order to fix this, Testsuite.hs has been made to depend on hpc
package, and after all tests has been run, HPC API is instructed to
write tix data to files rank<n>.tix.

Command line tool "hpc" could then be used to combine those into
single .tix file, which could be used to produce code coverage report.
Simple script "bin/coverage.sh" does all this automatically. Note:
script should be run from the toplevel project dir (where
bindings-mpi.cabal is residing).

Collective operations are split
-------------------------------
Collective operations in MPI usually take a large set of arguments
that include pointers to both input and output buffers. This fits
nicely in the C programming style:
* Pointers to send and receive buffers are declared
* if (my_rank == root) then (send buffer is allocated and filled)
* both pointers are passed to collective function, which ignores
  unallocated send buffer for all non-root processes.

In Haskell there is no simple way to declare pointers beforehand and
allocate memory for them to point to later. Plus, this would make for
a lousy programming style. Plus, we wanted to encourage users to
partially apply API calls both for convenience (see below) and for
simplifying reuse of already-allocated arrays (also see below).

Therefore it was decided to split most asymmetric collective calls in
two parts - sending and receiving. Thus "gather" become "sendGather"
and "recvGather", etc.

Order of arguments and terseness of programs
--------------------------------------------
It seems that very ofter programmer finds himself in the situation
where he wants to send/receive messages between the same processess
over and over again. This is true for both point-to-point modes of
communication and for collective operations.

Which is why we've chosen to change the order of arguments in most API
calls. We rearranged arguments in a way that would make partial
application of API calls natural. For example, if your rank 0 process
often sends single message msg1 to rank 1 and various messages to rank
2, you could define the following shortcuts:

sendMsg1 = send comm rank1 tag1 msg1
sendTo2 = send comm rank2

In-place receive vs new array allocation for Storable Array
-----------------------------------------------------------
When using StorableArray API in tight numeric loops, it is best to
reuse existing arrays and avoid penalties incurred by
allocation/deallocation of memory. Which is why destinations/receive
buffers in StorableArray API are specified exclusively as
(StorableArray i e).

If you'd rather allocate new array for a particular operation, you
could use withNewArray/withNewArray_:

Instead of (recv comm rank tag arr) you would write 
(arr <- withNewArray bounds $ recv comm rank tag), and new array would
be allocated, supplied as the target of the (recv) operation and
returned to you.

You could easily write your own convenience wrappers similar to
withNewArray. For example, you could create wrapper that would take an
array size as a simple number instead of range.

Rank checking in collective functions (Storable Array)
------------------------------------------------------
Collective operations that are split into separate send/recv parts
(see above) take "root rank" as an argument. Right now no safeguards
are in place to ensure that rank supplied to the send function is
corresponding to the rank of that process. We believe that it does not
worsen the general go-on-and-shoot-yourself-in-the-foot attitide of
the MPI API, but this could be changed in the future.

Planned API coverage
--------------------
Status:
+ - supported in both APIs
A - only in StorableArray API
S - only in Serializable API

First tier:
+    MPI_Send,
+    MPI_Recv
+    MPI_Bcast,
+    MPI_Allgather[v]
+    MPI_Barrier
+    MPI_Scatter[v]
+    MPI_Gather[v]
+    MPI_Alltoall[v]
+    MPI_Isend,
+    MPI_Wait,
     MPI_Reduce_scatter
+    MPI_Test,
+    MPI_Cancel
     MPI_Wtime,
     MPI_Wtick
     MPI_Iprobe
+    MPI_Ssend
     MPI_Probe

Second tier or under consideration:
     MPI_Op_create
     MPI_Pcontrol
     MPI_Type_commit
     MPI_Pack_external
     MPI_Register_datarep
     MPI_Op_free
     MPI_Query_thread
     MPI_Win_create,
     MPI_Put,
     MPI_Get,
     MPI_Accumulate

How to set up OpenMPI on 2 (3,4,..) nodes?
------------------------------------------
Quick intro for the impatient:
1)Set up OpenMPI on each node
{'fraction_non_alphanumeric': 0.08426670675787416, 'fraction_numerical': 0.003908892730962941, 'mean_word_length': 3.309685779073534, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4053734', 'n_tokens_mistral': 3501, 'n_tokens_neox': 3353, 'n_words': 1879}