<commit_before>= Welcome to the FELTOR project!
:source-highlighter: pygments
:toc: macro

(Please visit our project https://feltor-dev.github.io[Homepage] and
http://feltor-dev.github.io/doc/dg/html/modules.html[Documentation])

image::3dpic.jpg[3dsimulation]

FELTOR (Full-F ELectromagnetic code in TORoidal geometry) is both a
numerical library and a scientific software package built on top of it.

Its main physical target are plasma edge and scrape-off layer
(gyro-)fluid simulations. The numerical methods centre around
discontinuous Galerkin methods on structured grids. Our core level
functions are parallelized for a variety of hardware from multi-core cpu
to hybrid MPI{plus}GPU, which makes the library incredibly fast.
Note that the library ships with a multitude of test and benchmark programs.

https://zenodo.org/badge/latestdoi/14143578[image:https://zenodo.org/badge/14143578.svg[DOI]]
link:LICENSE[image:https://img.shields.io/badge/License-MIT-yellow.svg[License:
MIT]]

toc::[]

== 1. Quick start guide [[sec_quickstart]]
This guide discusses how to setup, test and benchmark the FELTOR library
installation.
We then present how to use the physical software package,
which requires additional software to be installed on the system.

=== System setup

Go ahead and clone the repository into any folder you like

[source,sh]
----
git clone https://www.github.com/feltor-dev/feltor
----

You also need to clone https://github.com/thrust/thrust[thrust] and
https://github.com/cusplibrary/cusplibrary[cusp] distributed under the
Apache-2.0 license. Also, we need Agner Fog's https://github.com/feltor-dev/vcl[vcl] library (GPL-3.0). So again in a folder of your choice

[source,sh]
----
git clone https://www.github.com/thrust/thrust
git clone https://www.github.com/cusplibrary/cusplibrary
git clone https://www.github.com/feltor-dev/vcl
----

____
Our code only depends on external libraries that are themselves openly
available.
____

Please see the following table for system requirements

[cols='3,10,14',options="header"]
|=======================================================================
|    | Minimum system requirements  | Recommended system requirements
| *CPU*     | Any         |support for AVX and FMA instruction set
| *Compiler*| gcc-4.9 or msvc-15 or icc-15.0 (C{plus}{plus}-11 standard)| OpenMP-4 support, avx, fma instruction set flags
| *GPU*     | - | NVidia GPU with compute-capability > 6 and nvcc-8.0
| *MPI*     | - | mpi installation compatible with compiler (must be cuda-aware in case hybrid MPI+GPU is the target system)
|=======================================================================
____
Our GPU backend uses the
https://developer.nvidia.com/cuda-zone[Nvidia-CUDA] programming
environment and in order to compile and run a program for a GPU a user
needs at least the nvcc-7.5 compiler (available free of charge) and a NVidia
GPU. However, we explicitly note here that due to the modular design of
our software a user does not have to possess a GPU nor the nvcc
compiler. The CPU version of the backend is equally valid and provides
the same functionality. Analogously, an MPI installation is only required if the user targets
a distributed memory system.
____

=== Running a FELTOR test or benchmark program

In order to compile one of the many test and benchmark codes
inside the FELTOR library you need to tell
the FELTOR configuration where the external libraries are located on
your computer. The default way to do this is to go into your `HOME`
directory, make an include directory and link the paths in this
directory

[source,sh]
----
cd ~
mkdir include
cd include
ln -s path/to/thrust/thrust
ln -s path/to/cusplibrary/cusp
ln -s path/to/vcl
----

____
If you do not like this, you can also create your own config file as
described link:config/README.md[here].
____

Now let us compile the first benchmark program.

[source,sh]
----
cd path/to/feltor/inc/dg

make blas_b device=omp #(for an OpenMP version)
#or
make blas_b device=gpu #(if you have a gpu and nvcc )
----


Run the code with

[source,sh]
----
./blas_b
----

and when prompted for input vector sizes type for example `3 100 100 10`
which makes a grid with 3 polynomial coefficients, 100 cells in x, 100
cells in y and 10 in z. If you compiled for OpenMP, you can set the
number of threads with e.g. `export OMP_NUM_THREADS=4`.
____
This is a
benchmark program to benchmark various elemental functions the library
is built on. Go ahead and vary the input parameters and see how your
hardware performs. You can compile and run any other program that ends
in `_t.cu` (test programs) or `_b.cu` (benchmark programs) in
`feltor/inc/dg` in this way.
____

Now, let us test the mpi setup
____
You can of course skip this if you
don't have mpi installed on your computer. If you intend to use the
MPI backend, an implementation library of the mpi standard is required.
Per default `mpic++` is used for compilation.
____

[source,sh]
----
cd path/to/feltor/inc/dg

make blas_mpib device=omp  # (for MPI+OpenMP)
# or
make blas_mpib device=gpu # (for MPI+GPU, requires CUDA-aware MPI installation)
----

Run the code with `$ mpirun -n '# of procs' ./blas_mpib` then tell how
many process you want to use in the x-, y- and z- direction, for
example: `2 2 1` (i.e. 2 procs in x, 2 procs in y and 1 in z; total
number of procs is 4) when prompted for input vector sizes type for
example `3 100 100 10` (number of cells divided by number of procs must
be an integer number). If you compiled for MPI{plus}OpenMP, you can set the
number of OpenMP threads with e.g. `export OMP_NUM_THREADS=2`.


=== Using FELTOR as a library

FELTOR's library is the *dg-library* (from discontinuous Galerkin). Note
that the library is **header-only**, which means that you just have to
include the relevant header(s) and you're good to go. For example in the
following program we compute the square L2 norm of a
function:

.test.cpp
[source,c++]
----
#include <iostream>
//include the basic dg-library
#include "dg/algorithm.h"
//optional: include the geometries expansion
#include "geometries/geometries.h"

double function(double x, double y){return exp(x)*exp(y);}
int main()
{
    //create a 2d discretization of [0,2]x[0,2] with 3 polynomial coefficients
    dg::CartesianGrid2d g2d( 0, 2, 0, 2, 3, 20, 20);
    //discretize a function on this grid
    const dg::DVec x = dg::evaluate( function, g2d);
    //create the volume element
    const dg::DVec vol2d = dg::create::volume( g2d);
    //compute the square L2 norm on the device
    double norm = dg::blas2::dot( x, vol2d, x);
    // norm is now: (exp(4)-exp(0))^2/4
    std::cout << norm <<std::endl;
    return 0;
}
----

To compile and run this code for a GPU use

[source,sh]
----
nvcc -x cu -Ipath/to/feltor/inc -Ipath/to/thrust/thrust -Ipath/to/cusplibrary/cusp test.cpp -o test
./test
----

Or if you want to use OpenMP and gcc instead of CUDA for the device
functions you can also use

[source,sh]
----
g++ -fopenmp -mavx -mfma -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_OMP -Ipath/to/feltor/inc -Ipath/to/thrust/thrust -Ipath/to/cusplibrary/cusp test.cpp -o test
export OMP_NUM_THREADS=4
./test
----

If you want to use mpi, just include the MPI header before any other
FELTOR header and use our convenient typedefs like so:

.test_mpi.cpp
[source,c++]
----
#include <iostream>
//activate MPI in FELTOR
#include "mpi.h"
#include "dg/algorithm.h"

double function(double x, double y){return exp(x)*exp(y);}
int main(int argc, char* argv[])
{
    //init MPI and create a 2d Cartesian Communicator assuming 4 MPI threads
    MPI_Init( &argc, &argv);
    int periods[2] = {true, true}, np[2] = {2,2};
    MPI_Comm comm;
    MPI_Cart_create( MPI_COMM_WORLD, 2, np, periods, true, &comm);
    //create a 2d discretization of [0,2]x[0,2] with 3 polynomial coefficients
    dg::CartesianMPIGrid2d g2d( 0, 2, 0, 2, 3, 20, 20, comm);
    //discretize a function on this grid
    const dg::MDVec x = dg::evaluate( function, g2d);
    //create the volume element
    const dg::MDVec vol2d = dg::create::volume( g2d);
    //compute the square L2 norm
    double norm = dg::blas2::dot( x, vol2d, x);
    //on every thread norm is now: (exp(4)-exp(0))^2/4
    //be a good MPI citizen and clean up
    MPI_Finalize();
    return 0;
}
----

Compile e.g. for a hybrid MPI {plus} OpenMP hardware platform with

[source,sh]
----
mpic++ -mavx -mfma -fopenmp -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_OMP -Ipath/to/feltor/inc -Ipath/to/thrust/thrust -Ipath/to/cusplibrary/cusp test_mpi.cpp -o test_mpi
export OMP_NUM_THREADS=2
mpirun -n 4 ./test_mpi
----

Note the striking similarity to the previous program. Especially the
line calling the dot function did not change at all. The compiler
chooses the correct implementation for you! This is a first example of a
__container free numerical algorithm__.

=== Using FELTOR on Windows
FELTOR has been developed mostly on Linux machines. Recently, it has
become possible to develop on Windows using https://www.visualstudio.com/[Microsoft Visual Studio].
____
Unfortunately, the msvc compiler only supports an outdated OpenMP version so
consider a performance penalty of approximately factor 2, when running the OpenMP backend on Windows.
____

For the download of the library and its dependencies described <<sec_quickstart,previously>> you may want
to activate Bash in Windows 10 or
 consider a software solution like https://desktop.github.com
____
The include paths need to be set in the project properties.
Please also enable the use of intrinsic functions, the AVX2 instruction set and
activate OpenMP support.
It is important to set the Platform to x64!
____

=== Running a FELTOR simulation

Now, we want to compile and run a simulation program. First, we have to
download and install some additional libraries for I/O-operations.

For data output we use the
http://www.unidata.ucar.edu/software/netcdf/[NetCDF] library under an
MIT - like license. The underlying https://www.hdfgroup.org/HDF5/[HDF5]
library also uses a very permissive license. Note that for the mpi
versions of applications you need to build hdf5 and netcdf with the
--enable-parallel flag. Do NOT use the pnetcdf library, which uses the
classic netcdf file format. Our JSON input files are parsed by
https://www.github.com/open-source-parsers/jsoncpp[JsonCpp] distributed
under the MIT license.
____
Some desktop applications in FELTOR use the
https://github.com/mwiesenberger/draw[draw library] (developed by us
also under MIT), which depends on OpenGL (s.a.
http://en.wikibooks.org/wiki/OpenGL_Programming[installation guide]) and
http://www.glfw.org[glfw], an OpenGL development library under a
BSD-like license.
____

As in Step 3 you need to create links to the jsoncpp library include
path (and optionally the draw library) in your include folder or provide
the paths in your config file. We are ready to compile now

[source,sh]
----
cd path/to/feltor/src/toefl # or any other project in the src folder

make toeflR device=gpu     # (compile on gpu or omp)
./toeflR <inputfile.json>  # (behold a live simulation with glfw output on screen)
# or
make toefl_hpc device=gpu  # (compile on gpu or omp)
./toefl_hpc <inputfile.json> <outputfile.nc> # (a single node simulation with output stored in a file)
# or
make toefl_mpi device=omp  # (compile on gpu or omp)
export OMP_NUM_THREADS=2   # (set OpenMP thread number to 1 for pure MPI)
echo 2 2 | mpirun -n 4 ./toefl_mpi <inputfile.json> <outputfile.nc>
# (a multi node simulation with now in total 8 threads with output stored in a file)
# The mpi program will wait for you to type the number of processes in x and y direction before
# running. That is why the echo is there.
----

A default input file is located in `path/to/feltor/src/toefl/input`. All
three programs solve the same equations. The technical documentation on
what equations are discretized, input/output parameters, etc. can be
generated as a pdf with `make doc` in the `path/to/feltor/src/toefl`
directory.


== 2. Documentation

The
http://feltor-dev.github.io/doc/dg/html/modules.html[documentation]
of the dG library was generated with
http://www.doxygen.org[Doxygen] and LateX. You can generate a local
version including informative pdf writeups on implemented numerical
methods directly from source code. This depends on the `doxygen`,
`libjs-mathjax` and `graphviz` packages and LateX. Type `make doc` in
the folder `path/to/feltor/doc` and open `index.html` (a symbolic link
to `dg/html/modules.html`) with your favorite browser.
Finally, also note the documentations of https://thrust.github.io/doc/modules.html[thrust]
and https://cusplibrary.github.io/[cusp].

We maintain tex files in every src folder for
technical documentation, which can be compiled using pdflatex with
`make doc` in the respective src folder.

For details on how FELTOR's internal Makefiles are configured please see the link:config/README.md[config] folder.


== 3. Authors, Acknowledgements, Contributions

FELTOR has been developed by <NAME> and <NAME>. Please see the list of https://feltor-dev.github.io/about[contributors]
and funding.
Also check out our https://feltor-dev.github.io[homepage]
for general information, wiki pages,
troubleshooting and guides on how to contribute.

== License

This project is licensed under the MIT license - see link:LICENSE[LICENSE] for details.

<commit_msg>rewrite info on jsoncpp and netcdf installation in README<commit_after>= Welcome to the FELTOR project!
:source-highlighter: pygments
:toc: macro

(Please visit our project https://feltor-dev.github.io[Homepage] and
http://feltor-dev.github.io/doc/dg/html/modules.html[Documentation])

image::3dpic.jpg[3dsimulation]

FELTOR (Full-F ELectromagnetic code in TORoidal geometry) is both a
numerical library and a scientific software package built on top of it.

Its main physical target are plasma edge and scrape-off layer
(gyro-)fluid simulations. The numerical methods centre around
discontinuous Galerkin methods on structured grids. Our core level
functions are parallelized for a variety of hardware from multi-core cpu
to hybrid MPI{plus}GPU, which makes the library incredibly fast.
Note that the library ships with a multitude of test and benchmark programs.

https://zenodo.org/badge/latestdoi/14143578[image:https://zenodo.org/badge/14143578.svg[DOI]]
link:LICENSE[image:https://img.shields.io/badge/License-MIT-yellow.svg[License:
MIT]]

toc::[]

== 1. Quick start guide [[sec_quickstart]]
This guide discusses how to setup, test and benchmark the FELTOR library
installation.
We then present how to use the physical software package,
which requires additional software to be installed on the system.

=== System setup

Go ahead and clone the repository into any folder you like

[source,sh]
----
git clone https://www.github.com/feltor-dev/feltor
----

You also need to clone https://github.com/thrust/thrust[thrust] and
https://github.com/cusplibrary/cusplibrary[cusp] distributed under the
Apache-2.0 license. Also, we need <NAME>'s https://github.com/feltor-dev/vcl[vcl] library (GPL-3.0). So again in a folder of your choice

[source,sh]
----
git clone https://www.github.com/thrust/thrust
git clone https://www.github.com/cusplibrary/cusplibrary
git clone https://www.github.com/feltor-dev/vcl
----

____
Our code only depends on external libraries that are themselves openly
available.
____

Please see the following table for system requirements

[cols='3,10,14',options="header"]
|=======================================================================
|    | Minimum system requirements  | Recommended system requirements
| *CPU*     | Any         |support for AVX and FMA instruction set
| *Compiler*| gcc-4.9 or msvc-15 or icc-15.0 (C{plus}{plus}-11 standard)| OpenMP-4 support, avx, fma instruction set flags
| *GPU*     | - | NVidia GPU with compute-capability > 6 and nvcc-8.0
| *MPI*     | - | mpi installation compatible with compiler (must be cuda-aware in case hybrid MPI+GPU is the target system)
|=======================================================================
____
Our GPU backend uses the
https://developer.nvidia.com/cuda-zone[Nvidia-CUDA] programming
environment and in order to compile and run a program for a GPU a user
needs at least the nvcc-7.5 compiler (available free of charge) and a NVidia
GPU. However, we explicitly note here that due to the modular design of
our software a user does not have to possess a GPU nor the nvcc
compiler. The CPU version of the backend is equally valid and provides
the same functionality. Analogously, an MPI installation is only required if the user targets
a distributed memory system.
____

=== Running a FELTOR test or benchmark program

In order to compile one of the many test and benchmark codes
inside the FELTOR library you need to tell
the FELTOR configuration where the external libraries are located on
your computer. The default way to do this is to go into your `HOME`
directory, make an include directory and link the paths in this
directory

[source,sh]
----
cd ~
mkdir include
cd include
ln -s path/to/thrust/thrust
ln -s path/to/cusplibrary/cusp
ln -s path/to/vcl
----

____
If you do not like this, you can also set the include paths in your own config file as
described link:config/README.md[here].
____

Now let us compile the first benchmark program.

[source,sh]
----
cd path/to/feltor/inc/dg

make blas_b device=omp #(for an OpenMP version)
#or
make blas_b device=gpu #(if you have a gpu and nvcc )
----


Run the code with

[source,sh]
----
./blas_b
----

and when prompted for input vector sizes type for example `3 100 100 10`
which makes a grid with 3 polynomial coefficients, 100 cells in x, 100
cells in y and 10 in z. If you compiled for OpenMP, you can set the
number of threads with e.g. `export OMP_NUM_THREADS=4`.
____
This is a
benchmark program to benchmark various elemental functions the library
is built on. Go ahead and vary the input parameters and see how your
hardware performs. You can compile and run any other program that ends
in `_t.cu` (test programs) or `_b.cu` (benchmark programs) in
`feltor/inc/dg` in this way.
____

Now, let us test the mpi setup
____
You can of course skip this if you
don't have mpi installed on your computer. If you intend to use the
MPI backend, an implementation library of the mpi standard is required.
Per default `mpic++` is used for compilation.
____

[source,sh]
----
cd path/to/feltor/inc/dg

make blas_mpib device=omp  # (for MPI+OpenMP)
# or
make blas_mpib device=gpu # (for MPI+GPU, requires CUDA-aware MPI installation)
----

Run the code with `$ mpirun -n '# of procs' ./blas_mpib` then tell how
many process you want to use in the x-, y- and z- direction, for
example: `2 2 1` (i.e. 2 procs in x, 2 procs in y and 1 in z; total
number of procs is 4) when prompted for input vector sizes type for
example `3 100 100 10` (number of cells divided by number of procs must
be an integer number). If you compiled for MPI{plus}OpenMP, you can set the
number of OpenMP threads with e.g. `export OMP_NUM_THREADS=2`.


=== Using FELTOR as a library

FELTOR's library is the *dg-library* (from discontinuous Galerkin). Note
that the library is **header-only**, which means that you just have to
include the relevant header(s) and you're good to go. For example in the
following program we compute the square L2 norm of a
function:

.test.cpp
[source,c++]
----
#include <iostream>
//include the basic dg-library
#include "dg/algorithm.h"
//optional: include the geometries expansion
#include "geometries/geometries.h"

double function(double x, double y){return exp(x)*exp(y);}
int main()
{
    //create a 2d discretization of [0,2]x[0,2] with 3 polynomial coefficients
    dg::CartesianGrid2d g2d( 0, 2, 0, 2, 3, 20, 20);
    //discretize a function on this grid
    const dg::DVec x = dg::evaluate( function, g2d);
    //create the volume element
    const dg::DVec vol2d = dg::create::volume( g2d);
    //compute the square L2 norm on the device
    double norm = dg::blas2::dot( x, vol2d, x);
    // norm is now: (exp(4)-exp(0))^2/4
    std::cout << norm <<std::endl;
    return 0;
}
----

To compile and run this code for a GPU use

[source,sh]
----
nvcc -x cu -Ipath/to/feltor/inc -Ipath/to/thrust/thrust -Ipath/to/cusplibrary/cusp test.cpp -o test
./test
----

Or if you want to use OpenMP and gcc instead of CUDA for the device
functions you can also use

[source,sh]
----
g++ -fopenmp -mavx -mfma -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_OMP -Ipath/to/feltor/inc -Ipath/to/thrust/thrust -Ipath/to/cusplibrary/cusp test.cpp -o test
export OMP_NUM_THREADS=4
./test
----

If you want to use mpi, just include the MPI header before any other
FELTOR header and use our convenient typedefs like so:

.test_mpi.cpp
[source,c++]
----
#include <iostream>
//activate MPI in FELTOR
#include "mpi.h"
#include "dg/algorithm.h"

double function(double x, double y){return exp(x)*exp(y);}
int main(int argc, char* argv[])
{
    //init MPI and create a 2d Cartesian Communicator assuming 4 MPI threads
    MPI_Init( &argc, &argv);
    int periods[2] = {true, true}, np[2] = {2,2};
    MPI_Comm comm;
    MPI_Cart_create( MPI_COMM_WORLD, 2, np, periods, true, &comm);
    //create a 2d discretization of [0,2]x[0,2] with 3 polynomial coefficients
    dg::CartesianMPIGrid2d g2d( 0, 2, 0, 2, 3, 20, 20, comm);
    //discretize a function on this grid
    const dg::MDVec x = dg::evaluate( function, g2d);
    //create the volume element
    const dg::MDVec vol2d = dg::create::volume( g2d);
    //compute the square L2 norm
    double norm = dg::blas2::dot( x, vol2d, x);
    //on every thread norm is now: (exp(4)-exp(0))^2/4
    //be a good MPI citizen and clean up
    MPI_Finalize();
    return 0;
}
----

Compile e.g. for a hybrid MPI {plus} OpenMP hardware platform with

[source,sh]
----
mpic++ -mavx -mfma -fopenmp -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_OMP -Ipath/to/feltor/inc -Ipath/to/thrust/thrust -Ipath/to/cusplibrary/cusp test_mpi.cpp -o test_mpi
export OMP_NUM_THREADS=2
mpirun -n 4 ./test_mpi
----

Note the striking similarity to the previous program. Especially the
line calling the dot function did not change at all. The compiler
chooses the correct implementation for you! This is a first example of a
__container free numerical algorithm__.

=== Using FELTOR on Windows
FELTOR has been developed mostly on Linux machines. Recently, it has
become possible to develop on Windows using https://www.visualstudio.com/[Microsoft Visual Studio].
____
Unfortunately, the msvc compiler only supports an outdated OpenMP version so
consider a performance penalty of approximately factor 2, when running the OpenMP backend on Windows.
____

For the download of the library and its dependencies described <<sec_quickstart,previously>> you may want
to activate Bash in Windows 10 or
 consider a software solution like https://desktop.github.com
____
The include paths need to be set in the project properties.
Please also enable the use of intrinsic functions, the AVX2 instruction set and
activate OpenMP support.
It is important to set the Platform to x64!
____

=== Running a FELTOR simulation

Now, we want to compile and run a simulation program. To this end, we have to
download and install some additional libraries for I/O-operations.

First, we need to install jsoncpp (distributed under the MIT License).
The easiest way to do this on Linux is to install `libjsoncpp-dev` through the package managment system. However, if you have
a GPU or if you do not have sudo privileges you have to install the library manually. That means you have to clone https://www.github.com/open-source-parsers/jsoncpp[JsonCpp] and follow the instructions in the README. After this, link the
include path
[source,sh]
----
cd ~/include
ln -s /usr/include/jsoncpp/json # if installed as a system library
# or
ln -s path/to/jsoncpp/include/json # if installed manually
----
or append the respective path as well as the path to the object library to the `INCLUDE` and `JSONLIB` variables as
described under link:config/README.md[config].

For data output we use the
http://www.unidata.ucar.edu/software/netcdf/[NetCDF-C] library under an
MIT - like license (we use the netcdf-4 file format).
The underlying https://www.hdfgroup.org/HDF5/[HDF5]
library also uses a very permissive license.
Both can be installed easily on Linux through the `libnetcdf-dev` package.
For a manual build follow the build instructions in the https://www.unidata.ucar.edu/software/netcdf/docs/getting_and_building_netcdf.html[netcdf-documentation].
Note that for the mpi
versions of applications you need to build hdf5 and netcdf with the
`--enable-parallel` flag. Do NOT use the pnetcdf library, which uses the
classic netcdf file format.

Some desktop applications in FELTOR use the
https://github.com/mwiesenberger/draw[draw library] (developed by us
also under MIT), which depends on OpenGL (s.a.
http://en.wikibooks.org/wiki/OpenGL_Programming[installation guide]) and
http://www.glfw.org[glfw3], an OpenGL development library under a
BSD-like license. Again, link `path/to/draw` in the `include` folder.

We are now ready to compile and run a simulation program

[source,sh]
----
cd path/to/feltor/src/toefl # or any other project in the src folder

make toeflR device=gpu     # (compile on gpu or omp)
./toeflR <inputfile.json>  # (behold a live simulation with glfw output on screen)
# or
make toefl_hpc device=gpu  # (compile on gpu or omp)
./toefl_hpc <inputfile.json> <outputfile.nc> # (a single node simulation with output stored in a file)
# or
make toefl_mpi device=omp  # (compile on gpu or omp)
export OMP_NUM_THREADS=2   # (set OpenMP thread number to 1 for pure MPI)
echo 2 2 | mpirun -n 4 ./toefl_mpi <inputfile.json> <outputfile.nc>
# (a multi node simulation with now in total 8 threads with output stored in a file)
# The mpi program will wait for you to type the number of processes in x and y direction before
# running. That is why the echo is there.
----

A default input file is located in `path/to/feltor/src/toefl/input`. All
three programs solve the same equations. The technical documentation on
what equations are discretized, input/output parameters, etc. can be
generated as a pdf with `make doc` in the `path/to/feltor/src/toefl`
directory.


== 2. Documentation

The
http://feltor-dev.github.io/doc/dg/html/modules.html[documentation]
of the dG library was generated with
http://www.doxygen.org[Doxygen] and LateX. You can generate a local
version including informative pdf writeups on implemented numerical
methods directly from source code. This depends on the `doxygen`,
`libjs-mathjax` and `graphviz` packages and LateX. Type `make doc` in
the folder `path/to/feltor/doc` and open `index.html` (a symbolic link
to `dg/html/modules.html`) with your favorite browser.
Finally, also note the documentations of https://thrust.github.io/doc/modules.html[thrust]
and https://cusplibrary.github.io/[cusp].

We maintain tex files in every src folder for
technical documentation, which can be compiled using pdflatex with
`make doc` in the respective src folder.

For details on how FELTOR's internal Makefiles are configured please see the link:config/README.md[config] folder.


== 3. Authors, Acknowledgements, Contributions

FELTOR has been developed by <NAME> and <NAME>. Please see the list of https://feltor-dev.github.io/about[contributors]
and funding.
Also check out our https://feltor-dev.github.io[homepage]
for general information, wiki pages,
troubleshooting and guides on how to contribute.

== License

This project is licensed under the MIT license - see link:LICENSE[LICENSE] for details.


{'fraction_non_alphanumeric': 0.09955520196723683, 'fraction_numerical': 0.011897443315372654, 'mean_word_length': 4.1040974529346625, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 28, 'https://': 43, 'lorem ipsum': 0, 'www.': 21, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '284515', 'n_tokens_mistral': 8879, 'n_tokens_neox': 8332, 'n_words': 3642}