<commit_before>         Running the Globus GridFTP Server With GFork
         ============================================

Running the globus-gridftp-server under gfork is almost identical to 
running it under xinetd.  First you need a configuration file:

service gridftp2
{
  env += GLOBUS_LOCATION=<path to GL>
  env += LD_LIBRARY_PATH=<path to GL>/lib
  server = <path to GL>/sbin/globus-gridftp-server
  server_args = -i 
  server_args += -d ALL -l <path to GL>/var/gridftp.log
  port = 5000
}

That portion is identical to xinetd.  In fact an existing xinetd 
configuration file should work.  Once you have a configuration file 
gfork is run with:

% gfork -c <path to config file>

When running GridFTP out of GFork the server should be run with a master
program.  The master program provides enhanced functionality such as
dynamic backend registration for striped servers, managed system memory 
pools and internal data monitoring for both striped and non striped
servers.  For information about what a gfork master program is see the
GFork documentaton.  To run with a master program the following two lines 
are needed.

  master = <path to GL>/libexec/gfs-gfork-master
  master_args = <options>

The first line tells GFork what master program to use, for the GridFTP
server we use gfs-gfork-master.  The second line provides options to
the master program

The full list of master options (to date, run the program with --help for
newer options) are:

 -b | --reg-cs <contact string>
                             Contact to the frontend registry.  This option
                             makes it a data node
 -df | --dn-file <path>      Path to a file containing the list of acceptable
                             DNs.  Default is system gridmap file
 -G | --gsi <bool>           Enable or disable GSI.  Default is on.
 -h | --help                 print the help message
 -l | --logfile <path>       Path to the logfile.
 -p | --port <int>           Port where server listens for connections.
 -s | --stripe-count <int>   The max number of stripes to give to each server.
                             0 is all avaiable.
 -u | --update-interval <int>
                             Number of seconds between registration updates.

Striped Servers
===============
GridFTP offers a powerful enhancement called striped servers.  In this 
mode a GridFTP server is setup with a single frontend and one or many 
backends.  All of the backends work in concert to transfer a single file
and thereby achieve high through put rates.  More information can be 
found about striped servers on the GridFTP web pages.  Here we describe
how to configure a frontend and all the backends for use as a striped 
server.

Frontend Configuration
----------------------
The frontend server described here is run using dynamic backends. 
We need additional options to both the GridFTP server itself and
to the master program.  The following lines are added:

  server_args += -dsi remote 
  master_args = -port 8588
  master_args += -df <path to gridmap file>

The first line is an additional argument to the GridFTP server.  It tells
the server that it will be operating in split mode (seperate frontend
and backend processes) and that it will be the frontend.  (Specifically
it tells the server to use the 'remote' DSI).

The second line tells the master program on which port it should listen
for backend registrations.  Backend services can then connect to this
port to notify the frontend of their existance.  By default a registration
is good for 10 minutes, but a backend is free to refresh its registration.
In this way a frontend is provided with the list of possible backends 
(stripes) which maybe used for a transfer.

The third line provides the master program with a list of authorized DNs.
Each line in the file must contain a GSI DN (certificate subject).
In order to register the backend
must authenticate and provide its DN.  The provided DN is checked against
this file.  In short the file is a list of DNs that may register with the
frontend.

If the master program is not given a -df option and is given the -G n option
then there is no registration security at all.


Backend Configuration
----------------------
Any striped server setup can have more than one backend service.  Furhter,
any one computer can run multiple backends.  Here we explain how to setup
a backend server.  These steps should be repeated for ever needed backend
instance.

A backend server can also be run with GFork, it just needs different 
options to both the GridFTP server and the master program.  A sample 
backend config file is show here:

service gridftp2
{
  env += GLOBUS_LOCATION=<path to GL>
  env += LD_LIBRARY_PATH=<path to GL>/lib

  server = <path to GL>/sbin/globus-gridftp-server
  server_args = -i  
  server_args += -dn
  master = <path to GL>/libexec/gfs-gfork-master
  master_args = -b localhost:8588
}

Notable additions to this file are:

  server_args += -dn
  master_args = -b localhost:8588

These first line tells the gridftp server that it will be a 'data node', 
which is another name for a backend.  The second line tells the master
program two things, first that it will be a master of a data node, and
second what the frontends registration contact point is.  Note that in
our example we have a hostname of 'localhost' and a port of '8588'.  
8588 is (and must be) the same port that was provided to the frontend's
master program in the previous step.

Once the configuration file is complete GFork is again run as:

% gfork -c <conf file>

This will start up the data node and the master program will register
itself to the frontend and refresh its registration every 5 minutes 
(default setting).


================================
Alternative Backend Registration
================================
This is not a recommended method, but it is provided for situations that may
need it.

Registration Client
-------------------
To register a backend user the program: 
$GLOBUS_LOCATION/libexec/gfs-dynbe-client

The options to it are as follows:

 -G | --gsi <bool>           Enable or disable GSI.  Default is on.
 -h | --help                 print the help message
 -m | --max <int>            set the maximum number of connections allowed to
                             the backend at one time
 -q | --quiet                print no output
 -t | --total <int>          set the maximum total number of connections
                             allowed to the backend


Registering a Backend
---------------------

Once GFork is started with the master program gfs-dynbe-client can be 
used to register backends with it.  The value for the -G option must be 
the same for both the client and the server.  It determines if any 
security will be used.  The registration program contacts the master 
program on its listening port.  If security is on, the master program 
checks the clients GSI distinguished name again the file pointed to by 
the -dn options to the master program.  If not the client is just 
trusted.  An example client invocation is:

% gfs-dynbe-client -G y <backend host>:<backend port> <master host>:<master port>

==============================
GridFTP with Memory Management
==============================

Another feature that gfork with the GridFTP plugin brings memory usage
limiting.  Under extreme client loads it is possible that GridFTP servers
require more memory than the system has.  Due to common kernel memory
allocation scheme known as optimistic provisioning this situation can 
lead to a full consumption of memory resources and thus triggering the
out of memory handler.  The OOM handler will kill processes in a difficult
to predict way, in order to free up memory.  This will leave the system 
in an unpredicatable and unstable state, therefor this is a situation that
we want to avoid.  

To control this situation the GridFTP gfork plugin has a memory limiting
option.  This will attempt to limit memory usage to a given value or
to the maximum amount of RAM in the system.  Most of the memory is given 
to the first few connections, but when the plugin detects that it is
overloaded, each session is limited to half the available memory.

To enable this feature one of two options must be passed to the master 
program:

   -m                : limits memory consuption to amount if RAM in the system
   -M <formated int> : limit memory to the give value.

Another important option should be provided in the gfork config file.  When
a client connects to gfork a gridftp server instance is executed.  This 
instance requires a certain amount of RAM.  If connections are comming in
to fast this can act as DOS attack.  Limiting the number of allowed
simultaneous connections will help the memory management algorithm do its
job.  This limit is set with:

    instance = <int>

We recommend a value of 100 or |RAM|/2M, whichever is smaller.

An example gfork config is:

service gridftp2
{
  instance = 100  
  env += GLOBUS_LOCATION=<path to GL>
  env += LD_LIBRARY_PATH=<path to GL>/lib
  server = <path to GL>/sbin/globus-gridftp-server
  server_args = -i
  server_args += -dn
  master = <path to GL>/libexec/gfs-gfork-master
  master_args = -M 512M
}



<commit_msg>referencing gfrok config files in doc<commit_after>         Running the Globus GridFTP Server With GFork
         ============================================

Running the globus-gridftp-server under gfork is almost identical to 
running it under xinetd.  First you need a configuration file:

service gridftp2
{
  env += GLOBUS_LOCATION=<path to GL>
  env += LD_LIBRARY_PATH=<path to GL>/lib
  server = <path to GL>/sbin/globus-gridftp-server
  server_args = -i 
  server_args += -d ALL -l <path to GL>/var/gridftp.log
  port = 5000
}

*note: example configuration file exist in $GLOBUS_LOCATION/gridftp???.gfork 

That portion is identical to xinetd.  In fact an existing xinetd 
configuration file should work.  Once you have a configuration file 
gfork is run with:

% gfork -c <path to config file>

When running GridFTP out of GFork the server should be run with a master
program.  The master program provides enhanced functionality such as
dynamic backend registration for striped servers, managed system memory 
pools and internal data monitoring for both striped and non striped
servers.  For information about what a gfork master program is see the
GFork documentaton.  To run with a master program the following two lines 
are needed.

  master = <path to GL>/libexec/gfs-gfork-master
  master_args = <options>

The first line tells GFork what master program to use, for the GridFTP
server we use gfs-gfork-master.  The second line provides options to
the master program

The full list of master options (to date, run the program with --help for
newer options) are:

 -b | --reg-cs <contact string>
                             Contact to the frontend registry.  This option
                             makes it a data node
 -df | --dn-file <path>      Path to a file containing the list of acceptable
                             DNs.  Default is system gridmap file
 -G | --gsi <bool>           Enable or disable GSI.  Default is on.
 -h | --help                 print the help message
 -l | --logfile <path>       Path to the logfile.
 -p | --port <int>           Port where server listens for connections.
 -s | --stripe-count <int>   The max number of stripes to give to each server.
                             0 is all avaiable.
 -u | --update-interval <int>
                             Number of seconds between registration updates.

Striped Servers
===============
GridFTP offers a powerful enhancement called striped servers.  In this 
mode a GridFTP server is setup with a single frontend and one or many 
backends.  All of the backends work in concert to transfer a single file
and thereby achieve high through put rates.  More information can be 
found about striped servers on the GridFTP web pages.  Here we describe
how to configure a frontend and all the backends for use as a striped 
server.

Frontend Configuration
----------------------
The frontend server described here is run using dynamic backends. 
We need additional options to both the GridFTP server itself and
to the master program.  The following lines are added:

  server_args += -dsi remote 
  master_args = -port 8588
  master_args += -df <path to gridmap file>

The first line is an additional argument to the GridFTP server.  It tells
the server that it will be operating in split mode (seperate frontend
and backend processes) and that it will be the frontend.  (Specifically
it tells the server to use the 'remote' DSI).

The second line tells the master program on which port it should listen
for backend registrations.  Backend services can then connect to this
port to notify the frontend of their existance.  By default a registration
is good for 10 minutes, but a backend is free to refresh its registration.
In this way a frontend is provided with the list of possible backends 
(stripes) which maybe used for a transfer.

The third line provides the master program with a list of authorized DNs.
Each line in the file must contain a GSI DN (certificate subject).
In order to register the backend
must authenticate and provide its DN.  The provided DN is checked against
this file.  In short the file is a list of DNs that may register with the
frontend.

If the master program is not given a -df option and is given the -G n option
then there is no registration security at all.


Backend Configuration
----------------------
Any striped server setup can have more than one backend service.  Furhter,
any one computer can run multiple backends.  Here we explain how to setup
a backend server.  These steps should be repeated for ever needed backend
instance.

A backend server can also be run with GFork, it just needs different 
options to both the GridFTP server and the master program.  A sample 
backend config file is show here:

service gridftp2
{
  env += GLOBUS_LOCATION=<path to GL>
  env += LD_LIBRARY_PATH=<path to GL>/lib

  server = <path to GL>/sbin/globus-gridftp-server
  server_args = -i  
  server_args += -dn
  master = <path to GL>/libexec/gfs-gfork-master
  master_args = -b localhost:8588
}

Notable additions to this file are:

  server_args += -dn
  master_args = -b localhost:8588

These first line tells the gridftp server that it will be a 'data node', 
which is another name for a backend.  The second line tells the master
program two things, first that it will be a master of a data node, and
second what the frontends registration contact point is.  Note that in
our example we have a hostname of 'localhost' and a port of '8588'.  
8588 is (and must be) the same port that was provided to the frontend's
master program in the previous step.

Once the configuration file is complete GFork is again run as:

% gfork -c <conf file>

This will start up the data node and the master program will register
itself to the frontend and refresh its registration every 5 minutes 
(default setting).


================================
Alternative Backend Registration
================================
This is not a recommended method, but it is provided for situations that may
need it.

Registration Client
-------------------
To register a backend user the program: 
$GLOBUS_LOCATION/libexec/gfs-dynbe-client

The options to it are as follows:

 -G | --gsi <bool>           Enable or disable GSI.  Default is on.
 -h | --help                 print the help message
 -m | --max <int>            set the maximum number of connections allowed to
                             the backend at one time
 -q | --quiet                print no output
 -t | --total <int>          set the maximum total number of connections
                             allowed to the backend


Registering a Backend
---------------------

Once GFork is started with the master program gfs-dynbe-client can be 
used to register backends with it.  The value for the -G option must be 
the same for both the client and the server.  It determines if any 
security will be used.  The registration program contacts the master 
program on its listening port.  If security is on, the master program 
checks the clients GSI distinguished name again the file pointed to by 
the -dn options to the master program.  If not the client is just 
trusted.  An example client invocation is:

% gfs-dynbe-client -G y <backend host>:<backend port> <master host>:<master port>

==============================
GridFTP with Memory Management
==============================

Another feature that gfork with the GridFTP plugin brings memory usage
limiting.  Under extreme client loads it is possible that GridFTP servers
require more memory than the system has.  Due to common kernel memory
allocation scheme known as optimistic provisioning this situation can 
lead to a full consumption of memory resources and thus triggering the
out of memory handler.  The OOM handler will kill processes in a difficult
to predict way, in order to free up memory.  This will leave the system 
in an unpredicatable and unstable state, therefor this is a situation that
we want to avoid.  

To control this situation the GridFTP gfork plugin has a memory limiting
option.  This will attempt to limit memory usage to a given value or
to the maximum amount of RAM in the system.  Most of the memory is given 
to the first few connections, but when the plugin detects that it is
overloaded, each session is limited to half the available memory.

To enable this feature one of two options must be passed to the master 
program:

   -m                : limits memory consuption to amount if RAM in the system
   -M <formated int> : limit memory to the give value.

Another important option should be provided in the gfork config file.  When
a client connects to gfork a gridftp server instance is executed.  This 
instance requires a certain amount of RAM.  If connections are comming in
to fast this can act as DOS attack.  Limiting the number of allowed
simultaneous connections will help the memory management algorithm do its
job.  This limit is set with:

    instance = <int>

We recommend a value of 100 or |RAM|/2M, whichever is smaller.

An example gfork config is:

service gridftp2
{
  instance = 100  
  env += GLOBUS_LOCATION=<path to GL>
  env += LD_LIBRARY_PATH=<path to GL>/lib
  server = <path to GL>/sbin/globus-gridftp-server
  server_args = -i
  server_args += -dn
  master = <path to GL>/libexec/gfs-gfork-master
  master_args = -M 512M
}




{'fraction_non_alphanumeric': 0.07029270659263652, 'fraction_numerical': 0.004420246886960272, 'mean_word_length': 3.1401472885516624, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7433075', 'n_tokens_mistral': 4970, 'n_tokens_neox': 4742, 'n_words': 2706}