<commit_before>For each job, the first thing to do is to start mem_io.  This is done using the `mem_io_start` command.  This will:
* create a run-specific mem_io configuration file,
* create a run-specific redis configuration file, and
* start the redis database in the background.

Each mem_io run is identified by a mem_io ID, which is intended to be unique.  it can either be specified on the command line using the `-mem_io_id` option, taken for the `PBS_JOBID` environment variable, or default to `default` (which is of course not unique, and will overwrite data from from previous runs with that mem_io ID).

The mem_io configuration file will be created in the current working directory and its name has the following pattern `mem_io_<mem_io_id>.conf`.  Since this file contains the password, care is taken that it is creaed with the correct file permissions.

The redis configuration file will be created in the current working directory and its name has the following pattern `redis_<mem_io_id>.conf`.  Since this file contains the password, care is taken that it is creaed with the correct file permissions.

Upon starting mem_io, you have to specify the number of I/O channels to use.  These are independent storage entries in the database, and each store operation will append information to a specific channel, identified by a channel ID.  In practice, the number of channels is typically equal to the number of files that would have been produced by the job if mem_io were not used.

Hence, starting mem_io with, e.g., 20 channels would be accomplished by:
```
mem_io_start  -nr_channels 20
```

The resulting on-disk database file will named according to the following pattern: `mem_io_<mem_io_id>.rdb`.

Note that mem_io is only started once per job, either in the PBS user prologue file, the worker prolog script, or at the start of your PBS job script.  Hence this command *should never* occor in a worker batch script!<commit_msg>Added documentation on redis log file<commit_after>For each job, the first thing to do is to start mem_io.  This is done using the `mem_io_start` command.  This will:
* create a run-specific mem_io configuration file,
* create a run-specific redis configuration file, and
* start the redis database in the background.

Each mem_io run is identified by a mem_io ID, which is intended to be unique.  it can either be specified on the command line using the `-mem_io_id` option, taken for the `PBS_JOBID` environment variable, or default to `default` (which is of course not unique, and will overwrite data from from previous runs with that mem_io ID).

The mem_io configuration file will be created in the current working directory and its name has the following pattern `mem_io_<mem_io_id>.conf`.  Since this file contains the password, care is taken that it is creaed with the correct file permissions.  The other mem_io commands `mem_io_store`, `mem_io_retrieve`, and `mem_io_stop` will use the mem_io configuration file generated by `mem_io_start`.

The redis configuration file will be created in the current working directory and its name has the following pattern `redis_<mem_io_id>.conf`.  Since this file contains the password, care is taken that it is creaed with the correct file permissions.

Upon starting mem_io, you have to specify the number of I/O channels to use.  These are independent storage entries in the database, and each store operation will append information to a specific channel, identified by a channel ID.  In practice, the number of channels is typically equal to the number of files that would have been produced by the job if mem_io were not used.

Hence, starting mem_io with, e.g., 20 channels would be accomplished by:
```
mem_io_start  -nr_channels 20
```

The resulting on-disk database file will named according to the following pattern: `redis_<mem_io_id>.rdb`.  Similarly, the log file created by redis will be named `redis_<mem_io_id>.log`.

The redis database is now running, and ready to store information fed to it by multiple `mem_io_store` processes.

Note that mem_io is only started once per job, either in the PBS user prologue file, the worker prolog script, or at the start of your PBS job script.  Hence this command *should never* occor in a worker batch script!

Note also that the data in the redis database file is not encrypted, the password is only used to authenticate connections with the database.  Hence it is important the the database file also has appropriate file permissions, something mem_io also takes care of.

Note that if you specify a mem_io ID explicitly when calling `mem_io_start`, that same ID has to be used for all calls of `mem_io_store`, `mem_io_retrieve`, and `mem_io_stop`.
{'fraction_non_alphanumeric': 0.04377390565235869, 'fraction_numerical': 0.0016999575010624734, 'mean_word_length': 4.637125748502994, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2056090', 'n_tokens_mistral': 1228, 'n_tokens_neox': 1201, 'n_words': 730}