<commit_before>## Extract text from file dump

1. Configuration - create a file `maldini/site/settings/local.py`:

   * `DATABASES`: django database configuration
   * `MALDINI_ROOT`: path to the dump
   * `ELASTICSEARCH_URL`: url to elasticsearch server
   * `ELASTICSEARCH_INDEX`: name of elasticsearch index where to index the data

2. Create file and folder entries in the database:

   ```shell
   $ ./manage.py walk
   $ ./manage.py walkfolders
   ```

3. Select documents for analysis. The argument to the `digestqueue` command is
   an SQL `WHERE` clause to choose which documents will be analyzed. `true`
   means all documents. They are added to the `digest` queue.

   ```shell
   $ ./manage.py digestqueue "true"
   ```

4. Run the `digest` worker to process. All documents successfully digested will
   be automatically added to the `index` queue.

   ```shell
   $ ./manage.py digestworker
   ```

5. Run the `index` worker to push digested documents to elasticsearch.

   ```shell
   $ ./manage.py indexworker
   ```

To digest a single document and view the JSON output:

```shell
$ ./manage.py digest 123
```
<commit_msg>update readme with msgconvert install instructions<commit_after>## Extract text from file dump

1. Configuration - create a file `maldini/site/settings/local.py`:

   * `DATABASES`: django database configuration
   * `MALDINI_ROOT`: path to the dump
   * `ELASTICSEARCH_URL`: url to elasticsearch server
   * `ELASTICSEARCH_INDEX`: name of elasticsearch index where to index the data

2. Create file and folder entries in the database:

   ```shell
   $ ./manage.py walk
   $ ./manage.py walkfolders
   ```

3. Select documents for analysis. The argument to the `digestqueue` command is
   an SQL `WHERE` clause to choose which documents will be analyzed. `true`
   means all documents. They are added to the `digest` queue.

   ```shell
   $ ./manage.py digestqueue "true"
   ```

4. Run the `digest` worker to process. All documents successfully digested will
   be automatically added to the `index` queue.

   ```shell
   $ ./manage.py digestworker
   ```

5. Run the `index` worker to push digested documents to elasticsearch.

   ```shell
   $ ./manage.py indexworker
   ```

To digest a single document and view the JSON output:

```shell
$ ./manage.py digest 123
```

## Setup .msg -> .eml converter

Current setup requires the `msgconvert` script to be accesible in the PATH.
Docs: http://www.matijs.net/software/msgconv/

```shell
$ cpan -i Email::Outlook::Message
```

{'fraction_non_alphanumeric': 0.10170838299562972, 'fraction_numerical': 0.006356773937226857, 'mean_word_length': 3.4253075571177503, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4053938', 'n_tokens_mistral': 781, 'n_tokens_neox': 719, 'n_words': 303}