<commit_before># consigliere

This is the beginnings of an app to process sosreport (and possibly SAR) data from an ElasticSearch warehouse.

## Getting the data

I was able to bulk export the sosreport indices using the [elasticdump](https://github.com/taskrabbit/elasticsearch-dump) utility; if you want to do this yourself, here's how to invoke elasticdump:

    for month in $(seq -f "%02g" 01 12) ; do
       elasticdump --all --debug --input=http://es-perf44.perf.lab.eng.bos.redhat.com:80/vos.sosreport-2014$month --output=vos.sosreport-2014$month.json
    done

For convenience, willb has also made a tarball of the extracted indices (available inside RHT; ask him for a reference).

## Preprocessing the data

Once you've extracted the JSON files to some directory, you can run the `sos-preprocess.py` script with the names of the JSON files as arguments.  Currently, this just splits each file by record type, but it might do more sophisticated preprocessing in the future.  You'll invoke it like this:

    % ../bin/sos-preprocess.py vos.sosreport-2014*.json

and the output will look something like this:

    processing vos.sosreport-201401.json...
     - writing cpuinfo records...
     - writing date records...
     - writing lsblk records...
     - writing dmidecode records...
     - writing slabinfo records...
     - writing ps records...
     - writing cmdline records...
     - writing installed-rpms records...
     - writing meminfo records...
     - writing lspci records...
     - writing lsmod records...<commit_msg>trivial README fixes<commit_after># consigliere

This is the beginning of an app to process sosreport (and possibly SAR) data from an ElasticSearch warehouse.

## Getting the data

It's possible to bulk export the sosreport indices using the [elasticdump](https://github.com/taskrabbit/elasticsearch-dump) utility; if you want to do this yourself, here's how to invoke elasticdump:

    for month in $(seq -f "%02g" 01 12) ; do
       elasticdump --all --debug --input=http://es-perf44.perf.lab.eng.bos.redhat.com:80/vos.sosreport-2014$month --output=vos.sosreport-2014$month.json
    done

For convenience, willb has also made a tarball of the extracted indices (available inside RHT; ask him for a reference).

## Preprocessing the data

Once you've extracted the JSON files to some directory, you can run the `sos-preprocess.py` script with the names of the JSON files as arguments.  Currently, this just splits each file by record type, but it might do more sophisticated preprocessing in the future.  You'll invoke it like this:

    % ../bin/sos-preprocess.py vos.sosreport-2014*.json

and the output will look something like this:

    processing vos.sosreport-201401.json...
     - writing cpuinfo records...
     - writing date records...
     - writing lsblk records...
     - writing dmidecode records...
     - writing slabinfo records...
     - writing ps records...
     - writing cmdline records...
     - writing installed-rpms records...
     - writing meminfo records...
     - writing lspci records...
     - writing lsmod records...
{'fraction_non_alphanumeric': 0.09273021001615508, 'fraction_numerical': 0.018093699515347336, 'mean_word_length': 3.0736842105263156, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '695359', 'n_tokens_mistral': 922, 'n_tokens_neox': 843, 'n_words': 358}