<commit_before>
* Midas

This is a collection of scripts used to correlate rankings of the one
million most visited sites per day and information from
CrunchBase [fn:1]. The final goal is to make a prediction whether an
enterprise will receive a funding of the type ~angel~, ~seed~ or ~a~
soon.

The several scripts are written to run under Hadoop.

Further documentation is provided in the ~doc~ folder via Sphinx.

* Footnotes

[fn:1] To fetch the data from CrunchBase another, separate script was
  used. It can be found on [[https://github.com/fuzzy-id/crawlcrunch]].
<commit_msg>better documentation<commit_after>
* Midas

This is a collection of scripts used to correlate rankings of the one
million most visited sites per day (Alexa Top1M files) and information
from CrunchBase [fn:1]. The final goal is to make a prediction whether
an enterprise will receive a funding of the type ~angel~, ~seed~ or
~a~ soon.

The several scripts are written to run under Hadoop.

Further documentation is provided in the ~doc~ folder via Sphinx.

* Footnotes

[fn:1] To fetch the data from CrunchBase another, separate package
  should be used. It can be found on
  [[https://github.com/fuzzy-id/crawlcrunch]].

{'fraction_non_alphanumeric': 0.06478405315614617, 'fraction_numerical': 0.004152823920265781, 'mean_word_length': 4.379464285714286, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2160692', 'n_tokens_mistral': 356, 'n_tokens_neox': 335, 'n_words': 179}