<commit_before>Runnable shell scripts
=======================
make_ratiolist.sh: qsub-able wrapper for freqratio.py
make_wordlist.sh: qsub-able wrapper for wordlist.py

Runnable Python scripts
========================
api_sample.py: A sample of using the codeswitchador API.
eval_codeswitch.py: Evaluate the performance of codeswitching models 0, 1.0, 1.5.
freqratio.py: Create a frequency ratio list from two wordlists.
wordlist.py: Create a wordlist from a corpus.

Libraries
==========
codeswitchador.py: Support for codeswitching detection.
lid_constants.py: Constants used by many files.
lidlists.py: Wordlists and paths used by idiotLID and wordlist-based models.
scalereader.py: Support for reading from common SCALE file formats (e.g., Jerboa output).
wordlistlid.py: Wordlist-based LID/CS models.

The Basics
==========
Most common things you'll need to do:
1. Create wordlists. See:
tools/make_eng_wordlist.sh
tools/make_spa_wordlist.sh

TODO: More things here!
<commit_msg>Fixes formatting in README.<commit_after>Runnable shell scripts
=======================
* make_ratiolist.sh: qsub-able wrapper for freqratio.py
* make_wordlist.sh: qsub-able wrapper for wordlist.py

Runnable Python scripts
========================
* api_sample.py: A sample of using the codeswitchador API.
* eval_codeswitch.py: Evaluate the performance of codeswitching models 0, 1.0, 1.5.
* freqratio.py: Create a frequency ratio list from two wordlists.
* wordlist.py: Create a wordlist from a corpus.

Libraries
==========
* codeswitchador.py: Support for codeswitching detection.
* lid_constants.py: Constants used by many files.
* lidlists.py: Wordlists and paths used by idiotLID and wordlist-based models.
* scalereader.py: Support for reading from common SCALE file formats (e.g., Jerboa output).
* wordlistlid.py: Wordlist-based LID/CS models.

The Basics
==========
Most common things you'll need to do:

1. Create wordlists. See:
   * tools/make_eng_wordlist.sh
   * tools/make_spa_wordlist.sh

TODO: More things here!

{'fraction_non_alphanumeric': 0.13399503722084366, 'fraction_numerical': 0.005955334987593052, 'mean_word_length': 5.127659574468085, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1905416', 'n_tokens_mistral': 648, 'n_tokens_neox': 618, 'n_words': 217}