Try using a KenLM model trie binary version, if not found try using the hash table binary version

Sync with data tooling repo, using edugp/kenlm models, updating viz to use quantiles for coloring and ad-hoc viz for the registry dataset

Run tokenizer before computing perplexity and format

Replicate default cc_net preprocessing at inference time on KenlmModel.get_perplexity

Format file

Add distiluse-base-multilingual-cased-v2

Add tests and fix issue when splitting into sentences, to grab the minimum number between total sentences and sample size, rather than total original documents and sample size

Check if file exists before attempting to remove

Remove also the '{language}.sp.model' file on failure

Add CLI and refactor

Remove corrupt KenLM model files.

Support visualizing both sentences and whole documents. Smooth down color assignment in visualization.