Removing any need for a dataframe in expander_general_stats; instead making sure to cache and load the small amount of details needed for this widget. Note I also moved around a couple functions -- same content, just moved -- so that it was easier for me to navigate through the code. I also pulled out a couple of sub-functions from larger functions, again to make the code easier to work with/understand, as well as helping to further modularize so we can limit what needs to be cached.

Splitting prepare_dataset into preparing the base dataset, and the tokenized dataset. This will help us to have further control over caching and loading data, eventually removing the storage of base dataset.

Updating NLTK requirements due to vulnerability in versions below 3.6.4: contained an inefficient Regular Expression and is vulnerable to regular expression denial of service attacks

Continuing cache minimizing in new repository. Please see https://github.com/huggingface/DataMeasurements for full history

:art: add line to file to bump ci

hate speech18 cache

hate speech 18 pmi file cache

Test to push up a simple cache file

:construction_worker: update CI to rebase

:bug: filter_vocab -> filter_words

:bug: really make sure log_files/ exists

:bug: add log_files dir if not exists

:rocket: add app

:rocket: add app and reqs

:construction_worker: add CI

:bug: remove line added by CoPilot

:memo: add README.md

:tada: init

Initial commit