<commit_before>phoxy
=====

The Phoenix event data set records social and political events
automatically extracted from news reporting. \code{phoxy} provides
functions to make acquiring and analyzing this event data easier. For more
information about Phoenix, see its website here:
 \url{http://phoenixdata.org/} and the website of the Open Event Data
Alliance here: \url{http://openeventdata.org/}.

Currently, phoxy includes two basic functions.

* `download_phoenix` takes a directory path and downloads the entire Phoenix
  dataset from [http://phoenixdata.org/data](http://phoenixdata.org/data) into
  that directory.

* `ingest_phoenix` reads in each daily text file from that directory, does some
  basic column name and class fixing, and combines them all into one large
  dataframe for easy analysis.
<commit_msg>Sometimes I mix up LaTeX and Markdown<commit_after>phoxy
=====

Download, ingest, and analyze the Phoenix event dataset.

The Phoenix event data set records social and political events
automatically extracted from news reporting. `phoxy` provides
functions to make acquiring and analyzing this event data easier. For more
information about Phoenix, see its website here:
[http://phoenixdata.org/](http://phoenixdata.org/) and the website of the Open Event Data
Alliance here: [http://openeventdata.org/](http://openeventdata.org).

Currently, phoxy includes two basic functions.

* `download_phoenix` takes a directory path and downloads the entire Phoenix
  dataset from [http://phoenixdata.org/data](http://phoenixdata.org/data) into
  that directory.

* `ingest_phoenix` reads in each daily text file from that directory, does some
  basic column name and class fixing, and combines them all into one large
  dataframe for easy analysis.

{'fraction_non_alphanumeric': 0.07430516165626773, 'fraction_numerical': 0.0, 'mean_word_length': 4.841059602649007, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5148979', 'n_tokens_mistral': 463, 'n_tokens_neox': 457, 'n_words': 214}