<commit_before>===========
rosette-api
===========

Python tools for use with the Basis Technology Rosette API

enrich.py:  sends JSON documents to Rosette API and enriches them with entity extraction


Usage
-----
python enrich.py [-h] [-o OUTPUTDIR] -k KEY filespec
	

Arguments
---------
-h requests help
-o specifies the directory where enriched files are written; the default is enriched
-k specifies the rosette API key, and is required
filespec must be the path to one or more json files 

<commit_msg>Minor updates<commit_after>===========
rosette-api
===========

Python tools for use with the Basis Technology Rosette API

enrich.py: Enrich json files using the Rosette API'


Usage
-----
python enrich.py [-h] [-o OUTPUTDIR] [-r RESPONSEDIR] -k KEY filespec
	

Arguments
---------
-o specifies the subdirectory where enriched files are written; default: enriched
-r specifies the subdirectory where responses from Rosette API are saved; default: 
responses
-k specifies the rosette API key
filespec must be the path to one or more json files 


Operation
---------
Enrich.py iterates through each json file, takes the fields "subject" and "body",
combines them, and sends them to Rosette API for entity extraction. (This will
be specified on the CLI in a future version.)

The response from Rosette is written out to a file of the same name, in the response 
subdirectory. 

Enrich.py then filters the entities to those with confidence > 0.1 and groups
them into lists by type. Finally, it merges those types with the input file, and 
writes the enriched file into the enriched subdirectory.


Notes
-----
Only Entity Extraction is currently supported.


{'fraction_non_alphanumeric': 0.0912912912912913, 'fraction_numerical': 0.0012012012012012011, 'mean_word_length': 3.773638968481375, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4953294', 'n_tokens_mistral': 477, 'n_tokens_neox': 433, 'n_words': 240}