<commit_before># Language with NuPIC

This directory contains scripts for running NLP-based applications using [NuPIC](https://github.com/numenta/nupic) and [Cortical.io's API](http://www.cortical.io/developers.html).

Additional requirements:

- [cortipy](https://github.com/numenta/cortipy)
- Python modules [enum](https://pypi.python.org/pypi/enum34), [pandas](http://pandas.pydata.org/), and [numpy](http://www.numpy.org/).
- You must have a valid REST API key from [Cortical.io](http://www.cortical.io/developers.html).

## Use cases
In this directory are scripts for some NLP experiments. They each use one or more of the following models:

- Keywords model gives a random (seeded) encoding to each word.
- Cortical.io word fingerprints model encodes each word with their API, and then makes a union representation over the words in a block of text.
- Cortical.io document fingerprints model uses their API to encode a block of text.
- Cortical.io windows model encodes with word-level fingerprints and builds progressive union representations as a "sliding window" over a sequence of text.
- HTM Network models use the Cortical.io word-level encoder, and then run the patterns through the Network API.
- Cortical.io classify endpoint model is an end-to-end classification tool in their API.

#### Text querying

Please refer to the [Imbu project directory](https://github.com/numenta/nupic.research/tree/master/projects/imbu).

#### Classification experiments

The experiments test the models on classifying labeled samples of text.

- run_classification_experiment.py includes two experiments:
	- incrementally increases the training size each trial; we expect to see the classification accuracies increase.
	- k-folds cross validation.
- run_buckets_experiment.py: an experiment to quantify model performance for the text querying task.
	


#### Additional examples

Please see the integration tests for the [classification models](https://github.com/numenta/nupic.research/blob/master/tests/nlp/integration/classification_models_validation_test.py) and the [Cortical.io encoders](https://github.com/numenta/nupic.research/blob/master/tests/nlp/integration/cio_encodings_test.py) for example usage.

The ["Fox-eats"](https://www.youtube.com/watch?v=X4XjYXFRIAQ&start=7084) demo is an example from the NuPIC 2013 Hackathon, using a previous version Numenta's NLP code.
<commit_msg>update nlp readme<commit_after># Language with NuPIC

This directory contains scripts for running NLP-based applications using [NuPIC](https://github.com/numenta/nupic) and [Cortical.io's API](http://www.cortical.io/developers.html).

Additional requirements:

- [cortipy](https://github.com/numenta/cortipy)
- Python modules [enum](https://pypi.python.org/pypi/enum34), [pandas](http://pandas.pydata.org/), and [numpy](http://www.numpy.org/).
- You must have a valid REST API key from [Cortical.io](http://www.cortical.io/developers.html).

## Use cases
In this directory are scripts for some NLP experiments. They each use one or more of the following models:

- Keywords model gives a random (seeded) encoding to each word.
- Cortical.io word fingerprints model encodes each word with their API, and then makes a union representation over the words in a block of text.
- Cortical.io document fingerprints model uses their API to encode a block of text.
- HTM Network models use the Cortical.io word-level encoder, and then run the patterns through the Network API.
- Cortical.io classify endpoint model is an end-to-end classification tool in their API.

#### Text querying

Please refer to the [Imbu project directory](https://github.com/numenta/nupic.research/tree/master/projects/imbu).

#### Classification examples

See the [hello_classification.py](https://github.com/numenta/nupic.research/blob/master/projects/nlp/hello_classification.py) and [simple_labels.py](https://github.com/numenta/nupic.research/blob/master/projects/nlp/simple_labels.py) scripts.
	


#### Additional examples

The ["Fox-eats"](https://www.youtube.com/watch?v=X4XjYXFRIAQ&start=7084) demo is an example from the NuPIC 2013 Hackathon, using a previous version Numenta's NLP code.

{'fraction_non_alphanumeric': 0.1013480982185845, 'fraction_numerical': 0.005296100144439095, 'mean_word_length': 4.803072625698324, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 14, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5358590', 'n_tokens_mistral': 1277, 'n_tokens_neox': 1202, 'n_words': 418}