<commit_before>#Introduction
This repository contains a Python script that demonstrates usage of the Rosette API Python bindings to summarize documents.

##Summarization Algorithm
The summarization algorithm operates on sentences, words, and named entities in a document extracted by the Rosette API.  Each sentence is assigned a score based on the contentful words and named entity mentions that occur in the sentence.  The score for each sentence is normalized by the length of the sentence in terms of the number of word tokens in the sentence.  Sentences are also penalized with a logarithmic falloff based on how close to the beginning of the document they occur (setences near the end of the document are penalized more than sentences toward the beginning).

##`summarize.py`
This script uses both the [`entities`](https://developer.rosette.com/features-and-functions#entity-extraction) and [`morphology/lemmas`](https://developer.rosette.com/features-and-functions#lemmas) endpoints to extract information about a document and uses that information to rank sentences by their content.  The script can then filter down to only the most contentful sentences to provide a gist of the original content.

###Installing Dependencies with Virtualenv
The script is written for Python 3.  If you are alright with installing external Python packages globally, you may skip this section.<commit_msg>remove introductory sentence<commit_after>#Introduction
The summarization algorithm operates on sentences, words, and named entities in a document extracted by the Rosette API.  Each sentence is assigned a score based on the contentful words and named entity mentions that occur in the sentence.  The score for each sentence is normalized by the length of the sentence in terms of the number of word tokens in the sentence.  Sentences are also penalized with a logarithmic falloff based on how close to the beginning of the document they occur (setences near the end of the document are penalized more than sentences toward the beginning).

##`summarize.py`
This script uses both the [`entities`](https://developer.rosette.com/features-and-functions#entity-extraction) and [`morphology/lemmas`](https://developer.rosette.com/features-and-functions#lemmas) endpoints to extract information about a document and uses that information to rank sentences by their content.  The script can then filter down to only the most contentful sentences to provide a gist of the original content.

###Installing Dependencies with Virtualenv
The script is written for Python 3.  If you are alright with installing external Python packages globally, you may skip this section.
{'fraction_non_alphanumeric': 0.044084400904295405, 'fraction_numerical': 0.0007535795026375283, 'mean_word_length': 5.217798594847775, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2906649', 'n_tokens_mistral': 610, 'n_tokens_neox': 581, 'n_words': 361}