<commit_before>REFERENCES:
http://www.jstor.org/stable/1418920 Miller & Selfridge 1950
10.1002/j.1538-7305.1951.tb01366.x Shannon 1951

This module shall allow the user to generate random sentences based on text by an author or from a genre of their choice.
The user shall choose whether to use a unigram, bigram or trigram (or more?) model.
The user shall supply some text, in the form of a text file, to be used as training data for the system.
The system shall contain functionality to randomly generate a sentence based on the n-gram probabilities learned from this training data
It shall also be able to return the x most common words from the text, and all n-grams that begin with a specified word.<commit_msg>Updated to match functionality<commit_after>See Jurafksy and Martin (2009) p126-8 for information about the idea.

This module shall allow the user to generate random sentences based on text by an author or from a genre of their choice.
The module shall provide a command line interface for the user to interact with.
The user shall choose whether to use a unigram, bigram, trigram or quadrigram  model.
The user shall supply some text, in the form of a text file, to be used as training data for the system.
The system shall contain functionality to randomly generate a sentence based on the n-gram probabilities learned from this training data
It shall also be able to return a specified number of most common words from the text, the history which has the most possible words after it,
and all n-grams that begin with a specified history.

The module consists of three files:
linked_list.py contains a class representing a linked list, and associated sorting, searching, insertion, deletion etc methods, and a class representing a node.
ngrams_list.py contains a class representing an n-gram language model learned from a text file, and the associated methods for providing the specified functionality.
interface.py contains a function to provide a command-line interface to the NGramModel class, for easier interaction.
{'fraction_non_alphanumeric': 0.03334968121628249, 'fraction_numerical': 0.022560078469838155, 'mean_word_length': 4.6353591160221, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '609943', 'n_tokens_mistral': 517, 'n_tokens_neox': 473, 'n_words': 321}