<jupyter_start><jupyter_text>Load text exampleWe use a story Russian children's writer Nosov.<jupyter_code>with open('text_statistics/data/text_example.txt', 'r', encoding='utf8') as f:
    text = f.read()<jupyter_output><empty_output><jupyter_text>Create an object<jupyter_code>stats = TextStatistics(text)<jupyter_output><empty_output><jupyter_text>First, we can take a look at main statistics- mean_sequence_lenght - mean number of words in sentence- mean_words_lenght - mean number of characters in word- n_unique - number of unique words- n_unique_forms - number of unique word forms (i.e. after normalisation)<jupyter_code>stats.calculate_main_statics()<jupyter_output><empty_output><jupyter_text>Frequency analysis of text. Returns Counter objectHere, we calculate frequencies of words and word n-gramms. Also, we can drow a bar of the most frequent words<jupyter_code>stats.get_frequent_statistics(max_n_gram=3, plot_most_frequent=False).most_common(10)
st = stats.get_frequent_statistics(max_n_gram=2, plot_most_frequent=True, n_freq=20)<jupyter_output><empty_output><jupyter_text>Morphological analysisWe can see what morphology author uses. All morphology statistics is provided in terms of pymorphy2 analyser.<jupyter_code>stats.get_morph_statistics()<jupyter_output><empty_output><jupyter_text>Lexical DensityLexical density is a special metric, that shows if text is from specific domain, based on terms dictionary<jupyter_code>terms = ['мальчик', 'ребенок', 'взмахнул', 'пряник', 'ребята']
terms_2 = ['анализ', 'ряд', 'свойство']

stats.get_lexical_density(terms), stats.get_lexical_density(terms_2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09428223844282238, 'fraction_numerical': 0.006082725060827251, 'mean_word_length': 8.508670520231213, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '327348', 'license': 'MIT', 'n_tokens_mistral': 569, 'n_tokens_neox': 551, 'n_words': 132, 'path': 'Usage example.ipynb', 'repo_name': 'AnastasiaMalysheva/text_statistical_analysis', 'size': '1644'}