<jupyter_start><jupyter_text>Search text with context<jupyter_code>text6.concordance("swallow")
text6.similar("Soldier")
text6.common_contexts(["oh", "very"])
text6.dispersion_plot(["swallow", "European", "it", "oh", "very"])
len(text6)
sorted(set(text6))<jupyter_output><empty_output><jupyter_text>Measure the lexical richness of the text ~ 12.8% of the tokens are unique in this text<jupyter_code>len(set(text6)) / len(text6)
text6.count("Allo")
sentence = ['Melby', 'is', 'the', 'best']
text5[16715:16735]
FreqDist(text6)<jupyter_output><empty_output><jupyter_text>"the set of all w such that w is an element of V (the vocabulary) and w has property P".{w | w ∈ V & P(w)}[w for w in V if p(w)]<jupyter_code>V = set(text1)
long_words = [w for w in V if len(w) > 16]
sorted(long_words)<jupyter_output><empty_output><jupyter_text>A collocation is a sequence of words that occur together unusually often. Thus red wine is a collocation, whereas the wine is not. A characteristic of collocations is that they are resistant to substitution with words that have similar senses; for example, maroon wine sounds definitely odd.To get a handle on collocations, we start off by extracting from a text a list of word pairs, also known as bigrams. This is easily accomplished with the function bigrams():<jupyter_code>#list(bigrams(['more', 'is', 'said', 'than', 'done']))
text6.collocations()
fdist = FreqDist(len(w) for w in text6)
fdist.most_common()
fdist.plot(cumulative=True)
[len(w) for w in text6]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1259016393442623, 'fraction_numerical': 0.019672131147540985, 'mean_word_length': 6.785714285714286, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '351020', 'license': 'Apache-2.0', 'n_tokens_mistral': 552, 'n_tokens_neox': 520, 'n_words': 172, 'path': 'Natural Language Processing.ipynb', 'repo_name': 'Sebbenbear/notebooks', 'size': '1525'}