<jupyter_start><jupyter_text>**Check of context of word**<jupyter_code>text1.concordance("monstrous")
text5.concordance("lol")<jupyter_output>Displaying 25 of 822 matches:
ast PART 24 / m boo . 26 / m and sexy lol U115 boo . JOIN PART he drew a girl w
ope he didnt draw a penis PART ewwwww lol & a head between her legs JOIN JOIN s
a bowl i got a blunt an a bong ...... lol JOIN well , glad it worked out my cha
e " PART Hi U121 in ny . ACTION would lol @ U121 . . . but appearently she does
30 make sure u buy a nice ring for U6 lol U7 Hi U115 . ACTION isnt falling for 
 didnt ya hear !!!! PART JOIN geeshhh lol U6 PART hes deaf ppl here dont get it
es nobody here i wanna misbeahve with lol JOIN so read it . thanks U7 .. Im hap
ies want to chat can i talk to him !! lol U121 !!! forwards too lol JOIN ALL PE
k to him !! lol U121 !!! forwards too lol JOIN ALL PErvs ... redirect to U121 '
 loves ME the most i love myself JOIN lol U44 how do u know that what ? jerkett
ng wrong ... i can see it in his eyes lol U20 = fiance Jerketts lmao wtf yah I 
cooler by the minute what 'd I miss ? lol noo there too much work ! why not ?? 
 that mean[...]<jupyter_text>What other words appear in a similar range of contexts?<jupyter_code>text1.similar("monstrous")<jupyter_output>true contemptible christian abundant few part mean careful puzzled
mystifying passing curious loving wise doleful gamesome singular
delightfully perilous fearless<jupyter_text>Contexts that are shared by two or more words.<jupyter_code>text2.common_contexts(["monstrous","very"])<jupyter_output>a_pretty am_glad a_lucky is_pretty be_glad<jupyter_text>Positional information of a word<jupyter_code>text4.dispersion_plot(["citizens", "democracy", "freedom", "duties", "America"])<jupyter_output><empty_output><jupyter_text>Here, each stripe representsan instance of a word, and each row represents the entire text.  Enerating some random text.<jupyter_code>text3.generate(length=10)<jupyter_output><empty_output><jupyter_text>Tokens: Count the words in a text<jupyter_code>len(text1)<jupyter_output><empty_output><jupyter_text>Hence, <NAME> by <NAME> 1851 has 260819 words and punctuations or "tokens".  **Word Type:To find distinct words.**<jupyter_code>set(text1)
sorted(set(text1))
len(set(text1))<jupyter_output><empty_output><jupyter_text>**Lexical richness: Number of unique words used in a text.**<jupyter_code>from __future__ import division
a=len(text1)/len(set(text1))
print("Average usage time of each word is",a)
text3.count("smote")
b=100*text3.count("smote")/len(text3)
print("Percentage of occurence of the given word is",b,"%")
text5.count("lol")
c=100*text5.count("lol")/len(text5)
print("Percentage of occurence of the given word is",c,"%")
def lexical_diversity(text):
  return len(text) / len(set(text))


def percentage(count, total):
  return 100 * count / total

percentage(text4.count('a'), len(text4))
lexical_diversity(text5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10030498136225008, 'fraction_numerical': 0.027787190782785496, 'mean_word_length': 5.530973451327434, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 1, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '344996', 'license': 'Apache-2.0', 'n_tokens_mistral': 1063, 'n_tokens_neox': 978, 'n_words': 357, 'path': 'NLP_Day1.ipynb', 'repo_name': 'Innovation-tech-max/NLP', 'size': '2963'}