<jupyter_start><jupyter_text>@font-face {font-family: "B Lotus"; src: url("//db.onlinewebfonts.com/t/1605a655ba0a3246ce5eca3eaff6c5c2.eot"); src: url("//db.onlinewebfonts.com/t/1605a655ba0a3246ce5eca3eaff6c5c2.eot?iefix") format("embedded-opentype"), url("//db.onlinewebfonts.com/t/1605a655ba0a3246ce5eca3eaff6c5c2.woff2") format("woff2"), url("//db.onlinewebfonts.com/t/1605a655ba0a3246ce5eca3eaff6c5c2.woff") format("woff"), url("//db.onlinewebfonts.com/t/1605a655ba0a3246ce5eca3eaff6c5c2.ttf") format("truetype"), url("//db.onlinewebfonts.com/t/1605a655ba0a3246ce5eca3eaff6c5c2.svgB Lotus") format("svg"); }بازیابی پیشرفته اطلاعات - درس پردازش زبان‌های طبیعی  آزمایشگاه پردازش هوشمند متن و زبان و علوم انسانی محاسباتی  http://language.ml  contact: ehsan.asgari [AT] sharif [dot] edu  semesters: Fall 2021 / Spring 2022 (Updated)   انگلیسی - پیش پردازش و بررسی متن         § نمونه‌ای از متن<jupyter_code>import random 
import tqdm
import nltk, re
from nltk.corpus import brown
from nltk import word_tokenize
import string

#nltk.download('brown')
all_categories = brown.categories()
all_categories
all_brown_sentences_religion = list(brown.sents(categories=['religion']))
all_brown_sentences_news = list(brown.sents(categories=['news']))

all_brown_tagged_sentences = list(brown.tagged_sents(categories=all_categories))
for x in random.sample(all_brown_sentences_religion, 30):
    print(' '.join(x))<jupyter_output>16 ) .
what next ? ?
we cover each of these with our blanket of peas and then at last we have octillion peas corresponding in number to the atoms in the body .
To the Zen monk the universe is still populated with `` spiritual beings '' who have to be appeased .
1 St. John 12 .
So we are the more prepared to give Parker the credit for having taken the right side in an unnecessary controversy , to salute his courage , and to pass on , happily forgetting both him and the entire episode .
20 ) .
The new birth is necessary because the natural man is spiritually dead and blind .
According to this doctrine , the universe was ruled by Heaven , T'ien -- as a natural force , or in the personification of a Supreme Sky-god -- governing all things by means of a process called the Tao , which can be roughly interpreted as `` the Order of the Universe '' or `` the Universal Way '' .
But , you see , those who are not mentioned will not resent it .
After this form of Indian Buddhism had been introduced[...]<jupyter_text>§ نرمالایز کردن<jupyter_code>def normalize_sentence(tokenized_sents, minimum_length=2, stopword_removal=True, stopwords_domain=[], lower_case=False, punctuation_removal=True):
    '''
    normalization function
    '''
    normalized_sents = tokenized_sents
    
    if stopword_removal:
        # Remove stopwords in English and also the given domain stopwords
        stopwords = [x.lower() for x in nltk.corpus.stopwords.words('english')]
        normalized_sents=[[word for word in sentence if (word.lower() not in stopwords_domain + stopwords)] for sentence in tokenized_sents ]

    if punctuation_removal:
        # Remove punctuations
        normalized_sents=[[word for word in sentence if word not in string.punctuation] for sentence in normalized_sents ]

    if lower_case:
        # Convert everything to lowercase and filter based on a min length
        normalized_sents=[[word.lower() for word in sentence if len(word)>minimum_length] for sentence in normalized_sents ]

    elif minimum_length>1:
        normalized_sents= [[word for word in sentence if len(word)>minimum_length] for sentence in normalized_sents ]        
        
    return normalized_sents
normalized_sentences_religion = normalize_sentence(all_brown_sentences_religion)
normalized_sentences_news = normalize_sentence(all_brown_sentences_news)
for x in normalized_sentences_news[0:3] + normalized_sentences_religion[0:3]:
    print(x)
    print()
from nltk.probability import FreqDist
import numpy as np
import itertools

mp_freqdist = FreqDist(itertools.chain(*normalized_sentences_religion))               # compute the frequency distribution
top20words=mp_freqdist.most_common(20)      # show the top 20 (word, frequency) pairs
print ('%-16s' % 'word', '%-16s' % 'Frequency','%-16s' %  '% of the total')
for topword in top20words:
    percent=(topword[1]/len(normalized_sentences_religion))*100
    print ('%-16s' % topword[0], '%-16s' % topword[1],'%-16s' %  percent)<jupyter_output>word             Frequency        % of the total  
God              131              7.634032634032634
world            90               5.244755244755245
one              87               5.06993006993007
may              78               4.545454545454546
new              77               4.487179487179487
would            68               3.9627039627039626
man              64               3.7296037296037294
could            59               3.4382284382284385
Christ           59               3.4382284382284385
also             56               3.263403263403263
life             55               3.205128205128205
must             54               3.146853146853147
church           51               2.972027972027972
Christian        50               2.9137529137529135
power            49               2.8554778554778557
members          49               2.8554778554778557
spirit           46               2.6806526806526807
many             46               2.6806526806526807
human[...]<jupyter_text>Religion<jupyter_code>sentence_words = list(itertools.chain(*normalized_sentences_religion))
print ('%-16s' % 'Number of words', '%-16s' % len(sentence_words))
print ('%-16s' % 'Number of unique words', '%-16s' % len(set(sentence_words)))
avg=np.sum([len(word) for word in sentence_words])/len(sentence_words)
print ('%-16s' % 'Average word length', '%-16s' % avg)
avg_sentence_length_c=np.mean([len(' '.join(sentence)) for sentence in sentence_words])
print ('%-16s' % 'Average sentence length in characters', '%-16s' % avg_sentence_length_c)
print ('%-16s' % 'Longest word', '%-16s' % sentence_words[np.argmax([len(word) for word in sentence_words])])<jupyter_output>Number of words  17145           
Number of unique words 6081            
Average word length 6.614289880431612
Average sentence length in characters 12.228579760863225
Longest word     Preparation-Inquirers'<jupyter_text>News<jupyter_code>sentence_words = list(itertools.chain(*normalized_sentences_news))
print ('%-16s' % 'Number of words', '%-16s' % len(sentence_words))
print ('%-16s' % 'Number of unique words', '%-16s' % len(set(sentence_words)))
avg=np.sum([len(word) for word in sentence_words])/len(sentence_words)
print ('%-16s' % 'Average word length', '%-16s' % avg)
avg_sentence_length_c=np.mean([len(' '.join(sentence)) for sentence in sentence_words])
print ('%-16s' % 'Average sentence length in characters', '%-16s' % avg_sentence_length_c)
print ('%-16s' % 'Longest word', '%-16s' % sentence_words[np.argmax([len(word) for word in sentence_words])])<jupyter_output>Number of words  50351           
Number of unique words 13970           
Average word length 6.407380191058768
Average sentence length in characters 11.814760382117536
Longest word     Scotch-Irish-Scandinavian<jupyter_text>§ توکنایزیشن<jupyter_code># Sample

text = 'The delta variant of COVID-19 is not done with the U.S.'

# http://stackoverflow.com/questions/36353125/nltk-regular-expression-tokenizer
pattern = r'''(?x)          # set flag to allow verbose regexps
        (?:[A-Z]\.)+        # abbreviations, e.g. U.S.A.
      | \w+(?:-\w+)*        # words with optional internal hyphens
      | \$?\d+(?:\.\d+)?%?\s?  # currency and percentages, e.g. $12.40, 82%
      | \.\.\.              # ellipsis
      | [][.,;"'?():_`-]    # these are separate tokens; includes ], [
    '''

pattern = re.compile(pattern)

nltk.regexp_tokenize(text, pattern)
normalized_sentences_religion[0:10]<jupyter_output><empty_output><jupyter_text>§ جمله‌بندی<jupyter_code># Example in NLTK

from nltk.tokenize import sent_tokenize

text="""Hi Tom, how are you? I’m fine, thanks. How are you? Great! thank you."""

sents=sent_tokenize(text)
for x in sents:
    print(x)

# Explore Spacy
#!pip install spacy
#!python -m spacy download en_core_web_sm

import spacy

nlp = spacy.load('en_core_web_sm') # or whatever model you have installed
doc = nlp(text)
sents = [sent for sent in doc.sents]
for x in sents:
    print(x)<jupyter_output>Hi Tom, how are you?
I’m fine, thanks.
How are you?
Great!
thank you.
Hi Tom, how are you?
I’m fine, thanks.
How are you?
Great!
thank you.<jupyter_text>§ تحلیل بسامد<jupyter_code>from nltk import FreqDist
import itertools
import pandas as pd

all_tokens_religion = list(itertools.chain(*normalized_sentences_religion))
all_tokens_news = list(itertools.chain(*normalized_sentences_news))


dataframe = {}

for opt in ['religion', 'news']:
     dataframe[opt] = FreqDist(eval(F"all_tokens_{opt}")).most_common(50)

freq_analysis = pd.DataFrame(dataframe)    
freq_analysis<jupyter_output><empty_output><jupyter_text>§ استفاده از lemmatization, stemming<jupyter_code>import nltk
from nltk.stem.porter import PorterStemmer
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
stemmer  = PorterStemmer()


def get_lemma_set(tok, opt=1):
    if opt ==1:
        return stemmer.stem(tok)
    if opt ==2:
        return lemmatizer.lemmatize(tok)
    if opt ==3:
        # write your own
        pass
opt = 2

religion_tokens_nonstop_lemstem =   [get_lemma_set(t, opt) for t in tqdm.tqdm(all_tokens_religion)]
news_tokens_nonstop_lemstem = [get_lemma_set(t, opt) for t in tqdm.tqdm(all_tokens_news)]

dataframe_nonstop_lemstem = {}

for opt in ['religion', 'news']:
     dataframe_nonstop_lemstem[opt] = FreqDist(eval(F"{opt}_tokens_nonstop_lemstem")).most_common(50)

freq_analysis_nonstop_lemstem = pd.DataFrame(dataframe_nonstop_lemstem)   
freq_analysis_nonstop_lemstem<jupyter_output>100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17145/17145 [00:01<00:00, 12446.07it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50351/50351 [00:00<00:00, 271495.51it/s]<jupyter_text>§ استفاده از POS-tags<jupyter_code>from nltk import word_tokenize, pos_tag
def create_data_sets(sentences):
    size = int(len(sentences) * 0.9)
    train_sents = sentences[:size]
    test_sents = sentences[size:]
    return train_sents, test_sents

def build_backoff_tagger (train_sents):
    t0 = nltk.DefaultTagger('NN')
    t1 = nltk.UnigramTagger(train_sents, backoff=t0)
    t2 = nltk.BigramTagger(train_sents, backoff=t1)
    return t2


def train_tagger(already_tagged_sents):
    train_sents, test_sents = create_data_sets(already_tagged_sents)
    ngram_tagger = build_backoff_tagger(train_sents)
    print ("%0.3f pos accuracy on test set" % ngram_tagger.evaluate(test_sents))
    return ngram_tagger    
brown_tagged_sents = brown.tagged_sents(categories=['adventure', 'belles_lettres', 'editorial', 'fiction', 'government', 'hobbies',
    'humor', 'learned', 'lore', 'mystery', 'religion', 'reviews', 'romance','science_fiction'])
brown_tagger = train_tagger(brown_tagged_sents)
brown_tagger.tag(globals()[F'normalized_sentences_{opt}'][0])
pos_tag(globals()[F'normalized_sentences_{opt}'][0])
import copy
dataframe_nonstop_lemstem_advjj = {}

for opt in tqdm.tqdm(['religion', 'news']):
    selected = []
    for sentence in globals()[F'normalized_sentences_{opt}']:
        tagged_sentence = brown_tagger.tag(sentence)
        for w, pos in tagged_sentence:
            if pos=='JJ':
                selected.append(w) 
    dataframe_nonstop_lemstem_advjj[opt]= list(FreqDist(selected).most_common(40))

dataframe_nonstop_lemstem_advjj = pd.DataFrame(dataframe_nonstop_lemstem_advjj)    
dataframe_nonstop_lemstem_advjj<jupyter_output><empty_output><jupyter_text>§ دیدن سیاق<jupyter_code>hafez_text = nltk.Text(religion_tokens_nonstop_lemstem)
hafez_text.concordance('Spirit')<jupyter_output>Displaying 25 of 25 matches:
 spirit matter term generally taken granted 
cisely one must modern world neither spirit matter refer generally agreed-upon e
t almost universal disbelief reality spirit century ago world spirit populous re
ief reality spirit century ago world spirit populous real world material entity 
man rock tree star world word matter spirit referred directly known reality comm
ategory reality large extent emptied spirit world entity previously populated ca
aim due credit objectification world spirit popular superstition certainly gone 
certainly gone far beyond experience spirit could justify support Science fully 
ty popular imagination peopled world spirit entity soon lost whatever status enj
nmixed blessing scientific debunking spirit world way successful thorough house 
y inadequate mean dealing experience spirit Although particular form conceptuali
imagination made response experience spirit undoubtedly defective raw experience
 leaf play spell kind spell exposur[...]<jupyter_text>§ عبارت یابی<jupyter_code>from nltk.corpus import inaugural
#nltk.download('inaugural')
print(inaugural.fileids())
print()
Lincoln = list(inaugural.words(inaugural.fileids()[18]))
Obama = list(inaugural.words(inaugural.fileids()[56]))
Trump = list(inaugural.words(inaugural.fileids()[57]))

Lincoln =' '.join(Lincoln)
Lincoln_sentences =  sent_tokenize(Lincoln)
Lincoln_sentences =  [x.split() for x in Lincoln_sentences]
Lincoln_sentences_tagged = [brown_tagger.tag(sent) for sent in Lincoln_sentences]

Obama =' '.join(Obama)
Obama_sentences =  sent_tokenize(Obama)
Obama_sentences =  [x.split() for x in Obama_sentences]
Obama_sentences_tagged = [brown_tagger.tag(sent) for sent in Obama_sentences]

Trump =' '.join(Trump)
Trump_sentences =  sent_tokenize(Trump)
Trump_sentences =  [x.split() for x in Trump_sentences]
Trump_sentences_tagged = [brown_tagger.tag(sent) for sent in Trump_sentences]
sentence_words = list(itertools.chain(*Lincoln_sentences))
print ('%-16s' % 'Number of words', '%-16s' % len(sentence_words))
print ('%-16s' % 'Number of unique words', '%-16s' % len(set(sentence_words)))
avg=np.sum([len(word) for word in sentence_words])/len(sentence_words)
print ('%-16s' % 'Average word length', '%-16s' % avg)
avg_sentence_length_c=np.mean([len(' '.join(sentence)) for sentence in sentence_words])
print ('%-16s' % 'Average sentence length in characters', '%-16s' % avg_sentence_length_c)
print ('%-16s' % 'Longest word', '%-16s' % sentence_words[np.argmax([len(word) for word in sentence_words])])
sentence_words = list(itertools.chain(*Obama_sentences))
print ('%-16s' % 'Number of words', '%-16s' % len(sentence_words))
print ('%-16s' % 'Number of unique words', '%-16s' % len(set(sentence_words)))
avg=np.sum([len(word) for word in sentence_words])/len(sentence_words)
print ('%-16s' % 'Average word length', '%-16s' % avg)
avg_sentence_length_c=np.mean([len(' '.join(sentence)) for sentence in sentence_words])
print ('%-16s' % 'Average sentence length in characters', '%-16s' % avg_sentence_length_c)
print ('%-16s' % 'Longest word', '%-16s' % sentence_words[np.argmax([len(word) for word in sentence_words])])
sentence_words = list(itertools.chain(*Trump_sentences))
print ('%-16s' % 'Number of words', '%-16s' % len(sentence_words))
print ('%-16s' % 'Number of unique words', '%-16s' % len(set(sentence_words)))
avg=np.sum([len(word) for word in sentence_words])/len(sentence_words)
print ('%-16s' % 'Average word length', '%-16s' % avg)
avg_sentence_length_c=np.mean([len(' '.join(sentence)) for sentence in sentence_words])
print ('%-16s' % 'Average sentence length in characters', '%-16s' % avg_sentence_length_c)
print ('%-16s' % 'Longest word', '%-16s' % sentence_words[np.argmax([len(word) for word in sentence_words])])
def chunkTechnicalTerm(sentence):
    grammar = r"""
      TECHTERM: {<JJ|NN>+<NN|CD>|<NN>}
    """
    cp = nltk.RegexpParser(grammar)
    return (cp.parse(sentence))
import copy

technical_terms = {}
opt_technical_terms = {}



for opt in tqdm.tqdm(['Lincoln', 'Obama','Trump']):
    technical_terms[opt] = []
    tagged = globals()[F'{opt}_sentences_tagged']
    for sentence in tagged:
        tree=chunkTechnicalTerm(sentence)
        for subtree in tree.subtrees():
            if subtree.label() == 'TECHTERM':
                technical_terms[opt].append(subtree.leaves())
for opt in tqdm.tqdm(['Lincoln', 'Obama','Trump']):
    opt_technical_terms[opt]=FreqDist([' '.join([x for x,y in sent]) for sent in technical_terms[opt] if len(sent)>1]).most_common(40)    
                                   
                                   

opt_technical_terms = pd.DataFrame(opt_technical_terms) 
opt_technical_terms<jupyter_output><empty_output><jupyter_text>§ WordNet<jupyter_code>from nltk.collocations import *
from string import *

def freq_normed_unigrams(sents):
    wnl = WordNetLemmatizer() # to get word stems
    
    tagged_POS_sents = [nltk.pos_tag(sent) for sent in sents] # tags sents
    
    normed_tagged_words = [wnl.lemmatize(word[0].lower()) for sent in tagged_POS_sents
                           for word in sent 
                           if word[0].lower() not in nltk.corpus.stopwords.words('english')
                           and word[0] not in punctuation # remove punctuation
                           and not re.search(r'''^[\.,;"'?!():\-_`]+$''', word[0])
                           and word[1].startswith('N')]  # include only nouns

    top_normed_unigrams = [word for (word, count) in nltk.FreqDist(normed_tagged_words).most_common(40)]
    return top_normed_unigrams
Trump_top_words = freq_normed_unigrams(Trump_sentences)
Obama_top_words = freq_normed_unigrams(Obama_sentences)
Lincoln_top_words = freq_normed_unigrams(Lincoln_sentences)
from nltk.corpus import wordnet as wn
from collections import defaultdict

def categories_from_hypernyms(termlist):
    hypterms = []
    hypterms_dict = defaultdict()
    for term in termlist:                  # for each term
        s = wn.synsets(term.lower(), 'n')  # get its nominal synsets
        for syn in s:                      # for each lemma synset
            for hyp in syn.hypernyms():    # It has a list of hypernyms
                hypterms.append(hyp.name())      # Extract the hypernym name and add to list
                if hyp.name() not in hypterms_dict:
                    hypterms_dict[hyp.name()] = list()
                hypterms_dict[hyp.name()].append(term)  # Extract examples and add them to dict
                
    hypfd = nltk.FreqDist(hypterms)             # After going through all the nouns, print out the hypernyms 
    for (name, count) in hypfd.most_common(25):  # that have accumulated the most counts (have seen the most descendents)
        print( name, '({0})'.format(count))
        print ('\t', ', '.join(set(hypterms_dict[name])))  # show the children found for each hypernym
        print ()
categories_from_hypernyms(Trump_top_words)
categories_from_hypernyms(Obama_top_words)
categories_from_hypernyms(Lincoln_top_words)<jupyter_output>person.n.01 (6)
	 slave, case, authority, party

time_period.n.01 (6)
	 term, time, year

concept.n.01 (4)
	 section, part, law

work.n.01 (4)
	 labor, duty, service

group.n.01 (3)
	 people

organization.n.01 (3)
	 union, institution, party

state.n.02 (3)
	 union, office, power

social_control.n.01 (3)
	 duty, administration, government

administrative_district.n.01 (2)
	 state

attribute.n.02 (2)
	 state, time

political_unit.n.01 (2)
	 union, state

executive_department.n.01 (2)
	 labor, state

beginning.n.05 (2)
	 institution, constitution

collection.n.01 (2)
	 hand, law

force.n.04 (2)
	 service, law

sexual_activity.n.01 (2)
	 union, congress

happening.n.01 (2)
	 union, case

container.n.01 (2)
	 case

activity.n.01 (2)
	 service, provision

number.n.01 (2)
	 majority, minority

age.n.03 (2)
	 majority, minority

administrative_unit.n.01 (2)
	 office, authority

goal.n.01 (2)
	 purpose, object

constituent.n.04 (2)
	 term, object

resoluteness.n.01 (2)
	 purpose, decision
{'fraction_non_alphanumeric': 0.12456506610995129, 'fraction_numerical': 0.04314544189283229, 'mean_word_length': 4.088265048052605, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '580422', 'license': 'Apache-2.0', 'n_tokens_mistral': 7775, 'n_tokens_neox': 6824, 'n_words': 1667, 'path': 'notebooks/English/ExploreYourTextCollection-English.ipynb', 'repo_name': 'language-ml/course-nlp-ir-1-text-exploring', 'size': '20118'}