<jupyter_start><jupyter_text>A few quantitative insights into Khamenei's speeches from 1997 to 2016 This notebook provides code used to generate some preliminary quantiative insights into a corpus of Khamenei's speeches, based on modern techniques in corpus/computational linguistics, and relying on two Python libraries, nltk and buzz.<jupyter_code># the main library we use for text processing; models text 'corpora'
from buzz import Corpus 
raw = Corpus('data/txt')
# parse the raw texts, already done, only takes ~1 minute, creates 'txt-parsed'
#parsed = raw.parse()
c = Corpus('data/txt-parsed')
# load corpus into memory for fast searching
l = c.load() 
# switch on the use of data visualisation in this notebook
%matplotlib inline
%pylab inline

# set size and colors for visualisations
import pandas as pd
import seaborn as sns
pylab.rcParams['figure.figsize'] = (12, 6)
colormap = sns.diverging_palette(220, 10, sep=10, n=20)<jupyter_output>Populating the interactive namespace from numpy and matplotlib<jupyter_text>The dataset is now inside a spreadsheet-like format, with one word and its features per line. We can take a look at some of the annotations:<jupyter_code># a slide of our dataset, with each word annotated grammatically
# The data is multiindexed, with text, then sentence, then word. Other texts could be added for comparative work
l[["w", "l", "p", "x", "f", "year"]]
# total words in our corpus
len(l)
# total number of verbs in our corpus
len(l.just.x.VERB)<jupyter_output><empty_output><jupyter_text>most common verb lemmas by year<jupyter_code>l.just.x.VERB.table(show="l", subcorpora="year")
# skip modals, like can, could, would, should, because these are very common verbs
verbs = l.skip.w("the").skip.p.MD.just.x.VERB
# show the base form (i.e. coming -> come) of verbs in this case
verbs = verbs.table(subcorpora="year", show="l", keyness="ll").iloc[:,:10]
verbs<jupyter_output><empty_output><jupyter_text>relative frequency of most common nouns by year<jupyter_code>noun = l.just.x.NOUN
# year 1998 is skipped, as the text from this year is very small
nouns = noun.skip.year('1998').table(subcorpora="year", relative = True, show="l").iloc[:,:10]
nouns
# same table only for 2009, when the "green movement" started
noun.just.year('2009').table(relative = True, show="l").iloc[:,:10]
# generate our heatmap, rounding to two decimal places
heat = sns.heatmap(nouns.round(2), annot=True, cmap=colormap)

# add title
heat.set_title('Noun heatmap by year')
plot = nouns.plot()
plot.set_title("Noun frequency by year")<jupyter_output><empty_output><jupyter_text>Concordancy tables Concordances can help us understand how a word is being deployed in context. It can help us move between quantitative and qualitative perspectives, ensuring that we are counting what we assume we are.<jupyter_code>l.just.w("Israel").conc(show=["w"])[["left", "match", "right","year"]]
# concordance table for the words America and States apperaing as the subject of the sentence
l.just.w("(America|States)$").just.f("nsubj").conc(show=["w", "f", "p"])[["left","match", "right", "year"]] 
l.just.l("regime").conc()[["left", "match","year"]]<jupyter_output><empty_output><jupyter_text>Word Counts<jupyter_code># counting the occurrences of JCPOA in the corpus, it's clear that the term came into existence in 2015
l.just.w("JCPOA").year.value_counts()
# Ahmadinejad was the president from 2005 to 2013
l.just.w("Ahmadinejad").year.value_counts()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12051649928263988, 'fraction_numerical': 0.01606886657101865, 'mean_word_length': 6.014084507042254, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '328338', 'license': 'MIT', 'n_tokens_mistral': 1204, 'n_tokens_neox': 1109, 'n_words': 391, 'path': 'Khamenei-analysis.ipynb', 'repo_name': 'mahsavafaie/Khamenei-Corpus', 'size': '3485'}