<jupyter_start><jupyter_text>Plot for overall Word Frequency of the TranscribesWe plot how often hand-craftet, topic relevant words occur in the transcribes of the lecture recodrings<jupyter_code>!pip install tika
!pip install textblob
import pandas as pd
import glob
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
from tika import parser
from textblob import TextBlob
import nltk
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
nltk.download('brown')<jupyter_output>[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.
[nltk_data] Downloading package brown to /root/nltk_data...
[nltk_data]   Unzipping corpora/brown.zip.<jupyter_text>The Transcibes are expected to be in a folder called Transcribes<jupyter_code>lectures_spoken = []
path = 'Transcribes/*.txt'
files=glob.glob(path)
for file in sorted(files):
    with open(file, 'r') as f:
        lectures_spoken.append(f.read())
lectures_spoken = " ".join(lectures_spoken)<jupyter_output><empty_output><jupyter_text>Create a texblob of the text. This is used to extract the noun phrases.<jupyter_code>blob_spoken = TextBlob(lectures_spoken)
freq_spoken = nltk.FreqDist(blob_spoken.noun_phrases)
relevant_words = ['bayes', 'frequentist', 'fairness', 'divergence', 'reproduc', 'regulariz', 'pca', 'principal c' 'bootstrap', 'nonlinear function', 'linear function', 'entropy', 'maximum likelihood estimat', 'significa', 'iid', 'bayes theorem', 'visualization', 'score function', 'dimensionality reduction', 'estimat', 'bayes', 'consumption', 'fisher', 'independence', 'logistic regression', 'bias', 'standard deviation', 'linear discriminant analysis', 'information matrix', 'null hypothesis', 'log likelihood', 'linear regression', 'hypothesis test', 'confidence', 'variance', 'sustainability', 'gaussian', 'linear model', 'climate', 'laplace', ]<jupyter_output><empty_output><jupyter_text>This function checks if a noun phrase is sufficiently similar to a relevant word(templates). Sufficiently similar is defined as that the template is a substring of the noun phrase.<jupyter_code>def convert_to_template(df_element, template):
  for template_element in template:
    if template_element in df_element:
      return template_element
  return "None"<jupyter_output><empty_output><jupyter_text>We first create a pandas dataframe of all the noun phrases and their frequencies in the transcribes. After that, we extract all words that are similar to a relevant word (as of the convert_to_template function). Then we group by the relevant words.<jupyter_code>df = pd.DataFrame.from_dict({"word": list(freq_spoken.keys()), "freq_spoken" : list(freq_spoken.values())})
df["word"] = df["word"].apply(lambda x: convert_to_template(x,relevant_words))
df = df.groupby(["word"]).sum().reset_index()
df = df[df["word"] != "None"].reset_index()
df = df.sort_values(by="freq_spoken",ascending=False)
plt.figure(figsize =(10, 10))
p1 = sns.barplot(x="freq_spoken", y="word",data=df, color="orange",ci="sd")
plt.xlabel("Word Frequency", size="x-large")
plt.ylabel("Relevant Words", size="xx-large")
p1.set_yticklabels(p1.get_ymajorticklabels(), size="x-large")
#p1.set_xticklabels(p1.get_xmajorticklabels(), size="medium")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12514450867052024, 'fraction_numerical': 0.002601156069364162, 'mean_word_length': 7.2601431980906925, 'pattern_counts': {'":': 1, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '347942', 'license': 'MIT', 'n_tokens_mistral': 1186, 'n_tokens_neox': 1127, 'n_words': 301, 'path': 'Notebooks/WordFrequencyTranscribes.ipynb', 'repo_name': 'Eoli-an/Exam-topic-prediction', 'size': '3460'}