<jupyter_start><jupyter_text>This notebook can be used to explore the dataset of policy documents with some prepared text mining/natural language procesing algorithms.It expects that a data frame has been produced by either the default pipeline or a custom analysis pipeline.Click on a cell and execute it with 'Shift + Enter' or 'Strg + Enter'.<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>The next cell reads in the preprocessed dataframe (after the default pipeline has run). Point the path to the location of the 'test.pkl' file.<jupyter_code>df = pd.read_pickle("results/test.pkl")<jupyter_output><empty_output><jupyter_text>The next cell displays the columns in the dataframe.<jupyter_code>df.columns<jupyter_output><empty_output><jupyter_text>The next cell counts the missing or Null values per column.<jupyter_code>for col in df.columns:
    nancount = len(df[df[col].isnull()])
    print("%d Null values in %s." %(nancount, col))
# Importing libraries for natural language processing
from gensim import corpora
from gensim import corpora, models, similarities
from nltk.tokenize import sent_tokenize, word_tokenize
import nltk
from nltk.corpus import stopwords
from nltk.text import Text
from itertools import chain
stop_en = stopwords.words('english')<jupyter_output><empty_output><jupyter_text>Additional stopwords can simply be added by filling in the list in the following cell, e.g.:`stop_en.extend(["extra_stopword1", "extra_stopword2"])`<jupyter_code>stop_en.extend([])
stop_en<jupyter_output><empty_output><jupyter_text>The next two cells extract the fulltext contents from the dataframe and prepare it for the text analysis.<jupyter_code># get a list of all fulltexts from the dataframe
fulltexts = df['fulltext']
fulltexts = [" ".join(ft) for ft in fulltexts]
# remove multiple linebreaks from fulltexts, this helps cleaning up artifacts from pdf extraction
fulltexts = [ft.replace('\n  \n', '\n') for ft in fulltexts]
fulltexts = [ft.replace('\n \n', '\n') for ft in fulltexts]
fulltexts = [ft.replace('\n\n', '\n') for ft in fulltexts]
fulltexts = [ft.replace('\n', ' ') for ft in fulltexts]<jupyter_output><empty_output><jupyter_text>The next cell makes a concordance analysis of a keyword, that can defined in the first line. Unfortunately, NLTK currently offers no support for multi-word concordance.<jupyter_code>keyword = "science"

def get_all_concordances(keyword, fulltext, pre = 5, post = 5):
    tokens = nltk.word_tokenize(fulltext)
    text = nltk.Text(tokens)
    c = nltk.ConcordanceIndex(text.tokens, key = lambda s: s.lower())
    concordances = []
    for offset in c.offsets(keyword):
        concordances.append(text.tokens[offset-pre:offset+post+1])
    result = [''.join([x+' ' for x in conc]) for conc in concordances]
    return result


results = get_all_concordances(keyword, fulltexts[0])
results
text = nltk.Text(nltk.word_tokenize(fulltexts[0]))
text.dispersion_plot(["research", "science", "open", "data", "open science", "Science", "Cloud"])
# lowercase and tokenize texts, returns a list of tokens (words) per fulltext
texts = [[word for word in nltk.wordpunct_tokenize(document.lower()) if word not in stop_en] for document in fulltexts]
# remove all tokens that appear only once in the whole corpus
from collections import defaultdict
frequency = defaultdict(int)
for text in texts:
    for token in text:
        frequency[token] += 1
texts = [[token for token in text if frequency[token] > 1] for text in texts]
# create a dictionary over all texts (ID to str map)
dictionary = corpora.Dictionary(texts)
# create a corpus by mapping all tokens to their IDs
corpus = [dictionary.doc2bow(text) for text in texts]
# create a Term Frequency - Inverse Document Frequency model of the corpus
tfidf = models.TfidfModel(corpus)
corpus_tfidf = tfidf[corpus]
# create an Latent Semantic Analysis (dimensionality reduction) model over the TF-IDF corpus
# vary the num_topics parameter
lsi = models.LsiModel(corpus_tfidf, id2word=dictionary, num_topics=10)
corpus_lsi = lsi[corpus_tfidf]
lsi.show_topics(num_words=10, formatted=False)
# create an Latent Dirichlet Allocation model over the TF-IDF corpus
# (probabilistic distributions of words over topics, and of topics over documents) 
# vary the num_topics parameter
train_corpus = corpus_tfidf[:-20]
test_corpus = corpus_tfidf[-20:]
for k in [2, 5, 10, 15, 20]:
    lda = models.LdaModel(train_corpus, num_topics=k, id2word=dictionary)
    corpus_lda = lda[corpus_tfidf]
    #lda.show_topics(num_words=10, formatted=False)
    print(lda.log_perplexity(test_corpus))
from itertools import chain
# split each fulltext into sentences, word tokenize each sentence
sentences = list(chain.from_iterable([sent_tokenize(ft) for ft in fulltexts]))
sentences = [word_tokenize(s) for s in sentences]
bigram_transformer = models.Phrases(sentences, min_count=3)
# create a word2vec model based on sentence, positioning words in a vector space
w2vModel = models.Word2Vec(bigram_transformer[sentences], size=100, window=5, min_count=3, workers=4, iter=50, sg=1)<jupyter_output>/home/chris/anaconda3/envs/masso/lib/python3.5/site-packages/gensim/models/phrases.py:248: UserWarning: For a faster implementation, use the gensim.models.phrases.Phraser class
  warnings.warn("For a faster implementation, use the gensim.models.phrases.Phraser class")<jupyter_text>The word2vec model can be used to look for words that are possible synonyms, or appear in a very similar context.<jupyter_code>w2vModel.most_similar_cosmul(topn=20, positive=['open_science'])
w2vModel.n_similarity(['open','science'], ['open','research'])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09775499381297507, 'fraction_numerical': 0.009545695598373696, 'mean_word_length': 5.908424908424909, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '544552', 'license': 'MIT', 'n_tokens_mistral': 1862, 'n_tokens_neox': 1809, 'n_words': 581, 'path': 'explorer.ipynb', 'repo_name': 'chreman/masso', 'size': '5657'}