<jupyter_start><jupyter_text>Master Thesis - Code examples & results  1. [Datasets & Hyperparameters](DatasetsAndHyperparameters) 2. [Functions & Overview](FunctionsAndOverview) 3. [Examples & Application](ExamplesAndApplication)Imports<jupyter_code>%load_ext nb_black
%matplotlib inline

import pandas as pd
import multiprocessing as mp
import statsmodels.api as sm
import networkx as nx
import pyLDAvis
import warnings

from sklearn.metrics import classification_report
from create_contingency_tables import createContingencyTables
from get_parameters import getParameters

# from run_tests import runTests
from clustering import clustering

warnings.filterwarnings("ignore")<jupyter_output>The nb_black extension is already loaded. To reload it, use:
  %reload_ext nb_black<jupyter_text>Datasets & Hyperparameters The datasets can be downloaded [here](https://tinyurl.com/banandit-thesis-datasets). I suggest to save them in a separate directory for which you can insert the path in DATASETS_PATH<jupyter_code># hyper-parameters and main dataset

# path datasets
DATASETS_PATH = "/home/jonas/Documents/GitHub/MasterThesis/datasets/finalDataset"

# path results
RESULTS_PATH = "/home/jonas/Documents/GitHub/MasterThesis/results/"

# corpus names (of finalDataset)
CORPORA = [
    "RM_2011_news_CNN.sqlite",
    "RM_2011_news_CTB.sqlite",
    "RM_2011_news_FXN.sqlite",
    "RM_2011_news_HFP.sqlite",
    "RM_2011_news_NBC.sqlite",
    "RM_2011_news_NYT.sqlite",
    "RM_2011_news_RET.sqlite",
    "RM_2011_news_WPO.sqlite",
    "RM_2011_news_WSJ.sqlite",
]
# hyper-parameters and calibration dataset

# path datasets
DATASETS_PATH = "/home/jonas/Documents/GitHub/MasterThesis/datasets/calibrationDataset"

# path results
RESULTS_PATH = "/home/jonas/Documents/GitHub/MasterThesis/results/"

# corpus names (of calibrationDataset)
CORPORA = [
    "RM_2012_news_HFP.sqlite",
    "RM_2012_news_NYT.sqlite",
    "RM_2012_news_CNN.sqlite",
    "RM_2012_news_RET.sqlite",
    "RM_2012_news_FXN.sqlite",
    "RM_2012_news_WSJ.sqlite",
]<jupyter_output><empty_output><jupyter_text>Functions & Overview Here the handling of the basic fucntions is explainedcreating contingency tables<jupyter_code>from create_contingency_tables import createContingencyTables

# initialize handler 
contingency_handler = createContingencyTables(DATASETS_PATH, CORPORA)

# function, that calculates a contingency table of a number of [N](int) co-occurring entities for a 
# [REFERENCE_ENTITY](str)

cont_tab = contingency_handler.build_contingency_table_from_single_topic(REFERENCE_ENTITY, N)


# function, that calculates a contingncy table of a preset list of [COOCC]urrence tuples (list) 
# for the model [MODEL](str)

cont_tab = contingency_handler.build_contingency_table_from_preset_cooccurrences(MODEL, COOCCS)

# handler for function multiple topics (arguments: list of entities, number of co-occurrences)
# function, that calculates a contingncy table of a number of [N](int) co-occurring entities 
# for a preset [ENTITY_LIST](list)  

cont_tab = contingency_handler.build_contingency_table_from_topic_list(ENTITY_LIST, N)<jupyter_output><empty_output><jupyter_text>extracting entities based on classic NLP<jupyter_code>from get_parameters import getParameters

# initialize handler for single corpus
param_handler = getParameters(DATASETS_PATH, CORPORA[0])

# functio, that creates a tf-idf table over the whole dictionary, used in the corpus
df_tf_idf = param_handler.create_tf_idf_table()

# function, that runs LDA on the selected corpus to extract [N](int) topics
# can be visualized with LDAvis or printed 
lda, corpus, dictionary = param_handler.LDA(N)

# visualize
vis = pyLDAvis.gensim.prepare(lda, corpus, dictionary, mds="mmds")
pyLDAvis.display(vis)

# print
lda.show_topics(num_topics=5, num_words=10)<jupyter_output><empty_output><jupyter_text>calculating the behavior of the chi-statistics over different reference entities and number of co-occurring entities<jupyter_code># for number of entities 
from run_tests import runTests
# initialize the handler to estimate an optimal [N]=number of co-occurring entities (int)
handler = runTests(DATASETS_PATH, CORPORA, REFERENCE_ENTITY, N)

# Init multiprocessing.Pool()
pool = mp.Pool(mp.cpu_count())

# create a df that pictures the behavior of the chi-statistics for the corpus-pairings in [CORPUS_PAIR](list)
# over the top [N](cf. handler) co-occurring entities
df = pd.concat(pool.map(handler.estimate_n, [single_tuple for single_tuple in CORPUS_PAIR]), axis=1)

# close pool
pool.close()
# for reference entities
from run_tests import runTests
# initialize the handler to estimate a feasible reference entity with [N]=number of co-occurring entities
handler = runTests(DATASETS_PATH, CORPORA, N)

# Init multiprocessing.Pool()
pool = mp.Pool(mp.cpu_count())

# create a df that pictures the behavior of the chi-statistics for all entitis witin an [ENTITIY_LIST](list)
# over all CORPORA
df = pd.concat(pool.map(handler.estimate_optimal_reference_entity, [entity for entity in ENTITY_LIST]), axis=1)
# close pool
pool.close()<jupyter_output><empty_output><jupyter_text>clustering the corpora based on their chi-statistics<jupyter_code># clustering 
from clustering import clustering 

# clusters the df of [hello_kitty_action_plan](func) after the [METRIC](str) 
# "geometric", "arithmetic", "argmax", "argmin" prints out clusters for different thresholds
clustering(df, METRIC)<jupyter_output><empty_output><jupyter_text>Examples & Application This section is based on the methodology used in my thesis.I ran four approaches of extracting entities for co-occurrences.All results I used in my thesis can be loaded and retraced[**TF-IFD**](tfidf)[**LDA**](lda)[**Reference Entity**](ref)[**Manual Selection**](man)[**Processing the entities**](proc)[**Clustering**](clu) **TF-IDF**    - [return to examples & application](ExamplesAndApplication)<jupyter_code># extract tf_idfs for all corpora
tf_idfs = []

for corpus in CORPORA:
    param_handler = getParameters(DATASETS_PATH, corpus)
    df = param_handler.create_tf_idf_table()
    tf_idfs.append(df)

# print top 10 entities with highest TF-IDF score
for df in tf_idfs:
    for i in df.iterrows():
        print(i[1].sort_values(ascending=False)[:5])

top_terms = [
    "deregulation",
    "latching",
    "katalin",
    "ﬂedged",
    "plumper",
    "plundered",
    "larders",
    "decmberists",
    "omitting",
]<jupyter_output><empty_output><jupyter_text>**LDA**  - [return to examples & application](ExamplesAndApplication)<jupyter_code># extract N topics for all corpora depict them with the top 10 entities
N = 5

for corpus in CORPORA:
    param_handler = getParameters(DATASETS_PATH, corpus)
    lda, corpus, dictionary = param_handler.LDA(N)
    # print
    print(lda.show_topics(num_topics=N, num_words=10))
    lda, corpus, dictionary = None, None, None

"""
# alternative (if Kernel shuts down)
# visualize
param_handler = getParameters(DATASETS_PATH, CORPORA[0])
lda, corpus, dictionary = param_handler.LDA(N)

vis = pyLDAvis.gensim.prepare(lda, corpus, dictionary, mds="mmds")
pyLDAvis.display(vis)

"""
# for LDA approach - created with LDAvis (lambda=0.4)

top_terms = [
    "people",
    "obama",
    "romney",
    "percent",
    "game",
    "state",
    "president",
    "police",
    "street",
]<jupyter_output><empty_output><jupyter_text>**Reference entity**  -  [return to examples & application](ExamplesAndApplication)<jupyter_code># for reference entities
N = 8

# initialize the handler to estimate a feasible reference entity with [N]=number of co-occurring entities
handler = runTests(DATASETS_PATH, CORPORA, N)

# Init multiprocessing.Pool()
pool = mp.Pool(mp.cpu_count())

# create a df that pictures the behavior of the chi-statistics for all entitis witin an [ENTITIY_LIST](list)
# over all CORPORA
df = pd.concat(pool.map(handler.estimate_optimal_reference_entity, [entity for entity in ENTITY_LIST]), axis=1)
# close pool
pool.close()
import matplotlib.pyplot as plt

# Import and visualize - same-slant news-outlet
df_same = pd.read_csv(RESULTS_PATH + "refEntity_sameSlant_n31.csv")

df_same.plot(kind="line", figsize=(7.5, 4))
plt.xlabel("n")
plt.ylabel("p-value")
plt.title(
    "Behavior of p-values in relation to the chosen\nreference entity - same-slant newsoutlet"
)
plt.show()

# Import and visualize - diff-slant news-outlet
df_diff = pd.read_csv(RESULTS_PATH + "refEntity_differentSlant_n31.csv")

df_diff.plot(kind="line", figsize=(7.5, 4))
plt.xlabel("n")
plt.ylabel("p-value")
plt.title(
    "Behavior of p-values in relation to the chosen\nreference entity - different-slant newsoutlet"
)
plt.show()
# for number of entities 
REFERENCE_ENTITY = "obama"
N = 31
# initialize the handler to estimate an optimal [N]=number of co-occurring entities (int)
handler = runTests(DATASETS_PATH, CORPORA, REFERENCE_ENTITY, N)

# Init multiprocessing.Pool()
pool = mp.Pool(mp.cpu_count())

# create a df that pictures the behavior of the chi-statistics for the corpus-pairings in [CORPUS_PAIR](list)
# over the top [N](cf. handler) co-occurring entities
df = pd.concat(pool.map(handler.estimate_n, [single_tuple for single_tuple in CORPUS_PAIR]), axis=1)

# close pool
pool.close()
# import dataset
df_same = pd.read_csv(RESULTS_PATH + "n_sameSlant_refObama.csv")

# Visualze dataset
df_same.plot(kind="line", figsize=(7.5, 7))
plt.ylabel("p-value")
plt.xlabel("n")
plt.title(
    "Behavior of p-values from chi-square test in relation to number of examined entities\n within same slant groups - reference entitiy: obama"
)
plt.legend()
plt.show()


# import dataset
df_same = pd.read_csv(RESULTS_PATH + "n_differentSlant_refObama.csv")

# Visualze dataset
df_same.plot(kind="line", figsize=(7.5, 7))
plt.ylabel("p-value")
plt.xlabel("n")
plt.title(
    "Behavior of p-values in relation to number of examined entities\n within same slant groups - reference entitiy: obama"
)
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>**Manual Selection**  - [return to examples & application](ExamplesAndApplication)<jupyter_code># first run for manual selection
top_terms = [
    "challenge",
    "obama",
    "romney",
    "dangerous",
    "administration",
    "corruption",
    "china",
    "oil",
    "iran",
]

# second run for manual selection
top_terms = [
    "challenge",
    "obama",
    "romney",
    "dangerous",
    "administration",
    "corruption",
    "illegal",
    "border",
    "fail",
]<jupyter_output><empty_output><jupyter_text>**Processing the entities**  -  - [return to examples & application](ExamplesAndApplication)creating a list of entity tuples for co-occurrences and test them for every constellation of corpus tuples<jupyter_code># create list of co-occurrence tuples from entity lists
preset_cooccurrences = []
for i in top_terms:
    for j in top_terms:
        if i != j:
            single_tuple = (i, j)
            preset_cooccurrences.append(single_tuple)
contingency_handler = createContingencyTables(DATASETS_PATH, CORPORA)
raw_cooccurrences = []

# run co-occurrence queries for every corpus 
def hello_kitty_action_plan(model):
    contingency_table = contingency_handler.build_contingency_table_from_preset_cooccurrences(
        model, preset_cooccurrences
    )
    return contingency_table




# open multiprocessing pool
pool = mp.Pool(mp.cpu_count())
# multiprocessing
raw_cooccurrences.append(pool.map(hello_kitty_action_plan, CORPORA))
# close pool
pool.close()

raw_cooccurrences = raw_cooccurrences[0]

# create contingency tables for every possible tuple of corpus and calculate chi-statistic and p_value
df = pd.DataFrame()
for i in range(len(raw_cooccurrences)):
    for j in range(len(raw_cooccurrences)):
        if i != j:
            contingency_table = pd.DataFrame()
            df_results = pd.DataFrame()
            description = (
                raw_cooccurrences[i][0].index[0][-10:-7]
                + "-"
                + raw_cooccurrences[j][0].index[0][-10:-7]
            )
            contingency_table = pd.concat(
                [raw_cooccurrences[i], raw_cooccurrences[j]], axis=0
            )
            contingency_table = sm.stats.Table(contingency_table)

            # calculate results + add them to dataframe
            results = contingency_table.test_nominal_association()

            df_results[description] = [
                raw_cooccurrences[i][0].index[0][-10:-7],
                raw_cooccurrences[j][0].index[0][-10:-7],
                description,
                results.statistic,
                results.pvalue,
            ]

            df_results = df_results.transpose()
            df_results = df_results.rename(
                columns={
                    0: "reference",
                    1: "comparison",
                    2: "combination",
                    3: str("chi_sq"),
                    4: str("p_value"),
                }
            )
            df = pd.concat([df, df_results], axis=0)


# save results to csv
# df.to_csv(RESULTS_PATH + "test.csv", index=False)
# load my results

# FILENAME = "ES_reference_entity_approach.csv"
# FILENAME = "ES_TF_IDF_approach.csv"
# FILENAME = "ES_LDA_approach.csv"
# FILENAME = "ES_manual_selection_approach1.csv"
FILENAME = "ES_manual_selection_approach2.csv"

df = pd.read_csv(RESULTS_PATH + FILENAME)
df.reset_index()
df
# add labels for evaluating accuracy, precision and recall
liberal = ["NYT", "HFP", "WPO"]
center = ["CNN", "RET", "NBC"]
conservative = ["CTB", "FXN", "WSJ"]

df["same_chi"] = 1
df["same_lit"] = 1


df["reference"] = 1
df["comparison"] = 1
df["ones"] = 1
for i in range(len(df)):
    df.reference[i] = df.combination[i][:3]
    df.comparison[i] = df.combination[i][-3:]


for i in range(len(df)):
    # adding labels, derived from the chi-test
    if df.p_value[i] <= 0.05:
        df.same_chi[i] = 1
    else:
        df.same_chi[i] = 0

    # adding labels, derived from literature
    if all(x in liberal for x in [df.reference[i], df.comparison[i]]) == True:
        df.same_lit[i] = 1
    elif all(x in center for x in [df.reference[i], df.comparison[i]]) == True:
        df.same_lit[i] = 1
    elif all(x in conservative for x in [df.reference[i], df.comparison[i]]) == True:
        df.same_lit[i] = 1
    else:
        df.same_lit[i] = 0
# split into two dataframes df_same and df_diff
df_same = pd.DataFrame()
df_diff = pd.DataFrame()

for i in range(len(df)):
    if all(x in liberal for x in [df.reference[i], df.comparison[i]]) == True:
        df_same = df_same.append(df.loc[i])
    elif all(x in center for x in [df.reference[i], df.comparison[i]]) == True:
        df_same = df_same.append(df.loc[i])
    elif all(x in conservative for x in [df.reference[i], df.comparison[i]]) == True:
        df_same = df_same.append(df.loc[i])
    else:
        df_diff = df_diff.append(df.loc[i])

df_same = df_same.reset_index()
df_diff = df_diff.reset_index()


# Results for overall precision
y_true = df.same_lit
y_pred = df.same_chi

target_names = ["diff", "same"]
print(classification_report(y_true, y_pred, target_names=target_names, digits=4))<jupyter_output>precision    recall  f1-score   support

        diff     0.7391    0.6296    0.6800        54
        same     0.2308    0.3333    0.2727        18

    accuracy                         0.5556        72
   macro avg     0.4849    0.4815    0.4764        72
weighted avg     0.6120    0.5556    0.5782        72<jupyter_text>**Clustering the corpora**  - [return to examples & application](ExamplesAndApplication)<jupyter_code># Visualize as undirected weighted graph.
# The edge, weights are combined and averaged 

df_new = pd.DataFrame()
G_new = nx.Graph()

for i in range(len(df)):
    k = pd.DataFrame()
    for j in range(len(df)):
        if (
            df.reference[i] == df.comparison[j]
            and df.reference[j] == df.comparison[i]
            and i != j
        ):
            l = [
                str(df.reference[i]),
                str(df.comparison[i]),
                ((df.chi_sq[i] + df.chi_sq[j])),
            ]
                      
            G_new.add_edge(
                str(df.reference[i]),
                str(df.comparison[i]),
                weight=(((df.chi_sq[i]+df.chi_sq[j])/2)*80),
            )
            
            k = k.append(l)
            k = k.transpose()
            df_new = df_new.append(k)

nx.spring_layout(G_new)
plt.figure(figsize=(6, 6))
plt.axis = "off"
nx.draw_networkx(G_new, with_labels=True, edge_color="gray", node_color="white")

df_new = df_new.rename(columns={0: "reference", 1: "comparison", 2: "summed_chi_sq"})

from clustering import clustering

# cluster arguments(df, mode=("geometric", "arithmetic", "argmax", "argmin"))
clustering(df, "geometric")<jupyter_output>threshold 0.5*c_omega
['FXN', 'WPO', 'WSJ']
['NYT', 'WPO', 'WSJ']
['RET', 'WPO', 'WSJ']
['WSJ', 'RET', 'WPO']


 threshold 0.75*c_omega
['WSJ', 'RET', 'WPO']


 threshold 0.8*c_omega
['WSJ', 'RET', 'WPO']


 threshold 0.9*c_omega
['WSJ', 'RET', 'WPO']
{'fraction_non_alphanumeric': 0.11567383386206086, 'fraction_numerical': 0.014216895781929268, 'mean_word_length': 3.9399303540336623, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 46, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '209652', 'license': 'Apache-2.0', 'n_tokens_mistral': 6104, 'n_tokens_neox': 5658, 'n_words': 1399, 'path': 'functions/.ipynb_checkpoints/examples_and_application-checkpoint.ipynb', 'repo_name': 'the-banandit/OmissionOrCommission', 'size': '17022'}