<jupyter_start><jupyter_text>Introduction..... Check to see if jupyter lab uses the correct python interpreter with '!which python'.It should be something like '/opt/anaconda3/envs/[environment name]/bin/python' (on Mac).If not, try this: https://github.com/jupyter/notebook/issues/3146issuecomment-352718675<jupyter_code>!where python<jupyter_output>C:\Users\fluhrer\AppData\Local\Continuum\anaconda3\envs\[new_env]\python.exe<jupyter_text>Install dependencies:<jupyter_code>install_packages = True
if install_packages:
    !conda install tensorflow=2 -y
    !conda install -c anaconda pandas -y
    !conda install -c conda-forge tensorflow-hub -y
    !conda install -c akode html2text -y
    !conda install -c conda-forge tqdm -y
    !conda install -c anaconda scikit-learn -y
    !conda install -c conda-forge matplotlib -y
    !conda install -c anaconda seaborn -y<jupyter_output>Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: C:\Users\fluhrer\AppData\Local\Continuum\anaconda3\envs\[new_env]

  added / updated specs:
    - tensorflow=2


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    _tflow_select-2.2.0        |            eigen           3 KB
    absl-py-0.11.0             |   py36haa95532_0         190 KB
    aiohttp-3.6.3              |   py36he774522_0         598 KB
    astor-0.8.1                |           py36_0          46 KB
    async-timeout-3.0.1        |           py36_0          12 KB
    blinker-1.4                |           py36_0          21 KB
    ca-certificates-2020.10.14 |                0         159 KB
    cachetools-4.1.1           |             py_0          12 KB
    click-7.1.2                |             py_0          67 KB
    googl[...]<jupyter_text>Imports<jupyter_code>#imports
import pandas as pd
import numpy as np
import os
import time
import tensorflow as tf
import tensorflow_hub as hub
import zipfile
from html2text import HTML2Text
from tqdm import tqdm
import re
from sklearn.metrics import pairwise_distances
from sklearn.preprocessing import normalize
import matplotlib.pyplot as plt
import seaborn as sns<jupyter_output><empty_output><jupyter_text>Set pandas print optionsThis will improve readability of printed pandas dataframe.<jupyter_code>pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.width', None)
pd.set_option('display.max_colwidth', None)<jupyter_output><empty_output><jupyter_text>Set global ParametersSet your parameters here:data_path: In this path put the data you have downloaded with YouTube Data Tools. output_path: Tghe files generated in this notebook will be saved here.url_dict: URLs to models on Tensorflow hub are saved here. Other models are available there.model_type: Define which model you would like to use. Choose one from url_dictnew_embeddings: If this is true, new embeddings will be generated and saved at output_path. Otherwise, embeddings are loaded from Disc.<jupyter_code>data_path = './Comments_Naomi_Seibt_Video.csv'
output_path = "./output/"

new_embeddings = True

url_dict = {
            'Transformer' : "https://tfhub.dev/google/universal-sentence-encoder-large/5",
            'DAN' : "https://tfhub.dev/google/universal-sentence-encoder/4",
            'Transformer_Multilingual': "https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/3"
}

model_type = 'Transformer' #@param ['DAN','Transformer','Transformer_Multilingual']<jupyter_output><empty_output><jupyter_text>Create output directoryTry to create the directory defined by output_path<jupyter_code>try:
   os.mkdir(output_path)
except OSError:
  print ("Creation of the directory %s failed" % output_path)
else:
  print ("Successfully created the directory %s " % output_path)<jupyter_output>Creation of the directory ./output/ failed<jupyter_text>Load DataLoad you data as a pandas dataframe<jupyter_code>if new_embeddings:
    data = pd.read_csv(data_path)
data.head()<jupyter_output><empty_output><jupyter_text>PreprocessingPreprocess your data:- Drop empty rows- Drop unused columns<jupyter_code>if new_embeddings:
    data = data.dropna(subset=['text', 'authorName']) # drop rows with no content
    data=data.drop(['id', 'replyCount','likeCount','authorChannelUrl','authorChannelId','isReplyTo','isReplyToName'],axis=1) # drop unused columns
    data.head()<jupyter_output><empty_output><jupyter_text>- remove HTML-tags, links and usernames<jupyter_code>if new_embeddings:
    # Remove HTML tags
    tqdm.pandas()
    h = HTML2Text()
    h.ignore_links = True
    data['cleaned'] = data['text'].progress_apply(lambda x: h.handle(x))
    print( "Removed HTML Tags.")

    # Remove links
    http_link_pattern = r'http\S+'
    bitly_link_pattern = r'bit.ly/\S+'
    data['cleaned'] = data['cleaned'].str.replace(http_link_pattern, '')
    data['cleaned'] = data['cleaned'].str.replace(bitly_link_pattern, '')
    print( "Removed Links.")<jupyter_output>100%|████████████████████████████████████████████████████████████████████████████| 1592/1592 [00:00<00:00, 6386.54it/s]<jupyter_text>Save or Load preprocessed dataSave your data afte preprocessing, or load preprocessed data from disc.<jupyter_code>if new_embeddings:
  data.to_pickle(output_path+'data_preprocessed'+'.pkl')
else:
  data = pd.read_pickle(output_path+'data_preprocessed'+'.pkl')
data.head()<jupyter_output><empty_output><jupyter_text>Produce Text Embeddings with Universal Sentence Encoder  Load ModelLoad the model from TF-hub<jupyter_code>hub_url = url_dict[model_type]
if new_embeddings:
  print("Loading model. This will take some time...")
  embed = hub.load(hub_url)<jupyter_output>Loading model. This will take some time...<jupyter_text>Embed DocumentsProduce embeddings of your documents.<jupyter_code>if new_embeddings:
    for k,g in data.groupby(np.arange(len(data))//200):
        if k == 0:
            embeddings = embed(g['cleaned'])
        else:
            embeddings_new = embed(g['cleaned'])
            embeddings = tf.concat(values=[embeddings,embeddings_new],axis = 0)
            print(k , end =" ")
    print("The embeddings vector is of fixed length {}".format(embeddings.shape[1]))

    np.save(output_path+'/embeddings'+model_type+'.npy', embeddings, allow_pickle=True, fix_imports=True)
else:
    embeddings = np.load(output_path+'/embeddings'+model_type+'.npy', mmap_mode=None, allow_pickle=False, fix_imports=True, encoding='ASCII')

embeddings.shape<jupyter_output>1 2 3 4 5 6 7 The embeddings vector is of fixed length 512<jupyter_text>Calculate Similarity Matrix with angular distance'Following Cer et al. (2018), we first computethe sentence embeddings u, v for an STS sentencepair, and then score the sentence pair similaritybased on the angular distance between the twoembedding vectors  d = − arccos (uv/||u|| ||v||).'<jupyter_code>from sklearn.metrics.pairwise import cosine_similarity
def cos_sim(input_vectors):
    similarity = cosine_similarity(input_vectors)
    return similarity
cosine_similarity_matrix = cos_sim(np.array(embeddings))
print(cosine_similarity_matrix)<jupyter_output>[[ 1.          0.4433955   0.05505778 ...  0.01702606  0.3150363
  -0.00370582]
 [ 0.4433955   0.9999998   0.01497677 ...  0.03613643  0.17931402
  -0.01068806]
 [ 0.05505778  0.01497677  1.0000002  ...  0.08898681  0.0761539
   0.07409675]
 ...
 [ 0.01702606  0.03613643  0.08898681 ...  0.9999999   0.13675542
   0.24166256]
 [ 0.3150363   0.17931402  0.0761539  ...  0.13675542  0.9999999
   0.06460285]
 [-0.00370582 -0.01068806  0.07409675 ...  0.24166256  0.06460285
   1.        ]]<jupyter_text>Plots Similarity Plot and print a heat map showing the semantic contextual similarity between comments.<jupyter_code>import seaborn as sns
def plot_similarity(labels, features, rotation):
  corr = np.inner(features, features)
  sns.set(font_scale=1.2)
  g = sns.heatmap(
      corr,
      xticklabels=labels,
      yticklabels=labels,
      vmin=0,
      vmax=1,
      cmap="YlOrRd")
  g.set_xticklabels(labels, rotation=rotation)
  g.set_title("Semantic Textual Similarity")

num_samples = 5
off_set = 10000

plot_similarity(data.iloc[off_set:off_set+num_samples]['cleaned'], embeddings[off_set:off_set+num_samples], 90)<jupyter_output>C:\Users\fluhrer\AppData\Local\Continuum\anaconda3\envs\[new_env]\lib\site-packages\seaborn\matrix.py:311: UserWarning: Attempting to set identical left == right == 0 results in singular transformations; automatically expanding.
  ax.set(xlim=(0, self.data.shape[1]), ylim=(0, self.data.shape[0]))
C:\Users\fluhrer\AppData\Local\Continuum\anaconda3\envs\[new_env]\lib\site-packages\seaborn\matrix.py:311: UserWarning: Attempting to set identical bottom == top == 0 results in singular transformations; automatically expanding.
  ax.set(xlim=(0, self.data.shape[1]), ylim=(0, self.data.shape[0]))<jupyter_text>Show neighbours of a comment Define which comment to analyze<jupyter_code>comment_index = 221
comment = data["cleaned"][comment_index]
comment_list = data["cleaned"].tolist()
print(comment)<jupyter_output>Many of you talk about scientists...most of them are sold to big
companies...they took huge amount of money if they support climate change
propaganda...if they have different opinion they dont get anything....this is
the reality..wake up<jupyter_text>Print similar comments.<jupyter_code>def get_top_similar(sentence, sentence_list, similarity_matrix, topN):
    # find the index of sentence in list
    index = sentence_list.index(sentence)
    # get the corresponding row in similarity matrix
    similarity_row = np.array(similarity_matrix[index, :])
    # get the indices of top similar
    indices = similarity_row.argsort()[-topN:][::-1]
    return [sentence_list[i] for i in indices]


for i, value in enumerate(get_top_similar(comment, comment_list, cosine_similarity_matrix, 10)):
  print("Top similar comment {}: {}".format(i+1, value))<jupyter_output>Top similar comment 1: Many of you talk about scientists...most of them are sold to big
companies...they took huge amount of money if they support climate change
propaganda...if they have different opinion they dont get anything....this is
the reality..wake up


Top similar comment 2: She is supported by heartland institute which gets 40% of it's funding from
ExxonMobil. If course they are trying to convince people that climate change
isn't serious, as it would ruin their business.


Top similar comment 3: This is what happens when people begin thinking for themselves and research
beyond the leftist echo chamber. They discover the real science and realise
they have been lied to about man made climate change.


Top similar comment 4: Anyone unsure about Climate Change go and watch Potholer54's channel, he knows
his stuff. Unlike these idiots in the video.


Top similar comment 5: Remember folks the people who are shouting about "Man made climate change" are
the same people who tell you [...]
{'fraction_non_alphanumeric': 0.1123960968558005, 'fraction_numerical': 0.04481387784604265, 'mean_word_length': 4.017679057116954, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 55, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '353490', 'license': 'CC0-1.0', 'n_tokens_mistral': 3915, 'n_tokens_neox': 3509, 'n_words': 941, 'path': 'Pipeline/svenja_fluhrer_assignment_4/Assignment-4_Language_model_2.ipynb', 'repo_name': 'Kosmopy/seminar_critical-social-media-analysis', 'size': '11068'}