<jupyter_start><jupyter_text>Text normalization using Gensim In this notebook we will load and preprocess the data from the `six_thousand_tweets.csv` file using the normalization techniques we talked about in the lecture.<jupyter_code># Importing the libraries
import pandas as pd
from gensim.utils import simple_preprocess
import nltk
nltk.download("wordnet")
nltk.download("stopwords")
from nltk import WordNetLemmatizer
from wordcloud import WordCloud<jupyter_output><empty_output><jupyter_text>Loading up the data Upload the `six_thousand_tweets.csv` file to this environment, and load it up using Pandas.<jupyter_code>tweets = pd.read_csv("six_thousand_tweets.csv")
tweets.head()<jupyter_output><empty_output><jupyter_text>The column `target` represents whether the tweet is saying something "positive" or something "negative".  Removing usernames and hashtags For the purposes of the analysis we plan to do, let's remove all usernames (i.e. all strings that contain an `@`)<jupyter_code>def clean_usernames(tweet_text):
  words = tweet_text.split()
  clean_words = [w for w in words if not ("@" in w or "#" in w)]
  return " ".join(clean_words)
tweets["clean_text"] = tweets["text"].map(clean_usernames)
# Sanity check.
for t in tweets["clean_text"]:
  if "@" in t:
    print(t)
  
  if "#" in t:
    print(t)<jupyter_output><empty_output><jupyter_text>Tokenizing We will use a library called `gensim`. `gensim` comes with a series of utilities, and among them is a simple tokenizer:<jupyter_code>preprocessed_texts = [
    simple_preprocess(t) for t in tweets["clean_text"]
]
for text, clean_text in zip(tweets["text"].iloc[:5], preprocessed_texts[:5]):
  print(text, clean_text)<jupyter_output>At home and watching Ellen - after a quite long time!  ['at', 'home', 'and', 'watching', 'ellen', 'after', 'quite', 'long', 'time']
@thehulkster thanx  ['thanx']
@lrobertsglobal I voted many times for BTO TCOB  o would have gone to nuit blanche for TCOB. Oh well ['voted', 'many', 'times', 'for', 'bto', 'tcob', 'would', 'have', 'gone', 'to', 'nuit', 'blanche', 'for', 'tcob', 'oh', 'well']
FYI - Wonderland's water WILL infect your piercings.  ['fyi', 'wonderland', 'water', 'will', 'infect', 'your', 'piercings']
@shineonmedia There's something wrong with your site. It's not letting me go on it. It said its temporarily unavailable  ['there', 'something', 'wrong', 'with', 'your', 'site', 'it', 'not', 'letting', 'me', 'go', 'on', 'it', 'it', 'said', 'its', 'temporarily', 'unavailable']<jupyter_text>Lemmatizing For lemmatizing, we can use several different tools:- The neural networks inside SpaCy...- NLTK and its lemmatizing utilities...For now, we will use `NLTK`'s WordNet lemmatizer<jupyter_code>import nltk
nltk.download("wordnet")
nltk.download("stopwords")

from nltk import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
corpus = [
  [lemmatizer.lemmatize(token) for token in doc] for doc in preprocessed_texts
]
for text, doc in zip(tweets["text"].iloc[:5], corpus[:5]):
  print(text, doc)<jupyter_output>At home and watching Ellen - after a quite long time!  ['at', 'home', 'and', 'watching', 'ellen', 'after', 'quite', 'long', 'time']
@thehulkster thanx  ['thanx']
@lrobertsglobal I voted many times for BTO TCOB  o would have gone to nuit blanche for TCOB. Oh well ['voted', 'many', 'time', 'for', 'bto', 'tcob', 'would', 'have', 'gone', 'to', 'nuit', 'blanche', 'for', 'tcob', 'oh', 'well']
FYI - Wonderland's water WILL infect your piercings.  ['fyi', 'wonderland', 'water', 'will', 'infect', 'your', 'piercings']
@shineonmedia There's something wrong with your site. It's not letting me go on it. It said its temporarily unavailable  ['there', 'something', 'wrong', 'with', 'your', 'site', 'it', 'not', 'letting', 'me', 'go', 'on', 'it', 'it', 'said', 'it', 'temporarily', 'unavailable']<jupyter_text>Removing stopwords For this particular application (which is just a visualization of positive and negative tweets), let's remove the stopwords.**Each library has its own family of stopwords.** In this example we will use the list inside NLTK.<jupyter_code>english_stopwords = set(nltk.corpus.stopwords.words("english"))
corpus = [
   [word for word in doc if word not in english_stopwords] for doc in corpus
]
corpus_positive = [
    doc for i, doc in enumerate(corpus) if tweets.loc[i, "target"] == 1
]

corpus_negative = [
    doc for i, doc in enumerate(corpus) if tweets.loc[i, "target"] == 0
]<jupyter_output><empty_output><jupyter_text>Visualizing the tweets using wordclouds<jupyter_code>def count_words(corpus):
  """
  Returns {word: frequency} for all words in
  all documents of the corpus.
  """
  counts = {}
  for doc in corpus:
    for word in doc:
      if word not in counts:
        counts[word] = 1
      else:
        counts[word] += 1
  
  return counts<jupyter_output><empty_output><jupyter_text>A wordcloud is a great tool for visualizing words in terms of their frequency.<jupyter_code>from wordcloud import WordCloud
positive_counts = count_words(corpus_positive)
negative_counts = count_words(corpus_negative)

wordcount = WordCloud(
    background_color="white",
    max_words=100,
    width=500,
    height=400).generate_from_frequencies(positive_counts)
wordcount.to_image()
wordcount = WordCloud(
    background_color="white",
    max_words=100,
    width=500,
    height=400).generate_from_frequencies(negative_counts)
wordcount.to_image()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.132277081798084, 'fraction_numerical': 0.004789977892409728, 'mean_word_length': 5.588592233009709, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '666256', 'license': 'MIT', 'n_tokens_mistral': 1889, 'n_tokens_neox': 1802, 'n_words': 588, 'path': 'day05/IN CLASS - Full - Preprocessing and visualizing tweets.ipynb', 'repo_name': 'flight505/Applied_AI_IT_Uni', 'size': '5428'}