<jupyter_start><jupyter_text>Dota Dataset Notebook 2 - Preprocessing and Feature Engineering This notebook covers:* Additional preprocessing* Additional features* Word embeddings<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import langdetect as ld
from textblob import TextBlob
import warnings
warnings.filterwarnings('ignore')<jupyter_output><empty_output><jupyter_text>Parts of Notebook 1* Finding the language of each text and getting the English rows<jupyter_code>df = pd.read_csv('dota2_chat_messages.csv', nrows=50000)
df.head()
# Labeling languages
langs = np.zeros(len(df)).astype(str)
i = -1
for message in df['text'].values:
    i += 1
    try:
        langs[i] = ld.detect(message)
    except:
        continue
df['language'] = langs
# Fixing some languages due to acronyms
lang_fix = df.copy()
lang_fix = lang_fix.mask(df['text'].str.contains('(ez)|(Ez)|(EZ)'), 'en')
lang_fix = lang_fix.mask(df['text'].str.contains('(lol)|(Lol)|(LOL)'), 'en')
lang_fix = lang_fix.mask(df['text'].str.contains('(gg)|(Gg)|(GG)'), 'en')
lang_fix = lang_fix.mask(df['text'].str.contains('(ty)|(Ty)|(TY)'), 'en')
lang_fix = lang_fix.mask(df['text'].str.contains('(xD)|(XD)'), 'en')
lang_fix = lang_fix.mask(df['text'].str.contains('[Rr]eport'), 'en')
lang_fix = lang_fix.mask(df['text'].str.contains('STUPID|[Ss]tupid'), 'en')
lang_fix = lang_fix.mask(df['text'].str.contains('[Ff][Uu][Cc][Kk]|[Ss]hit'), 'en')
lang_fix = lang_fix.mask(df['text'].str.contains('[Nn][Oo][Oo][Bb]'), 'en')
lang_fix = lang_fix.mask(df['text'].str.contains('retard|RETARD'), 'en')
lang_fix = lang_fix.mask(df['text'].str.contains('pls|stfu|omg|OMG|wtf|WTF|wp|guys|kill|KILL|god|feed|FEED|btw'),'en')
lang_fix = lang_fix.mask(df['text'].str.contains('idiot|IDIOT|defend|dumb|end'), 'en')
lang_fix = lang_fix.mask(df['text'].str.contains('good|game|nice|thx|THX'), 'en')
df['language'] = lang_fix['language']
eng = df[df['language']=='en'].drop('language', axis=1)
eng.head()<jupyter_output><empty_output><jupyter_text>______  Start of Notebook 2 Work  Additional Preprocesssing* Dropping links* Dropping stop words* Dropping words that do not appear often<jupyter_code>eng.head()
# Current Version 1 df
eng.shape
# Dropping links
eng = eng.drop(eng[eng['text'].str.contains("(\.tv)")].index).drop(eng[eng['text'].str.contains("(\.com)")].index)
eng.shape<jupyter_output><empty_output><jupyter_text>Recurring issues caused by the dataset of 21 million rows are due to limited space and long runtimes. A possible way to combat this may be to drop stop words. Stop words are commonly used words, such as "the", "a", "an", "in", etc. Natural Language Toolkit's list of stop words will be used.<jupyter_code># Loading stop words
import nltk
from nltk.corpus import stopwords
print(stopwords.words('english'))<jupyter_output>['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', '[...]<jupyter_text>For now, the stop words "you" and "you're" will not be taken out. It may be useful to see if words are directed at other teammates in the future.<jupyter_code>stopwords = stopwords.words('english')
stopwords.remove("you")
stopwords.remove("you're")
stopwords.remove("yourself")
stopwords[:10]
print("There will be {} different stop words dropped.".format(len(stopwords)))
# Dropping stop words
def drop_stop_words(text):
    """Drops stop words from a line of text."""
    text = text.split(" ")
    nonstop_words = [word for word in text if word not in stopwords]
    string = ""
    for word in nonstop_words:
        string += word + " "
    return string[:len(string)-1]

eng['text'] = eng['text'].apply(drop_stop_words)
# Finding words that are only used once or twice
words = []
for row in eng['text'].str.split(" "):
    for word in row:
        words.append(word)
word_counts = pd.Series(words).str.lower().value_counts()
rare_words = word_counts[word_counts < 3].index
rare_words[:30]
print("There will be {} different rarely used words dropped.".format(len(rare_words)))
def drop_rare_words(text):
    """Drops rare words from a line of text."""
    text = text.split(" ")
    nonrare_words = [word for word in text if word.lower() not in rare_words]
    string = ""
    for word in nonrare_words:
        string += word + " "
    return string[:len(string)-1]

eng['text'] = eng['text'].apply(drop_rare_words)<jupyter_output><empty_output><jupyter_text>Additional Feature Engineering* Explicit keywords* Word embedding → word2vecLike before, **this is to explore different features, methods to gain different features, and information** we can gain from them. No features are finalized within this notebook, along with the first.  Use of offensive language "**Description: A list of 1,300+ English terms that could be found offensive.** The list contains some words that many people won't find offensive, but it's a good start for anybody wanting to block offensive or profane terms on their Site."* From Luis von Ahn's Research Group: _https://www.cs.cmu.edu/~biglou/resources/bad-words.txt_<jupyter_code># list of potentially offensive words from the link above
bad_words = pd.read_fwf("bad-words.txt", header=None)

# dropping the words 'dead', 'death', 'sniper', and 'doom' (reasons explained further below this cell)
bad_words = bad_words[bad_words[0] != 'dead'][bad_words[0] != 'death'][bad_words[0] != 'sniper'][bad_words[0] != 'doom'].iloc[:,0].values
bad_words[:30]
print("Number of offensive words considered:", len(bad_words))
# Converting it back to the original dataset
offense = eng.copy()
offense.head()
def offensive_word(text):
    return [word for word in text.split(" ") if word in bad_words]

offense['offensive word'] = offense['text'].apply(offensive_word)
offense['num offensive words'] = offense['offensive word'].apply(len)
offense.sort_values('num offensive words', ascending=False).head(15)<jupyter_output><empty_output><jupyter_text>For now, the number of offensive words **seems to be a suitable indicator of toxicity.**Future possibility: putting a degree of offensiveness to certain words (ie. "ass" can be less toxic than "retard", and therefore carry less weight)* These weights were later included in the use of Tfidf.  EDA on Offensive Language<jupyter_code>print("Out of {} messages, {} messages contain offensive language."
      .format(len(offense), len(offense[offense['num offensive words'] > 0])))
offense.groupby('num offensive words').size()
used_words = []
for array in offense['offensive word']:
    for word in array:
        used_words.append(word)

print("Most commonly used offensive words:")
pd.Series(used_words).value_counts().head(10)<jupyter_output>Most commonly used offensive words:<jupyter_text>Some commonly used "offensive" words were: "dead", "death", "sniper", and "doom". Given these messages are within gaming contexts, these terms may be highly used without toxicity. For example, a player can tell their team that the enemy is dead. In addition, there are two heroes called Sniper and Doom. These words were dropped when loading the initial bad_word list after looking at the context of these words, shown below.<jupyter_code>offense[offense['text'].str.contains('dead')].tail(3)
offense[offense['text'].str.contains('death')].tail(3)
offense[offense['text'].str.contains('sniper')].tail(3)
offense[offense['text'].str.contains('doom')].tail(3)<jupyter_output><empty_output><jupyter_text>**By match and by player**<jupyter_code># Number of offensive words per match
bad_word_sum = offense.groupby('match')['num offensive words'].sum().sort_values()
bad_word_sum
print("On average, there is/are {} offensive word(s) said per match.".format(round(bad_word_sum.mean())))
by_player = offense.groupby(['match', 'slot'])['num offensive words'].sum().sort_values()
by_player
print("On average, there is/are {} offensive word(s) said per player.".format(round(by_player.mean())))<jupyter_output>On average, there is/are 0 offensive word(s) said per player.<jupyter_text>**Time**<jupyter_code>only_offense = offense[offense['num offensive words'] > 0]
print("Among the messages with at least one offensive word, the average time of usage is at the {} minute mark."
      .format(round(only_offense['time'].mean() / 60)))
# Negative time
time_copy = offense.copy()
prop = offense[offense['time'] < 0].sort_values("num offensive words")['num offensive words'].mean()
print("Proportion of messages with offensive words (negative time):", round(prop, 3))
# Within 0-15 minutes
prop = offense[(offense['time'] >= 0) & (offense['time'] < 900)].sort_values("num offensive words")['num offensive words'].mean()
print("Proportion of messages with offensive words (0-15 mins):", round(prop, 3))
# Within 15-30 minutes
prop = offense[(offense['time'] >= 900) & (offense['time'] < 1800)].sort_values("num offensive words")['num offensive words'].mean()
print("Proportion of messages with offensive words (15-30 mins):", round(prop, 3))
# Over 30 minutes
prop = offense[(offense['time'] >= 1800)].sort_values("num offensive words")['num offensive words'].mean()
print("Proportion of messages with offensive words (30+ mins):", round(prop, 3))<jupyter_output>Proportion of messages with offensive words (30+ mins): 0.072<jupyter_text>Word2Vec Word Embedding* **Word Embeddings** create "a representation for words that capture their _meanings, semantic relationships, and the different contexts_ they are used in."* **Word2vec** is a combination of the continuous bag of words and skip-gram models. Both of these models associate weights to word. CBOW predicts probabilities of words given a context. Skip-gram predicts the context given a word.<jupyter_code>messages = eng['text'].str.lower().str.split(" ").values
from gensim.models import Word2Vec

# loading the Word2Vec model
model = Word2Vec(sg=1, min_count=1, window=3, size=100, workers=4)
model.build_vocab(messages)
model.train(sentences=messages, total_examples=model.corpus_count, epochs=model.epochs)
model.most_similar('trash')
list(model.wv.vocab)[:10]
similar = model.similarity('trash', 'garbage')
print("Similarity between 'trash' and 'garbage' is {}".format(similar))<jupyter_output>Similarity between 'trash' and 'garbage' is 0.9980788826942444
{'fraction_non_alphanumeric': 0.16626180836707152, 'fraction_numerical': 0.010976158344579398, 'mean_word_length': 6.270111183780249, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 45, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '470143', 'license': 'Apache-2.0', 'n_tokens_mistral': 3735, 'n_tokens_neox': 3573, 'n_words': 1067, 'path': 'student-projects/fall-2020/GGWP-Identify-Toxic-Behavior-in-Gaming/Notebooks/Model Formation/ggwp2_preprocess_and_features.ipynb', 'repo_name': 'UCBerkeley-SCET/DataX-Berkeley', 'size': '11115'}