<jupyter_start><jupyter_text>Negative News Neural Nets Project: Classifying Adverse Media Articles using Machine Learning Algorithms In this notebook, conda environment with Python 3.86 is used. Some libraries, such as spacy and nltk may require installation if your machine does not have them. You can use the steps below to install spaCy. If something goes awry, feel free to use pip/do some stackoverflow search to complete the installation. The last two parts will be required later on in the notebook, they are not essential spaCy packages. - conda install -c conda-forge spacy  - conda install -c conda-forge spacy-lookups-data  - python -m spacy download en_core_web_sm  - pip install spacy-langdetect  - conda install -c conda-forge wordcloud On the other hand, installing nltk packages will be easy, just look at the error to understand what needs to be downloaded using nltk.download(...). I have already provided the download code for punkt package and I don't think anything is required beside that.  Dataset Preparation Before doing any EDA, null value imputation, necessary dataset checks etc, we need to form the whole training dataset by combining the AM and NAM articles together. The latter one will include the random articles as well.Let's begin with importing necessary/potentially useful stuff... Some of them below may not be used at all in the future, so the list below is tentative.<jupyter_code>import warnings
warnings.simplefilter("ignore", UserWarning)
import pandas as pd
import numpy as np
import json
import spacy
import matplotlib.pyplot as plt
%matplotlib inline

# For regular expressions
import re

# For handling strings
import string

# For performing mathematical operations
import math
# Uncomment this if you're using linux
# !ls 
# Let's get an overview of what our folder contains..
!dir<jupyter_output>Volume in drive C is Win 10
 Volume Serial Number is CA9A-F06E

 Directory of C:\Users\canberk\Desktop\ut-ml-adverse-media-main

12/07/2020  05:51 PM    <DIR>          .
12/07/2020  05:51 PM    <DIR>          ..
11/23/2020  06:08 PM           110,455 .ipynb
11/23/2020  06:09 PM    <DIR>          .ipynb_checkpoints
11/21/2020  04:43 PM         3,752,073 adverse_media_training.csv.zip
11/21/2020  07:29 PM         3,097,536 cleaned&tokenized_text.csv
12/07/2020  05:51 PM           134,333 Data Preprocessing&Baselines.ipynb
11/21/2020  04:43 PM         3,630,748 EDA - Kristjan's Original.ipynb
11/23/2020  06:04 PM         3,740,422 EDA.ipynb
10/24/2015  07:35 PM     5,646,236,541 glove.840B.300d.txt
12/07/2020  04:36 PM     2,176,768,927 glove.840B.300d.zip
11/21/2020  04:43 PM             1,073 LICENSE
11/21/2020  04:43 PM         3,764,231 non_adverse_media_training.csv.zip
11/21/2020  04:43 PM                21 README.md
              11 File(s)  7,841,236,360 bytes
               3 Di[...]<jupyter_text>We can see that the data required are in zipped format. Let's read them with pandas.<jupyter_code>am = pd.read_csv('adverse_media_training.csv.zip')
nam = pd.read_csv('non_adverse_media_training.csv.zip')<jupyter_output><empty_output><jupyter_text>Let's check the labels in both datasets. We may(/will :)) encounter some typos among them.<jupyter_code>print(am.label.unique())
print()
print(am.label.value_counts())
print(nam.label.unique())
print()
print(nam.label.value_counts())<jupyter_output>['nam' nan 'am' 'neither' 'random' 'doubt' 'delete']

nam        285
am          19
doubt       13
delete       9
neither      7
random       3
Name: label, dtype: int64<jupyter_text>Both datasets are not pure in their essence. We need to transfer some rows between them and drop the unnecessary rows having labels such as 'delete', 'neither' etc.<jupyter_code># Creating the AM dataset for train
am_confirmed = am.loc[(am.label == 'am') | (am.label == 'am ')]
am_confirmed = pd.concat([am_confirmed, nam.loc[nam.label == 'am']])
am_confirmed.shape
# Creating NAM dataset for train
nam_confirmed = nam.loc[(nam.label == 'nam') | (nam.label == 'random')]
nam_confirmed = pd.concat([nam_confirmed, am.loc[(am.label == 'nam') | (am.label == 'random')]])
nam_confirmed.shape
# Let us also append the necessary labels. Actually, we can also modify the label column in both datasets directly.
am_confirmed['is_adverse_media'] = 1
nam_confirmed['is_adverse_media'] = 0
# Creating the train dataset
train = pd.concat([am_confirmed, nam_confirmed])
print(train.shape)
print()
print(train['is_adverse_media'].value_counts())
# Ratio of AM to NAM class
print('Ratio of AM to NAM articles:', round(411/318, 2))<jupyter_output>Ratio of AM to NAM articles: 1.29<jupyter_text>Our dataset may turn out to be small for now, but thankfully it is not imbalanced very much. **After adding Oskar's json data, dataset imbalance will be a problem.** Anyway, let's take a quick look into what type of columns the train set has, and get some sumamry statistics on the labels.<jupyter_code>train.info()
train.is_adverse_media.describe()<jupyter_output><empty_output><jupyter_text>Since our basic task is to classify text articles, we will need the *article* and *is_adverse_media* columns for the sentiment analysis task. Later on, if we decide to add an entity recognition task or turn this into a multilabel classification problem, we will need some other columns like *entity_name* as well.  Data Preprocessing Until now, we have created the training dataset in its crude form. In this part we will filter the training data and check the articles column for null values or non-english text.<jupyter_code>train.head()<jupyter_output><empty_output><jupyter_text>Let's drop the unnecessary columns from the training dataset. (**Question for Kristjan:** Do the columns 'url, full_response and title' necessary for any other extra analysis in the future?)<jupyter_code># Keep only needed columns
train = train.loc[:, ['entity_name', 'entity_type', 'url', 'article', 'full_response', 'explanation', 'title', 'is_adverse_media']]

train.describe(include='all')<jupyter_output><empty_output><jupyter_text>Now we can narrow our focus a little bit more... Let's check if there are any nulls in article & is_adverse_media columns.<jupyter_code>sum(train.article.isna()), sum(train.is_adverse_media.isna())<jupyter_output><empty_output><jupyter_text>We gotta do one last check before tokenizing the articles, we need to check if there are any non-english text managed to slip in during the data collection process. spaCy can do this with its langdetect module, hope you succeeded in installing it.<jupyter_code>#!pip install spacy-langdetect
# Make sure we only have English articles
from spacy_langdetect import LanguageDetector

nlp = spacy.load('en_core_web_sm')
nlp.add_pipe(LanguageDetector(), name='language_detector', last=True)

# Let's create an example doc object first to test spaCy's LanguageDetector.
text = 'This is an english text. Ja see on eestikeelne lause. بوغيث	'
doc = nlp(text)
# document level language detection. Think of it like average language of the document!
print(doc._.language)
# sentence level language detection
for sent in doc.sents:
    print(sent, sent._.language)<jupyter_output>{'language': 'et', 'score': 0.9999948922362369}
This is an english text. {'language': 'en', 'score': 0.9999966325242425}
Ja see on eestikeelne lause. {'language': 'et', 'score': 0.9999937782572812}
بوغيث	 {'language': 'ar', 'score': 0.9999945969262914}<jupyter_text>The language detector acts like a weirdo while trying to understand the average language of the document. Classifying the whole text as 85 percent Estonian is a bit too much in my opinion. Let's test it on an actual article in the train dataset.<jupyter_code>example = train.article[8]
example # Clearly this one's in English.
doc = nlp(example)
print(doc._.language)<jupyter_output>{'language': 'en', 'score': 0.9999962549118911}<jupyter_text>With long texts like the articles we collected, spaCy does a good job.Now, let's check the whole dataset to see if any non-english article exists.<jupyter_code>train['article'].apply(lambda article: nlp(article)._.language['language']).unique()<jupyter_output><empty_output><jupyter_text>All of our articles are in English. We can now move on to creating tokens. Before applying any vectorizer, we need to create tokens from our articles by cleaning them from punctuation, empty spaces etc. The helper function below will use some regex commands to handle all those, besides transforming all the letters to lowercase.<jupyter_code># The regex below can be modified later on.
def lemmatize(article):
    article = re.sub(r'http\S+', '', article)
    article = re.sub(r"#(\w+)", '', article)
    article = re.sub(r"@(\w+)", '', article)
    article = re.sub(r'[^\w\s]', '', article)
    article = re.sub(r'\w*\d\w*','', article)
    article = re.sub(' +',' ', article)
    article = article.strip().lower()
    
    # nlp = spacy.load('en_core_web_sm', disable=['parser', 'ner'])
    doc = nlp(article)
    lemmatized_article = " ".join([token.lemma_ for token in doc if (token.is_stop==False)]) 
    
    return lemmatized_article<jupyter_output><empty_output><jupyter_text>Again, let's use the lemmatize function on an example to see what it does.<jupyter_code>example = train.article[8]
lemmatized = lemmatize(example)

print('Before Lemmatization:')
print()
print(example)
print()

print('After Lemmatization:')
print()
print(lemmatized)
print()<jupyter_output>Before Lemmatization:

<NAME>, who is scheduled to be sentenced June 29 for perpetrating history's biggest Ponzi scheme, is just be the latest in a long line of industry titans turned crooks

CRIMINAL EXECUTIVE OFFICER
<NAME>

CEO: ImClone
Convicted: October 15, 2002 of securities fraud, bank fraud, obstruction of justice, and perjury
Known for his networking skills as much as for his scientific expertise, immunologist <NAME> founded ImClone in 1984. The New York-based biotech firm remained relatively unknown until 1999, when it announced the creation of Erbitux — a cancer-fighting drug so promising it convinced pharmaceutical giant Bristol-Myers to purchase $1 billion of ImClone stock in one of the largest biotechnology partnerships in U.S. history. But when the Food and Drug Administration rejected the drug, Waksal alerted several relatives and friends to dump their stock as soon as possible — before the FDA's decision had been made public. Waksal's father and daughter[...]<jupyter_text>Let's copy our train data and apply lemmatization on the articles belonging to the copy.<jupyter_code>data = train[['article', 'is_adverse_media']].copy()
data = data.reset_index()
data = data.drop(['index'], axis=1)
print('Shape of our DataFrame:', data.shape)

data.head()
data['lemmatized_articles'] = data['article'].map(lemmatize)
data.head()
data = data[['is_adverse_media', 'lemmatized_articles']] # These are the only columns that we need for modeling
data = data.sample(frac = 1) # Let us not forget to shuffle the rows before train_test_split

print('Shape of our DataFrame:', data.shape)

data.head()<jupyter_output>Shape of our DataFrame: (729, 2)<jupyter_text>Let's save this cleaned dataframe as a csv file for later usage.<jupyter_code># Don't run this line again, the csv file is already created. This is just for explanatory purposes.
# data[['is_adverse_media', 'lemmatized_articles']].to_csv('./cleaned_lemmatized_text.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09470727639778811, 'fraction_numerical': 0.035987009567278155, 'mean_word_length': 4.819203268641471, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '149304', 'license': 'MIT', 'n_tokens_mistral': 3753, 'n_tokens_neox': 3393, 'n_words': 1319, 'path': 'Data/Data Prep&Preprocessing.ipynb', 'repo_name': 'kristjanr/ut-ml-adverse-media', 'size': '11408'}