<jupyter_start><jupyter_text>LIAR LIAR Dataset Clasification (Fake-News Analysis) Let’s look at the steps that we will follow in this course.- Problem Statement- Hypothesis Generation- Getting the system ready and loading the data- Understanding the data- Exploratory Data Analysis (EDA)- Univariate Analysis- Bivariate Analysis- Missing value and outlier treatment- Evaluation Metrics for classification problems-    Model Building : Part I- Logistic Regression using stratified k-folds cross validation- Feature Engineering- Model Building : Part II- Logistic Regression- Decision tree- Random Forest- XGBoostLets look at each section in detail. It is a classification problem where we have to predict whether a News Statement would be classified as Fake or Not. In a classification problem, we have to predict discrete values based on a given set of independent variable(s). Classification can be of two types:- Binary Classification : In this classification we have to predict either of the two given classes. For example: classifying the gender as male or female, predicting the result as win or loss, etc.- Multiclass Classification : Here we have to classify the data into three or more classes. For example: classifying a movie's genre as comedy, action or romantic, classify fruits as oranges, apples, or pears, etc.<jupyter_code># importing packages
import pandas as pd 
import numpy as np                     # For mathematical calculations 
import seaborn as sns                  # For data visualization 
import matplotlib.pyplot as plt        # For plotting graphs 
%matplotlib inline 
import warnings                        # To ignore any warnings warnings.filterwarnings("ignore")
##################
import seaborn as sns
from sklearn.preprocessing import MinMaxScaler
#from sklearn.cross_validation import train_test_split
from sklearn.model_selection import train_test_split

from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import f1_score
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
from nltk.stem.porter import PorterStemmer

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import cross_val_score
import nltk
import re
import codecs
import seaborn as sns
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
#from sklearn.cross_validation import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import f1_score<jupyter_output><empty_output><jupyter_text>Data For this practice problem, we have been given three CSV files: train, test and sample submission.- Train file will be used for training the model, i.e. our model will learn from this file. It contains all the independent variables and the target variable.- Test file contains all the independent variables, but not the target variable. We will apply the model to predict the target variable for the test data.- Sample submission file contains the format in which we have to submit our predictions. - Let's start with loading the dataPS: The data was in 'tsv' format which is then converted into 'csv' format later for analysing purpose.The Dataset available for the research purpose containg train,test and validation sets.<jupyter_code># converting train, test, and valid datasets into csv format
#train
tsv_file='/content/train.tsv'
csv_table=pd.read_table(tsv_file,sep='\t')
csv_table.to_csv('train_converted.csv',index=False)
#test
tsv_file1='/content/test.tsv'
csv_table1=pd.read_table(tsv_file1,sep='\t')
csv_table1.to_csv('test_converted.csv',index=False)
#valid
tsv_file2='/content/valid.tsv'
csv_table2=pd.read_table(tsv_file2,sep='\t')
csv_table2.to_csv('valid_converted.csv',index=False)<jupyter_output><empty_output><jupyter_text>Reading Data<jupyter_code>train=pd.read_csv('/content/train_converted.csv')
test=pd.read_csv('/content/test_converted.csv')
valid=pd.read_csv('/content/valid_converted.csv')
train.columns
# Naming columns
train.columns=["ID","Label","Statement","Subject(s)","Speaker","Speaker's job title","state info","party affiliation","barely true counts","false counts","half true counts","mostly true counts","pants on fire counts","the context (venue/location of speech or statement)"]
train.head()
train.shape
test.shape
valid.shape
# Naming columns 
test.columns=["ID","Label","Statement","Subject(s)","Speaker","Speaker's job title","state info","party affiliation","barely true counts","false counts","half true counts","mostly true counts","pants on fire counts","the context (venue/location of speech or statement)"]
# Naming columns 
valid.columns=["ID","Label","Statement","Subject(s)","Speaker","Speaker's job title","state info","party affiliation","barely true counts","false counts","half true counts","mostly true counts","pants on fire counts","the context (venue/location of speech or statement)"]<jupyter_output><empty_output><jupyter_text>Let’s make a copy of train and test data so that even if we have to make any changes in these datasets we would not lose the original datasets.<jupyter_code>train_copy=pd.read_csv('train.csv').copy()
test_copy=pd.read_csv('test.csv').copy()
valid_copy=pd.read_csv('valid.csv').copy()
train.head(10)
test.head(10)
valid.head(10)<jupyter_output><empty_output><jupyter_text>Loaded the datasets. Let's move to Univariate Analysis - Combining train and validation sets here. Later We will use validation techniques<jupyter_code>train = pd.concat([train, valid])
train.shape
# Dropping Target Variable from test set
Label_test = test.iloc[:,[1]]
test=test.drop(['Label'], axis=1)<jupyter_output><empty_output><jupyter_text>Univariate Analysis<jupyter_code>print(train.dtypes)
print(test.dtypes)
# shape of datasets
train.shape, test.shape
train['Label'].value_counts(normalize=True)
# Read as percentage after multiplying by 100
train['Label'].value_counts().plot.bar()<jupyter_output><empty_output><jupyter_text>On Looking at the datsets, we identified that there are 3 Data Types:- Continuous : ID, barely true counts, false counts, half true counts, mostly true counts, pants on fire counts.- Categorical : Label, Subject(s), Speaker, Speaker's job title, State info, party affiliation, the context (venue/location of speech or statement)- Text : Statement Let's go for Continuous Data Type Exploration.We know that we use BarGraphs for Categorical variable, Histogram or ScatterPlot for continuous variables. - Continuous Variables<jupyter_code># barely true counts
plt.figure(figsize=(20, 6))
sns.countplot(train["barely true counts"])
plt.title('barely true counts')
plt.show()
# false counts
plt.figure(figsize=(20, 6))
sns.countplot(train["false counts"])
plt.title('false counts')
plt.show()
# half true counts
plt.figure(figsize=(20, 6))
sns.countplot(train["half true counts"])
plt.title('half true counts')
plt.show()
# mostly true counts
plt.figure(figsize=(20, 6))
sns.countplot(train["mostly true counts"])
plt.title('mostly true counts')
plt.show()
# pants on fire counts 
plt.figure(figsize=(20, 6))
sns.countplot(train["pants on fire counts"])
plt.title('pants on fire counts')
plt.show()
# Pairplot for cross visualisation of continuous variables
plt.figure(figsize=(30,30))
sns.pairplot(train, diag_kind='kde');<jupyter_output>/usr/local/lib/python3.6/dist-packages/statsmodels/nonparametric/kde.py:447: RuntimeWarning: invalid value encountered in greater
  X = X[np.logical_and(X > clip[0], X < clip[1])] # won't work for two columns.
/usr/local/lib/python3.6/dist-packages/statsmodels/nonparametric/kde.py:447: RuntimeWarning: invalid value encountered in less
  X = X[np.logical_and(X > clip[0], X < clip[1])] # won't work for two columns.<jupyter_text>- Spread of Target Variable<jupyter_code>cnt_srs = train['Label'].value_counts()
plt.figure(figsize=(18,7))
sns.barplot(cnt_srs.index, cnt_srs.values, alpha=0.8)
plt.ylabel('Frequency', fontsize=12)
plt.xlabel('Label', fontsize=12)
plt.show()<jupyter_output><empty_output><jupyter_text>Data Preprocessing - Checking Missing Values<jupyter_code>def missing_value(df):
    total = df.isnull().sum().sort_values(ascending = False)
    percent = (df.isnull().sum()/df.isnull().count()*100).sort_values(ascending=False)
    missing_df = pd.concat([total, percent], axis=1, keys = ['Total', 'Percent'])
    return missing_df
x_train = missing_value(train)
x_train
x_test = missing_value(test)
x_test<jupyter_output><empty_output><jupyter_text>We need to treat the missing values here. It will reduce the model accuracy otherwise.<jupyter_code># Filling Missing Values with Unknown Variable
values = {"Speaker's job title":'Unknown'}
train = train.fillna(value=values)
test = test.fillna(value=values)
#valid = valid.fillna(value=values)
# Filling Missing Values with Unknown Variable
values = {"state info":'Unknown1'}
train = train.fillna(value=values)
test = test.fillna(value=values)
#valid = valid.fillna(value=values)
train['the context (venue/location of speech or statement)'].fillna(train['the context (venue/location of speech or statement)'].mode()[0], inplace=True)
test['the context (venue/location of speech or statement)'].fillna(test['the context (venue/location of speech or statement)'].mode()[0], inplace=True)
#valid['the context (venue/location of speech or statement)'].fillna(valid['the context (venue/location of speech or statement)'].mode()[0], inplace=True)
train['pants on fire counts'].fillna(train['pants on fire counts'].median(), inplace=True)
train['mostly true counts'].fillna(train['mostly true counts'].median(), inplace=True)
train['half true counts'].fillna(train['half true counts'].median(), inplace=True)
train['false counts'].fillna(train['false counts'].median(), inplace=True)
train['barely true counts'].fillna(train['barely true counts'].median(), inplace=True)
train['party affiliation'].fillna(train['party affiliation'].mode()[0], inplace=True)
train['Speaker'].fillna(train['Speaker'].mode()[0], inplace=True)
train['Subject(s)'].fillna(train['Subject(s)'].mode()[0], inplace=True)
# Checking again for missing values
y_train = missing_value(train)
y_train
# Checking again for missing values
y_test = missing_value(test)
y_test
# Checking again for missing values
#y_valid = missing_value(valid)
#y_valid<jupyter_output><empty_output><jupyter_text>- Targeting Statement Column<jupyter_code>train.shape
#removing non-letter symbols and converting text in 'Statement' to lowercase 
train['Statement'] = train['Statement'].apply(lambda x: "".join(re.sub(r"[^A-Za-z\s]", '',str(x))))
test['Statement'] = test['Statement'].apply(lambda x: "".join(re.sub(r"[^A-Za-z\s]",'',str(x))))
#valid['Statement'] = valid['Statement'].apply(lambda x: "".join(re.sub(r"[^A-Za-z\s]",'',str(x))))

# lower casing the Text
train['Statement'] = train['Statement'].apply(lambda x: " ".join(x.lower() for x in x.split()))
test['Statement'] = test['Statement'].apply(lambda x: " ".join(x.lower() for x in x.split()))
#valid['Statement'] = valid['Statement'].apply(lambda x: " ".join(x.lower() for x in x.split()))
#Removing punctuations
#adding characters list which needs to remove that is PUNCTUATION
punc = ['.', ',', '"', "'", '?', '!', ':', ';', '(', ')', '[', ']', '{', '}',"%",'/','<','>','br','�','^','XX','XXXX','xxxx','xx']
#removing extra characters
train['Statement'] = train['Statement'].apply(lambda x: " ".join(x for x in x.split() if x not in punc))
test['Statement'] = test['Statement'].apply(lambda x: " ".join(x for x in x.split() if x not in punc))
#valid['Statement'] = valid['Statement'].apply(lambda x: " ".join(x for x in x.split() if x not in punc))
import nltk
nltk.download('stopwords')
#removal of stopwords
from nltk.corpus import stopwords
stop = stopwords.words('english')
train['Statement'] = train['Statement'].apply(lambda x: " ".join(x for x in x.split() if x not in stop))
test['Statement'] = test['Statement'].apply(lambda x: " ".join(x for x in x.split() if x not in stop))
#valid['Statement'] = valid['Statement'].apply(lambda x: " ".join(x for x in x.split() if x not in stop))
#common words removal
freq_train = pd.Series(' '.join(train['Statement']).split()).value_counts()[:10]
freq_test = pd.Series(' '.join(test['Statement']).split()).value_counts()[:10]
#freq_valid = pd.Series(' '.join(valid['Statement']).split()).value_counts()[:10]

freq_train = list(freq_train.index)
train['Statement'] = train['Statement'].apply(lambda x: " ".join(x for x in x.split() if x not in freq_train))
freq_test = list(freq_test.index)
test['Statement'] = test['Statement'].apply(lambda x: " ".join(x for x in x.split() if x not in freq_test))
#freq_valid = list(freq_valid.index)
#valid['Statement'] = valid['Statement'].apply(lambda x: " ".join(x for x in x.split() if x not in freq_valid))
#rare words removal
freq_train_rare = pd.Series(' '.join(train['Statement']).split()).value_counts()[-10:]
freq_train_rare = list(freq_train_rare.index)
train['Statement'] = train['Statement'].apply(lambda x: " ".join(x for x in x.split() if x not in freq_train_rare))

freq_test_rare = pd.Series(' '.join(test['Statement']).split()).value_counts()[-10:]
freq_test_rare = list(freq_test_rare.index)
test['Statement'] = test['Statement'].apply(lambda x: " ".join(x for x in x.split() if x not in freq_test_rare))

#freq_valid_rare = pd.Series(' '.join(valid['Statement']).split()).value_counts()[-10:]
#freq_valid_rare = list(freq_valid_rare.index)
#valid['Statement'] = valid['Statement'].apply(lambda x: " ".join(x for x in x.split() if x not in freq_valid_rare))
#STEMMING
st = PorterStemmer()
train['Statement']=train['Statement'].apply(lambda x: " ".join([st.stem(w) for w in x.split()]))
test['Statement']=test['Statement'].apply(lambda x: " ".join([st.stem(w) for w in x.split()]))
train.head()
# Removing .json extension
train['ID'] = train['ID'].str.rstrip('.json')
test['ID'] = test['ID'].str.rstrip('.json')
#ID = train.iloc[:,[1]]
#train=train.drop(['ID'], axis=1)
train.columns, test.columns
X = train.loc[:,['Speaker',"Speaker's job title",'state info','party affiliation','the context (venue/location of speech or statement)']]
X1 = test.loc[:,['Speaker',"Speaker's job title",'state info','party affiliation','the context (venue/location of speech or statement)']]
X3 = pd.get_dummies(X1)
X2 = pd.get_dummies(X)   # gives (1451, 221)  
#test = pd.get_dummies(test)    # gives (1459, 206)  
x, x1 = X2.align(X3, join='inner', axis=1)  # inner join
x, x1 = X2.align(X3, join='inner', axis=1)  # inner join
x.shape, x1.shape
#x = x_data(train)  #train set
#x1 = x_data(test)
#x.shape, x1.shape
train.head()
rest_df_train = train.loc[:,['ID','barely true counts','false counts',"half true counts",'mostly true counts','pants on fire counts']]
rest_df_test = test.loc[:,['ID','barely true counts','false counts',"half true counts",'mostly true counts','pants on fire counts']]
rest_df_train.head(), rest_df_test.head()
#NOw We will convert the target variable into LabelEncoder
y_train = train.loc[:,['Label']]
labelencoder1 = LabelEncoder()
labelencoder1.fit(y_train.values)
y_train=labelencoder1.transform(y_train)
#pd.DataFrame(y_train).nunique()
# Converting the Text into list
stat_train = train['Statement'].tolist()
stat_test = test['Statement'].tolist()

# Adding train and test data
lis_stat = stat_train + stat_test 
#Fitting TFIDF on summary
tfv_sum = TfidfVectorizer(max_features=5000, lowercase=True, analyzer='word',
 stop_words= 'english',ngram_range=(1,1))
# Fitting TF-IDF to both training and test sets (semi-supervised learning)
tfv_sum.fit(lis_stat)
print(tfv_sum.vocabulary_)
print(tfv_sum.idf_)
xtrain_tfv_sum =  tfv_sum.transform(stat_train).toarray()
xtest_tfv_sum = tfv_sum.transform(stat_test).toarray()
type(xtrain_tfv_sum)
xtrain_tfv_sum.shape, xtest_tfv_sum.shape
x_train = np.concatenate((x, xtrain_tfv_sum, rest_df_train),axis = 1)
x_test = np.concatenate((x1, xtest_tfv_sum, rest_df_test),axis = 1)
x_train.shape
x_test.shape
# Fitting Naive Bayes to the Training set
from sklearn.naive_bayes import GaussianNB
classifier1 = GaussianNB()
classifier1.fit(x_train, y_train)
y_pred1 = classifier1.predict(x_test)
from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB()
clf.fit(x_train, y_train)
y_pred2 = clf.predict(x_test)
from sklearn.linear_model import LogisticRegression
classifier=LogisticRegression(solver='lbfgs',multi_class='multinomial',random_state=0,C=10)
classifier.fit(x_train,y_train)
y_pred = classifier.predict(x_test)
ans = labelencoder1.inverse_transform(y_pred)
type(ans)
ans = pd.DataFrame(ans)
id1=test.loc[:,['ID']]
final_ans = [id1, ans]
final_ans = pd.concat(final_ans, axis=1)
final_ans.columns = ['ID', 'Label']
final_ans.to_csv('NLPv1.csv',index=False)
######################################################################################################<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14961613997500448, 'fraction_numerical': 0.008510385050288638, 'mean_word_length': 6.631244323342416, 'pattern_counts': {'":': 2, '<': 47, '<?xml version=': 0, '>': 47, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '191694', 'license': 'Apache-2.0', 'n_tokens_mistral': 5502, 'n_tokens_neox': 5291, 'n_words': 1482, 'path': '06November2019.ipynb', 'repo_name': 'ssunwalka01/LIAR-LIAR-Dataset-FAKE_NEWS_Classification', 'size': '16803'}