<jupyter_start><jupyter_text><jupyter_code><jupyter_output><empty_output><jupyter_text>Spooky Author IdentificationShare code and discuss insights to identify horror authors from their writings<jupyter_code>from google.colab import files
files.upload()
!pip install -q kaggle
!mkdir -p ~/.kaggle
!cp kaggle.json ~/.kaggle/
!chmod 600 ~/.kaggle/kaggle.json
!kaggle competitions download -c spooky-author-identification
import pandas as pd
import numpy as np
from tqdm import tqdm
from keras.models import Sequential
from keras.layers.recurrent import LSTM, GRU
from keras.layers.core import Dense, Activation, Dropout
from keras.layers.embeddings import Embedding
from keras.layers.normalization import BatchNormalization
from keras.utils import np_utils
from keras.layers import GlobalMaxPooling1D, Conv1D, MaxPooling1D, Flatten, Bidirectional, SpatialDropout1D
from keras.preprocessing import sequence, text
from keras.callbacks import EarlyStopping


from sklearn import metrics, pipeline
from sklearn.model_selection import GridSearchCV

from nltk import word_tokenize
from nltk.corpus import stopwords
stop_words = stopwords.words('english')
!unzip train.zip
train = pd.read_csv('train.csv')
print(train.shape)
train.head()
train.info()   # there is no single cell is null<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 19579 entries, 0 to 19578
Data columns (total 3 columns):
id        19579 non-null object
text      19579 non-null object
author    19579 non-null object
dtypes: object(3)
memory usage: 459.0+ KB<jupyter_text>The problem requires us to predict the author, i.e. EAP, HPL and MWS given the text. In simpler words, text classification with 3 different classes.<jupyter_code>#We use the LabelEncoder from scikit-learn to convert text labels to integers, 0, 1 2
from sklearn import preprocessing
print(train.author.values)
lbl_enc = preprocessing.LabelEncoder()
y = lbl_enc.fit_transform(train.author.values)
print(y)
#Before going further it is important that we split the data into training and validation sets. We can do it using train_test_split from the model_selection module of scikit-learn.
from sklearn.model_selection import train_test_split
xtrain, xvalid, ytrain, yvalid = train_test_split(train.text.values, y, 
                                                  stratify=y, 
                                                  random_state=42, 
                                                  test_size=0.1, 
                                                  shuffle=True)
print (xtrain.shape)
print (xvalid.shape)<jupyter_output>(17621,)
(1958,)<jupyter_text>Building Basic ModelsLet's start building our very first model.Our very first model is a simple TF-IDF (Term Frequency - Inverse Document Frequency) followed by a simple Logistic Regression. (1.) logistic regression TF-IDF<jupyter_code># Always start with these features. They work (almost) everytime!
from sklearn.feature_extraction.text import TfidfVectorizer
tfv = TfidfVectorizer(min_df=3,  
                      max_features=None, 
                      strip_accents='unicode', 
                      analyzer='word',
                      token_pattern=r'\w{1,}',
                      ngram_range=(1, 3),
                      use_idf=1,smooth_idf=1,
                      sublinear_tf=1,
                      stop_words = 'english')


list(xtrain)[:5]
# Fitting TF-IDF to both training and test sets (semi-supervised learning)
tfv.fit(list(xtrain) + list(xvalid))  # Take whole document 
xtrain_tfv =  tfv.transform(xtrain)  #find the frequency of each word in particular chunks of Document
xvalid_tfv = tfv.transform(xvalid)   #find the frequency of each word in particular chunks of Document
print(len(xtrain[0]))
print(len(xtrain_tfv[0].data))  # removed 119 words(Stopwords,stems ,punctuation ,etc.) from the first page of the Whole document
xtrain_tfv[0].data
# Fitting a simple Logistic Regression on TFIDF
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(C=1.0)
clf.fit(xtrain_tfv, ytrain)
predictions = clf.predict_proba(xvalid_tfv)
def multiclass_logloss(actual, predicted, eps=1e-15):
    """Multi class version of Logarithmic Loss metric.
    :param actual: Array containing the actual target classes
    :param predicted: Matrix with class predictions, one probability per class
    """
    # Convert 'actual' to a binary array if it's not already:
    if len(actual.shape) == 1:
        actual2 = np.zeros((actual.shape[0], predicted.shape[1]))
        for i, val in enumerate(actual):
            actual2[i, val] = 1
        actual = actual2

    clip = np.clip(predicted, eps, 1 - eps)
    rows = actual.shape[0]
    vsota = np.sum(actual * np.log(clip))
    return -1.0 / rows * vsota
  
print ("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))<jupyter_output>logloss: 0.626<jupyter_text>`And there we go. We have our first model with a multiclass logloss of 0.626.``But we are greedy and want a better score. Lets look at the same model with a different data.``Instead of using TF-IDF, we can also use word counts as features. This can be done easily using CountVectorizer from scikit-learn.` CountVectorizer<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer
ctv = CountVectorizer(analyzer='word',
                      token_pattern=r'\w{1,}',
                      ngram_range=(1, 3), 
                      stop_words = 'english')


# Fitting Count Vectorizer to both training and test sets (semi-supervised learning)
ctv.fit(list(xtrain) + list(xvalid))
xtrain_ctv =  ctv.transform(xtrain) 
xvalid_ctv = ctv.transform(xvalid)
# Fitting a simple Logistic Regression on Counts
clf = LogisticRegression(C=1.0)
clf.fit(xtrain_ctv, ytrain)
predictions = clf.predict_proba(xvalid_ctv)

print ("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))<jupyter_output>/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)
/usr/local/lib/python3.6/dist-packages/sklearn/linear_model/logistic.py:469: FutureWarning: Default multi_class will be changed to 'auto' in 0.22. Specify the multi_class option to silence this warning.
  "this warning.", FutureWarning)<jupyter_text>(2.) Naive Bayes<jupyter_code># Fitting a simple Naive Bayes on TFIDF
from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB()
clf.fit(xtrain_tfv, ytrain)
predictions = clf.predict_proba(xvalid_tfv)

print ("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))<jupyter_output>logloss: 0.578<jupyter_text>`Good performance! But the logistic regression on counts is still better! What happens when we use this model on counts data instead?`<jupyter_code># Fitting a simple Naive Bayes on Counts
clf = MultinomialNB()
clf.fit(xtrain_ctv, ytrain)
predictions = clf.predict_proba(xvalid_ctv)

print ("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))<jupyter_output>logloss: 0.485<jupyter_text>(3.) SVM `nOW ,We must try SVM on this dataset.``Since SVMs take a lot of time, we will reduce the number of features from the TF-IDF using"Singular Value Decomposition" before applying SVM.``Also, note that before applying SVMs, we must standardize the data.``SVM works good on small dataset`<jupyter_code># Apply SVD, I chose 120 components. 120-200 components are good enough for SVM model.
from sklearn import decomposition
from sklearn.decomposition import TruncatedSVD
svd = decomposition.TruncatedSVD(n_components=120)
svd.fit(xtrain_tfv)   # Maximum 120 word vectors
xtrain_svd = svd.transform(xtrain_tfv)
xvalid_svd = svd.transform(xvalid_tfv)
print(len(xtrain_tfv[0].data))  # TF IDF LENGTH , length before Decomposition
print(len(xtrain_svd[0]))      # LENGTH AFTER DECOMPOSITION
print(xtrain_svd[0])
# Scale the data obtained from SVD. Renaming variable to reuse without scaling.
from sklearn import preprocessing
scl = preprocessing.StandardScaler()
scl.fit(xtrain_svd)
xtrain_svd_scl = scl.transform(xtrain_svd)
xvalid_svd_scl = scl.transform(xvalid_svd)
print(xtrain_svd_scl[0])
# Fitting a simple SVM
from sklearn.svm import SVC
clf = SVC(C=1.0, 
          probability=True) # since we need probabilities
clf.fit(xtrain_svd_scl, ytrain)
predictions = clf.predict_proba(xvalid_svd_scl)

print ("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))<jupyter_output>logloss: 0.734<jupyter_text>`Looks like SVM doesn't perform well on this data...!Before moving further, lets apply the most popular algorithm on Kaggle: xgboost!` (4.) XGBOOST<jupyter_code># Fitting a simple xgboost on tf-idf
import xgboost as xgb

clf = xgb.XGBClassifier(max_depth=7, 
                        n_estimators=200, 
                        colsample_bytree=0.8, 
                        subsample=0.8, 
                        nthread=10, 
                        learning_rate=0.1)

clf.fit(xtrain_tfv.tocsc(), ytrain)

predictions = clf.predict_proba(xvalid_tfv.tocsc())

print ("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))
# Fitting a simple xgboost on CountVectorizer
clf = xgb.XGBClassifier(max_depth=7, 
                        n_estimators=200, 
                        colsample_bytree=0.8, 
                        subsample=0.8, 
                        nthread=10, 
                        learning_rate=0.1)

clf.fit(xtrain_ctv.tocsc(), ytrain)

predictions = clf.predict_proba(xvalid_ctv.tocsc())

print ("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))
# Fitting a simple xgboost on tf-idf svd features with all the partameters
clf = xgb.XGBClassifier(max_depth=7, n_estimators=200, colsample_bytree=0.8, 
                        subsample=0.8, nthread=10, learning_rate=0.1)
clf.fit(xtrain_svd, ytrain)
predictions = clf.predict_proba(xvalid_svd)

print ("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))
# Fitting a simple xgboost on tf-idf svd features only with thread
clf = xgb.XGBClassifier(nthread=10)
clf.fit(xtrain_svd, ytrain)
predictions = clf.predict_proba(xvalid_svd)

print ("logloss: %0.3f " % multiclass_logloss(yvalid, predictions))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08881676010622602, 'fraction_numerical': 0.020458345628012197, 'mean_word_length': 3.8281101614434947, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '636293', 'license': 'MIT', 'n_tokens_mistral': 3466, 'n_tokens_neox': 3245, 'n_words': 918, 'path': 'Kaggle/NLP-CLSSIFICATION-Spooky_Author_Identification/code.ipynb', 'repo_name': 'priyanshgupta1998/Machine_learning', 'size': '10167'}