<jupyter_start><jupyter_text>CV Assignment  In this project, I've implemented a binary classifier and a bidirectional LSTM network. The dataset used consists of various tweets.<jupyter_code># packages rerquired for data preprocessing and representation
import pandas as pd
import seaborn as sns
import tensorflow as tf
import matplotlib.pyplot as plt
import re
import nltk
import numpy as np
# packages required for building the model
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential 
from keras.layers import Dense, Dropout
from keras.layers.embeddings import Embedding 
from keras.layers.convolutional import Conv1D
from keras.layers.convolutional import MaxPooling1D
from tensorflow.keras.layers import Bidirectional, LSTM
from tensorflow.keras.optimizers import Adam
from keras.preprocessing import sequence
# package required for computation of metrics
from sklearn.metrics import matthews_corrcoef, confusion_matrix, accuracy_score, f1_score
# reading the given dataset
df = pd.read_csv('CVAssignmentDataset.csv')
df.dropna(axis = 0, inplace = True)
df.head()<jupyter_output><empty_output><jupyter_text>The following code blocks contain codes which are used for Exploratory Data Analysis<jupyter_code>df.shape
df.info()
sns.countplot(x='Class', data = df)
plt.show
df['Class'].value_counts()
length = df['Text'].str.len()
plt.hist(length, bins=20, label="none")
plt.show()<jupyter_output><empty_output><jupyter_text>ConclusionsTotal number of rows = 86460 Number of values under class 0 : 44392 Number of values under class 1 : 42068The data preprocessing phase begins now<jupyter_code>#converting entries under 'Text' to string type
df['Text']=df['Text'].apply(str)
# function used to remove unwanted text patterns
def remove_pattern(input_text, pattern):
    r = re.findall(pattern, input_text)
    for i in r:
        input_text = re.sub(i, '', input_text)
    return input_text
df['processed_tweet'] = np.vectorize(remove_pattern)(df['Text'], "@[\w]*") #removing special characters(except #)
df['processed_tweet'] = df['processed_tweet'].str.replace("[^a-zA-Z#]", " ") #removing everything except alphabets and #
df['processed_tweet'] = df['processed_tweet'].str.replace(r'http\S+', " ") #removing links
# removing words of shorter length
df['processed_tweet'] = df['processed_tweet'].apply(lambda x: ' '.join([w for w in x.split() if len(w)>3])) 
# packages required for further data preprocessing
from nltk import word_tokenize, FreqDist
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
nltk.download
nltk.download('stopwords')
from nltk.tokenize import TweetTokenizer
#Let's have a look at the processed data till now
df['processed_tweet'].head()
# converting the processed_tweet column in  pandas to a numpy array
dataArray = df.processed_tweet.to_numpy()
dataList = []
lemma = nltk.WordNetLemmatizer()

for data in dataArray:
    data = data.lower() 
    data = nltk.word_tokenize(data)
    #removal of stop-words(here stopwords.words('english') contains a set of frequently appearing stop words.
    stop_words = set(stopwords.words('english')) 
    data = [w for w in data if not w in stop_words]
    data = [lemma.lemmatize(word) for word in data] #Lemmatizing(removing inflectional endings and returning the base forms of words)
    data = " ".join(data)
    dataList.append(data)


dataArray = np.asarray(dataList) #updating dataArray with the obtained list
df['processed_tweet'] = dataArray
df['processed_tweet']
# tokenizing the processed words using TweetTokenizer
from nltk.tokenize.casual import TweetTokenizer
tokenizer = TweetTokenizer(reduce_len=True)

df['processed_tweet'] = df.processed_tweet.apply(tokenizer.tokenize)

# converting the tokens to frequency distribution rankings
topWords = 10000 # chosen as given in the problem statement 
freq_dist = FreqDist(w for t in df.processed_tweet for w in t)
tweetTerms = [te for te, cnt in freq_dist.most_common(topWords)]
df.processed_tweet = df.processed_tweet.apply(lambda tweet:[tweetTerms.index(te) if te in tweetTerms else 0 for te in tweet])
# Let's now have a look at the tokenized and ranked dataset
df.processed_tweet
# padding is done to bring the processed dataset to a length of 50, as given in the problem statement
x = df.processed_tweet
ipLength = 50
x = sequence.pad_sequences(x, maxlen = ipLength)
print(x.shape)

# here the dataset is assigned to a variable x, which will be further used for training and testing.
# This is how our dataset(with values) looks like, after padding
df.processed_tweet
# now the labels are assigned to a variable y
y = df.Class
# splitting the pre-processed data into training and testing datasets
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(x,y, test_size=0.25, random_state=42)
# converting the splitted data into arrays
x_train = np.asarray(x_train)
y_train = np.asarray(y_train)
x_test = np.asarray(x_test)
y_test = np.asarray(y_test)

# type conversion to float
x_train = x_train.astype(float)
y_train = y_train.astype(float)
x_test = x_test.astype(float)
y_test = y_test.astype(float)
#creating the model as given in the model architecture image in the problem statement
def baseline_model():
    
    model = Sequential() # one input tensor and one output tensor
    embeddingVectorLength = 32 # assigned as given in the image
    
    model.add(layers.Input(shape = ipLength, dtype='float64')) # first layer(input)
    model.add(Embedding(topWords, embeddingVectorLength, input_length = ipLength))
    
    model.add(Conv1D(filters = 32, kernel_size = 3, padding = 'same', activation = 'relu')) #kernel size is chosen as 3 pertaining to normal convention
    model.add(MaxPooling1D(pool_size=2))
    
    model.add(Bidirectional(LSTM(256, return_sequences=True)))
    model.add(Dropout(0.5))
    
    model.add(Bidirectional(LSTM(256)))
    model.add(Dropout(0.5))
    
    model.add(Dense(1024, activation='relu'))
    model.add(Dropout(0.5))
    
    model.add(Dense(512, activation='relu'))
    model.add(Dropout(0.5))
    
    model.add(Dense(1, activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    return model
    
# let's run the summary of the model, and start training
model = baseline_model()
print(model.summary())
model.fit(x_train, y_train, epochs=3, batch_size=64, validation_split=0.33, verbose=1)<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding (Embedding)        (None, 50, 32)            320000    
_________________________________________________________________
conv1d (Conv1D)              (None, 50, 32)            3104      
_________________________________________________________________
max_pooling1d (MaxPooling1D) (None, 25, 32)            0         
_________________________________________________________________
bidirectional (Bidirectional (None, 25, 512)           591872    
_________________________________________________________________
dropout (Dropout)            (None, 25, 512)           0         
_________________________________________________________________
bidirectional_1 (Bidirection (None, 512)               1574912   
________________________________________________________[...]<jupyter_text>After multiple trials, the number of epochs is chosen to be 3 and validation_split is assigned a value of 0.33(pertaining to standard convention). After observing the increase in accuracy and reduction in loss, it is inferred that the model works fine. It is also noted that the validation loss increases, over the epochs. It was observed that for more number of epochs, the validation loss fluctuates (i.e) reduces and later increases which is an indication of overfitting. This is a common phenomenon, observed in many models. Hence to avoid that, the number of epochs is reduced to 3 and the test_size is also reduced. This leads to increased accuracy of predicted values and lesser chances of overfitting<jupyter_code>y_pred = model.predict(x_test, use_multiprocessing=True)
print('Accuracy {}'.format(accuracy_score(y_test, np.round(y_pred))))
print('Confusion Matrix {}'.format(confusion_matrix(y_test, np.round(y_pred))))
print('Matthews Correlation Coefficient {}'.format(matthews_corrcoef(y_test, np.round(y_pred))))
print('F1 score {}'.format(f1_score(y_test, np.round(y_pred))))<jupyter_output>F1 score 0.7149658862219603
{'fraction_non_alphanumeric': 0.09585853885528153, 'fraction_numerical': 0.018496975337366217, 'mean_word_length': 5.193804034582133, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '455164', 'license': 'MIT', 'n_tokens_mistral': 2667, 'n_tokens_neox': 2480, 'n_words': 805, 'path': 'CV_assignment.ipynb', 'repo_name': 'Charaan27/computer-vision-assignment', 'size': '8596'}