<jupyter_start><jupyter_text>Hi reader, this is a small and simple guide to RNNs, I will discuss all the basic requirements that you need to get started with RNNs from under neath concepts to code implementation.I will attatch some more resource at the bottom of this post. Table of Content: What is RNNs?The R in RNN,stands for recurrent which literally means repeating. Now you will think what is reapting here? Here reapting reffers to the reapting structurre of RNNs. There are different kind of data avilable, like normal tabular data with different features,image data,time series data, text data etc. From them time series data and text data are the data which has a sequence in it.Let me discuss the meaning of sequence for text data. It's very noticeable that every sentance has a sequence of words, and if you change the sequence of that sentance then the sentance wont has same meaning. Because of this reason we cant use MLPs on text data (there are other reasons too like large parameter size,size of i/p word and o/p word can be different etc.).Now, a RNN have many structure as per the problem requirments like- one to one, one to many,many to one,many to many as fig 1.But, here i will be discussing many to one network.But, keep in mind that a single RNN cell is same in every where, just they are placed in different ways to satisfy the problem requirments.  Many to One RNN: The structure of a many to many looks like fig2.Application of many to one networks are spam detection, sentiment analysis etc. Now lets dive deep and understand the bare bones of RNN. Every vertical block with input and output arrows is called a RNN cell, at the beginning it uses a vector with zeros which is named here as $O_0$, this is only for showing the repeating structure of RNN. the circles in the blocks are activation functions.And if you check out carefully,you will observe that there are three type of weights.$W$ which are with input word vector, $W'$ is with RNN cell o/p and finally $W''$ is at the final RNN cell which helps to make the prediction.O/p of RNN cells are taken as an i/p for the next RNN cell,this is to keep the sequence information preserved. Now, $O_i$ is,> \begin{equation} O_i = f(X_iW+O_iW') \end{equation}And at the end, the output of the last RNN cell and the $W''$ are taken and passed through an activation function(here sigmoid is taken) to generate the prediction, $Y' = S(O_nW'')$. And rest of the things are taken care by backpropagation and gradient descent stuff, normal model training.> Note, description of all symbols are given in the image itself.  Utils<jupyter_code>import warnings
warnings.filterwarnings("ignore")                     #Ignoring unnecessory warnings

import numpy as np                                  #for large and multi-dimensional arrays
import pandas as pd                                 #for data manipulation and analysis
import nltk                                         #Natural language processing tool-kit

from nltk.corpus import stopwords                   #Stopwords corpus
from nltk.stem import PorterStemmer                 # Stemmer

from sklearn.feature_extraction.text import CountVectorizer          #For Bag of words
from sklearn.feature_extraction.text import TfidfVectorizer          #For TF-IDF
from gensim.models import Word2Vec                                   #For Word2Vec

from tensorflow.keras.layers import Embedding
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.models import Sequential
from tensorflow.keras.preprocessing.text import one_hot
from tensorflow.keras.layers import LSTM
from tensorflow.keras.layers import Dropout
from tensorflow.keras.layers import Dense<jupyter_output><empty_output><jupyter_text>Loading Data Using Pandas:<jupyter_code>import pandas as pd
pd.pandas.set_option('display.max_columns',None)
pd.pandas.set_option('display.max_rows',None)
df = pd.read_csv('resume_data2.csv')
df.head()
df.drop(['Unnamed: 0'],inplace=True,axis=1)<jupyter_output><empty_output><jupyter_text>Some Basic Exploratory Data Analysis:<jupyter_code>df.head()
df.shape
df.isna().sum()<jupyter_output><empty_output><jupyter_text>Data Preprocessing: Here, we are dropping the data points where `UseId`,`"ProfileName`,`Time`,`Text` are same and keeping the first data point.<jupyter_code># df1 = df1.drop_duplicates(subset={"UserId","ProfileName","Time","Text"}, keep='first', inplace=False)<jupyter_output><empty_output><jupyter_text>Here, we are keeping the values where `HelpfulnessNumerator` <= `HelpfulnessDenominator`.<jupyter_code># df1 = df1[df1['HelpfulnessNumerator']<=df1['HelpfulnessDenominator']]<jupyter_output><empty_output><jupyter_text>As we have taken this dataset as a NLP problem, so we are only considering the text column as the i/p data and `score` as the labels.<jupyter_code># list1 = list(df['score'])
# list2 = list(df['cv_txt'])

# score_df = pd.DataFrame(list1,columns = ['score'])
# text_df = pd.DataFrame(list2,columns = ['cv_txt'])

# df2 = pd.concat([text_df,score_df],axis=1)
# df2.head()<jupyter_output><empty_output><jupyter_text>Checking out some of the text which starts from 500 index and ends at 550 index.<jupyter_code>for i in range(1,3):
    print(df['cv_txt'].values[i])
    print('-----------------------------------------------------')
# df3 = df2.head(50000)
# df3.head()<jupyter_output><empty_output><jupyter_text>Here we have created a function which will mark the score from 1 to 3 as negative and 4 to 5 as positive. A we are going to do binary classification that's why we are keeping the labels to positive and negative.<jupyter_code># def partition(x):
#     if x < 3:
#         return 'negative'
#     return 'positive'

# score_upd = df3['score']
# t = score_upd.map(partition)
# df3['score']=t

# df3.head()
df.isna().sum()<jupyter_output><empty_output><jupyter_text>Here, df_x is the i/p text data and df_y is the i/p lable.<jupyter_code>df_x = df['cv_txt']
df_y = df['score']
stop_words = set(stopwords.words('english'))
len(stop_words) #finding stop words<jupyter_output><empty_output><jupyter_text>Here we are doing the real pre-processing,which are like- keeping only the alphabets,making all the alphabets lower,removing all the stop word.<jupyter_code>import re
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
snow = nltk.stem.SnowballStemmer('english')

corpus = []
for i in range(0, len(df)):
    review = re.sub('[^a-zA-Z]', ' ', df['cv_txt'][i])
    review = review.lower()
    review = review.split()
    
    review = [snow.stem(word) for word in review if not word in stopwords.words('english')]
    review = ' '.join(review)
    corpus.append(review)
corpus[1]
df_x = corpus
type(df_x)<jupyter_output><empty_output><jupyter_text>This is an important step, here we are creating word vectors by doing one hot encoing, and we are only taking 5000 words as the dictionary.<jupyter_code>voc_size=5000
onehot_repr=[one_hot(words,voc_size)for words in corpus] 
type(onehot_repr)<jupyter_output><empty_output><jupyter_text>Padding is one of the most important part before feeding the data to the model. In simple way padding is a way to keep the input size same for all i/p text by adding zeros at the front. Here we are considering each i/p text corpus will be of 400 words.<jupyter_code>sent_length=400
embedded_docs=pad_sequences(onehot_repr,padding='pre',maxlen=sent_length)
print(embedded_docs)<jupyter_output>[[   0    0    0 ... 1730 4180   32]
 [   0    0    0 ... 1055 2683 2402]
 [   0    0    0 ... 4087  530  616]
 ...
 [   0    0    0 ... 1551  244 4276]
 [1645 1314 1686 ...  244  244 4276]
 [   0    0    0 ... 1647  244 4276]]<jupyter_text>This is the model build using tensorflow 2.0. Think about the whole model like this,first its a sequencial model starting with a `Embedding` layer (wich is deep learning version of word to vec,or more intuitively WV is an example of `Embedding` and we are use it because we can be trained,it's not static like WV) and input size is 5000x40 and output size of 400x40.Now we created a dropout layer using `Dropout()` which will reduce the chance of overfitting. After that we are taking 100 layers of LSTM and feeding the output of Embedding leyer to each of the LSTM layer . Now, each of the `LSTM` layer will spit out a scaller value and then we will be stacking them out,because of that the output size after `LSTM` layer would be 100. Ten we againg perform a Dropout layer. At the end we have a single activation unit of `sigmoid` (line 8) which will take the 100 sized vector and give a prediction or in general words give a scaler value between 0 to 1. Now we are setting the optimixer as `adam` and evaluation matrics as `accurecy` and loss as `binary_crossentropy`(this is because we have taken this as a binary classification problem) using the `model.compile()` method.At the 10th line we are printing the model architecture using `model.summary()` method.<jupyter_code>## Creating model
embedding_vector_features=40
model=Sequential()
model.add(Embedding(voc_size,embedding_vector_features,input_length=sent_length))
model.add(Dropout(0.3))
model.add(LSTM(100))
model.add(Dropout(0.3))
model.add(Dense(1,activation='sigmoid'))
model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])
print(model.summary())<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding (Embedding)        (None, 400, 40)           200000    
_________________________________________________________________
dropout (Dropout)            (None, 400, 40)           0         
_________________________________________________________________
lstm (LSTM)                  (None, 100)               56400     
_________________________________________________________________
dropout_1 (Dropout)          (None, 100)               0         
_________________________________________________________________
dense (Dense)                (None, 1)                 101       
=================================================================
Total params: 256,501
Trainable params: 256,501
Non-trainable params: 0
__________________________________________________[...]<jupyter_text>Some preprocessing before feeding the data. We are doing lable encoding of the `score` column.<jupyter_code># from sklearn.preprocessing import LabelEncoder
# encode = LabelEncoder()
# df_y2 = encode.fit_transform(df_y)
# type(df_y2)<jupyter_output><empty_output><jupyter_text>This is a important part, where we car converting our data to nd arrays as we cant just input a pandas data frame.<jupyter_code>import numpy as np
X_final=np.array(embedded_docs)
y_final=np.array(df_y)<jupyter_output><empty_output><jupyter_text>Train Test Split:<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, Y_train, Y_test = train_test_split(X_final, y_final, test_size=0.2, random_state=42)
#we are feeding the 
model.fit(X_train,Y_train,validation_data=(X_test,Y_test),epochs=100,batch_size=64)
y_pred_lstm = model.predict_classes(X_test)
from sklearn.metrics import accuracy_score
accuracy_score(Y_test,y_pred_lstm)<jupyter_output><empty_output><jupyter_text>TF-IDF:<jupyter_code>final_tf = df_x
tf_idf = TfidfVectorizer(ngram_range=(1,2),max_features=5000)
tf_data = tf_idf.fit_transform(final_tf)
print(tf_data[1])
pickle.dump(tf_idf, open("tfidf_vectorizer.pkl", "wb"))
v = pickle.load(open('tfidf_vectorizer.pkl','rb'))
tf_data.get_shape()
features = tf_idf.get_feature_names()
features[:10]
from sklearn.preprocessing import LabelEncoder
encode = LabelEncoder()
df_y1 = encode.fit_transform(df_y)
type(df_y1)
# pd.DataFrame(['df_y1'])
SEED =42
from sklearn.model_selection import train_test_split
x_train,x_val,y_train,y_val = train_test_split(tf_data,df_y,test_size = 0.2,random_state=SEED)
from sklearn.naive_bayes import MultinomialNB
nb_model = MultinomialNB().fit(x_train, y_train)

y_pred_nb=nb_model.predict(x_val)
from sklearn.metrics import accuracy_score
accuracy_score(y_val, y_pred_nb)<jupyter_output><empty_output><jupyter_text>Rf:<jupyter_code>from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier()
rf_model = rf.fit(x_train,y_train)
y_pred_rf = rf_model.predict(x_val)
accuracy_score(y_val,y_pred_rf)
import pickle
pickle.dump(rf_model,open('rf_score_model.pkl','wb'))
#TESTING WITH SOME DATE
test = 'last updat st june debarghya das debarghyada com fb co dd deedi fb com dd cornel edu educ cornel univers meng comput scienc dec ithaca ny cornel univers bs comput scienc may ithaca ny colleg engin magna cum laud cum gpa major gpa la martinier boy grad may kolkata india link facebook dd github deedyda linkedin debarghyada youtub deedydash twitter debarghya das quora debarghya das coursework graduat advanc machin learn open sourc softwar engin advanc interact graphic compil practicum cloud comput evolutionari comput defend comput network machin learn undergradu inform retriev oper system artifici intellig practicum function program comput graphic practicum research asst teach asst x unix tool script skill program line java shell python javascript ocaml matlab rail latex line c c css php assembl familiar io android mysql experi facebook softwar engin jan present new york ny coursera kpcb fellow softwar engin intern june sep mountain view ca applic chosen kpcb fellow led ship yoda admin interfac new phoenix platform full stack develop wrote review code js use backbon jade stylus requir scala use play googl softwar engin intern may aug mountain view ca work youtub caption team javascript python plan design develop full stack add edit automat speech recognit caption product creat backbon js like framework caption editor phabric open sourc contributor team leader jan may palo alto ca ithaca ny phabric use daili facebook dropbox quora asana creat meme generat php shell led team mit cornel ic london uhelsinki project research cornel robot learn lab research jan jan ithaca ny work ashesh jain prof ashutosh saxena creat planit tool learn larg scale user prefer feedback plan robot trajectori human environ cornel phonet lab head undergradu research mar may ithaca ny led develop quicktongu first ever breakthrough tongu control game prof sam tilsen aid linguist research award top kpcb engin fellow st microsoft code competit cornel nation jump trade challeng finalist th cs cach race bot tournament nd cs biannual intra class bot tournament nation indian nation mathemat olympiad inmo finalist public jain das saxena planit crowdsourc approach learn plan path larg scale prefer feedback tech report icra press tilsen das b mckee real time articulatori biofeedback electromagnet articulographi linguist vanguard press'
snow = nltk.stem.SnowballStemmer('english')

corpus_test = []
# for i in range(0, len(df)):
review = re.sub('[^a-zA-Z]', ' ', test)
review = review.lower()
review = review.split()
    
review = [snow.stem(word) for word in review if not word in stopwords.words('english')]
review = ' '.join(review)
corpus_test.append(review)
final_tf_test = corpus_test
# tf_idf_test = TfidfVectorizer(ngram_range=(1,2),max_features=5000)
tf_data_test = v.transform(final_tf_test)
tf_data_test.get_shape()
loaded_model = pickle.load(open('rf_score_model.pkl', 'rb'))
result = loaded_model.predict(tf_data_test)
print(result)<jupyter_output>[6]<jupyter_text>rf with rscv:<jupyter_code># Number of trees in random forest
n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]
# Number of features to consider at every split
max_features = ['auto', 'sqrt']
# Maximum number of levels in tree
max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]
max_depth.append(None)
# Minimum number of samples required to split a node
min_samples_split = [2, 5, 10]
# Minimum number of samples required at each leaf node
min_samples_leaf = [1, 2, 4]
# Method of selecting samples for training each tree
bootstrap = [True, False]



random_grid = {'n_estimators': n_estimators,
               'max_features': max_features,
               'max_depth': max_depth,
               'min_samples_split': min_samples_split,
               'min_samples_leaf': min_samples_leaf,
               'bootstrap': bootstrap}

cls_rf_rscv = RandomForestClassifier()
from sklearn.model_selection import RandomizedSearchCV

random_search_rf = RandomizedSearchCV(cls_rf_rscv, random_grid,n_iter=5, n_jobs=1, cv=5,verbose=2)
random_search_rf.fit(x_train,y_train)
random_search_rf.best_params_
base_model = RandomForestClassifier(n_estimators= 2000,
                                     min_samples_split= 10,
                                     min_samples_leaf= 1,
                                     max_features= 'sqrt',
                                     max_depth= 90,
                                     bootstrap= True,
                                    random_state = SEED)
base_model.fit(x_train, y_train)
y_pred_rf_rscv = base_model.predict(x_val)
# y_pred_rf_rscv = rf_rscv_model.predict(x_val)
accuracy_score(y_val,y_pred_rf_rscv)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08555839331321105, 'fraction_numerical': 0.017819828186672857, 'mean_word_length': 4.5739242963442255, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '538951', 'license': 'MIT', 'n_tokens_mistral': 5427, 'n_tokens_neox': 5083, 'n_words': 1856, 'path': 'Notebooks/classification model training notebook.ipynb', 'repo_name': 'streamlit-badge-bot/Resume-analyzer', 'size': '17228'}