<jupyter_start><jupyter_text>Question  2 : Data preparation<jupyter_code>posPath='aclImdb/train/pos'
arr = []
for filename in tqdm(os.listdir(posPath)):
    path= os.path.join(posPath,filename)
    with open(path) as f:
        review = f.readlines()
        arr.append(review)
        
posArr = np.array(arr)
posArr = np.insert(posArr, 1, 1, axis=1)
negPath='aclImdb/train/neg'
arr = []
for filename in tqdm(os.listdir(negPath)):
    #print(filename)
    #path='imdb/train/neg/0_3.txt'
    path= os.path.join(negPath,filename)
    with open(path) as f:
        review = f.readlines()
        arr.append(review)

negArr = np.array(arr)
negArr = np.insert(negArr, 1, 0, axis=1)
print(posArr.shape)
print(negArr.shape)<jupyter_output>(12500, 2)
(12500, 2)<jupyter_text>2 .1 Prepare the data so it can be the input for a machine learning model (samples, target)<jupyter_code># we will join both dataframes
df1 = pd.DataFrame(data = posArr, columns = ['review','label'])
df2 = pd.DataFrame(data = negArr, columns = ['review','label'])
reviews_df = df1.append(df2)
print(df1.head(2))
print(df2.head(2))
reviews_df.head()
reviews_df.groupby('label').count()<jupyter_output><empty_output><jupyter_text>2.2 Clean and preprocess the text<jupyter_code>def pre_process(text):
    # lowercase
    text = text.lower()
    # tags
    text = re.sub('&lt;/?.*?&gt;',' &lt;&gt; ',text)
    # special characters and digits
    text=re.sub('(\\d|\\W)+',' ',text)
    
    return text

reviews_df['review'] = reviews_df['review'].apply(lambda x:pre_process(x))
reviews_df.head()<jupyter_output><empty_output><jupyter_text>3 Train a machine learning classifier capable of predicting the sentiment of a review 3.1 Preprocessing & Machine Learning ( three approuches was tried)  3.1.2 simple TF-IDF (Term Frequency - Inverse Document Frequency) followed by a simple Logistic Regression.<jupyter_code># FIRST: we split the data
x = reviews_df.review
y = reviews_df.label
xtrain, xvalid, ytrain, yvalid = train_test_split(x, y, 
                                                  stratify=y, 
                                                  random_state=42, 
                                                  test_size=0.3, shuffle=True)
# SECOND: Transform reviews to TFIDF - vectorizer
tfv = TfidfVectorizer(min_df=3,  max_features=None, 
            strip_accents='unicode', analyzer='word',token_pattern=r'\w{1,}',
            ngram_range=(1, 3), use_idf=1,smooth_idf=1,sublinear_tf=1,
            stop_words = 'english')

# THIRD : Fitting TF-IDF to both training and validation data sets.
tfv.fit(list(xtrain) + list(xvalid))
xtrain_tfv =  tfv.transform(xtrain) 
xvalid_tfv = tfv.transform(xvalid)
# FOURTH : Fitting a simple Logistic Regression on TFIDF
clf_lr = LogisticRegression(C=1.0) # initiate model instance
clf_lr.fit(xtrain_tfv, ytrain) # fit model 


# FIFTH : evaluate model performance 
print('Accuracy Score;',clf_lr.score(xvalid_tfv,yvalid))
confusion_matrix(yvalid, clf_lr.predict(xvalid_tfv))
plot_confusion_matrix(clf_lr, xvalid_tfv, yvalid) 
print(classification_report(yvalid, clf_lr.predict(xvalid_tfv)))<jupyter_output>Accuracy Score; 0.8784
              precision    recall  f1-score   support

           0       0.89      0.86      0.88      3750
           1       0.86      0.90      0.88      3750

    accuracy                           0.88      7500
   macro avg       0.88      0.88      0.88      7500
weighted avg       0.88      0.88      0.88      7500<jupyter_text>3.1.3 using word counts as features ( CountVectorizer) followed by a simple Logistic Regression.<jupyter_code># FIRST : initaite CountVectorizer
ctv = CountVectorizer(analyzer='word',token_pattern=r'\w{1,}',
            ngram_range=(1, 3), stop_words = 'english')

# SECOND : Fitting Count Vectorizer to both training and validation sets
ctv.fit(list(xtrain) + list(xvalid))
xtrain_ctv =  ctv.transform(xtrain) 
xvalid_ctv = ctv.transform(xvalid)

#THIRD : Fitting a simple Logistic Regression on Counts
clf_lr2 = LogisticRegression(C=1.0 , max_iter = 200)
clf_lr2.fit(xtrain_ctv, ytrain)

# FIFTH : evaluate model performance 
print('Accuracy Score;',clf_lr2.score(xvalid_ctv,yvalid))
confusion_matrix(yvalid, clf_lr2.predict(xvalid_ctv))
plot_confusion_matrix(clf_lr2, xvalid_ctv, yvalid) 
print(classification_report(yvalid, clf_lr2.predict(xvalid_ctv)))<jupyter_output>Accuracy Score; 0.8756
              precision    recall  f1-score   support

           0       0.88      0.87      0.87      3750
           1       0.87      0.89      0.88      3750

    accuracy                           0.88      7500
   macro avg       0.88      0.88      0.88      7500
weighted avg       0.88      0.88      0.88      7500<jupyter_text>3.1.4 SVMs using the TF-IDF using Singular Value Decomposition ( to incraese performance)<jupyter_code># FIRST : Apply SVD, I chose 150 components. we reduse to enhance performance. 
svd = decomposition.TruncatedSVD(n_components=150)
svd.fit(xtrain_tfv)
xtrain_svd = svd.transform(xtrain_tfv)
xvalid_svd = svd.transform(xvalid_tfv)

# SECOND : Scale the data obtained from SVD. Renaming variable to reuse without scaling.
scl = preprocessing.StandardScaler()
scl.fit(xtrain_svd)
xtrain_svd_scl = scl.transform(xtrain_svd)
xvalid_svd_scl = scl.transform(xvalid_svd)

# THIRD : Fitting a simple SVM
clf_svd = SVC(C=1.0, probability=True) 
clf_svd.fit(xtrain_svd_scl, ytrain)

## FOURTH: evaluate model performance 
print('Accuracy Score;',clf_svd.score(xvalid_svd_scl,yvalid))
confusion_matrix(yvalid, clf_svd.predict(xvalid_svd_scl))
plot_confusion_matrix(clf_svd, xvalid_svd_scl, yvalid) 
print(classification_report(yvalid, clf_svd.predict(xvalid_svd_scl)))<jupyter_output>Accuracy Score; 0.8689333333333333
              precision    recall  f1-score   support

           0       0.85      0.89      0.87      3750
           1       0.88      0.85      0.87      3750

    accuracy                           0.87      7500
   macro avg       0.87      0.87      0.87      7500
weighted avg       0.87      0.87      0.87      7500<jupyter_text>4.Evaluate your model extracting precision, recall and f1  all the model have been evaluated in the previous step.  5.how the prediction of your model for one review<jupyter_code>index = 4348
print(xvalid[index])
print(yvalid[index])
print(clf_lr.predict(xvalid_tfv[index])) # prediction according to linear regression TF-IDF 
print(clf_lr2.predict(xvalid_ctv[index]))  # prediction according to linear regression CountVectorizer 
print(clf_svd.predict(xvalid_svd_scl[index].reshape(1, -1)))  # prediction according to SVM<jupyter_output>['0']<jupyter_text>6. Can you extract general topics from this corpus of texts? (Optional) Latend Dirichlet Allocation<jupyter_code>from sklearn.decomposition import LatentDirichletAllocation
# !python -m spacy download en_core_web_sm
nlp = spacy.load('en_core_web_sm')  # load spacy model


# define function to tokenize
def spacy_tokenizer(sentence):
    return [word.lemma_ for word in nlp(sentence) if not (word.like_num or word.is_stop or word.is_punct or word.is_space or len(word)==1)]


# toknize the words and remove stop words 
vectorizer = CountVectorizer(tokenizer = spacy_tokenizer, min_df=2)
data_vectorized = vectorizer.fit_transform(tqdm(reviews_df.review))
# most frequent words
word_count = pd.DataFrame({'word': vectorizer.get_feature_names(), 'count': np.asarray(data_vectorized.sum(axis=0))[0]})
word_count.sort_values('count', ascending=False).set_index('word')[:20].sort_values('count', ascending=True).plot(kind='barh')
# initaite Latent Dirichlet Allocation
lda = LatentDirichletAllocation(n_components=50, random_state=0)
# fit LDA to the data
lda.fit(data_vectorized)
# print top topics 

def print_top_words(model, vectorizer, n_top_words):
    feature_names = vectorizer.get_feature_names()
    for topic_idx, topic in enumerate(model.components_):
        message = "\nTopic #%d: " % topic_idx
        message += " ".join([feature_names[i]
                             for i in topic.argsort()[:-n_top_words - 1:-1]])
        print(message)
    print()
print_top_words(lda, vectorizer, n_top_words=25)<jupyter_output>Topic #0: br good film like time go play great know work man think love story little find end people movie scene want life year come get

Topic #1: film hardy stan silent laurel star davy lou br comedy grace studio ollie buster marion slapstick role chaplin taylor mary scene shepherd warner mgm gag

Topic #2: movie action good game film fight great story scene character like original time play art fan martial see watch fu well kung sequence plot lot

Topic #3: br film play man character story action like time scene good seagal way know star find peter set get bakshi david cagney fall lead role

Topic #4: ben br film gay canadian parker park arthur metal brosnan sport south stone zizek kubrick baseball grey canada brother pierce chaplin barrymore owl day example

Topic #5: br life film love movie like people man way feel character woman new real come good think find watch live time story want girl work

Topic #6: man western stewart br play film town good james get character story kill[...]<jupyter_text>7. Answer the following questions:  7.1 Have you observed any limitations in this dataset?<jupyter_code># this database is very well balanced.<jupyter_output><empty_output><jupyter_text>7.2 What other techniques or methods that you haven’t used could you apply in any of the stages of the NLP pipeline to improve the performance of the model?<jupyter_code># we can use deep learning to learn the needed features from the text.<jupyter_output><empty_output><jupyter_text>7.3 In case you haven’t answered question 6, what technique could you use if you wanted to extract topics from the corpus of texts?<jupyter_code># question 6 was answered<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09409612844223147, 'fraction_numerical': 0.0339396402804593, 'mean_word_length': 3.693371483071054, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '487002', 'license': 'MIT', 'n_tokens_mistral': 3467, 'n_tokens_neox': 3265, 'n_words': 893, 'path': 'AdvancedDataAnalysis/Final Assignment/Advanced Data Analysis - Assignment 2.ipynb', 'repo_name': 'robretoarenal/BTS2', 'size': '9841'}