<jupyter_start><jupyter_text>**Resolução Lista 3**---<NAME> <NAME><jupyter_code>#from google.colab import files
#uploaded = files.upload()
# upando train_data e test_data_solution<jupyter_output><empty_output><jupyter_text>Questão 1Escolha 10 documentos da base, e determine seu documento mais parecido(que não pode ser ele mesmo), usando:a) Representação vetorial CountVectorizer com similaridade do cosseno;b) Representação vetorial TF-IDF com similaridade do cosseno.Mostre pelo menos um trecho de cada par de documentos parecidos que justifiqueessa semelhança.<jupyter_code>import pandas as pd
import numpy as np
import random
import re
import nltk
from IPython.core.display import display, HTML
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import LinearSVC
from matplotlib import pyplot as plt
from sklearn.metrics import plot_confusion_matrix
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
import warnings
warnings.filterwarnings('ignore')
nltk.download('stopwords')
nltk.download('punkt')
df = pd.read_csv('train_data.txt', sep =':::', header = None, engine = 'python', nrows = 6000)
df.columns = ['id', 'title', 'genre', 'description']
df2 = pd.read_csv('test_data_solution.txt', sep =':::', header = None, engine = 'python', nrows = 6000)
df2.columns = ['id', 'title', 'genre', 'description']
df3 = pd.concat([df, df2], ignore_index=True)
df3.drop('id', axis=1, inplace=True)
comedy = df3.loc[df3['genre'].str.contains('comedy')]
drama = df3.loc[df3['genre'].str.contains('drama')]
dataset = pd.concat([comedy, drama], ignore_index=True)
def preprocess(description):
  description = re.sub(r'\w*\d\w*', '', description) #remove todas as palavras que contêm números
  description = re.sub(r'[^a-zA-Z ]', '', description.lower())
  return re.sub(r'\s+', ' ', description) #retira espaços repetidos
dataset['description'] = dataset.description.apply(preprocess)
stop_words = set(nltk.corpus.stopwords.words('english'))
def remove_stopwords(description):
    tokenized_text = nltk.word_tokenize(description, language='english')
    return " ".join([token for token in tokenized_text if token not in stop_words])
dataset['description'] = dataset.description.apply(remove_stopwords)
dataset
tfidf_X = TfidfVectorizer().fit_transform(dataset.description)
count_X = CountVectorizer().fit_transform(dataset.description)

tfidf_similarity_matrix = cosine_similarity(tfidf_X)
count_similarity_matrix = cosine_similarity(count_X)
np.fill_diagonal(tfidf_similarity_matrix, 0.0)
np.fill_diagonal(count_similarity_matrix, 0.0)

result = [
    {
        'doc': dataset.iloc[i],
        'tfidf_similar_doc': dataset.iloc[tfidf_similarity_matrix[i, :].argmax()],
        'tfidf_similarity': tfidf_similarity_matrix[i, :].max(),
        'count_similar_doc': dataset.iloc[count_similarity_matrix[i, :].argmax()],
        'count_similarity': count_similarity_matrix[i, :].max(),
    }
    for i in [random.randint(0, len(dataset)) for _ in range(10)]
]    
%%html
<style>
    .col {
        height: 100%;
        padding-left: 16px;
        padding-right: 16px;
        padding-top: 8px;
        padding-bottom: 8px;
    }
</style><jupyter_output><empty_output><jupyter_text>1.a) Representação vetorial CountVectorizer com similaridade do cosseno; 1.b) Representação vetorial TF-IDF com similaridade do cosseno.<jupyter_code>for r in result:
    title = r['doc'].title if r['doc'].title != '' else r['doc'].description[:150] + '...'
    movie = '<span>%(title)s</span> <span>%(description)s</span>' 
    desc_length = 300
    
    tfidf_content = movie % ({
        'title': r['tfidf_similar_doc'].title,
        'description': r['tfidf_similar_doc'].description[:desc_length] + '...',
    })
    count_content = movie % ({ 
        'title': r['count_similar_doc'].title,
        'description': r['count_similar_doc'].description[:desc_length] + '...',
    })
    
    display(HTML('''
    <div style="margin: 8px; border: 1px solid lightgray">
        <div style="text-align: center;background-color: whitesmoke;padding: 8px"><u r['doc'].title >''' + title + '''</u></div>
        <div>
            <div>
                <div class="col">
                <div><strong>TfidfVectorizer: </strong>''' + str(r['tfidf_similarity'])[:6] + '''</div>
                <div>''' + tfidf_content + '''</div>
                </div>
            </div>
            <div>
                <div class="col">
                <div><strong>CountVectorizer: </strong>''' + str(r['count_similarity'])[:6] + '''</div>
                <div>''' + count_content + '''</div>
                </div>
            </div>
        </div>
    </div>
    '''))<jupyter_output><empty_output><jupyter_text>Questão 2Elabore um problema de classificação binária de textos coerente com suabase.a) Determine o rótulo dos documentos (separando os documentos em classesbem definidas).b) Extraia as representações vetoriais com CountVectorizer e TF-IDF.c) Treine um classificador baseado em cada uma das duas representações vetoriais e Regressão Logística usando validação cruzada com 70% das amostras selecionadas para treino e 30% para teste. Exiba as matrizes de confusão,métricas de acurácia, precisão, recall e F1 score.d) Faça o mesmo para o classificador Naive-Bayes.e) Faça o mesmo para um outro classificador de sua preferência (pesquise nabiblioteca Scikit-learn).f) Compare os 6 resultados.  2.a) Determine o rótulo dos documentos (separando os documentos em classes bem definidas).<jupyter_code>lbl_to_ind = {
    ' comedy ': 0, ' drama ': 1, 
}
y = [lbl_to_ind[l] for l in dataset.genre.values[:4669]]<jupyter_output><empty_output><jupyter_text>2.b) Extraia as representações vetoriais com CountVectorizer e TF-IDF.<jupyter_code>tfidf_X_train, tfidf_X_test, tfidf_y_train, tfidf_y_test = train_test_split(tfidf_X, y, test_size=0.3)
count_X_train, count_X_test, count_y_train, count_y_test = train_test_split(count_X, y, test_size=0.3)<jupyter_output><empty_output><jupyter_text>2.c) Treine um classificador baseado em cada uma das duas representações vetoriais e Regressão Logística usando validação cruzada com 70% das amostras selecionadas para treino e 30% para teste. Exiba as matrizes de confusão, métricas de acurácia, precisão, recall e F1 score.<jupyter_code># Regressão Logística com tfidf
cl = LogisticRegression().fit(tfidf_X_train.toarray(), tfidf_y_train)
tfidf_y_pred = cl.predict(tfidf_X_test.toarray())
plot_confusion_matrix(cl, tfidf_X_test.toarray(), tfidf_y_test)
plt.show()
accuracy_lr_tfidf = accuracy_score(tfidf_y_test, tfidf_y_pred)
precision_lr_tfidf = precision_score(tfidf_y_test, tfidf_y_pred)
recall_lr_tfidf = recall_score(tfidf_y_test, tfidf_y_pred)
f1_lr_tfidf = f1_score(tfidf_y_test, tfidf_y_pred)

print("Regressão Logísitca e TF-IDF\nAcurácia: {}\nPrecisão: {}\nRecall: {}\nF1 Score: {}\n".format(accuracy_lr_tfidf, precision_lr_tfidf, recall_lr_tfidf, f1_lr_tfidf))
# Regressão Logística com countvectorizer
cl = LogisticRegression().fit(count_X_train.toarray(), count_y_train)
count_y_pred = cl.predict(count_X_test.toarray())
plot_confusion_matrix(cl, count_X_test.toarray(), count_y_test)
plt.show()
accuracy_lr_count = accuracy_score(count_y_test, count_y_pred)
precision_lr_count = precision_score(count_y_test, count_y_pred)
recall_lr_count = recall_score(count_y_test, count_y_pred)
f1_lr_count = f1_score(count_y_test, count_y_pred)

print("Regressão Logísitca e CountVectorizer\nAcurácia: {}\nPrecisão: {}\nRecall: {}\nF1 Score: {}\n".format(accuracy_lr_count, precision_lr_count, recall_lr_count, f1_lr_count))<jupyter_output>Regressão Logísitca e CountVectorizer
Acurácia: 0.7844396859386152
Precisão: 0.8004094165813715
Recall: 0.8796400449943758
F1 Score: 0.8381564844587354<jupyter_text>2.d) Faça o mesmo para o classificador Naive-Bayes.<jupyter_code># Naive Bayes com tfidf

cl_nb = MultinomialNB().fit(tfidf_X_train.toarray(), tfidf_y_train)
tfidf_y_pred_nb = cl_nb.predict(tfidf_X_test.toarray())
plot_confusion_matrix(cl_nb, tfidf_X_test.toarray(), tfidf_y_test)
plt.show()
accuracy_nb_tfidf = accuracy_score(tfidf_y_test, tfidf_y_pred_nb)
precision_nb_tfidf = precision_score(tfidf_y_test, tfidf_y_pred_nb)
recall_nb_tfidf = recall_score(tfidf_y_test, tfidf_y_pred_nb)
f1_nb_tfidf = f1_score(tfidf_y_test, tfidf_y_pred_nb)

print("Naive Bayes e TF-IDF\nAcurácia: {}\nPrecisão: {}\nRecall: {}\nF1 Score: {}\n".format(accuracy_nb_tfidf, precision_nb_tfidf, recall_nb_tfidf, f1_nb_tfidf))
# Naive Bayes com countvectorizer

cl_nb = MultinomialNB().fit(count_X_train.toarray(), count_y_train)
count_y_pred_nb = cl_nb.predict(count_X_test.toarray())
plot_confusion_matrix(cl_nb, count_X_test.toarray(), count_y_test)
plt.show()
accuracy_nb_count = accuracy_score(count_y_test, count_y_pred_nb)
precision_nb_count = precision_score(count_y_test, count_y_pred_nb)
recall_nb_count = recall_score(count_y_test, count_y_pred_nb)
f1_nb_count = f1_score(count_y_test, count_y_pred_nb)

print("Naive Bayes e CountVectorizer\nAcurácia: {}\nPrecisão: {}\nRecall: {}\nF1 Score: {}\n".format(accuracy_nb_count, precision_nb_count, recall_nb_count, f1_nb_count))<jupyter_output>Naive Bayes e CountVectorizer
Acurácia: 0.793718772305496
Precisão: 0.7824858757062146
Recall: 0.9347581552305961
F1 Score: 0.8518708354689901<jupyter_text>2.e) Faça o mesmo para um outro classificador de sua preferência (pesquise na biblioteca Scikit-learn).<jupyter_code># SVM com tfidf

cl_svm = LinearSVC().fit(tfidf_X_train.toarray(), tfidf_y_train)
tfidf_y_pred_svm = cl_svm.predict(tfidf_X_test.toarray())
plot_confusion_matrix(cl_svm, tfidf_X_test.toarray(), tfidf_y_test)
plt.show()
accuracy_svm_tfidf = accuracy_score(tfidf_y_test, tfidf_y_pred_svm)
precision_svm_tfidf = precision_score(tfidf_y_test, tfidf_y_pred_svm)
recall_svm_tfidf = recall_score(tfidf_y_test, tfidf_y_pred_svm)
f1_svm_tfidf = f1_score(tfidf_y_test, tfidf_y_pred_svm)

print("SVM e TF-IDF\nAcurácia: {}\nPrecisão: {}\nRecall: {}\nF1 Score: {}\n".format(accuracy_svm_tfidf, precision_svm_tfidf, recall_svm_tfidf, f1_svm_tfidf))
# SVM com countvectorizer

cl_svm = LinearSVC().fit(count_X_train.toarray(), count_y_train)
count_y_pred_svm = cl_svm.predict(count_X_test.toarray())
plot_confusion_matrix(cl_svm, count_X_test.toarray(), count_y_test)
plt.show()
accuracy_svm_count = accuracy_score(count_y_test, count_y_pred_svm)
precision_svm_count = precision_score(count_y_test, count_y_pred_svm)
recall_svm_count = recall_score(count_y_test, count_y_pred_svm)
f1_svm_count = f1_score(count_y_test, count_y_pred_svm)

print("SVM e CountVectorizer\nAcurácia: {}\nPrecisão: {}\nRecall: {}\nF1 Score: {}\n".format(accuracy_svm_count, precision_svm_count, recall_svm_count, f1_svm_count))<jupyter_output>SVM e CountVectorizer
Acurácia: 0.7630264097073519
Precisão: 0.7953340402969247
Recall: 0.843644544431946
F1 Score: 0.8187772925764192<jupyter_text>2.f) Compare os 6 resultados<jupyter_code>lr_count_results = [accuracy_lr_count, precision_lr_count, recall_lr_count, f1_lr_count]
lr_tfidf_results = [accuracy_lr_tfidf, precision_lr_tfidf, recall_lr_tfidf, f1_lr_tfidf]
nb_count_results = [accuracy_nb_count,precision_nb_count, recall_nb_count,f1_nb_count]
nb_tfidf_results = [accuracy_nb_tfidf, precision_nb_tfidf,recall_nb_tfidf, f1_nb_tfidf]
svm_count_results = [accuracy_svm_count,precision_svm_count, recall_svm_count,f1_svm_count]
svm_tfidf_results = [accuracy_svm_tfidf,precision_svm_tfidf, recall_svm_tfidf,f1_svm_tfidf]

data_results2 = {
    'Regression with Count': lr_count_results,
    'Regression with TF-IDF': lr_tfidf_results,
    'Naive Bayes with Count': nb_count_results,
    'Naive Bayes with TF-IDF': nb_tfidf_results,
    'SVM with Count': svm_count_results,
    'SVM with TF-IDF': svm_tfidf_results,
}

index_values2 = ['accuracy', 'precision', 'recall', 'f1']
pd.set_option('display.max_columns', None)

df_results2 = pd.DataFrame(data_results2, index=index_values2)

df_results2
df_results2.plot()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11633622808454858, 'fraction_numerical': 0.02654432246436179, 'mean_word_length': 6.227353463587922, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '360667', 'license': 'MIT', 'n_tokens_mistral': 4963, 'n_tokens_neox': 4700, 'n_words': 862, 'path': 'pln_lista3.ipynb', 'repo_name': 'audreyemmely/pln', 'size': '12255'}