<jupyter_start><jupyter_text>Week 4:  Text Classification Project NLP - Naive Bayes (Lyrics from Songs) Table of Contents1. Objective of the Project2. Data Collection    - Download HTML File    - Getting list of urls    -Extracing lyrics from songs3. Preprocessing of Data (Bag of Words, Tokenization and Vectorization)    - Analyzing Data    - Build a count vectorizer    - Sparse Matrix    - TF-Idf Vectors4. Modelling with Naive Bayes Classifier    - Compute IDF    - Compute Score    - Train Test Split    - Calculating accuracy and generating classification report from test data    - Predicting Another Song5. Modelling with Other Classifiers    - Logistic Regression    - Decision Tree    - Random Forest 6. Word Cloud  1. Objective of the ProjectThe goal of this project is to test whether Naive Bayer classifier (and also other classifier) can successfully predict the Artist, based on the lyrics of their songs. Two artists are deliberately choosen to separate the classification problem according to the music they made. ![sheeran](sheeran2.jpg)source: https://wallpapersafari.com/ed-sheeran-lyrics-wallpaper/  2. Data Collection Importing Libraries<jupyter_code>import os
from os import path
import re
import spacy
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from PIL import Image

from bs4 import BeautifulSoup

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import  MultinomialNB
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn import metrics
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier

from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator<jupyter_output><empty_output><jupyter_text>Downloading the HTML File<jupyter_code>base_url='https://www.azlyrics.com'
path = 'songs/'
lyrics_path = path + 'lyrics/'
def load_file(filename):
    with open(filename,'r') as file:
        return file.read()
def create_link_list(filename,link_list_filename):
    lyrics = load_file(path + filename)
    soup = BeautifulSoup(lyrics, 'html.parser')
    rows = soup.body.find_all(class_='listalbum-item')
    links = []
    for div in rows:
        if "listalbum-item" in div.get("class", []):
            for item in div.find_all('a', href=True):
                links.append(base_url + item['href'][2:])
    with open(path + link_list_filename,"w") as file:
        for link in links:
            file.write(link+"\n")
create_link_list('azsheeran.html','sheeranlinks.txt')
create_link_list('azmetallica.html','metallicalinks.txt')<jupyter_output><empty_output><jupyter_text>3. Preprocessing of Data<jupyter_code>def load_song(filename):
    song_page=""
    with open(lyrics_path + filename, "r") as file:
        song_page = file.read().encode('utf8')
    song_soup = BeautifulSoup(song_page, 'html.parser')
    label = song_soup.find(class_="ringtone")
    return label.find_next_sibling('div').text.strip().replace('\n',' ').replace('â€™',"'")
corpus = []
titles = []

for filename in os.listdir(lyrics_path):
    corpus.append(load_song(filename))
    titles.append(filename.split('.')[1])
    
#corpus
titles
model = spacy.load('en_core_web_sm')
model
def clean_lyrics(lyrics):
    word_list = re.findall('\w+',lyrics)
    worded_lyrics = ' '.join(word_list)
    tokenised_lyrics = model(worded_lyrics)
    clean_lyrics = []
    for token in tokenised_lyrics:
        if not token.is_stop:
            clean_lyrics.append(token.lemma_)
    super_clean_lyrics = []
    for word in clean_lyrics:
        if len(word) > 2:
            super_clean_lyrics.append(word)
    return ' '.join(super_clean_lyrics)
clean_corpus = []
for song in corpus:
    clean_corpus.append(clean_lyrics(song))<jupyter_output><empty_output><jupyter_text>Build a count vectorizer<jupyter_code>cv = CountVectorizer()
cv.fit(clean_corpus)
vec_corpus = cv.transform(clean_corpus)
vec_corpus
vec_corpus.todense()<jupyter_output><empty_output><jupyter_text>Sparse Matrix<jupyter_code>df = pd.DataFrame(vec_corpus.todense(), columns=cv.get_feature_names(),index=titles)
df.info(verbose=True)<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 20 entries, amievil to typicalaverage
Data columns (total 528 columns):
 #   Column         Dtype
---  ------         -----
 0   abuse          int64
 1   acting         int64
 2   action         int64
 3   addicted       int64
 4   adrenaline     int64
 5   affect         int64
 6   alive          int64
 7   alright        int64
 8   analyze        int64
 9   angel          int64
 10  aren           int64
 11  armor          int64
 12  army           int64
 13  ass            int64
 14  attack         int64
 15  attention      int64
 16  avenge         int64
 17  average        int64
 18  await          int64
 19  away           int64
 20  baby           int64
 21  bad            int64
 22  bag            int64
 23  bang           int64
 24  bear           int64
 25  befall         int64
 26  bell           int64
 27  belong         int64
 28  bitch          int64
 29  blade          int64
 30  blame          int64
 31  bleed          int64[...]<jupyter_text>Tf-Idf Vectors<jupyter_code>df['feel']
tf = TfidfTransformer()
tf.fit(vec_corpus)
tf_corpus = tf.transform(vec_corpus)
df2 = pd.DataFrame(tf_corpus.todense(),  columns=cv.get_feature_names(), index=titles)
df2<jupyter_output><empty_output><jupyter_text>Effect of tfidf<jupyter_code>df2['world']
df.sum(axis=1)<jupyter_output><empty_output><jupyter_text>4. Modelling with Naive Bayes Classifier<jupyter_code>cvect = CountVectorizer(stop_words='english')
word_count = cvect.fit_transform(corpus)
word_count.shape<jupyter_output><empty_output><jupyter_text>Compute IDF<jupyter_code>tfidf_tran = TfidfTransformer()
tfidf_tran.fit(word_count)
idf_df = pd.DataFrame(tfidf_tran.idf_, index=cvect.get_feature_names(), columns=["idf_weights"])
idf_df.sort_values(by=["idf_weights"])<jupyter_output><empty_output><jupyter_text>Compute TFIDF score (Term Frequency Inverse Document Frequency)<jupyter_code>tfidf_mat=tfidf_tran.transform(word_count)
tfidf_mat
tfidf_mat_dense = tfidf_mat.todense()
tfidf_mat_dense
tfidf_df = pd.DataFrame(tfidf_mat_dense, columns=cvect.get_feature_names(),index=titles)
tfidf_df<jupyter_output><empty_output><jupyter_text>Train Test Split<jupyter_code>artist = ['metallica'] * 10 + ['sheeran'] * 10
X = tfidf_mat_dense
y = np.array(artist)
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.5,random_state=42)
X_train.shape, X_test.shape, y_train.shape, y_test.shape
mnb_model = MultinomialNB(alpha=0.01)
mnb_model.fit(X_train, y_train)
y_pred = mnb_model.predict(X_test)
y_pred<jupyter_output><empty_output><jupyter_text>Calculating accuracy and generating classification report from test data<jupyter_code>score = accuracy_score(y_test, y_pred)
print("accuracy:   %0.3f" % score)

print(metrics.classification_report(y_test, y_pred, target_names=['Metallica','Sheeran']))

print("confusion matrix:")
con_mat = metrics.confusion_matrix(y_test, y_pred)
print(con_mat)
fig = plt.figure(figsize=(6, 6))
ax = plt.subplot()
sns.heatmap(con_mat, annot=True,  cbar=True, \
                cmap = 'Greens', vmax = 1,\
               xticklabels=['Metallica', 'Sheeran'],\
                yticklabels=['Metallica', 'Sheeran'],\
                linecolor='white',\
               linewidth=2, square = True, fmt = '.0f', robust = True); 
plt.xlabel('Predicted')
plt.ylabel('Actual')<jupyter_output><empty_output><jupyter_text>Predicting another song<jupyter_code>new_song = "When your legs don't work like they used to before And I can't sweep you off of your feet Will your mouth still remember the taste of my love Will your eyes still smile from your cheeks "
new_song_vect = cvect.transform([new_song]).todense()
new_song_tfidf = tfidf_tran.transform(new_song_vect).todense()
mnb_model.predict(new_song_tfidf)
new_song2 = "Make his fight on the hill in the early day Constant chill deep inside Shouting gun, on they run through the endless grey On they fight, for the right, yes, but who's to say? For a hill men would kill, why? They do not know Stiffened wounds test their pride Men of five, still alive through the raging glow Gone insane from the pain that they surely know"
new_song_vect = cvect.transform([new_song2]).todense()
new_song_tfidf = tfidf_tran.transform(new_song_vect).todense()
mnb_model.predict(new_song_tfidf)<jupyter_output><empty_output><jupyter_text>5. Using Other Classification Models<jupyter_code>models = [LogisticRegression(), DecisionTreeClassifier(), RandomForestClassifier()]

model_names=['Logistic Regression', 'Decision Tree Classifier', 'Random Forest Classifier']


accuracy=[]
d={}

for model in range (len(models)):
    clf=models[model]
    clf.fit(X_train,y_train)
    y_pred = clf.predict(X_test)
    accuracy.append(clf.score(X_test,y_test))
    
d={'Model Names':model_names, 'Accuracy' :accuracy}
data = pd.DataFrame(d).sort_values(by='Accuracy', ascending=False)
data<jupyter_output><empty_output><jupyter_text>6. Word Cloud<jupyter_code>clean_sheeran_corpus = clean_corpus.copy()[10:20]
corpus1 = clean_sheeran_corpus 
text = " ".join(corpus1)
stopwords = set(STOPWORDS)


wc = WordCloud(background_color="white", 
    width = 1000, 
    height = 1000, 
    max_words=2000, 
    stopwords=stopwords, 
    contour_width=3, 
    contour_color='steelblue')

wc.generate(text)

plt.figure(figsize = (8,8))
plt.imshow(wc, interpolation='bilinear')
plt.axis("off")
plt.figure()

#text = " ".join(corpus1)
ed_mask = np.array(Image.open("ed1.png"))
stopwords = set(STOPWORDS)
font = 'Mom.ttf'

wc = WordCloud(font_path = font,
    background_color='white', 
    max_words=2000, 
    mask=ed_mask,
    stopwords=stopwords,
    collocations=False,
    contour_width=3, 
    contour_color='green')

wc.generate(text)

plt.imshow(ed_mask, cmap=plt.cm.gray, interpolation='bilinear')
plt.axis("off")
plt.show()

plt.figure(figsize = (15,15))
plt.imshow(wc, interpolation='bilinear')
plt.axis("off")
plt.figure()

clean_metallica_corpus = clean_corpus.copy()[0:9]
corpus2 = clean_metallica_corpus
text2 = " ".join(corpus2)
metal_mask = np.array(Image.open("metallica3.jpg"))
stopwords = set(STOPWORDS)

wc2 = WordCloud(background_color='black', 
    max_words=2000, 
    mask=metal_mask,
    stopwords=stopwords,
    collocations=False,
    contour_width=3, 
    contour_color='white')

wc2.generate(text)

plt.imshow(metal_mask, cmap=plt.cm.gray, interpolation='bilinear')
plt.axis("off")
plt.show()
plt.figure(figsize = (8,8))
plt.imshow(wc2, interpolation='bilinear')
plt.axis("off")
plt.figure()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10709665496211421, 'fraction_numerical': 0.019774533357974498, 'mean_word_length': 4.403394907638543, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 62, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '641346', 'license': 'MIT', 'n_tokens_mistral': 3951, 'n_tokens_neox': 3722, 'n_words': 821, 'path': 'week4/TextClassificationProject_Kristine.ipynb', 'repo_name': 'kristine-wiesner/spiced_projects', 'size': '10822'}