<jupyter_start><jupyter_text>Baysian Classification and Naive Bayes for genre classification using lyrics In this notebook we look at the problem of classifying songs to three genres (rap, rock and country) based on a simple binary bag of words representation. First we load the data and then we take a look at it. Using our implementation of discrete random variables we generate new random songs. Finally we show how classification can be performed using Bayes Rule. The data comes for the lyrics of songs from the Million Song DataSet and was created for an assignment in my course on MIR. The data layout and the way the classifier is implemented is not general and not optimal but done for pedagogical purposes.<jupyter_code>%matplotlib inline 
import matplotlib.pyplot as plt
import pickle 
import numpy as np


# load some lyrics bag of words data, binarize, separate matrix rows by genre 
data = np.load('data.npz')
a = data['arr_0']
a[a > 0] = 1
labels = np.load('labels.npz')
labels = labels['arr_0']
dictionary = pickle.load(open('dictionary.pck','rb'), encoding='latin1')
word_indices = [  41, 1465,  169,  217, 1036,  188,  260,  454,  173,  728,  163,
        151,  107,  142,   90,  141,  161,  131,   86,   73,  165,  133,
         84,  244,  153,  126,  137,  119,   80,  224]
words = [dictionary[r] for r in word_indices]

# binary row vectors separate by genre 
ra_rows = a[0:1000,:]
ro_rows = a[1000:2000,:]
co_rows = a[2000:3000,:] 
print(ra_rows.shape, ro_rows.shape, co_rows.shape)


plt.imshow(a, aspect='auto', cmap='gray')<jupyter_output>(1000, 30) (1000, 30) (1000, 30)<jupyter_text>Let's calculate the word probability vector for each genre and then look at the most probable words for each genre in our data as well as how particular songs are represented as bag of words<jupyter_code># calculate word counts for each genre 
word_probs_ra = (ra_rows.sum(axis=0).astype(float) + 1.0) / 1001.
word_probs_ro = (ro_rows.sum(axis=0).astype(float) + 1.0) / 1001.
word_probs_co = (co_rows.sum(axis=0).astype(float) + 1.0) / 1001.

word_probs_ro
words
#let's look at the bag of words for three particular songs 
track_id = 500
print("RAP 342:",[words[i] for i,r in enumerate(ra_rows[track_id]) if r==1])
print("ROCK 342:",[words[i] for i,r in enumerate(ro_rows[track_id]) if r==1])
print("COUNTRY 342:",[words[i] for i,r in enumerate(co_rows[track_id]) if r==1])
# let's look at the k most probable words for each genre 
k = 10 
[[words[x] for x in np.argpartition(word_probs_ra, -k)[-k:]],
 [words[x] for x in np.argpartition(word_probs_ro, -k)[-k:]],
 [words[x] for x in np.argpartition(word_probs_co, -k)[-k:]]]

#word_probs_ra
 
funny_list = ['a','b','c']
print(funny_list)

for (i,w) in enumerate(funny_list): 
    print(i,w)<jupyter_output>['a', 'b', 'c']
0 a
1 b
2 c<jupyter_text>Now let's generate some random songs represented as bag of words using the calculated word probabilities.<jupyter_code>print('Random rap', [w for (i,w) in enumerate(words) if np.greater(word_probs_ra, np.random.rand(30))[i]])
print('Random rock', [w for (i,w) in enumerate(words) if np.greater(word_probs_ro, np.random.rand(30))[i]])
print('Random country', [w for (i,w) in enumerate(words) if np.greater(word_probs_co, np.random.rand(30))[i]])<jupyter_output>Random rap ['shit', 'yo', 'again', 'our', 'good', 'littl', 'well']
Random rock ['ya', 'our', 'away']
Random country ['wait', 'our']<jupyter_text>Now let's look at classifying songs using a naive Bayes multinomial classifier. We simply calculate the likelihood for each genre independently by taking the products of the genre dependent word probabilities. The genere with the highest likelihood is selected as the predicted class.<jupyter_code># calcuate likelihood separately for each word 
# using naive bayes assumption and multiply 
# typically a sum of log-likelihoods is used 
# rather than a multiplication. 
def likelihood(test_song, word_probs_for_genre): 
    probability_product = 1.0 
    for (i,w) in enumerate(test_song): 
        if (w==1): 
            probability = word_probs_for_genre[i]
        else: 
            probability = 1.0 - word_probs_for_genre[i]
        probability_product *= probability 
    return probability_product 

def predict(test_song): 
    scores = [likelihood(test_song, word_probs_ra), 
             likelihood(test_song, word_probs_ro),
             likelihood(test_song, word_probs_co)]
    labels = ['rap', 'rock', 'country']
    return labels[np.argmax(scores)]

def predict_set(test_set, ground_truth_label): 
    score = 0 
    for r in test_set: 
        if predict(r) == ground_truth_label: 
            score += 1
    # convert to percentage 
    return score / 10.0 

# predict a random country track 
track_id = np.random.randint(1000)
print("Random track id", track_id)
test_song = co_rows[track_id]
print(predict(test_song))
# Let's evaluate how well our classifier does on the training set 
# A more proper evaluation would utilize cross-validation 

print("Rap accuracy% = ", predict_set(ra_rows, 'rap'))
print("Rock accuracy% = ", predict_set(ro_rows, 'rock'))
print("Country accuracy% = ", predict_set(co_rows, 'country'))
            
test_song = co_rows[track_id]

print(test_song)
scores = [likelihood(test_song, word_probs_ra), 
             likelihood(test_song, word_probs_ro),
             likelihood(test_song, word_probs_co)]
print(scores)<jupyter_output>[4.4908121666597401e-09, 2.8457883260190305e-08, 1.2226159956175496e-07]
{'fraction_non_alphanumeric': 0.11470911086717893, 'fraction_numerical': 0.045371386754482254, 'mean_word_length': 4.494472361809045, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '256087', 'license': 'CC0-1.0', 'n_tokens_mistral': 2045, 'n_tokens_neox': 1814, 'n_words': 594, 'path': 'data_mining_naive_bayes.ipynb', 'repo_name': 'gtzan/mir_book', 'size': '5466'}