<jupyter_start><jupyter_text>Downloading the Dataset<jupyter_code>!git clone https://github.com/kaazima/Dataset.git<jupyter_output>Cloning into 'Dataset'...
remote: Enumerating objects: 10942, done.[K
remote: Counting objects: 100% (2/2), done.[K
remote: Compressing objects: 100% (2/2), done.[K
remote: Total 10942 (delta 0), reused 0 (delta 0), pack-reused 10940[K
Receiving objects: 100% (10942/10942), 1.84 GiB | 36.28 MiB/s, done.
Resolving deltas: 100% (510/510), done.
Checking out files: 100% (10001/10001), done.<jupyter_text>Rename the directories & their paths<jupyter_code>import os, sys

dir_tree = '/content/Dataset/MillionSongSubset'

for dir_path, dir_names, file_names in os.walk(dir_tree):
    for file_name in file_names:
        try:
            os.rename(os.path.join(dir_path, file_name), os.path.join(dir_tree, file_name))
        except OSError:
            print ("Could not move %s " % os.join(dir_path, file_name))<jupyter_output><empty_output><jupyter_text>Compiling a Title-Artist table<jupyter_code>import pandas as pd

def make_artist_table(base):

# Get file names

    files = [os.path.join(base,fn) for fn in os.listdir(base) if fn.endswith('.h5')]
    data = {'file':[], 'artist':[], 'title':[]}

    # Add artist and title data to dictionary
    for f in files:
        store = pd.HDFStore(f)
        title = store.root.metadata.songs.cols.title[0]
        artist = store.root.metadata.songs.cols.artist_name[0]
        data['file'].append(os.path.basename(f))
        data['title'].append(title.decode("utf-8"))
        data['artist'].append(artist.decode("utf-8"))
        store.close()
    
    # Convert dictionary to pandas DataFrame
    df = pd.DataFrame.from_dict(data, orient='columns')
    df = df[['file', 'artist', 'title']]
    return df
base = '/content/Dataset/MillionSongSubset'
df = make_artist_table(base)
df.tail()<jupyter_output><empty_output><jupyter_text>Downloading Lyrics<jupyter_code>df['lyrics'] = pd.Series('', index=df.index)
df.tail()
import urllib
import bs4
from bs4 import BeautifulSoup, NavigableString
          
def songlyrics(artist, title):
    artist = urllib.parse.quote(artist.lower().replace(' ','_'))
    title = urllib.parse.quote(title.lower().replace(' ','_'))
    try:
        lyrics = urllib.request.urlopen('https://www.lyricsmania.com/%s_lyrics_%s.html' % (title,artist))
    except:
        return ''
    text = lyrics.read()
    soup = bs4.BeautifulSoup(text)
    if not soup.find(class_ = 'lyrics-body'):
        return ''
    lyrics = [x.strip() if isinstance(x, NavigableString) else x.text.strip() for x in soup.find(class_ = 'lyrics-body').contents[2:]]
    if not lyrics:
        return ''
    string='\n'.join(lyrics)
    return string
lyr = songlyrics('Mastodon','Deep Sea Creature')
print(lyr)<jupyter_output>Knowing right

Learning wrong

What you're feeling is pressure

Pulsate new blood

I've seen things you've heard never before

Bones aged in dust

Buy your bite take the body

Calm

Shutting down

You built me

I knew it

I'll never lie liar

You fed me

I chewed it

I'll never lie liar

I follow you covering me

Spirits in pieces

Crumbled and burnt<jupyter_text>Adding lyrics to the Dataframe<jupyter_code>!pip install pyprind
import pyprind

pbar = pyprind.ProgBar(df.shape[0])
for row_id in df.index:
    lyr = songlyrics(df.loc[row_id]['artist'],df.loc[row_id]['title'])
    df.loc[row_id,'lyrics'] = lyr
    pbar.update()
print('downloaded Lyrics for %s songs' %sum(df.lyrics!=''))
df.head()<jupyter_output>downloaded Lyrics for 91 songs<jupyter_text>Remove rows where lyrics are not available<jupyter_code>df = df[df.lyrics!='']
print("No of songs: %s" %sum(df.lyrics!=''))
df.head()<jupyter_output>No of songs: 91<jupyter_text>Remove the songs that are not in English<jupyter_code>import nltk
nltk.download('words')

def eng_ratio(text):
    # Returns the ratio of non-English to English words from a text
    english_vocab = set(w.lower() for w in nltk.corpus.words.words()) 
    text_vocab = set(w.lower() for w in text.split() if w.lower().isalpha()) 
    if len(text_vocab)==0:
        # Can't find the value because of the presence of division by 0. So, returning a value 1
        return 1
    unusual = text_vocab.difference(english_vocab)
    diff = len(unusual)/len(text_vocab)
    return diff
text1 = 'This is a test'
print(eng_ratio(text1))
text2 = 'Ye jaanch karne ke liye hai'
print(eng_ratio(text2))
text3 = 'This is a test hai'
print(eng_ratio(text3))
text4 = '   \n '
print(eng_ratio(text4))<jupyter_output>0.0
0.8333333333333334
0.2
1<jupyter_text>text1 is 0% non-English ( It is Englishtext2 is 83.33% non-English (>50%) => It is Non-Englishtext3 is 20% non-English ( It is Englishtext4 is empty. So it is assumed to be 100% non-English (>50%) => It is Non-English<jupyter_code>pbar = pyprind.ProgBar(df.shape[0])

before = df.shape[0]
for row_id in df.index:
    text = df.loc[row_id]['lyrics']
    diff = eng_ratio(text)
    if diff >= 0.5:
        df = df[df.index != row_id]
    pbar.update()
after = df.shape[0]
rem = before - after
print('%s have been removed.' %rem)
print('%s songs remain in the dataset.' %after)
df.head()
df.to_csv('/content/df_lyr_backup.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13671949589459614, 'fraction_numerical': 0.02692381134237159, 'mean_word_length': 4.105263157894737, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 33, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '208931', 'license': 'MIT', 'n_tokens_mistral': 2034, 'n_tokens_neox': 1917, 'n_words': 477, 'path': 'ML_models/Mood_Based_Recommender.ipynb', 'repo_name': 'Nuthi-Sriram/SongRecommendationSystem', 'size': '5237'}