<jupyter_start><jupyter_text>MMT ProjectMy aim is to find associations between a person's score in Big-5 personality assessment and the emotions depicted by lyrics of their favourite songs.For this purpose, I have collected data from more than 50 participants. This data consists of* demographic data* big-5 personality assessment data* name and artist of 3 to 10 favourite songs Process1. Gather lyrics of songs2. Analyze the emotions depected in lyrics3. Analyze associations between the Big-5 personality scores and emotions depected in lyrics1. I will automate the lyrics gathering process by using `lyrics-extractor` package, which searches the song on Google and scrapes the lyrics shown on the results page. I also tried other packages for searching lyrics, but the problem with them is that they are not tolerant to spelling mistakes and incomplete words. The data was collected by circulating a Google form, there are high chances of misspelled and incomplete song names.  After extracting the lyrics, we need to remove any labelling brackets like *\[chorus\]* or *\[bridge\]* as they may contain meaningful words which may affect the analysis of emotions. For this purpose I will use regular expressions.2. To analyze the emotions depected in lyrics, I will use Tone Analyzer service by IBM Watson. This service takes in the lyrics and return the scorings of lyrics in 13 different components which consist of Big-5 components O,C,E,A,N under social category and anger, disgust, fear, joy, and sadness under emotional category and analytical, confident, and tentative under language category.3. Once we have all the scores for each song, we need to calculate cummulative scores for each participant. To calculate cummulative scores, we will do average of each emotion type accross all the favourite songs. This will give us a score vector against each participant.  Where ever the score is less than 0.5, we will make it zero. It's done because according to the IBM watson documents, it's difficult to percieve the emotions which score less than 0.5.<jupyter_code>import pandas as pd
"""
Load dataset which contains `id`, `songnames`, `songartist`. Always load latest 
version of dataset which may contain lyrics and scores fetched previously. This 
will save time and any quota imposed on third-party services which we will use.
"""
songs = pd.read_csv("/content/drive/My Drive/acads/MMT/MMT project/collected data/songs_dataset.csv")<jupyter_output><empty_output><jupyter_text>Fetch and process the lyrics<jupyter_code>'''
Load external package to fetch lyrics for each song in dataset.

Extracting lyrics of songs by using `lyrics_extractor` API
(https://pypi.org/project/lyrics-extractor/#description)

We will use `lyrics_extractor` API for this purpose. It is tolerant to spelling 
mistakes and incomplete words because it does a google search for the given data,
google autocorrects and autocompletes misspelled or incomplete words. Which is
very useful for us because it takes away the tiring task of manually checking and 
correcting spelling of each entry in dataset. 
'''
!pip install lyrics-extractor

# Example
# song_title, song_lyrics = extract_lyrics.get_lyrics("Its you	<NAME>")
'''
Extract and store lyrics of songs.

Go through each row in songs dataset and add lyric to each row. 
To save time and Google search requests quota, skip rows which already have lyrics.
'''
from lyrics_extractor import Song_Lyrics
extract_lyrics = Song_Lyrics("","") # insert the Google api keys here

for i in songs.index:
  if len(str(songs['lyrics'][i])) < 15: # is true if the corresponding lyrics coloumn is empty in dataset
    try:  # fetch lyrics of song from the Internet
      title, lyrics = extract_lyrics.get_lyrics(str(songs['songnames'][i])+" "+str(songs['songartists'][i]))
      if lyrics == '':  # if `extract_lyrics` is unable to fetch/find lyrics of song, print song details
        print(i, str(songs['songnames'][i])+" "+str(songs['songartists'][i]), " lyrics not found!!!")
      songs['lyrics'][i] = lyrics
    except: # print song details with keyword `ERROR!!` if try fails
      print(i, songs['email'][i], songs['songnames'][i], songs['songartists'][i], "ERROR!!")

"""
Just to play safe, save the current dataframe to CSV. This way we are free to
play with current dataframe without any worry of destroying the df by mistake. 
"""

songs.to_csv('songs_dataset.csv')
'''
Now we have lyrics in place, let's do some necessary preprocessing.

We noticed some labels here and there in the fetched lyrics.
Let's remove labelling brackets with labels from lyrics because labels may 
contain words which will unncessarily influence the sentiment of the lyrics.

examples of labels: [chorus], [bridge]
'''
import re
regex1 = '\[[a-zA-Z 0-9]*\]'  # select words like [chorus], [bridge]

for i in songs.index:
  try:
    lyrics = str(songs['lyrics'][i])
    lyrics = re.sub(regex1, ' ', lyrics)
    songs['lyrics'][i] = lyrics
  except:
    print("something went wrong!!!")
songs.head()<jupyter_output><empty_output><jupyter_text>Analyze tone of lyricsWe will use **IBM Watson Tone Analyzer** tool for this purpose.<jupyter_code>"""
IBM Watson Tone Analyzer API is available as a package in pip.
In order to use it, you need to signup to Watson, create a service in 
ToneAnalyzer and get the credentials.
"""

!pip install --upgrade "ibm-watson>=4.3.0"
from ibm_watson import ToneAnalyzerV3
from ibm_cloud_sdk_core.authenticators import IAMAuthenticator
import json
authenticator = IAMAuthenticator('') # insert Watson API key here
'''
The service with version '2017-09-21' can return results for the following tone IDs:
anger, fear, joy, and sadness (emotional tones); analytical,
confident, and tentative (language tones). The service returns
results only for tones whose scores meet a minimum threshold of 0.5.

The service with version '2016-05-19' can return results for the following tone IDs of the different categories: 
for the emotion category: anger, disgust, fear, joy, and sadness; 
for the language category: analytical, confident, and tentative; 
for the social category: openness_big5, conscientiousness_big5,
extraversion_big5, agreeableness_big5, and emotional_range_big5. 
The service returns scores for all tones of a category, regardless of their values.
'''
version='2017-09-21'
tone_analyzer = ToneAnalyzerV3(
    version=version,
    authenticator=authenticator
)

tone_analyzer.set_service_url('https://api.eu-gb.tone-analyzer.watson.cloud.ibm.com')
'''
Function to get sentiment scores of lyrics. Based on version of the tone analyzer, 
it will return the factors.
parameter: `lyrics`
return: JSON object returned by watson API.
'''
def analyzeLyrics(lyrics):
  try:
    tone_analysis = tone_analyzer.tone(
        {'text': lyrics},
        content_type='application/json',
        sentences=False # sentencewise analysis not required
    ).get_result()
  except:
    print(i, songs['id'][i], songs['songnames'][i], 'unable to fetch scores!!!')
  return json.dumps(tone_analysis, indent=2)
# example input
# Let's check if our function is working fine or not.
i = 436
results = analyzeLyrics(songs['lyrics'][i])
print(songs['id'][i], songs["songnames"][i])
r_dict = json.loads(results)

""" use this snippet when you with version '2017-09-21' of Tone Analyzer """
for tones in r_dict['document_tone']['tones']:
  print(tones['tone_id'], ':', tones['score'])

# """ use this snippet with version '2016-05-19' of Tone Analyzer """
# for category in r_dict['document_tone']['tone_categories']:
#   for scores in category['tones']:
#     print(scores['tone_id'], ':', scores['score'])
# add new columns to DF 'songs'
songs_with_scores = pd.concat(\
                              [songs, \
                               pd.DataFrame(\
                                columns = [ 'scoringDone','anger','fear',\
                                           'joy', 'sadness', 'analytical', 'confident',\
                                           'tentative'])])
# songs_with_scores = songs
songs_with_scores.head()
# iterate through each row in DF songs and calculate the scores for lyrics.
# add the scores back to DF in corresponding song row

for i in songs_with_scores.index:
  results_dict=[]
  try:
    lyrics = songs_with_scores['lyrics'][i]
    if(len(lyrics)>15 and pd.isna(songs_with_scores['scoringDone'][i])):
      results = analyzeLyrics(lyrics)
      results_dict = json.loads(results)
      for category in results_dict['document_tone']['tone_categories']:
        for scores in category['tones']:
          songs_with_scores.at[i, scores['tone_id']] = scores['score']
      songs_with_scores.at[i, 'scoringDone'] = 1
  except:
    print(i, songs['id'][i], songs['songnames'][i], "unable to put scores in df!!!")
songs_with_scores.head()<jupyter_output><empty_output><jupyter_text>Prepare the data for correlation analysis<jupyter_code># Drop columns which are not required in Data analysis
id_lyrics_scores = songs_with_scores.drop(columns=["Unnamed: 0","scoringDone","songnames", \
                                                    "songartists", "lyrics"],\
                                           errors='ignore')
# # group lyrics scores into positive emotions and negative emotions by adding columns

# using apply function to create a new column 
# id_lyrics_scores['negativity'] = id_lyrics_scores.apply(lambda row: (row.anger+row.disgust+row.fear+row.sadness)/4, axis = 1)
id_lyrics_scores.head()
# watson documentation states that any score below 0.5 is unpercievable by humans.
# such scores should be floored to 0.
for i in id_lyrics_scores.columns:
  if(i not in ["id"]):
    id_lyrics_scores.loc[id_lyrics_scores[i] < 0.5, i] = 0
id_lyrics_scores.head()
"""
Here we find mean lyrics scores for each participant.
"""

# Find mean and sum of lyrics scores over each participant
lyrics_scores_mean = id_lyrics_scores.groupby(['id']).mean()
lyrics_scores_mean.head()
# let's get big-5 scores and demographic data here. Later we use these scores and lyrics scores to analyze associations between them
big5_scores = pd.read_csv("/content/drive/My Drive/acads/MMT/MMT project/collected data/big5_continuous_scores.csv")
demographic_data = pd.read_csv("/content/drive/My Drive/acads/MMT/MMT project/collected data/demographics.csv")

# and merge `lyrics_scores_mean` and above DFs by 'id'
df1 = pd.merge(left=demographic_data, right=big5_scores, left_on='id', right_on='id')
big5_continuous_with_lyrics_mean = pd.merge(left=df1, right=lyrics_scores_mean, left_on='id', right_on='id')

big5_continuous_with_lyrics_mean.head()
"""
Let's save the curent DF to save time in future.
"""
big5_continuous_with_lyrics_mean.to_csv("big5_continuous_with_lyrics_mean.csv")<jupyter_output><empty_output><jupyter_text><jupyter_code>"""
now accessing the dataset which we saved in previous block :P :D
"""
big5_continuous_with_lyrics_mean = pd.read_csv("/content/drive/My Drive/acads/MMT/MMT project/collected data/big5_continuous_with_lyrics_mean.csv")
"""
Now let's prepare the dataset for Spearman rho analysis. 
We need to label the categorical with numbers in order to use the algorithm. 
"""
object_data = big5_continuous_with_lyrics_mean.select_dtypes(include="object")
numeric_data = big5_continuous_with_lyrics_mean.select_dtypes(exclude="object")

# Import label encoder 
from sklearn import preprocessing 
  
# label_encoder object knows how to understand word labels. 
label_encoder = preprocessing.LabelEncoder() 
object_data = object_data.apply(label_encoder.fit_transform)
data  = pd.merge(left=object_data, right=numeric_data, left_on=numeric_data.index, right_on=object_data.index)
data.drop(columns=["id"])<jupyter_output><empty_output><jupyter_text>Correlation Analysis<jupyter_code>"""
Now we have our desired df with us, let's do some analysis over it.
In our data, some variables are numeric(like big-5 score, emotional tone scores),
some are ordinal(like age, musical preference) and some are categorical(like sex, musical preferences).
For this reason we will use "Spearman's rho correlation" which is a non-parametric analysis.
"""

"""
H0: Big-5 personality type of a person is not assiciated with the emotion depicted
    in lyrics of their favourite songs.
H1: Big-5 personality type of a person is associated with the emotion depicted in 
    in lyrics of their favourite songs.
"""

from scipy.stats import pointbiserialr, spearmanr, chisquare, mannwhitneyu, kendalltau, pearsonr

current_method = [pointbiserialr, spearmanr, chisquare, mannwhitneyu, pearsonr]

big5_traits = ['agreeableness', 'extraversion',	'conscientiousness',\
               'neuroticism',	'openness']
demographics = ['age group','sex', 'instumental or lyrical music',	'listen to english lyrics']
emotions = ['anger',	'fear',	'joy',	'sadness',	'analytical',\
                  'confident',	'tentative']

# Find correlations and p-value for aspects of data. For example: 
# between demographics and big5_traits
for i in demographics:
  for j in big5_traits:
    if(i is not j):
      i_score = data[i]
      j_score = data[j]
      pbc = current_method[1](i_score, j_score)
      if(pbc[1]<=0.06 and (pbc[0]>=0.3 or pbc[0]<=-0.3)):
        print(i, j, pbc)


# Plot correlation matrix
matrix = data.corr(method="spearman")
matrix = matrix.drop(index=['id', 'key_0'],columns=['id', 'key_0'])
matrix<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09734777736271946, 'fraction_numerical': 0.009712364587224505, 'mean_word_length': 4.565904365904366, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 29, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '439989', 'license': 'MIT', 'n_tokens_mistral': 4109, 'n_tokens_neox': 3884, 'n_words': 1455, 'path': 'MMT_Project_with_watson.ipynb', 'repo_name': 'raghavmittal101/music_mind_tech_project', 'size': '13388'}