<jupyter_start><jupyter_text>Main notebook: computing correlations between offensiveness and popularityIn this notebook we are trying to answer our previous questions: is there a correlation between (positive or negative) between the popularity and the offensiveness of a song ? Let's start by importing our libraries:<jupyter_code>import pandas as pd
import numpy as np
import sqlite3
import matplotlib.pyplot as plt
%matplotlib inline

# This variable should be set to false, unless you are running this notebook without the
# generated datasets
first_time = False<jupyter_output><empty_output><jupyter_text>Remember, we have computed the song word count at three levels of "swear word categories":<jupyter_code>offensiveness_rating = pd.read_pickle("../pickles/offensiveness_rating_structured")
offensiveness_rating.head()<jupyter_output><empty_output><jupyter_text>Now, the problem is that we have tuples as index but we would like to have all these categories in different columns. So we do some processing to take care of that:<jupyter_code>def process_tuple(t):
    
    def process_tuple_elem(elem):
        if elem == "nan":
            return np.nan
        else:
            return elem[1:-1]
    return list(map(process_tuple_elem, t[1:-1].replace(" ", "").split(",")))
index_offensiveness = list(map(process_tuple, offensiveness_rating.index.tolist()))
value_offensiveness = list(offensiveness_rating.values)
data_non_flat = list(zip(index_offensiveness, value_offensiveness))
columns = ["track_id", "category", "strength", "target", "value"]
track_off = pd.DataFrame.from_records([elem[0] + [elem[1]] for elem in data_non_flat], columns=columns)
track_off = track_off.set_index(["track_id", "category", "strength", "target"])
track_off.head()<jupyter_output><empty_output><jupyter_text>That's it, we have our multi-index structure. Now, we have the choice: absolute or relative count of swear words in a song ? This choice is rather difficult. But, in some way, even if a song is longer than another one, we can (for now) consider that simply having more swear words is equivalent to being more vulgar. So let's simply count the number of swear words per song:<jupyter_code>%%time 
track_offensiveness = track_off.groupby(by="track_id").apply(lambda frame: frame.reset_index().dropna(thresh = 3).value.sum())
track_offensiveness = track_offensiveness.fillna(0)<jupyter_output><empty_output><jupyter_text>And here is our frame relating a song with its offensiveness (so, for example, the song with track ID 'TRAAAAV128F421A322' contains one swear word)<jupyter_code>track_offensiveness.head()<jupyter_output><empty_output><jupyter_text>Let's save this:<jupyter_code>if first_time:
    track_offensiveness.to_csv("../datasets/track_offensiveness.csv")
else:
    track_offensiveness = pd.read_csv("../datasets/track_offensiveness.csv", header=-1)
    track_offensiveness.columns = ["track_id", "offensiveness"]<jupyter_output><empty_output><jupyter_text>We take only the columns we are interested in:<jupyter_code>song_info = pd.read_csv("../datasets/total.csv")[["track_id", "genre", "song_hotness",
                                                            "song_id", "title", "year"]]<jupyter_output><empty_output><jupyter_text>English songsWe now want to do our analysis for english songs only so we read the corresponding track IDs. The process of "detecting" which songs are in english is described in [this notebook](handling_languages.ipynb).<jupyter_code>english_songs = pd.read_csv("../datasets/tracks_english.csv", header=-1)[1]
print("We have {} english songs.".format(english_songs.shape[0]))
english_songs = english_songs.to_frame()
english_songs.columns = ["track_id"]
english_songs.head()
english_track_offensiveness = english_songs.merge(track_offensiveness.reset_index(), on="track_id")
english_track_offensiveness.drop("index", axis=1, inplace=True)
print("We have {} songs.".format(english_track_offensiveness.shape[0]))
english_track_offensiveness.columns = ["track_id", "abs_offensiveness"]
english_track_offensiveness.head()
song_info_offensiveness = song_info.merge(english_track_offensiveness.reset_index(), on="track_id")
song_info_offensiveness = song_info_offensiveness.rename(columns={0: "abs_offensiveness"})
song_info_offensiveness.head()<jupyter_output><empty_output><jupyter_text>That's good, we have more than 200000 english songs with absolute offensiveness value. The two other values we will be the most focused on are the song hotness and the year. So we need to keep those where the year is not 0 and the song hotness is not nan. For now, let's proceed by keeping only datapoints with positive hotness value.  Keep positive hotness datapoints:<jupyter_code>song_info_offensiveness_hotness = song_info_offensiveness[song_info_offensiveness.song_hotness > 0]<jupyter_output><empty_output><jupyter_text>And now, let's compute the correlation for all songs independent of year:<jupyter_code>song_info_offensiveness_hotness.song_hotness.corr(song_info_offensiveness_hotness["abs_offensiveness"])<jupyter_output><empty_output><jupyter_text>So we do not really have a correlation which means something here. Let's try by keeping the hotness = 0 values (if we suppose that those songs were "complete flops"):<jupyter_code>song_info_offensiveness.song_hotness.corr(song_info_offensiveness.abs_offensiveness)<jupyter_output><empty_output><jupyter_text>This does not tell us much neither...So let's include the year and see what we get:<jupyter_code>song_info_YOH = song_info_offensiveness_hotness[song_info_offensiveness_hotness.year > 0]
song_info_YOH.head()<jupyter_output><empty_output><jupyter_text>And the correlation for songs where we have the year is:<jupyter_code>song_info_YOH.song_hotness.corr(song_info_YOH.abs_offensiveness)<jupyter_output><empty_output><jupyter_text>Still not significant, but it increased (even if the increase is very small)<jupyter_code># group by:
year_correlation = song_info_YOH.groupby(by="year").apply(lambda frame: frame.song_hotness.corr(frame.abs_offensiveness))

# we drop the years where we have too few songs and we rename the columns
year_correlation = year_correlation.dropna()
year_correlation = year_correlation.to_frame().reset_index()
year_correlation.columns = ["year", "correlation"]
plt.figure(figsize=(16, 10))
plt.title("Correlation between song offensiveness and popularity over time")
plt.ylabel("Correlation")
plt.xlabel("Time")
plt.plot(year_correlation.year.tolist(), year_correlation.correlation.tolist())
plt.show()<jupyter_output><empty_output><jupyter_text>Category, strength and target-based correlationsUntil now we computed the general offensiveness of a song. But we would also like to see how the popularity and the offensiveness relate in terms of categories of swear words. So let's proceed by defining a function for getting the offensiveness only based on one combination of category, strength and target:<jupyter_code>def correlate_for_swear_attributes(track_off, category="none", strength="none", target="none", plot_now=True):
    """
    This method plots the date correlations for the specified parameters. If the 3 are "none"
    then the correlations will be computed based on all swear words.
    """
    def get_off_count_based_on(track_off, category, strength, target):
        track_off_nidx = track_off.reset_index().fillna("none")
        return track_off_nidx.query("(category == @category and strength == @strength and target == @target) or\
                                (category == 'none' and strength == 'none' and target == 'none')").replace("none", np.nan)
    def get_off_based_on(track_off, category, strength, target):
        count_off = get_off_count_based_on(track_off, category, strength, target)
        count_off_non_na = count_off[count_off.duplicated("track_id", keep=False)].dropna(thresh=3)
        count_off_non_na.loc[:, "offensiveness"] = count_off_non_na["value"]
        count_off_na = count_off.drop_duplicates("track_id",keep=False)
        count_off_na.insert(0, "offensiveness", 0)
        return pd.concat([count_off_non_na, count_off_na])[["track_id", "offensiveness"]]
    
    all_swear_words = category == "none" and strength == "none" and target == "none"
    
    english_songs = pd.read_csv("../datasets/tracks_english.csv", header=-1)[1].to_frame()
    english_songs.columns = ["track_id"]
    song_info = pd.read_csv("../datasets/total.csv")[["track_id", "genre", "song_hotness",
                                                            "song_id", "title", "year"]]
    label = "all"
    if all_swear_words:
        # We want all categories/strength/targets:
        print("Correlations for all swear words:")
        if first_time:
            track_off_group = track_off.groupby(by="track_id").apply(lambda frame: frame.reset_index().dropna(thresh = 3).value.sum())
            track_off_group = track_off_group.fillna()
        else:
            track_off_group = pd.read_csv("../datasets/track_offensiveness.csv", header=-1)
            track_off_group.columns = ["track_id", "offensiveness"]
    else:
        print("Correlations for category = {}, strength = {} and target = {}".format(category, strength, target))
        label = "{}, {}, {}".format(category, strength, target)
        track_off_group = get_off_based_on(track_off, category, strength, target)
        
    english_track_offensiveness = english_songs.merge(track_off_group.reset_index(), on="track_id")
    song_info_offensiveness = song_info.merge(english_track_offensiveness, on="track_id")
    song_info_offensiveness = song_info_offensiveness[["track_id", "song_hotness", "year", "offensiveness"]]
    song_info_offensiveness = song_info_offensiveness.dropna().query("year > 0 and song_hotness > 0")
    year_correlation = song_info_offensiveness.groupby(by="year").apply(lambda frame: frame.song_hotness.corr(frame.offensiveness))
    year_correlation = year_correlation.dropna()
    year_correlation = year_correlation.to_frame().reset_index()
    year_correlation.columns = ["year", "correlation"]
    
    plt.plot(year_correlation.year.tolist(), year_correlation.correlation.tolist(), label = label)
    if plot_now:
        plt.show()
    # return year_correlation
    # english_track_offensiveness.columns = ["track_id", "abs_offensiveness"]
    
#correlate_for_swear_attributes(track_off, "non-discriminatory", "mild", "none")<jupyter_output><empty_output><jupyter_text>Now let's go over all the category-strength-target combinations and plot the correlations for each of them:<jupyter_code>all_combinations = track_off.reset_index()[["category", "strength", "target"]].drop_duplicates()
all_combinations = all_combinations.fillna("none")
for i in range(all_combinations.shape[0]):
    row = all_combinations.iloc[i]
    correlate_for_swear_attributes(track_off, row["category"], row["strength"], row["target"])<jupyter_output>Correlations for category = non-discriminatory, strength = mild and target = none<jupyter_text>We see that they all follow the same trend: there is a kind of correlation in the beginning and then it simply tends to 0. We might be tempted to think that, for music in the mid-10th century, popularity is more correlated with offensiveness because of the Second World War but this is probably wrong. In the following we will see that we have very little data for those years so it is normal that the correlations are much stronger as soon as one song has one word that is interpreted as a swear word by our algorithms (sometimes it makes mistakes because the lyrics data: we will see an example where the lyrics contain the words Whoo - hoo which doesn't mean anything but MusixMatch put the "hoo" as "ho", which is a swear word in our data).Let us take a few of them and put them in out report<jupyter_code>combinations_to_show = all_combinations.sample(8)
combinations_to_show.head()
plt.figure(figsize=(16, 10))
print(combinations_to_show.shape)
for i in range(combinations_to_show.shape[0]):
    row = combinations_to_show.iloc[i]
    correlate_for_swear_attributes(track_off, row["category"], row["strength"], row["target"], plot_now=False)
plt.title("Correlations over time for different swear words categories")
plt.xlabel("Time")
plt.ylabel("Offensiveness - hotness correlation")
plt.legend()
plt.show()<jupyter_output>(8, 3)
Correlations for category = discriminatory, strength = mild and target = sexuality
{'fraction_non_alphanumeric': 0.10218566013388176, 'fraction_numerical': 0.0049197515928703925, 'mean_word_length': 5.60628662759723, 'pattern_counts': {'":': 2, '<': 67, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '554913', 'license': 'MIT', 'n_tokens_mistral': 3910, 'n_tokens_neox': 3732, 'n_words': 1016, 'path': 'Project/main_nb.ipynb', 'repo_name': 'lhk/epfl_ada', 'size': '12399'}