<jupyter_start><jupyter_text>Tidy<jupyter_code>import pandas as pd 
import numpy as np
data = pd.read_csv('music.csv')
data.head()<jupyter_output><empty_output><jupyter_text>Drop Extraneous ColumnsThe latitude and longitude columns were not needed since there was a categorical artist location variable. We also drop similar because it is a recomendation of similar artists, and is not in relation to the actual song observations. Plus, we dropped tatums_confidence and tatums_start columns because we will analyze beats in the beat related variables, and do not need to worry about tatums. Finally we delete the artist_mbtags and artist_mbtags_count because we will use the hotness levels as a measure of popularity.<jupyter_code>data = data.drop(columns = ['latitude', 'longitude', 'similar', 'tatums_confidence', 'tatums_start', 'artist_mbtags', 'artist_mbtags_count'])<jupyter_output><empty_output><jupyter_text>Appropriate Categorical Values The Echonest software encoded the song keys, modes, and time signatures using integers, but because they will be used as categories in the analysis they were put back into more appropriate terms for observation.<jupyter_code>new_vals = {'mode': {0: "Minor", 1: "Major", -1: np.nan}, 
            'key': {0: "C", 1: "C#", 2: "D", 3: "D#", 4: "E", 5: "F", 6: "F#", 7: "G", 8: "G#", 9: "A", 10: "A#", 11: "B"},
           'time_signature': {-1: np.nan, 1: "Complex or Changing", 4: "4/4", 5: "5/4", 6: "6/4", 7: "7/4"}}
data.replace(new_vals, inplace= True)<jupyter_output><empty_output><jupyter_text>Missing Values<jupyter_code>null_data = len(data[data['song.hotttnesss'].isnull()])
null_data
data['year'] = data['year'].replace(0, np.nan)
data['artist.id'] = data['artist.id'].replace(-1, np.nan)
data['release.id'] = data['release.id'].replace(-1, np.nan)
data['location']= data['location'].replace('Not available', np.nan)
data['familiarity'] = data['familiarity'].fillna((data['familiarity'].mean(skipna = True)))
data['song.hotttnesss']=data['song.hotttnesss'].fillna(0)<jupyter_output><empty_output><jupyter_text>Here we needed to adjust the familiarity value because there were missing values and it was a quantitative variable. I set missing values of song hotness to 0 because they were songs not popular enough to be measured by the scaling program and there were so many values missing that using the average would likely not have been appropriately represenative.<jupyter_code>assert len(data[data['artist.hotttnesss'].isnull()]) == 0
assert len(data[data['bars_confidence'].isnull()]) == 0
assert len(data[data['bars_start'].isnull()]) ==0
assert len(data[data['beats_confidence'].isnull()]) ==0
assert len(data[data['beats_start'].isnull()]) ==0
assert len(data[data['duration'].isnull()]) ==0
assert len(data[data['end_of_fade_in'].isnull()]) ==0
assert len(data[data['familiarity'].isnull()]) ==0
assert len(data[data['key_confidence'].isnull()]) ==0
assert len(data[data['loudness'].isnull()]) ==0
assert len(data[data['mode_confidence'].isnull()]) ==0
assert len(data[data['song.hotttnesss'].isnull()]) ==0
assert len(data[data['start_of_fade_out'].isnull()]) ==0
assert len(data[data['tempo'].isnull()]) ==0
assert len(data[data['terms_freq'].isnull()]) ==0
assert len(data[data['time_signature_confidence'].isnull()]) ==0<jupyter_output><empty_output><jupyter_text>Rename ColumnsHere I renamed some of the columns that were not very clear as to what variable was being described. The general format is that if there is a period in the variable name, the variable is refrencing what is before the period. Example : 'artist.hotness' is a hotness popularity rating of an artist, not the song that is the row observation.<jupyter_code>data = data.rename(columns = {'artist.id': 'artist_id', 'artist.name': 'artist_name', 'artist.hotttnesss': 'artist_hotness', 'familiarity':'artist_familiarity', 'location': 'artist_location', 'terms': 'genre', 'release.id': 'release_id', 'release.name': 'release_name', 'song.id': 'song_id', 'terms_freq': 'artist_genre_freq', 'song.hotttnesss': 'song_hotness', 'terms_weight': 'artist_terms_weight', 'year': 'release_year', 'title':'song_title'})<jupyter_output><empty_output><jupyter_text>Checking Type ContinuityFirst I had to change some float values to integers for discrete quantitative variables.After this, the following tests ensure that the data types are what we expect, and are the same through each column.<jupyter_code>data['key'] = data['key'].astype('object')
data['mode'] = data['mode'].astype('object')
data['release_id'] = data['release_id'].astype('int')
data['time_signature'] = data['time_signature'].astype('object')
data['release_year'] = data['release_year'].astype('object')
assert np.dtype(data['artist_hotness']) == 'float'
assert np.dtype(data['artist_id']) == 'object'
assert np.dtype(data['artist_name']) == 'object'
assert np.dtype(data['bars_confidence']) == 'float'
assert np.dtype(data['bars_start']) == 'float'
assert np.dtype(data['beats_confidence']) == 'float'
assert np.dtype(data['beats_start']) == 'float'
assert np.dtype(data['duration']) == 'float'
assert np.dtype(data['end_of_fade_in']) == 'float'
assert np.dtype(data['artist_familiarity']) == 'float'
assert np.dtype(data['key']) == 'object'
assert np.dtype(data['key_confidence']) == 'float'
assert np.dtype(data['artist_location']) == 'object'
assert np.dtype(data['loudness']) == 'float'
assert np.dtype(data['mode']) == 'object'
assert np.dtype(data['mode_confidence']) == 'float'
assert np.dtype(data['release_id']) == 'int'
assert np.dtype(data['release_name']) == 'object'
assert np.dtype(data['song_hotness']) == 'float'
assert np.dtype(data['song_id']) == 'object'
assert np.dtype(data['start_of_fade_out']) == 'float'
assert np.dtype(data['tempo']) == 'float'
assert np.dtype(data['genre']) == 'object'
assert np.dtype(data['artist_genre_freq']) == 'float'
assert np.dtype(data['time_signature']) == 'object'
assert np.dtype(data['time_signature_confidence']) == 'float'
assert np.dtype(data['song_title']) == 'object'
assert np.dtype(data['release_year']) == 'object'<jupyter_output><empty_output><jupyter_text>One to One RelationshipsHere there are no one to one relationships as the only thing unique to each song is the song.id. Every other feature is able to potentially applied to multiple observations.<jupyter_code>assert len(data['song_id'].unique()) == 10000
data['song_id'].describe(include = 'all')<jupyter_output><empty_output><jupyter_text>One to Many Relationships In this data, since there were so many columns to consider, there were a great number of one to many relationships. The following subsets are a few of these one to many relationships which will be used to answer questions of interest.<jupyter_code>song_popularity =data[['song_id', 'song_hotness']]
song_year = data[['song_id', 'release_year']]
song_genres = data[['song_id', 'genre']]
song_tempo = data[['song_id', 'tempo']]
song_key = data[['song_id', 'key']]
song_mode = data[['song_id', 'mode']]
song_length = data[['song_id', 'duration']]<jupyter_output><empty_output><jupyter_text>Many to Many Relationships In the music dataset, there are repeats for genre across observations, and the other features also have repeat values. This can create a many to many relationship which can be useful in predicting relationships between features.<jupyter_code>genre_loudness = data[['genre', 'loudness']]
genre_mode = data[['genre', 'mode']]
songpop_artistpop = data[['song_hotness', 'artist_hotness']]<jupyter_output><empty_output><jupyter_text>Save Final Sets<jupyter_code>data.to_json('tidy_music.json', orient = 'records')
song_popularity.to_json('song_popularity.json', orient = 'records')
song_year.to_json('song_year.json', orient = 'records')
song_genres.to_json('song_genres.json', orient = 'records')
song_tempo.to_json('song_tempo.json', orient = 'records')
song_key.to_json('song_key.json', orient = 'records')
song_mode.to_json('song_mode.json', orient = 'records')
song_length.to_json('song_length.json', orient = 'records')
genre_loudness.to_json('genre_loudness.json', orient = 'records')
genre_mode.to_json('genre_mode.json', orient = 'records')
songpop_artistpop.to_json('songpop_artistpop.json', orient = 'records')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.158641600970285, 'fraction_numerical': 0.006913280776228017, 'mean_word_length': 7.744432661717921, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '317354', 'license': 'CC-BY-3.0', 'n_tokens_mistral': 2781, 'n_tokens_neox': 2675, 'n_words': 711, 'path': 'PortfolioContent/DATA301Final/03-Tidy.ipynb', 'repo_name': 'danield2255/danield2255.github.io', 'size': '8245'}