<jupyter_start><jupyter_text>Musical Theatre Eras: An Analysis BackgroundMusical Theatre is an art form that dates since Ancient Greece where there were plays with song and dance. However, Musical Theatre in the format we conceive musicals today has been around since the late 1800's and has greatly evolved ever since. When studying Musical Theatre, we usually discuss its periods. However, there is no consensus about how many periods or eras exist. The only common thread is the Golden Age that most people agree encompasses some period between the 1940's and 1960's.This analysis will focus on understanding Musical Theatre eras Research questions and/or hypotheses Q1: Can we identify how many Musical Theatre eras exist using audio features from songs?  Q2:  Can we classify Musical Theatre songs into three distinct periods: Early Age (pre-Golden Age), Golden Age, and Modern/Contemporary (Post-Golden Age)  Q3: Hypothesis: - Contemporary musicals' songs have longer duration than the other periods.- Contemporary musicals' songs have more speechiness than the other periods.- Contemporary musicals' songs have more loudness than the other periods.- Early Age musicals' songs have slower tempo than the most recent periods.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt  
import os<jupyter_output>Bad key "text.kerning_factor" on line 4 in
/Users/andreiasodrenichols/miniconda3/lib/python3.7/site-packages/matplotlib/mpl-data/stylelib/_classic_test_patch.mplstyle.
You probably need to get an updated matplotlibrc file from
https://github.com/matplotlib/matplotlib/blob/v3.1.3/matplotlibrc.template
or from the matplotlib source distribution<jupyter_text>Get DataFor this project, we are using a dataset I created in 2017 that contains features from musical theater songs. The dataset is available [here](https://github.com/andreiabds/orpheus). The dataset was originally created using data from: - Wikipedia: list of musical theater titles, with composer, lyricist, and year - Spotify: music features dataset from - stlyrics.com: lyrics We used the titles from the musicals to download music features from songs of albuns that matched the musical titles. To clean the dataset so it would only contain songs from musicals, we used [Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance) to match song titles per album of the dataset with song titles per album in the stlyrics website, which is a lyrics website that is curated to contain only soundtrack lyrics.<jupyter_code>#Create folder to save figures
os.makedirs('figures', exist_ok=True)
#download data into current directory
!curl -O https://raw.githubusercontent.com/andreiabds/orpheus/master/data/COMPLETE_DATA.csv
#load dataset
musical_songs_features = pd.read_csv('COMPLETE_DATA.csv')
musical_songs_features.head()<jupyter_output><empty_output><jupyter_text>Data Cleaning and EDA The dataset needs a little bit of data cleaning. We are dropping an empty song. There is one musical () that had two Broadway productions listed on Wikpedia, thus turning the feature year in our dataset as a string and limiting our capacity of placing this musical into a specific era. Chose the earlier date to represent its year.We also filled missing audio features with a very high number (99). Usually these features range from 0.0 to 1.0.We also fixed the data types of the columns, turning most columns into floats.<jupyter_code># Data cleaning
musical_songs_features = musical_songs_features.dropna()
musical_songs_features=musical_songs_features.replace({'year': '1971/1999'}, 1971)
musical_songs_features = musical_songs_features.replace('None',99 )

# Cleaning data types
column_types = {'Unnamed: 0': 'int', 
              'sp_track_id': 'object', 
              'wikipedia_title': 'object', 
              'danceability': 'float',
              'energy': 'float' , 
              'key': 'float', 
              'loudness': 'float', 
              'mode': 'float', 
              'speechiness' : 'float', 
              'acousticness': 'float',
              'instrumentalness': 'float', 
              'liveness': 'float', 
              'valence': 'float', 
              'tempo': 'float', 
              'duration_ms': 'float',
              'time_signature': 'float', 
              'year': 'int', 
              'lyrics': 'object', 
              'composer_label': 'object', 
              'lyricist_label': 'object'}


for key in column_types:
    musical_songs_features[key]= musical_songs_features[key].astype(column_types[key])


musical_songs_features.describe()<jupyter_output><empty_output><jupyter_text>Looking at the distribution of musicals over the years, we notice that we have a good spread of musicals across all decades, with more concentration of musicals in the most recent decades.<jupyter_code>musical_songs_features.groupby('year')['wikipedia_title'].nunique().plot(title='Number of Musicals per Year')
#save figure
plt.gcf() # get current figure - helps with savefig
plt.savefig('figures/MusicalsPerYear.jpg')<jupyter_output><empty_output><jupyter_text>Q1:  How many musical eras are there?In this section, we will use the audio features and attempt to make clusters of musicals emerge. We are assuming the the clusters would represent the different periods or eras that we can organize musicals.<jupyter_code>audio_features = ['danceability',
           'energy', 
           'speechiness', 
           'acousticness',
           'instrumentalness', 
           'duration_ms',
           'key', 
           'loudness', 
           'mode', 
           'liveness', 
           'valence', 
           'tempo', 
           'time_signature']<jupyter_output><empty_output><jupyter_text>We K-means, a simple clustering algorithm, to create clusters of musical songs. We first need to scale our data using the MinMaxScaler. Then, we will ran our clustering algorithm with different number of clusters. Finally, we will use the Elbow Method to identify the optimal number of clusters.<jupyter_code>from sklearn.cluster import KMeans
from sklearn.preprocessing import MinMaxScaler
from scipy.spatial.distance import cdist 

scaler = MinMaxScaler()
X = scaler.fit_transform(musical_songs_features[audio_features])


# The code below is from geeksforgeeks.org/elbow-method-for-optimal-value-of-k-in-kmeans/
distortions = [] 
mapping1 = {} 
K = range(1,10) 
  
for k in K: 
    #Building and fitting the model 
    kmeanModel = KMeans(n_clusters=k).fit(X) 
    kmeanModel.fit(X)     
      
    distortions.append(sum(np.min(cdist(X, kmeanModel.cluster_centers_, 
                      'euclidean'),axis=1)) / X.shape[0]) 
  
    mapping1[k] = sum(np.min(cdist(X, kmeanModel.cluster_centers_, 
                 'euclidean'),axis=1)) / X.shape[0] 

plt.plot(K, distortions, 'bx-') 
plt.xlabel('Values of K') 
plt.ylabel('Distortion') 
plt.title('The Elbow Method using Distortion') 

#save figure
plt.gcf() # get current figure - helps with savefig
plt.savefig('figures/KMeans_ElbowMethod.jpg')<jupyter_output><empty_output><jupyter_text>ResultsThere is no clear optimal number of clusters based on the Elbow Method. Therefore, we were not able to use audio features data to identify how many musical eras exist.    Q2: Can we use audio features to classify Musical Theatre songs into three distinct periods? We will train a classifier to assign songs into three different categories: Early Age (pre-Golden Age), Golden Age, and Modern/Contemporary (Post-Golden Age). We will consider Golden Era to be between 1943 and 1968, as this seems to be a common denomination. [[1]](https://www.chicagostagereview.com/history-of-musical-theatre-timeline/)  [[2]](https://trace.tennessee.edu/cgi/viewcontent.cgi?referer=https://www.google.com/&httpsredir=1&article=1093&context=gamut)Songs from musicals prior to 1943 will be labeled 'Early Age'.Songs from musicals post 1968 will be labeled 'Modern Contemporary'.<jupyter_code>conditions = [musical_songs_features['year'] > 1968,
              musical_songs_features['year'] < 1943]

# User Friendly Name for musical period
choices = ['Modern/Contemporary', 'Early Age']
musical_songs_features['musical_period'] = np.select(conditions, choices, default='Golden Age')

# creating target for classifier
choices_target = [3, 1]
musical_songs_features['target'] = np.select(conditions, choices_target, default=2)<jupyter_output><empty_output><jupyter_text>Looking at the distribution of the songs between the three different periods, we can notice that the majority of songs belong to the Modern/Contemporary Age.Early Age has the least song, however, it still has a significant amount of songs.<jupyter_code>y = musical_songs_features['musical_period'].value_counts().values
x = musical_songs_features['musical_period'].value_counts().index.values

fig, ax = plt.subplots()
ax.bar(x, y)
ax.set_xticks(x)

plt.title('# Songs per Musical Period')
#save figure
plt.gcf() # get current figure - helps with savefig
plt.savefig('figures/SongsPerPeriod.jpg')<jupyter_output><empty_output><jupyter_text>Since Modern/Contemporary Age is our majority class, we will consider a good baseline accuracy for our model to be above 55.94%. That is, our model will have better performance than a model that always picks the majority class.<jupyter_code>modern_songs = musical_songs_features['musical_period'].value_counts()['Modern/Contemporary']
all_songs = np.sum(musical_songs_features['musical_period'].value_counts().values)

modern_songs/all_songs<jupyter_output><empty_output><jupyter_text>We train a Random Forest Classifier with 60% of our data. Our results are good! We get an accuracy score of almost 70% Looking further into other metrics, we notice that our classifier performs better on songs from Modern/Contemporary Musicals (3), while it performs worst on songs from Early Age (1).<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn import metrics

# preparing data, train/test split -
X = musical_songs_features[audio_features]
y = musical_songs_features['target']
X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.4, random_state=42)

# training Random Forest Classifier
clf = RandomForestClassifier(class_weight ="balanced" , n_estimators=100, max_depth=20, min_samples_split=2, random_state=4)
clf.fit(X_train, y_train)

# Testing results on test set, print metrics
pred = clf.predict(X_test)
print(metrics.classification_report(y_test, pred, digits=3))<jupyter_output>precision    recall  f1-score   support

           1      0.636     0.197     0.301       320
           2      0.607     0.611     0.609      1055
           3      0.754     0.846     0.797      1751

    accuracy                          0.700      3126
   macro avg      0.666     0.551     0.569      3126
weighted avg      0.692     0.700     0.683      3126<jupyter_text>Looking at the feature importance from our classifier, we find it surprising that acousticness appeared at the top as one of the most important features.At the same time, it is interesting that features that we will explore in our hypothesis also appear at the top (e.g., speechiness, duration_ms, loudness).<jupyter_code>importances = clf.feature_importances_
important = pd.DataFrame(audio_features)
important['feature_rank'] = importances
important.sort_values(by='feature_rank', ascending=False).reset_index(drop=True)<jupyter_output><empty_output><jupyter_text>Results:Yes! We can use audio features to classify our musical songs into three distinct periods. However, our classifier has better results with songs from Modern/Contemporary Musicals.   Q3: Hypothesis Tests Plotting the median value of each audio feature per period, so we can have a visual to compare across periods.<jupyter_code>median = musical_songs_features.groupby('musical_period').median()


n = round((len(audio_features) + 1)/2)

fig, axs = plt.subplots(n, 2, figsize=(20,40))
x = median.index.values

for i in range(len(audio_features)):
    feature_name = audio_features[i]
    if i%2 == 0:
        x2=0
        x1 = round(i/2)
    else:
        x2=1
        x1 = round((i-1)/2)
    axs[x1,x2].bar(x, median[feature_name])
    axs[x1,x2].set_xticks(x)
    axs[x1,x2].title.set_text('median of ' + feature_name)
    
    
#save figure
plt.gcf() # get current figure - helps with savefig
plt.savefig('figures/MedianAudioFeaturesPerPeriod.jpg')<jupyter_output><empty_output><jupyter_text>We separate the songs into different datasets so we can perform our statistical test. We chose the [MannWhitney U test](https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test) because it is a great option to test populations of different sizes, given that we have a very unbalanced distribution of songs across the three periods.<jupyter_code># Dataset for hypothesis about modern/contemporary musicals
contemporary_musicals = musical_songs_features[musical_songs_features['target']==3]
non_contemporary_musicals = musical_songs_features[musical_songs_features['target']!=3]

# Dataset for hypothesis about early musicals
early_musicals = musical_songs_features[musical_songs_features['target']==1]
non_early_musicals = musical_songs_features[musical_songs_features['target']!=1]
from scipy.stats import mannwhitneyu<jupyter_output><empty_output><jupyter_text>SpeechinessSpeechiness detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g. talk show, audio book, poetry), the closer to 1.0 the attribute value. Values above 0.66 describe tracks that are probably made entirely of spoken words. Values between 0.33 and 0.66 describe tracks that may contain both music and speech, either in sections or layered, including such cases as rap music. Values below 0.33 most likely represent music and other non-speech-like tracks. [[source]](https://developer.spotify.com/documentation/web-api/reference/object-model/audio-features-object)- H0: Modern/Contemporary musicals songs have smaller or equal speechiness to songs from other periods- H1: Modern/Contemporary musicals songs have greater speechiness to songs from other periods Result : We reject the null hypothesis.<jupyter_code>print(mannwhitneyu(contemporary_musicals['speechiness'],non_contemporary_musicals['speechiness'], alternative='greater' ))

plt.bar(x, median['speechiness'])
plt.title('median of speechiness')
#save figure
plt.gcf() # get current figure - helps with savefig
plt.savefig('figures/MedianPerPeriodSpeechiness.jpg')<jupyter_output>MannwhitneyuResult(statistic=7936671.0, pvalue=1.7105330773238287e-05)<jupyter_text>LoudnessThe overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks. Loudness is the quality of a sound that is the primary psychological correlate of physical strength (amplitude). Values typical range between -60 and 0 db.[[source]](https://developer.spotify.com/documentation/web-api/reference/object-model/audio-features-object)- H0: Modern/Contemporary musicals songs have smaller or equal loudness to songs from other periods- H1: Modern/Contemporary musicals songs have greater loudness to songs from other periods Result : We reject the null hypothesis.<jupyter_code>print(mannwhitneyu(contemporary_musicals['loudness'],non_contemporary_musicals['loudness'], alternative='greater' ))

plt.bar(x, median['loudness'])
plt.title('median of loudness')
#save figure
plt.gcf() # get current figure - helps with savefig
plt.savefig('figures/MedianPerPeriodLoudness.jpg')<jupyter_output>MannwhitneyuResult(statistic=9721498.0, pvalue=3.4289161779179547e-109)<jupyter_text>DurationThe duration of the track in milliseconds.[[source]](https://developer.spotify.com/documentation/web-api/reference/object-model/audio-features-object)- H0: Modern/Contemporary musicals songs have smaller or equal duration to songs from other periods- H1: Modern/Contemporary musicals songs have greater duration to songs from other periods Result : We reject the null hypothesis.<jupyter_code>print(mannwhitneyu(contemporary_musicals['duration_ms'],non_contemporary_musicals['duration_ms'], alternative='greater' ))

plt.bar(x, median['duration_ms'])
plt.title('median of duration_ms')
#save figure
plt.gcf() # get current figure - helps with savefig
plt.savefig('figures/MedianPerPeriodDuration.jpg')<jupyter_output>MannwhitneyuResult(statistic=7777283.5, pvalue=0.005642394558058948)<jupyter_text>TempoThe overall estimated tempo of a track in beats per minute (BPM). In musical terminology, tempo is the speed or pace of a given piece and derives directly from the average beat duration. [[source]](https://developer.spotify.com/documentation/web-api/reference/object-model/audio-features-object)- H0: Modern/Contemporary musicals songs have greater (faster) or equal tempo to songs from other periods- H1: Modern/Contemporary musicals songs have smaller (slower) tempo to songs from other periods Result : We reject the null hypothesis.<jupyter_code>print(mannwhitneyu(early_musicals['tempo'],non_contemporary_musicals['tempo'], alternative='less' ))

plt.bar(x, median['tempo'])
plt.title('median of tempo')
#save figure
plt.gcf() # get current figure - helps with savefig
plt.savefig('figures/MedianPerPeriodTempo.jpg')<jupyter_output>MannwhitneyuResult(statistic=1316964.0, pvalue=0.200646191184347)
{'fraction_non_alphanumeric': 0.0867692661077294, 'fraction_numerical': 0.021247272309635926, 'mean_word_length': 4.836126005361931, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 64, 'https://': 12, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '634183', 'license': 'MIT', 'n_tokens_mistral': 5355, 'n_tokens_neox': 4959, 'n_words': 1675, 'path': 'Musical Theater Era Analysis.ipynb', 'repo_name': 'andreiabds/MusicalTheatreEras', 'size': '17414'}