<jupyter_start><jupyter_text>Probably Interesting Data - <NAME> (2762306) and <NAME> (1083734) Audio features of songs ranging from 1922 to 2011 Build a heuristic to model the data using mixture models of probability distributions programmaticallyhttps://www.kaggle.com/uciml/msd-audio-features  Madness MethodWe will first investigate the data available in the kaggle dataset for Timbre from various songs over time to identify the trends in the data - looking for interesting features in the data.Once we have determined the values of focus, we will normalize the data using a Guassian Normal Distribution Function.With the normalized model data, we will plot the values, looking for interesting trends in the data. We will remove data that is uninteresting (few characteristics) or which go out of bounds from the more 'normal' dataset.At the end, with our final collection of data parameters, we will plot a mean function against the data to see where the moderate pattern lies.<jupyter_code>import pandas as pd
import numpy as np
import datetime as dt
import seaborn as sns
import matplotlib.pyplot as plt
import math
from statistics import mean, stdev<jupyter_output><empty_output><jupyter_text>Read in the data<jupyter_code>df = pd.read_csv("../data/external/audio-features-of-songs-ranging-from-1922-to-2011.zip", compression="zip")
df.head(1)<jupyter_output><empty_output><jupyter_text>Feature Engineering<jupyter_code>df[df.columns[0]].unique()
hist = df.hist(column = df.columns[0])
plt.plot(df['label'], df['TimbreAvg1'], 'ro')
plt.show()<jupyter_output><empty_output><jupyter_text>Based on observation from distribution of songs over years, we plan to trim dataset to focus on solely songs produced in the year 2000 or later.  This still provides us a large amount of data to work with.<jupyter_code>df2 = df[['label', 'TimbreAvg1']].copy()
hist2 = df2.hist(column = df2.columns[1])<jupyter_output><empty_output><jupyter_text>Work the data into a Mixed Guassian Model  Guassian TransformationThe following function, gauss_transform, goes through the dataset and normalizes all of the values using a Gaussian distribution function: $\Large \mathcal{N}(x|\mu,\sigma^2) = \frac{1}{\sigma \sqrt{2 \pi}}    e^{-\frac{(x - \mu)^2}{2 (\mu^2)}}$<jupyter_code>def gauss_transform(col):
    result_arr = []
    
    mu = mean(col)
    sig = stdev(col)
    frac_part = 1/(sig*math.sqrt(2*math.pi))
    for val in col:
        exponent_part = ((val - mu)**2) / (2*(mu**2)) * -1
        result_arr.append(frac_part * math.exp(exponent_part))
        
    return result_arr<jupyter_output><empty_output><jupyter_text>Maximum Likelihood for Gaussian DistributionAs per Bishop, the simple form for calculating likelihood is:$\Large \mu = \frac{1}{N}\Sigma_{n=1}^{N}x_n$ a.k.a. $\Large \pi_k$<jupyter_code>def likelihood(col):    
    return sum(col) / len(col)<jupyter_output><empty_output><jupyter_text>ObservationHaving looked at the data, we need to be able to bin the dataOur focus, based on feature engineering, will be the Averages<jupyter_code>## Expects two column dataframe
## date, value and the increment size
## size is number of years to bin
## Min is 1922 (start 1920), Max is 2011 (stop 2020)
## Bins and averages values
def bin_data(df, size):
    ## Init bins
    dates = [i for i in range(1920,2030,size)]
    values = [0 for _ in range(1920,2030,size)]
    
    ## track count and bin index
    count = 0
    i = 0
    
    for yr, val in df.values:
        if int(yr) < int(dates[i+1]):
            count += 1
            values[i] += val
        else:
            values[i] /= count
            count = 0
            i += 1
            values[i] += val
            
    return [dates, values]
## Each column is numbered with 'TimbreAvg' as the prefix
## label column is the date (year)
## Init
binsize = 10
avestart = 1
avestop = 13
avestep = 1
binned_df = pd.DataFrame()
for i in range(avestart,avestop,avestep):
    temp_df = df[['label', 'TimbreAvg'+str(i)]].copy()
    temp_df = temp_df.sort_values(['label'], ascending=True)
    bin_array = bin_data(temp_df,binsize)
    if 'Year' not in binned_df:
        binned_df['Year'] = bin_array[0]
    binned_df['TimbreAvg'+str(i)] = bin_array[1]<jupyter_output><empty_output><jupyter_text>Perform Guassian Transformation on each column (TimbreAve*)<jupyter_code>## Init
gauss_df = pd.DataFrame()
likely = [0 for _ in range(avestart,avestop,avestep)]
gauss_df['Year'] = binned_df['Year']
for i in range(avestart,avestop,avestep):
    ## Pass each column to the transformation function
    gauss_df['TimbreAvg'+str(i)] = gauss_transform(binned_df['TimbreAvg'+str(i)].values.tolist())
    likely[i-1] = likelihood(gauss_df['TimbreAvg'+str(i)].values.tolist())<jupyter_output><empty_output><jupyter_text>Generate the mixed model - Mixture of Gaussians$\Large p(x) = \Sigma_{k=1}^{K}\pi_k \mathcal{N}(x|\mu_k,{\small \Sigma}_k)$<jupyter_code>## Go through the dataframe and tabulate
## Init
mixed = [0 for _ in range(gauss_df.shape[0])]
counter = 0
mixed_df = pd.DataFrame()
for index, row in gauss_df.iterrows():
    for i in range(len(likely)):
        mixed[counter] += row['TimbreAvg'+str(i+1)] *likely[i]
    counter += 1
mixed_df['Year'] = gauss_df['Year']
mixed_df['MIXED'] = mixed<jupyter_output><empty_output><jupyter_text>Plot the data, including the mixture<jupyter_code>print(gauss_df.head(1))
ys = ['TimbreAvg'+str(i+1) for i in range(gauss_df.shape[0]+1)]
ys.append('MIXED')
gauss_df['MIXED'] = mixed_df['MIXED']
gauss_df.plot(x='Year', y=ys)
plt.show()<jupyter_output><empty_output><jupyter_text>Based on the skewing above - we could either modify the $\pi$ value (likelihood) for the TimbreAvg 7 values. For simplicity, we will remove the attribute from the trending. Note - this did not result in a re-factoring of the mixed gaussian value, which should be done to be 'correct'.<jupyter_code>ys.remove('TimbreAvg7')
gauss_df['MIXED'] = mixed_df['MIXED']
gauss_df.plot(x='Year', y=ys)
plt.show()<jupyter_output><empty_output><jupyter_text>Now removing TimbreAvg1<jupyter_code>ys.remove('TimbreAvg1')
gauss_df['MIXED'] = mixed_df['MIXED']
gauss_df.plot(x='Year', y=ys)
plt.show()<jupyter_output><empty_output><jupyter_text>Now removing TimbreAvg11<jupyter_code>ys.remove('TimbreAvg11')
gauss_df['MIXED'] = mixed_df['MIXED']
gauss_df.plot(x='Year', y=ys)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12448979591836734, 'fraction_numerical': 0.019623233908948195, 'mean_word_length': 4.943097014925373, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 55, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '238453', 'license': 'MIT', 'n_tokens_mistral': 2266, 'n_tokens_neox': 2149, 'n_words': 610, 'path': 'notebooks/01-dahrla-prob-interesting-data-timbre.ipynb', 'repo_name': 'daltonhahn/eecs738-probably_interesting_data', 'size': '6380'}