<jupyter_start><jupyter_text>Scoring Analysis Motivation: Determine a way to evaluate speech detection numerically.Given a (fabricated) y_pred that indicated whether a give sample was speech or not, it was comparedto a ground truth y. I manually fabricated the y_pred so that I could compare the various scoring approachesunder various scenarios.One of the more challenging scenarios was that I wanted to penalize the prediction when they were "too simple".For example, predicting continuous speech when in fact there were short bursts of silence.Since time series data is not independent, many common scoring methods are not suitable, e.g.:* F1, MSE, Recall, Precision, etc.One approach was to process the time series data such that series dependency was embedded in each element sothat typical scoring methods could be used. Specifically, by counting the duration between event transitions and thenuse the typical scoring methods on that data.Another approach was to convolve using a generic boolean shape (this make it easier to convert to a standard scale) ResultsAdding a penalty for differences in variance combined with convolution seemed to offer the best result.Example:```ConvVar(var_penalty=20).score(y_pred, y)```Tha variance penalty (var_penalty) adds a scoring penalty if the variances of `y_pred` and `y` differ. This penaltyhelps compensate for high scores where the predictions are too simplistic (example: predicting all false, when ground truthactually had a short spike of trues)Convolution generally performed better than the typical scoring method, but just convolving over the general shapedid not sufficiently penalize overly simplistic models.The standard scoring methods did not perform particularly well. Processing the data to embed the series dependency didnot significantly improve the performance either. Strategies yet to be considered: Dynamic Time WarpingMy initial attempts at using DTW did not result in a usefulmetric. The resultant DTW path was trivial. More investigation is needed to determine if it was due to poorimplementation or whether the data was not well suited for DTW.  Create Sample Actual vs Prediction data sets and Scorers<jupyter_code>import matplotlib.pyplot as plt
import matplotlib.font_manager as mfm
import numpy as np
import pandas as pd
from sklearn.metrics import f1_score, precision_score

from scorers import ConvVar, bconvolve_score, ticker_r2, r2_score_wrapper, ticker_mse, mse_wrapper

font_path = '/Users/dev/Fonts/Noto_Sans_KR/NotoSansKR-Regular.otf'
k_prop = mfm.FontProperties(fname=font_path)
plt.style.use('dark_background')

def make_from_runs(runs):
    """
    Takes a list of tuples where the first element is the number of times to insert the second element.
    r = [(3, True), (2, False)]
    make_from_runs(r)
    #=>
    [ True, True, True, False, False ]

    :param runs: List of tuple (integer, value)
    :return: A numpy array of values as defined by the input tuple.
    """
    return np.concatenate([np.repeat(val, num) for num, val in runs])

# Scoring strategies for comparison
SCORE_FNS = {
    'conv': bconvolve_score,
    'conv_var': ConvVar(var_penalty=20).score,
    'tick r2': ticker_r2,
    'r2': r2_score_wrapper,
    'prec': precision_score,
    'f1': f1_score,
    'tick mse': ticker_mse,
    'mse': mse_wrapper,
}

SCORE_NAMES = list(SCORE_FNS.keys())


# Make Sample data sets (y_sims) to investigate scoring strategies
y_sims = {}
y1 = [
    {
        'id': '0-0',
        'vals': make_from_runs([(8, False), (112, True), (8, False)]),
        'manual score': 1.0
    },
    {
        'id': '0-1L',
        'vals': make_from_runs([(7, False), (113, True), (8, False)]),
        'manual score': 0.99
    },
    {
        'id': '0-1R',
        'vals': make_from_runs([(8, False), (113, True), (7, False)]),
        'manual score': 0.99
    },
    {
        'id': '0-2^',
        'vals': make_from_runs([(8, False), (54, True), (4, False), (54, True),(8, False)]),
        'manual score': 0.90
    },
    {
        'id': '0-3^^^',
        'vals': make_from_runs([(8, False), (16, True), (4, False), (8, True), (16, False),(8, True),
                                   (8, False),
                                   (8, True), (16, False), (8, True), (4, False), (16, True), (8, False)]),
        'manual score': 0.30
    }
]

y2 = [
    {
        'id': '1-0',
        'vals': make_from_runs([(8, False), (54, True), (4, False), (54, True),(8, False)]),
        'manual score': 1.0
    },
    {
        'id': '1-1_',
        'vals': make_from_runs([(8, False), (112, True), (8, False)]),
        'manual score': 0.5
    },
    {
        'id': '1-2L',
        'vals': make_from_runs([(8, False), (52, True), (4, False), (56, True), (8, False)]),
        'manual score': 0.99
    },
    {
        'id': '1-3LL',
        'vals': make_from_runs([(8, False), (50, True), (4, False), (58, True), (8, False)]),
        'manual score': 0.97
    },
    {
        'id': '1-4^^',
        'vals': make_from_runs([(8, False), (52, True), (2, False),(4, True), (2,False), (52, True), (8, False)]),
        'manual score': 0.7
    },
    {
        'id': '1-5^^^',
        'vals': make_from_runs([(8, False), (16, True), (4, False), (8, True), (16, False),(8, True),
                                   (8, False),
                                   (8, True), (16, False), (8, True), (4, False), (16, True), (8, False)]),
        'manual score': 0.55
    }
]

y_sims['all speech'] = y1
y_sims['one dip'] = y2

scores = {}
score_names = ['man'] + SCORE_NAMES<jupyter_output><empty_output><jupyter_text>Visualize the Sample DataConcise plots of the sample (boolean) data. Red X -> False, Green Dots -> TrueThe labels are a bit cryptic, but identify the various scenarios.<jupyter_code>for name, ys in y_sims.items():

    plt_rows = len(ys)
    plt_cols = 1
    fig, full_axs  = plt.subplots(plt_rows, plt_cols, squeeze=False, figsize=(20,1.5))
    fig.suptitle(f'{name}', x=0, y=1.7)
    axs = full_axs[:,0]
    base_data = ys[0]['vals']

    scores[name] = {}
    for i, data in enumerate(ys):
        y = data['vals']
        data_id = data['id']
        major_ticks = np.linspace(0,len(y),1+len(y)//8)
        minor_ticks = np.linspace(0,len(y),len(y)+1)

        y_val = np.repeat(-i, len(y)) # negative so our base plot is on top
        xT = np.where(y == True)[0]
        xF = np.where(y == False)[0]

        # calculate scores
        score_list = [data['manual score']] + [SCORE_FNS[score_name](base_data, y) for score_name in SCORE_NAMES]
        # store for later use
        scores[name][data_id] = score_list

        # Plot sample data
        title = f'{data_id}'
        # x=-0.1, y=0.5
        axs[i].set_title(f'{title}', position=(-0.03,0), y=-0.6)
        axs[i].set_xticks(major_ticks)
        axs[i].set_xticks(minor_ticks, minor=True)
        axs[i].set_xticklabels([])
        axs[i].set_yticks([])
        axs[i].set_yticklabels([])
        axs[i].grid(axis='x', color='gray', linestyle='--', linewidth=0.5)

        axs[i].scatter(xT, y_val[xT], marker='o', color='green')
        axs[i].scatter(xF, y_val[xF], marker='x', color='red')


    fig.subplots_adjust(left=0.0,
                    bottom=0.0,
                    right=1.0,
                    top=1.2,
                    wspace=0.,
                    hspace=1.8)<jupyter_output>/Users/dev/Projects/speech-scoring-analysis-ml/scorers.py:90: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))
/Users/dev/Projects/speech-scoring-analysis-ml/scorers.py:90: RuntimeWarning: overflow encountered in exp
  return 1/(1 + np.exp(-x))<jupyter_text>Scorer ResultsAll scoring methods were converted to a scale between 0 (bad) and 1 (good).Caveat: This is an attempt to get a **rough** comparison between different scoring stragies. There are several "hacks"to get the data to be apples to apples, and it's quite possible that those hacks have distorted the comparison.However, as I had a rough idea of what reasonable scores should look like (the manually provided score). I was mainlyinterested in something that generated values similar to my manual scores that could then be implemented as acustom scorer in the actual implementation. No warranties are provided or implied, this is just documentation ofwhat I did and why and setting up a framework, so I can change my mind or re-evaluate things in the future. Scorers:|score id| description ||--|--||man|Manually generated score based on my personal feeling of what a reasonable score would be||conv| Maximum value when convolving the general prediction shape on the general ground truth shape.||conv_var| Same as conv, but with an added penalty based on difference between  the prediction's and ground truth's variance.||tick r2| R2 score on the event duration count rather than the base data || r2 | R2 score for the prediction vs ground truth || prec | Precision score for the prediction vs ground truth || f1 | F1 score for the prediction vs ground truth ||tick mse| MSE score on the event duration count rather than the base data, but with the MSE normalized as a score 0-1 || mse | MSE score for the prediction vs ground truth, but with the MSE normalized as a score 0-1 |<jupyter_code># Plot scores
score_dfs = [pd.DataFrame.from_dict(scores[name], orient='index', columns=score_names) for name in scores.keys()]
scores_rows = len(scores.values())

f2, axs2  = plt.subplots(scores_rows, 1, squeeze=False, figsize=(20,16))

for i in range(len(score_dfs)):
    axs = axs2[:,0]
    score_dfs[i].plot.bar(ax=axs[i])
    display(score_dfs[i])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12472693227920524, 'fraction_numerical': 0.024029959429938626, 'mean_word_length': 3.3015659955257273, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '448060', 'license': 'MIT', 'n_tokens_mistral': 3183, 'n_tokens_neox': 3032, 'n_words': 1070, 'path': 'scorers.ipynb', 'repo_name': 'forforf/boolean-series-scorers', 'size': '9613'}