<jupyter_start><jupyter_text>Load and Synthesize Data from CSVThis process loads each summary CSV file, creates a few derived metrics, and then merges each system's CSV into a single global dataset that can be sliced and diced by system, benchmark, or any other way.  We are now caching the processed CSV in HDF5 format to speed up initial data ingest at the beginning of each analysis.  Delete the `CACHE_FILE` to re-generate this cache (e.g., when the contents of the CSV are updated).<jupyter_code>filtered_df = abcutils.sc18paper.load_dataset()<jupyter_output><empty_output><jupyter_text>Demonstrate a Single Test PlatformLook at one combination of (compute system, file system, benchmark) to show what this UMAMI analysis can do. Define Input Parameters<jupyter_code>plot_metric = 'darshan_normalized_perf_by_max'

group_by = ['_test_platform', '_benchmark_id']

print("plot_metric =", abcutils.CONFIG['metric_labels'].get(plot_metric, plot_metric))
print("date_start =", abcutils.sc18paper.DATE_START.isoformat())
print("date_end =", abcutils.sc18paper.DATE_END.isoformat())
# Width of simple moving average (SMA) short/long windows
short_window = pandas.Timedelta(days=14)
long_window = pandas.Timedelta(days=49)

print("Short window will average over %s measurements at a time" % short_window)
print("Long window will average over %s measurements at a time" % long_window)<jupyter_output><empty_output><jupyter_text>Classification We actually only attempt to correlate against a fixed subset of the features.  This is a time-saving measure; the features in `umami_row_order` were determined by performing an unguided correlation against everything and only selecting those features which showed some degree of correlation _and_ were not degenerate of other features.<jupyter_code>CORRELATION_METRICS = abcutils.CONFIG['umami_row_order']
CORRELATION_METRICS<jupyter_output><empty_output><jupyter_text>Generate or load calculated contributorsThis can take an inconvenient amount of time, so we cache the results to `contributors.hdf`<jupyter_code>all_contributors = None

# Load the cached contributors list if available
try:
    all_contributors = pandas.read_hdf('contributors.hdf5', 'contributors')
except IOError:
    pass

# Otherwise recalculate the contributors list
if all_contributors is None:
    grouped_df = filtered_df.groupby(by=group_by)
    for group in grouped_df.groups:
        example_df = grouped_df.get_group(group)

        intercepts = abcutils.features.sma_intercepts(example_df,
                                                      plot_metric,
                                                      short_window=short_window,
                                                      long_window=long_window)

        loci = abcutils.features.generate_loci_sma(example_df,
                                                   plot_metric,
                                                   mins=True,
                                                   maxes=False,
                                                   short_window=short_window,
                                                   long_window=long_window)
        regions = list(abcutils.features.intercepts_to_region(example_df, intercepts))

        for region in regions:
            contributors = abcutils.classify.identify_contributors(region=region,
                                                    target_column=plot_metric,
                                                    target_index=region[plot_metric].idxmin(),
                                                    correlate_columns=CORRELATION_METRICS,
                                                    want_good=False,
                                                    classifier='minmax')
            if all_contributors is None:
                all_contributors = contributors
            else:
                all_contributors = pandas.concat((all_contributors, contributors))

    # Cache the contributors list for the next time
    all_contributors.index = numpy.arange(len(all_contributors))
    all_contributors.to_hdf('contributors.hdf5', key='contributors', mode='w', format='fixed', complevel=9, complib='zlib')
print("Number of target indices:", len(all_contributors['target_index'].unique()))<jupyter_output><empty_output><jupyter_text>Apply filters to remove very high p-value measurements from the dataset.  These cause problems when performing significance testing later on, since they dilute the significance of the results.<jupyter_code># Also drop any contributors who lack statistical confidence because of duplicate values
significant_contributors = all_contributors[all_contributors['pvalue'] < 0.10]
print("Discarding %d contributors with p-values < 0.10" % (len(all_contributors) - len(significant_contributors)))

# Keep all data and let p-values speak for themselves
#significant_contributors = all_contributors

print("Number of contributors remaining:", len(significant_contributors))
print("Number of target indices ('bad' jobs):", len(significant_contributors['target_index'].unique()))
print("Number of unclassified jobs:", (all_contributors.groupby(['target_index']).sum()['target_metric_matches'] < 1.0).sum())<jupyter_output><empty_output><jupyter_text>The following table is the total number of observations broken down by file system.  For example, `fs_ave_mds_cpu` for `cscratch@cori-knl` = `102.0` means that there were 102 cases where poor performance was observed on `cscratch@cori-knl` while at the same time the `fs_ave_mds_cpu` metric was available.<jupyter_code>metric_observation_counts = pandas.pivot_table(significant_contributors,
                                               values='target_metric_matches',
                                               index=['metric_name'],
                                               columns=['_test_platform'],
                                               aggfunc=lambda x: (~numpy.isnan(x)).sum()).fillna(0.0)
metric_observation_counts<jupyter_output><empty_output><jupyter_text>The following table shows how many of the observations in the previous table were actually implicated (tagged) as being correlated with poor performance.<jupyter_code>tagged_metric_counts = pandas.pivot_table(significant_contributors,
                                          values='target_metric_matches',
                                          index=['metric_name'],
                                          columns=['_test_platform'],
                                          aggfunc=numpy.sum).fillna(0.0)
tagged_metric_counts<jupyter_output><empty_output><jupyter_text>We then divide the number of times each metric was implicated (tagged) as correlating with poor performance by the total number of times that metric was observed on each file system.  The result is the fraction of times each metric was observed to correlate with poor performance on a per-file system basis.<jupyter_code>contributor_distribution = tagged_metric_counts.div(metric_observation_counts.sum(axis=1), axis=0)
contributor_distribution = contributor_distribution.reindex(index=contributor_distribution.sum(axis=1).sort_values(ascending=False).index)
try:
    del contributor_distribution.columns.name
except AttributeError:
    pass
contributor_distribution<jupyter_output><empty_output><jupyter_text>We then take the total number of times each metric was tagged across _all file systems_ (across an entire row in the previous table) and divide it by the total number of observations of that metric to calculate the fraction of observations where each metric was tagged as being correlated with poor performance.<jupyter_code>def draw_stacked_bars(contributor_distribution, legendprops=None):
    _legendprops = {}
    if legendprops:
        _legendprops.update(legendprops)
    
    row_sums = contributor_distribution.sum(axis=1)

    fig, ax = matplotlib.pyplot.subplots(figsize=(8,4))

    contributor_distribution.plot.bar(stacked=True, ax=ax, width=0.90)
    ax.grid()
    ax.set_ylim(0, 0.5)
    ax.set_axisbelow(True)

    xticklabels = [abcutils.CONFIG['umami_rows'].get(x.get_text(), x.get_text()) for x in ax.get_xticklabels()]
    ax.set_xticklabels(xticklabels, rotation=30, ha='right')
    ax.set_xlabel("")
    ax.set_ylabel("Fraction of tests")
    handles, labels = ax.get_legend_handles_labels()
    ax.legend(handles, [abcutils.CONFIG['platform_labels'].get(x, x) for x in labels], **_legendprops)

    for index, x_value in enumerate(ax.get_xticks()):
        ax.annotate("%d%%" % (100.0 * row_sums[index]), xy=(x_value, row_sums[index] + 0.02),
                    ha='center',
                    backgroundcolor='#FFFFFFAA')

    return ax<jupyter_output><empty_output><jupyter_text>Assert confidenceWe use the binomial test to calculate the p-values of each fraction of tests asserted above.<jupyter_code>results = []
for metric in contributor_distribution.index.values: # ['fs_ave_oss_cpu']: #
    prob_success = 1.0
    prob_failure = 1.0

    successes = []
    failures = []

    num_matches = 0
    metric_measures = significant_contributors[significant_contributors['metric_name'] == metric]
    
    for row in metric_measures.itertuples():
        if row.target_metric_matches:
            num_matches += 1
            prob_success *= row.pvalue
            successes.append(row.pvalue)
        else:
            prob_failure *= (1.0 - row.pvalue)
            failures.append(row.pvalue)

    pick_n = num_matches
    out_of = len(metric_measures)
    
    if not successes:
        continue

    # what is the probability that we observe pick_n / out_of jobs with this
    # tagged metric given the probability of encountering a tagged metric
    # if there's no relationship between this metric being tagged and each
    # job's performance?
    #
    # binomial test: assume the null hypothesis is TRUE
    #   1. pick the highest p-value observed for this metric - that is the
    #      case where the null hypothesis is most likely to be true
    #   2. perform the binomial test to see what the odds are of observing
    #      pick_n **or more** tagged metrics if the null hypothesis is true?
    probability = numpy.max(successes)
    pvalue = scipy.stats.binom_test(pick_n,
                                    out_of,
                                    probability,
                                    alternative='greater')
    
    result = collections.OrderedDict({})
    result['metric'] = metric
    result['pick_n'] = pick_n
    result['out_of'] = out_of
    result['probability_used'] = probability
    result['calculated_pvalue'] = pvalue

    results.append(result)

binomial_results = pandas.DataFrame.from_dict(results).set_index('metric')
binomial_results
ax = draw_stacked_bars(contributor_distribution.loc[binomial_results.index])
# relabel legend
for text in ax.legend().get_texts():
    text_str = text.get_text()
    text.set_text(abcutils.config.CONFIG['platform_labels_public'].get(text_str, text_str))

ax.legend(fontsize="small")<jupyter_output><empty_output><jupyter_text>Shade off the statistically insignificant metrics<jupyter_code># Build a mapping from metrics to rectangles
reverse_metric_map = {}
for key, val in abcutils.CONFIG['umami_rows'].items():
    reverse_metric_map[val] = key
    if 'CF' in val:
        reverse_metric_map[val.replace(' CF', ' Contention')] = key

# Find all rectangles corresponding to each metric
rectangle_map = {}
xticks = ax.get_xticks()
xticklabels = [x.get_text() for x in ax.get_xticklabels()]
for child in ax.get_children():
    if isinstance(child, matplotlib.patches.Rectangle) and child.get_width() == 0.9:
        child_x = int(round(child.xy[0] + child.get_width() / 2))
        key = reverse_metric_map[xticklabels[child_x]]
        if key not in rectangle_map:
            rectangle_map[key] = []
        rectangle_map[key].append(child)

# Actually apply a grey box over the box of each metric that is not statistically significant
for row in binomial_results.itertuples():
    if row.calculated_pvalue > 0.10:
        for rectangle in rectangle_map[row.Index]:
            rectangle.set_color("#DDDDDD")
            rectangle.set_edgecolor('#DDDDDD')

# relabel legend
new_labels = []
for text in ax.legend().get_texts():
    text_str = text.get_text()
    new_text_str = abcutils.config.CONFIG['platform_labels_public'].get(text_str, text_str)
    print("Replacing %s with %s" % (text_str, new_text_str))
    new_labels.append(new_text_str)

ax.legend(labels=new_labels, fontsize="small")
ax.xaxis.grid(False)
ax.get_figure().savefig('figs/contributors-bad-by-system-grey.pdf', bbox_inches='tight', transparent=True)
ax.get_figure()<jupyter_output><empty_output><jupyter_text>Also provide a less confusing version of the plot without the per-file system resolution.<jupyter_code>for row in binomial_results.itertuples():
    if row.calculated_pvalue > 0.10:
        for rectangle in rectangle_map[row.Index]:
            rectangle.set_color("#DDDDDD")
            rectangle.set_edgecolor('#DDDDDD')
    else:
        for rectangle in rectangle_map[row.Index]:
            rectangle.set_color("C0")
            rectangle.set_edgecolor('C0')
ax.get_legend().set_visible(False)
ax.get_figure().savefig('figs/contributors-bad-grey.pdf', bbox_inches='tight')
ax.get_figure()<jupyter_output><empty_output><jupyter_text>Then only show the metrics that are statistically significant at all.<jupyter_code>ax = draw_stacked_bars(contributor_distribution.loc[(binomial_results['calculated_pvalue'] < 0.10).values],
                      legendprops={
                          'loc': 'upper right',
                          'bbox_to_anchor': (1.01, 1.03),
                          'labelspacing': 0.4
                      })
ax.set_xticklabels([x.get_text().replace(' CF', '\nContention') for x in ax.get_xticklabels()], rotation=30)
ax.xaxis.grid(False)
ax.get_figure().savefig('figs/contributors-bad-by-system.pdf', bbox_inches='tight')
binomial_results.sort_values('calculated_pvalue')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08928571428571429, 'fraction_numerical': 0.006331815594763802, 'mean_word_length': 3.193615751789976, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 55, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 45, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '43193', 'license': 'BSD-3-Clause-LBNL', 'n_tokens_mistral': 4076, 'n_tokens_neox': 3858, 'n_words': 1009, 'path': 'sc18_figure10.ipynb', 'repo_name': 'NERSC/tokio-abcutils', 'size': '14056'}