<jupyter_start><jupyter_text>This Colab demonstrates how to use the FuzzBench analysis library to show experiment results that might not be included in the default report.   Get the dataEach report contains a link to the raw data,  e.g., see the bottom of our sample report: https://www.fuzzbench.com/reports/sample/index.html.Find all our published reports at https://www.fuzzbench.com/reports.<jupyter_code>!wget https://www.fuzzbench.com/reports/sample/data.csv.gz<jupyter_output><empty_output><jupyter_text>Get the code<jupyter_code># Install requirements.
!pip install Orange3 pandas scikit-posthocs scipy seaborn
# Get FuzzBench
!git clone https://github.com/google/fuzzbench.git
# Add fuzzbench to PYTHONPATH.
import sys; sys.path.append('fuzzbench')<jupyter_output><empty_output><jupyter_text>Experiment results<jupyter_code>import pandas
from fuzzbench.analysis import experiment_results, plotting
from IPython.display import SVG, Image

# Load the data and initialize ExperimentResults.
experiment_data = pandas.read_csv('data.csv.gz')
fuzzer_names = experiment_data.fuzzer.unique()
plotter = plotting.Plotter(fuzzer_names)
results = experiment_results.ExperimentResults(experiment_data, '.', plotter)<jupyter_output><empty_output><jupyter_text>Top level results<jupyter_code>results.summary_table<jupyter_output><empty_output><jupyter_text>Rank by median on benchmarks, then by average *rank*<jupyter_code># The critial difference plot visualizes this ranking
SVG(results.critical_difference_plot)
results.rank_by_median_and_average_rank.to_frame()<jupyter_output><empty_output><jupyter_text>Rank by pair-wise statistical test wins on benchmarks, then by average rank<jupyter_code>results.rank_by_stat_test_wins_and_average_rank.to_frame()<jupyter_output><empty_output><jupyter_text>Rank by median on benchmarks, then by avereage normalized score<jupyter_code>results.rank_by_median_and_average_normalized_score.to_frame()<jupyter_output><empty_output><jupyter_text>Rank by average rank on benchmarks, then by avereage rank<jupyter_code>results.rank_by_average_rank_and_average_rank.to_frame()<jupyter_output><empty_output><jupyter_text>Benchmark level results<jupyter_code># List benchmarks
benchmarks = {b.name:b for b in results.benchmarks}
for benchmark_name in benchmarks.keys(): print(benchmark_name)
sqlite = benchmarks['sqlite3_ossfuzz']
SVG(sqlite.violin_plot)
SVG(sqlite.coverage_growth_plot)
SVG(sqlite.mann_whitney_plot)
# Show p values
sqlite.mann_whitney_p_values<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08697378872120731, 'fraction_numerical': 0.0007942811755361397, 'mean_word_length': 8.917322834645669, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 4, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '55566', 'license': 'Apache-2.0', 'n_tokens_mistral': 867, 'n_tokens_neox': 834, 'n_words': 172, 'path': 'analysis/notebooks/example.ipynb', 'repo_name': 'jviszlai/fuzzbench', 'size': '2518'}