<jupyter_start><jupyter_text>Alignment Quality - Ext. Data Figure 2This notebook creates the plots presented in Ext. Data Figure 2 of the FlyWire paper.<jupyter_code>import os
import dill
import warnings
import pandas as pd
import numpy as np

warnings.filterwarnings("ignore")
from flywirepaper import plotting<jupyter_output><empty_output><jupyter_text>Define relevant paths and create directories<jupyter_code>data_dir = f"{os.path.realpath('')}/../data"
figure_dir = f"{os.path.realpath('')}/../figures/ext_figure_2/"
figure_source_dir = f"{os.path.realpath('')}/../figures/ext_figure_source_2/"

if not os.path.exists(figure_dir):
    os.makedirs(figure_dir)
    
if not os.path.exists(figure_source_dir):
    os.makedirs(figure_source_dir)<jupyter_output><empty_output><jupyter_text>Loading and relative CPC This csv contains precomputed chunked pearson correlation (CPC) measurements between neighboring slices for both the realigned volume (v14.1, "aligned" in the table below) and the original volume (v14, "raw" in the table below). The table below contains the summed CPC values per slice.<jupyter_code>cpc_df = pd.read_csv(f"{data_dir}/fafb_v141_v14_delta_cpc.csv")
cpc_df = cpc_df.iloc[4:-1] # We exlude slices without data
cpc_df.head(5)<jupyter_output><empty_output><jupyter_text>Here, we calculate the relative cpc change:<jupyter_code>cpc_df["rel_cpc_change"] = (cpc_df["aligned sum"] - cpc_df["raw sum"]) / cpc_df["raw sum"]<jupyter_output><empty_output><jupyter_text>Due to missing slices, summed CPCs can be zero leading to nan values after the division above.<jupyter_code>cpc_df = cpc_df[~np.isnan(cpc_df["rel_cpc_change"])]<jupyter_output><empty_output><jupyter_text>a: relative CPC change per slice<jupyter_code>plotting.scatter_plot(cpc_df["z"], 
                      cpc_df["rel_cpc_change"],
                      figsize=(12, 6), marker_size=2,
                      yrange=[-.1, .5],
                      xlabel="z section", 
                      ylabel="Relative CPC improvement",
                      path=f"{figure_dir}/total_cpc_over_sections.png")<jupyter_output>findfont: Font family ['sans-serif'] not found. Falling back to DejaVu Sans.
findfont: Font family ['sans-serif'] not found. Falling back to DejaVu Sans.<jupyter_text>b: relative CPC change as histogram<jupyter_code>plotting.hist_plot(cpc_df["rel_cpc_change"], 
                   figsize=(5, 6), 
                   nbins=200, 
                   horizontal_lines=[0],
                   normalize=False,
                   ylabel="Relative CPC improvement", 
                   xlabel="Section count",
                   orientation="horizontal", 
                   yrange=[-.1, .5],
                   path=f"{figure_dir}/total_cpc_over_sections_hist_h.png")
cpc_df.to_csv(f"{figure_source_dir}/ab_source.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11212653778558876, 'fraction_numerical': 0.010896309314586995, 'mean_word_length': 3.735440931780366, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '466481', 'license': 'Apache-2.0', 'n_tokens_mistral': 989, 'n_tokens_neox': 931, 'n_words': 198, 'path': 'flywirepaper/Alignment Quality - Ext. Data Figure 2.ipynb', 'repo_name': 'seung-lab/FlyWirePaper', 'size': '2845'}