<jupyter_start><jupyter_text>LIVECell Fluorescence cell count benchmarkThis notebook contains a reference implementation of the evaluation of the fluorescence cell count benchmark in "LIVECell - A large-scale dataset for label-free live cell segmentation" by Edlund et. al. Given data of predicted and fluorescence-based cell count, the evaluation consists of two parts:1. R2 between predicted and fluorescence-based counts in images with fewer than 1600 cells per image (roughly corresponding to full confluency).2. The point which the linear relationship breaks. This test works by comparing the residuals of a linear vs. a non-linear regression model of the fluorescence-based counts as a function of the predicted ones.<jupyter_code>import pandas as pd
import numpy as np
import ipywidgets
from IPython.core.display import display
import matplotlib.pyplot as plt

from sklearn.metrics import r2_score
from scipy import stats
from sklearn.linear_model import LinearRegression
from sklearn.neighbors import KNeighborsRegressor<jupyter_output><empty_output><jupyter_text>First we define our functions.1. `get_counts_from_excel_file` reads the counts from the specific Excel-file format we used for the manuscripts. This is preferrably replaced by whatever format you like.2. `linearity_cutoff_test` contains the test for when linearity breaks.<jupyter_code>def get_counts_from_excel_file(sheet_name, excel_file):
    """ Load data from Excel-file and flatten to 1D-arrays. """
    
    sheet = excel_file.parse(sheet_name, index_col=1)
    sheet = sheet.rename(columns={sheet.columns[0]: 'time'})

    nc_cols = [col for col in sheet.columns if 'Image' in col]
    model_cols = [col for col in sheet.columns if not col in nc_cols and col != 'time']

    nc_flat = sheet[nc_cols].values.flatten()
    model_flat = sheet[model_cols].values.flatten()

    nc_is_nan = np.isnan(nc_flat)
    model_is_nan = np.isnan(model_flat)
    any_is_nan = nc_is_nan | model_is_nan

    nc_flat = nc_flat[~any_is_nan]
    model_flat = model_flat[~any_is_nan]
    return nc_flat, model_flat


def linearity_cutoff_test(
    fluorescence_counts,
    prediction_counts,
    start_threshold = 500,
    increment = 1,
    p_cutoff = 1e-5, 
    n_neighbors=5
):
    """ Test when linearity breaks. 
    
    While the maximum number of objects per image is increased incrementally, 
    the fluorescence-based counts are regressed as a function of the predicted
    counts using linear regression and KNN-regression (default 5 neighbors). 
    
    Then the null hypothesis of equally sized residuals is tested using a 
    Levene's test. If the null hypothesis is rejected, the fit is considered
    non-linear. 
    
    Parameters
    ----------
    fluorescence_counts : array
        1D-array of ints containing fluorescence-based counts
    prediction_counts : array
        1D-array ints containing predicted counts
    start_threshold : int
        Maximum number of objects per image to start incrementing from (default 500)
    increment : int
        Number of objects per image to increment with (default 1)
    p_cutoff : float
        p-value cutoff to reject null hypothesis (default 1E-5)
    n_neighbors : int
        Number of neighbors in KNN-regression.
        
    Returns
    -------
    int
        Number of objects per image where null hypothesis was first rejected.
    """

    for test_threshold in range(start_threshold, int(nc_flat.max()), increment):
        below_test_threshold = fluorescence_counts < test_threshold
        y = fluorescence_counts[below_test_threshold]

        prediction_counts_2d = np.atleast_2d(prediction_counts[below_test_threshold]).T
        linear_model = LinearRegression().fit(prediction_counts_2d, y)
        knn_model = KNeighborsRegressor(n_neighbors).fit(prediction_counts_2d, y)
        linear_pred_nc = linear_model.predict(prediction_counts_2d)
        knn_pred_nc = knn_model.predict(prediction_counts_2d)

        knn_residal = (y - knn_pred_nc)
        linear_residual = (y - linear_pred_nc)
        test_result = stats.levene(knn_residal, linear_residual)
        if test_result.pvalue < p_cutoff:
            break
            
    return test_threshold<jupyter_output><empty_output><jupyter_text>Pick file to analyze.<jupyter_code>uploader = ipywidgets.FileUpload(accept='.xlsx', multiple=False)
display(uploader)<jupyter_output><empty_output><jupyter_text>Run tests<jupyter_code>if not uploader.value:
    print('Pick file using file-picker first')
else:
    first_key = next(key for key in uploader.value)
    excel_file = pd.ExcelFile(uploader.value[first_key]['content'], engine='openpyxl')
    sheet_names = excel_file.sheet_names

    threshold = 1600

    for sheet_name in sheet_names:
        cell_type, model_name = sheet_name.split('-', 1)
        print(f'{cell_type} - {model_name} model')
        nc_flat, model_flat = get_counts_from_excel_file(sheet_name, excel_file)

        below_threshold = nc_flat < threshold
        r2 = r2_score(nc_flat[below_threshold], model_flat[below_threshold])
        linearity_cutoff = linearity_cutoff_test(nc_flat, model_flat)
        print(f'R2 below {threshold} objects = {r2:.3f}')
        print(f'Linearity break, n objects = {linearity_cutoff}')
        print()<jupyter_output>A549 - Anchor-free model
R2 below 1600 objects = 0.980
Linearity break, n objects = 2031

A549 - Anchor-based model
R2 below 1600 objects = 0.985
Linearity break, n objects = 1403

A172 - Anchor-free model
R2 below 1600 objects = 0.942
Linearity break, n objects = 1948

A172 - Anchor-based model
R2 below 1600 objects = 0.977
Linearity break, n objects = 1328
{'fraction_non_alphanumeric': 0.07137796670208997, 'fraction_numerical': 0.01930570315267446, 'mean_word_length': 3.793718166383701, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '613846', 'license': 'MIT', 'n_tokens_mistral': 1858, 'n_tokens_neox': 1698, 'n_words': 521, 'path': 'code/Fluorescence cell count evaluation.ipynb', 'repo_name': 'sartorius-research/LIVECell', 'size': '5646'}