<jupyter_start><jupyter_text>Count and Coverage Analysis  Declare count and coverage DataFrames.<jupyter_code>counts = pd.DataFrame(df.iloc[:, 2:].sum().reset_index())
counts.columns = ["method_name", "count"]
coverage = pd.DataFrame((~(df.iloc[:, 2:] == 0)).sum().reset_index())
coverage.columns = ["method_name", "coverage"]
coverage["coverage"] = coverage["coverage"] / len(df)
counts.nlargest(25, "count")
counts.nsmallest(25, "count")
coverage.nlargest(25, "coverage")
coverage.nsmallest(25, "coverage")
coverage.to_csv("coverage.csv")
middle_core_length = len(counts[counts["count"] > 2])
print(
    "If we take out functions that have not been used by more than two notebooks,"
    + f" we are left with a very large middle core of {middle_core_length} functions."
)
counts[counts["count"] > 2].nsmallest(15, "count")<jupyter_output><empty_output><jupyter_text>Visualize the count and coverage as histograms.<jupyter_code>import matplotlib.pyplot as plt

fig, axes = plt.subplots(nrows=2, ncols=1)
counts.hist(bins=50, ax=axes[0])
coverage.hist(bins=50, ax=axes[1])<jupyter_output><empty_output><jupyter_text>Breakdown of coverage percentage<jupyter_code>coverage["bins"] = pd.cut(
    coverage.coverage,
    bins=(0, 0.01, 0.05, 0.1, 0.65),
    labels=["0-1%", "1-5%", "5-10%", "10-65%"],
    right=False,
)
coverage["bins"].value_counts()<jupyter_output><empty_output><jupyter_text>Number of functions in a notebook<jupyter_code>num_func = df.iloc[:, 2:-1].sum(axis=1)
num_func.mean()
num_func.median()
num_func.max()<jupyter_output><empty_output><jupyter_text>Cumulative Distribution based on Rank The idea behind computing cumulative coverage is that we want to find the _least_ common function for each script. If we know the least common function for each script, then we know at what point our cumulative coverage will cover that script entirely.The algorithm is:1. Sort the columns by descreasing coverage (most to least common).2. For each row, find the rank (rank 1 = most common) of the least common function, which is the `least_common_rank`.3. Group by and count the `least_common_ranks` of all columns. This gives us a mapping of `least_common_rank` to `count`. When sorted by rank, the mapping of `least_common_rank` to `count` tells us, for each function __x__ (if iterating in increasing rank) how many __additional__ scripts are covered by __x__.4. With the mapping sorted by rank, we compute the cumulative sum, which gives us a mapping of `least_common_rank` to `cumulative_count` where `cumulative_count` is now the number of scripts that are covered by `least_common_rank` functions or __less__. As a sanity check, the `cumulative_count` of the least common function should be equal to the total number of scripts.5. With the cumulative sum mapping, we can divide each `count` by the total number of scripts to get a cumulative percentage.<jupyter_code># (1) Get the top functions, sorted.
ordered_top_funcs = list(
    coverage.sort_values("coverage", ascending=False)["method_name"]
)
print(len(ordered_top_funcs))

top_funcs_df = df[ordered_top_funcs]
top_funcs_df
"""
Helper that returns the rank of the least common function
that is in a row/script. This rank corresponds to the rank
where the entire script's functions are covered.
"""


def least_common_function_rank(row):
    for i, val in reversed(list(enumerate(row))):
        if val != 0:
            return int(i + 1)
# (2) For each row, find the rank (rank 1 = most common) of the least common function, which is the least_common_rank.
least_common_ranks = top_funcs_df.apply(
    lambda row: least_common_function_rank(row), axis=1
)
least_common_ranks
"""
(3) Group by and count the least_common_ranks of all columns. This gives us a mapping of
least_common_rank to count. When sorted by rank, the mapping of least_common_rank to
count tells us, for each function x (if iterating in increasing rank) how many 
additional scripts are covered by x.

NOTE: This does not have to be monotonically increasing.
Imagine the function rank is: 1. read_csv 2. plot 3. mean 4. to_csv and there are
only two types of users, ones who use read_csv and plot, and ones who use read_csv,
mean, to_csv. read_csv is the most common function, used in all scripts. However, its
least_common_ranks_count would be 0, since everyone who uses read_csv also uses
another less common function.
"""
least_common_ranks_count = least_common_ranks.groupby(least_common_ranks).count()
least_common_ranks_count
"""
(4) With the mapping sorted by rank, we compute the cumulative sum, which gives 
us a mapping of least_common_rank to cumulative_count where cumulative_count is
now the number of scripts that are covered by least_common_rank functions or 
less. As a sanity check, the cumulative_count of the least common function
should be equal to the total number of scripts.
"""

least_common_ranks_cumulative_count = (
    least_common_ranks_count.cumsum()
)  # Get the cumulative sum of the series.
print(least_common_ranks_cumulative_count.iloc[-1])
assert (
    least_common_ranks_cumulative_count.iloc[-1] == df.shape[0]
), "The cumulative sum of the least common function should be equal to the number of rows."
least_common_ranks_cumulative_count
"""
(5) With the cumulative sum mapping, we can divide each count
by the total number of scripts to get a cumulative percentage.
"""

coverage_rank_cumulative_perc = least_common_ranks_cumulative_count / df.shape[0] * 100
coverage_rank_cumulative_perc
k = 3
pct = coverage_rank_cumulative_perc.iloc[k]
print(f"Top k func: {pct:.2f}%, {int(pct*df.shape[0]/100)} notebooks")
k = 50
pct = coverage_rank_cumulative_perc.iloc[k]
print(f"Top k func: {pct:.2f}%, {int(pct*df.shape[0]/100)} notebooks")
k = 179
pct = coverage_rank_cumulative_perc.iloc[k]
print(f"Top k func: {pct:.2f}%, {int(pct*df.shape[0]/100)} notebooks")
import seaborn as sns

sns.set_theme()
coverage_rank_cumulative_perc.plot()

y_pt = coverage_rank_cumulative_perc.iloc[50]

plt.xlabel("Top k pandas functions")
plt.ylabel("% of notebooks completely covered")
plt.annotate(
    f"Top 50 pandas functions only \ncovers ~{int(y_pt)}% of notebook",
    xy=(50, y_pt),
    xycoords="data",
    xytext=(57, y_pt - 25),
    textcoords="data",
    arrowprops=dict(facecolor="black", shrink=0.05),
)

plt.hlines(y_pt, xmin=0, xmax=50, linestyle="dotted", color="red")
plt.vlines(50, ymin=0, ymax=y_pt, linestyle="dotted", color="red")
plt.plot(50, y_pt, marker=".", markersize=10, color="red")

y_pt = 99
x_pt = 177
plt.hlines(y_pt, xmin=0, xmax=x_pt, linestyle="dotted", color="brown")
plt.vlines(x_pt, ymin=0, ymax=y_pt, linestyle="dotted", color="brown")
plt.plot(x_pt, y_pt, marker=".", markersize=10, color="brown")
plt.annotate(
    f"170+ functions\n required to\n support 99%+\n notebooks",
    xy=(x_pt, y_pt),
    xycoords="data",
    xytext=(x_pt + 10, y_pt - 35),
    textcoords="data",
    arrowprops=dict(facecolor="black", shrink=0.05),
)


plt.xlim(0, 250)
plt.ylim(0, 105)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11161926539945691, 'fraction_numerical': 0.021294840645991138, 'mean_word_length': 5.546304957904583, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '326486', 'license': 'Apache-2.0', 'n_tokens_mistral': 2508, 'n_tokens_neox': 2329, 'n_words': 752, 'path': 'notebook/2-pandas-usage-analysis.ipynb', 'repo_name': 'ponder-org/pandas-API-analysis', 'size': '6997'}