<jupyter_start><jupyter_text>IntroductionIn this notebook, I am going to compare the use of histograms and ECDFs in interpreting data.Histograms are probably the most popular distribution visualization tool. But it comes fraught with one major hidden danger that can be decieving: binning bias.Using the default binning in matplotlib, we get the following plot for draws from a standard normal distribution:<jupyter_code># Set random seed for deterministic reproducibility.
np.random.seed(42)

# Generate a single Gaussian distribution, but with 
data = np.random.normal(loc=0, scale=1, size=(20))
plt.hist(data)
plt.show()<jupyter_output><empty_output><jupyter_text>If we increased the number of bins, each bin gets smaller:<jupyter_code>plt.hist(data, bins=100)
plt.show()<jupyter_output><empty_output><jupyter_text>Suddenly, this standard normal distribution looks a bit more like a Laplace distribution centered on -0.5 - which we know is incorrect!Moreover, it's difficult to intuit important statistical properties of the data: e.g. the range (min-max), inter-quartile range (25th-75th percentile), the median. They are simply not immediately visible from the histogram.What if, instead, we looked at the ECDF of the data?<jupyter_code>def ecdf(data):
    x, y = np.sort(data), np.arange(1, len(data)+1) / len(data)
    return x, y
x, y = ecdf(data)
p = [25, 50, 75]
fractions = np.array(p)/100
percs = np.percentile(data, p, interpolation='lower')
plt.scatter(x, y, label='data')
plt.vlines(percs, min(y), fractions, linestyles='--')
plt.hlines(fractions, min(x), percs, linestyles='--')<jupyter_output><empty_output><jupyter_text>A few things are immediately obvious! We can see read off the range approx. $(-1.0, 0.5)$ as containing approximately 50% of the probability density associated with the data.<jupyter_code>x, y = ecdf(data)
plt.scatter(x, y, label='data')
x_n, y_n = ecdf(np.random.normal(0, 1, size=20000))
plt.plot(x_n, y_n, color='orange', label='normal', ls='--')
x_l, y_l = ecdf(np.random.laplace(-0.5, 1, size=20000))
plt.plot(x_l, y_l, color='red', label='laplace', ls='--')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>By comparing the ECDF of the data against a standard normal CDF and a shifted laplace distribution, we can immediately see that the shifted laplace might have a poorer distributional fit compared to the standard normal distribution.  Bimodal DistributionsBimodal distributions can be easily discovered in an ECDF. It manifests as a plateau in the data.Let's create a bimodal gaussian mixture, which are well-separated.<jupyter_code>mode1 = np.random.normal(loc=0, scale=1, size=1000)
mode2 = np.random.normal(loc=4, scale=1, size=100)
data_bimodal = np.concatenate([mode1, mode2])
fig = plt.figure(figsize=(6, 4))
ax1 = fig.add_subplot(211)
ax2 = fig.add_subplot(212)

ax1.hist(data_bimodal)
ax1.set_title('histogram')
x, y = ecdf(data_bimodal)
ax2.set_title('ecdf')
ax2.plot(x, y)
dy = y[1:] - y[:-1]
idxs = np.argsort(dy)
idxs = idxs[idxs > 820]
ax2.vlines(x=x[idxs[0]], ymin=min(y), ymax=y[idxs[0]])
ax2.hlines(y=y[idxs[0]], xmin=min(x), xmax=x[idxs[0]])
plt.tight_layout()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12774639045825487, 'fraction_numerical': 0.027306967984934087, 'mean_word_length': 6.0198237885462555, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '497214', 'license': 'MIT', 'n_tokens_mistral': 1198, 'n_tokens_neox': 1104, 'n_words': 339, 'path': 'Untitled.ipynb', 'repo_name': 'ericmjl/ecdf-guide', 'size': '3186'}