<jupyter_start><jupyter_text>NumericalThere are many more options when it comes to analyzing a numerical variable as most, if not all, descriptive statistics were designed around numerical variables. The plotting options are greater as well. Still, just because something is numeric, doesn't mean that descriptive statistics automatically make sense. Consider US annual GDP from 1901-2018, does it make sense to calculate the *mean* GDP? What does that even signify? What if the mean latitude and longitude is in the middle of a lake?With those caveats in mind, there are a host of things we can do with numerical data, which are usually measurements or counts of some kind.We start by generating some synthetic data:<jupyter_code>data = pd.DataFrame(np.concatenate([stats.norm.rvs( 90.0, 9.0, 100), stats.norm.rvs(120.0, 12.0, 100)]))
data.columns = ["X"]<jupyter_output><empty_output><jupyter_text>As previously mentioned, the EDA of a variable should begin with a description of the variable and any relevant domain knowledge. You should also add anything from the problem or question that might specifically relevant to this variable. You can then start with parametric and non-parametric descriptive statistics. Absent domain knowledge, it's a bit difficult to say anything about the descriptive statistics. However, you can at least make note of important relationships and notice if the `min` or `max` are "weird" and if there are missing values.Pandas has a `describe` function that calculates many of these descriptive statistics. The downside is that the function returns a Series which makes accessing individual values somewhat tiresome. If `describe` doesn't return what you want, you can always calculate your own descriptive statistics. Overtime, you may want to develop a function that calculates descriptive statistics that you find useful and shows them in a format you like.<jupyter_code>ds = data[ "X"].describe()
pd.DataFrame(ds)<jupyter_output><empty_output><jupyter_text>We do not have missing values (the count is the same as n). The min is 73.6 and the max is 149.3, neither of which suggest something is "weird" with the data. The median (50%) is slightly lower than the mean which suggests there is a leftward skew (and some larger values on the right).Here are a few additional descriptive statistics:<jupyter_code>def ex(stats, stat):
    return stats.loc[[stat]].values[0]

print("Range=", ex(ds, "max") - ex(ds, "min"))
print("IQR  =", ex(ds, "75%") - ex(ds, "25%"))
print("COV  =", ex(ds, "std")/ex(ds, "mean"))<jupyter_output>Range= 89.10472535412697
IQR  = 32.19147921637675
COV  = 0.17758271003288673<jupyter_text>The Coefficient of Variation (*not* the variance) is 17.2%.Discussing descriptive statistics can be difficult without domain knowledge or an actual problem or question but at the very least you can walk through the individual statistics.  Box PlotWhen we turn to graphical techniques, the options open up. There is a graphical equivalent of the "Tukey's Five" descriptive statistics called the boxplot. Boxplots can have a lot of configuration options so it's worth looking into them in order to make boxplots that highlight the data you want to highlight in the way you want it highlighted.The box represents the extent of Q1 and Q3. The line in the middle is the median (Q2). If `showfliers` is True, then the "whiskers" extend $1.5 \times IQR$ and any extreme value will be beyond that, if there is one. If `showfliers` is False, the "whiskers" extend to the min and max. Note that this is matplotlib specific, any other library you use may be different.Here are two boxplots, one regular box plot and one with the actual data points added with a bit of "jitter" to handle the problem of overplotting (when many identical or similar observations are plotted to the same location on the chart).<jupyter_code>figure = plt.figure(figsize=(20, 6))

axes = figure.add_subplot(1, 2, 1)
patch = axes.boxplot( data[ "X"], labels=["X"], showfliers=True, patch_artist=True, zorder=1)
eda.restyle_boxplot( patch)
axes.set_title("Distribution of X")
axes.set_ylabel("Values")

axes = figure.add_subplot(1, 2, 2)
patch = axes.boxplot( data[ "X"], labels=["X"], patch_artist=True, zorder=1)
eda.restyle_boxplot( patch)
x = np.random.normal(1, 0.01, size=len(data["X"]))
axes.plot(x, data[ "X"], 'o', alpha=0.4, color="DimGray", zorder=2)
axes.set_title("Distribution of X")
axes.set_ylabel("Values")

plt.show()
plt.close()<jupyter_output><empty_output><jupyter_text>Do not just copy these charts. Everything you do during EDA should be done with some thought. Why are you doing this? Write it down in the notes.Whether or not you choose to show the data will depend on a number of factors include the size of the data set and whether or not it tells a story. It is possible to add so much data that there is little benefit to showing the data or the data may follow the boxplot so closely that adding the data is not informative. In other cases, the data may deviate from what is being described by the descriptive statistics that it is worth showing the data. We will see a case of this later.Of course, you may have to do it to see if it tells a story. That's the **E** part in **E**DA. But remember the criticism that EDA is isn't rigorous. Be methodical. Each chart is an experiment; make notes about each experiment, whether or not you learn something and if you do, what you learn. There's nothing worse than coming back to a chart in two weeks and thinking, "huh, this is interesting but why did I do it?".  HistogramsWe have already seen the venerable histogram. It represents a different way of viewing the distribution of the data as the approximation of a probability density. Density? Yes. We've talked about probability distributions such as $P(A)$ when $A$ is a set of values (a categorical variable). Technically, this is a probability *mass*.When we switch to $X$ as a numeric value, we run into a problem. Theoretically, a numeric value has an infinite number of values (depends on the resolution of the measuring device and the ability of computers to represent floating point numbers but we can ignore these). Because there are an infinite number of values, the probability of any given value is 0! Enter densities.A probability *density* represents probability as area under the curve between any two points. The entire area under the curve sums to 1 as any probability distribution must. But we instead of representing the probability of $P(X=x_1)$, the density can tell us $P(x_0 \leqslant X \leqslant x_1)$.The technique was developed by <NAME>. The histogram assigns each observation to a fixed number of "bins" that are then represented as bars as in a bar chart. There are important differences between histograms and bar charts:| Element     | Bar Chart        | Histogram         ||-------------|------------------|-------------------|| Data Type   | Categorical      | Numerical         || Bar Spacing | Space            | No space          || Y-axis      | Counts, Percents | Counts, Densities || Sortable?   | Yes              | No!               |The bars in a Histogram are intrinsically ordered and cannot be reordered. The number (or size) of bins relative to the characteristics of the data can affect the efficacy of the histogram.Histograms can express absolute counts or densities.<jupyter_code>figure = plt.figure(figsize=(20, 6))

axes = figure.add_subplot(1, 2, 1)
axes.hist( data[ "X"], color="DimGray")
axes.set_ylabel( "Count")
axes.set_xlabel( "X")
axes.set_title("Count Histogram of X; default bins")

axes = figure.add_subplot(1, 2, 2)
axes.hist( data[ "X"], density=True, color="DimGray")
axes.set_ylabel( "Density")
axes.set_xlabel( "X")
axes.set_title("Density Histogram of X; default bins")
plt.show()
plt.close()<jupyter_output><empty_output><jupyter_text>When using your tool/library's histogram feature, you should read the documentation to find out how it determines the bin size (if it tells you at all). There are a number of algorithms available that purport to determine "the best" bin widths but you should consider looking at different bin sizes on your own to reveal interesting structure in your data.Not every library works the same way. For example, R takes bin widths whereas Python takes the actual bin edges. If we want to look at bins that are 1 unit in width, we have to calculate those bin edges:<jupyter_code>bins = [70 + i for i in range( 0, 80, 1)]

figure = plt.figure(figsize=(10, 6))

axes = figure.add_subplot(1, 1, 1)
axes.hist( data[ "X"], bins=bins, density=True, color="DimGray")
axes.set_title("Density Histogram of X; 1 unit bins")
axes.set_ylabel( "Density")
axes.set_xlabel( "X")
plt.show()
plt.close()<jupyter_output><empty_output><jupyter_text>This changes the visual shape of the distribution quite a bit but you have to be careful when comparing this histogram to other histograms of the same variable because the y axis has changed. If you concentrate on the *shape*, tt might be very useful to plot your histograms as a matrix of charts so you can easily compare them side by side.There are number of formulas for calculating the "best" number of bins or the bin width. They are not the same although you can perform approximate conversions between them:$$k=\lceil\frac{max~x - min~x}{h}\rceil$$where k is the number of bins and h is the bin width or:$$h=\lfloor\frac{k}{max~x - min~x}\rfloor$$The binning rules are:** Square Root Rule** $k = \sqrt{n}$** Sturges' Formula** $k = \lceil log_2~n + 1\rceil$This is the default in R's `ggplot2` but not necessarily the "best" choice.** Rice Rule** $k = \lceil 2n^{1/3}\rceil$** Doane's Formula**$k = 1 + log_2~n + log_2(1 + \frac{\lvert g_1 \rvert}{\sigma_{g_1}})$$\sigma_{g_1} = \sqrt{\frac{6(n-2)}{(n+1)(n+3)}}$** Scott's Normal Reference Rule**$h=\frac{3.5\hat{\sigma}}{n^{1/3}}$** Freedman-Diaconis' Rule**$h=2\frac{IQR(x)}{n^{1/3}}$where IQR is the interquartile range (Q3 - Q1).If you want to get a sense of the differences, you can easily make up some synthetic data and plot a histogram of it using each formula.Despite there not being a "best" choice, Freedman-Diaconis' Rule is often used. Here's an example:<jupyter_code>def freeman_diaconis( data):
    quartiles = stats.mstats.mquantiles( data, [0.25, 0.5, 0.75])
    iqr = quartiles[2] - quartiles[ 0]
    n = len( data)
    h = 2.0 * (iqr/n**(1.0/3.0))
    return int( h)

h = freeman_diaconis( data[ "X"])
print("Freeman Diaconis bin width is {}".format(h))
bins = [70 + i for i in range( 0, 80, h)]

figure = plt.figure(figsize=(10, 6))

axes = figure.add_subplot(1, 1, 1)
axes.hist( data[ "X"], bins=bins, density=True, color="DimGray")
axes.set_title("Density Histogram of X; Freeman Diaconis bins")
axes.set_ylabel( "Density")
axes.set_xlabel( "X")

plt.show()
plt.close()<jupyter_output>Freeman Diaconis bin width is 11
{'fraction_non_alphanumeric': 0.09449760765550239, 'fraction_numerical': 0.018310636731689364, 'mean_word_length': 4.611254517294785, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '222090', 'license': 'MIT', 'n_tokens_mistral': 3320, 'n_tokens_neox': 3024, 'n_words': 1509, 'path': 'fundamentals_2018.9/eda/matrix/numerical.ipynb', 'repo_name': 'topseer/APL_Great', 'size': '10874'}