<jupyter_start><jupyter_text>Distributions> A brief introduction to distributions in statistics.- toc: true - badges: true- comments: true- categories: [Statistics, EDA] In this blog, we will discuss all about distributions from Data Science & EDA perspective. We will be using "diamonds" dataset from seaborn library.<jupyter_code>import seaborn as sns
import pandas as pd

data = sns.load_dataset('diamonds')
data.head()<jupyter_output><empty_output><jupyter_text>Dataset & basic EDAEvery dataset consists of samples (or records) and for every sample we record multiple features. Here, every sample is a diamond & for every diamond we have recorded *carat, cut, color, clarity, depth, table, price, x, y & z* features as we can see in the table above.Its a good practice to check the shape of the dataset, missing values & unique values, even before we start the exploration process.<jupyter_code>data.shape<jupyter_output><empty_output><jupyter_text>Wow! we have a lot of data . . .<jupyter_code>data.isnull().sum()<jupyter_output><empty_output><jupyter_text>We don't have any missing values. This is good news.Now lets look at unique values . . .<jupyter_code>data.nunique()<jupyter_output><empty_output><jupyter_text>A feature can be either *categorical* or *continuous*.- **Categorical Features**: Finite & Distinct values. Here, `cut`, `color` and `clarity` are categorical features.- **Continuous Features**: Infinite & continuous values.Its extremely important to identify your categorical & continuous features because they require different treatment when it comes to analysis & pre-processing.   Frequency TableNow that we know our categorical & continuous features, lets start the exploration . . . we will start with `cut` feature<jupyter_code>data.cut.values<jupyter_output><empty_output><jupyter_text>There are 53940 values in total & 5 unique values. But this is not very useful. So lets create a **frequency table** out of it<jupyter_code>data.cut.value_counts()<jupyter_output><empty_output><jupyter_text>Now we can clearly see how many diamonds we have for each possible value of *cut*. This is much better than randomly throwing a list of ~54K values at someone. Similarly, lets explore other columns . . .<jupyter_code>data.color.value_counts()
data.clarity.value_counts()<jupyter_output><empty_output><jupyter_text>So far, we have very few unique values. And still we are not able to make such sense out of these *frequency tables*. Imagine, if we have a lot of unique values then it will quickly become unmanageable.  Bar PlotRemember, humans are visual creatures. So lets create a *bar plot* describing above table.<jupyter_code>tmp = data.clarity.value_counts()
p = sns.barplot(x=tmp.index, y=tmp.values)
p.set(xlabel='Clarity', ylabel='Count');<jupyter_output><empty_output><jupyter_text>Again, this is much better. We can clearly see people prefer *VS2* & *SI1* over other category of *clarity*.Using these three things (data, frequency table, & bar plot) we can answer a lot of interesting question about our data. For example:- What is most & least frequent category?- How many more people bought *VS2* compared to *VS1*? Things get really interesting once you start repeating the same process for other categorical features as well. Not just other categories, you can also repeat the process for any subset of data.  Comparing Bar plotsSay We want to see how *clarity* changes for *Ideal* cut of diamonds<jupyter_code>ideal = data[data.cut == "Ideal"]
ideal.shape
tmp = ideal.clarity.value_counts()
p = sns.barplot(x=tmp.index, y=tmp.values)
p.set(xlabel='Clarity', ylabel='Count');<jupyter_output><empty_output><jupyter_text>Now lets, see how *clarity* is distributed for *Good* cut of diamonds.<jupyter_code>good = data[data.cut == "Good"]
good.shape
tmp = good.clarity.value_counts()
p = sns.barplot(x=tmp.index, y=tmp.values)
p.set(xlabel='Clarity', ylabel='Count');<jupyter_output><empty_output><jupyter_text>As you can see most other categories are same but in *Good* cut diamonds, *SI1* is the most popular category where as in *Ideal* cut diamonds, *VS2* is the most popular category.Lets compare both these distributions by plotting them together . . .<jupyter_code>#collapse-hide
tmp1 = pd.DataFrame(ideal.clarity.value_counts())
tmp1.reset_index(inplace=True)
tmp1['cut'] = 'Ideal'
tmp1.columns = ['clarity', 'count', 'cut']

tmp2 = pd.DataFrame(good.clarity.value_counts())
tmp2.reset_index(inplace=True)
tmp2['cut'] = 'Good'
tmp2.columns = ['clarity', 'count', 'cut']

tmp = pd.concat([tmp1, tmp2])

sns.barplot(x=tmp.clarity, y=tmp['count'], hue=tmp.cut);<jupyter_output><empty_output><jupyter_text>Huh 🤔, its pretty obvious that we have many more *Ideal* diamonds than *Good* diamonds. Its not a fair comparison, because of bigger sample size, *Ideal* diamonds are completely overshadowing the *Good* diamonds. For a fair comparison instead of comparing absolute frequency values, we should compare the percentage values.  Probability Mass Functions (PMFs)Count Plots are not good for comparing distributions. Hence we need to normalize them. These plots are know as **Probability Mass functions (PMFs)**.Lets see how we can normalize our frequency tables to make PMF plots<jupyter_code>ideal.clarity.value_counts()/len(ideal)<jupyter_output><empty_output><jupyter_text>pretty simple, right? If you just divide your absolute values by total number of samples (i.e length of the corresponding dataframe) then you will get percentage values. The above table shows that 23.5% of *Ideal* cut diamond has *VS2* clarity.Lets now try to plot these percentage values.<jupyter_code>#collapse-hide
tmp1 = pd.DataFrame(ideal.clarity.value_counts()/len(ideal))
tmp1.reset_index(inplace=True)
tmp1['cut'] = 'Ideal'
tmp1.columns = ['clarity', 'prob', 'cut']

tmp2 = pd.DataFrame(good.clarity.value_counts()/len(good))
tmp2.reset_index(inplace=True)
tmp2['cut'] = 'Good'
tmp2.columns = ['clarity', 'prob', 'cut']

tmp = pd.concat([tmp1, tmp2])

sns.barplot(x=tmp.clarity, y=tmp.prob, hue=tmp.cut);<jupyter_output><empty_output><jupyter_text>WOW! this is huge difference. Take a minute to compare both the plot. Can you spot the difference? Hint: look at y-axis.Clearly, *SI1*, *SI2* and *I1* are more popular in *Good* diamonds than *Ideal* diamonds. This was not visible when we tried to compare them using absolute values. > Important: Whenever comparing two or more distributions make sure you **normalize** them before comparison.Wonderful, now we know how to compare two distributions using bar plots. But we cannot use bar plot if we have a lot unique values or for continuous features.For example, lets try to make a bar plot for *carat* feature<jupyter_code>tmp = data.carat.value_counts()
sns.barplot(x=tmp.index, y=tmp.values);<jupyter_output><empty_output><jupyter_text>The labels on the x-axis are not at all readable. There are so many small values that are hardly visible. Also, a lot of unique values make it difficult to understand the plot. We will see, how we can use histograms to alleviate this problem.  HistogramInstead of creating a separate bar for every unique value, we create bins for small ranges of values and put all the values in that range inside the bin.<jupyter_code>sns.histplot(data=data, x='carat', bins=50);<jupyter_output><empty_output><jupyter_text>But histograms are not perfect. Changing the number of bins can result in drastically different plot. It can also obscure some meaningful insight from the data. Here is example<jupyter_code>sns.histplot(data=data, x='carat', bins=100);<jupyter_output><empty_output><jupyter_text>The plot will 100 bins looks very different from the one with 50 bins. Histograms works great when you well defined bins. For example: all age 18 & 60 is senior citizen. In other cases, try to avoid them.   Probability Density Functions (PDFs)For simplicity, you can think of PDFs are histograms with a lot of bins & then some smoothing to reduce the impact of missing values & noise. Technical Stuff (not important)Technically, PDFs are derivatives of Cumulative Distribution Functions (CDFs). CDFs are nothing but cumulative sum of PMFs. In physics, Density of a substance is defined as mass per unit volume. No matter how the mass or volume changes, their ratio (i.e density) will always remain constant. Similarly, no matter how many data points you select from the distribution and the absolute values of these data points, the PDF will (more or less) look the same. Hence, PDF is a very apt name. Density is the way you summaries substances & also distributions. PDF is the standard way to describe any distribution.---PDFs are very expensive to compute & often times don't have real solutions. Kernel density estimation (KDE) is an algorithm that takes a sample and finds an appropriately smooth PDF that fits the data. You can read details at [wikipedia](http://en.wikipedia.org/wiki/Kernel_density_estimation).Don't worry about all these technical details, you can easily make KDE plot using seaborn.<jupyter_code>sns.kdeplot(data=data, x='carat');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08936825885978428, 'fraction_numerical': 0.006053268765133172, 'mean_word_length': 5.671806167400881, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 71, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '470361', 'license': 'Apache-2.0', 'n_tokens_mistral': 2717, 'n_tokens_neox': 2593, 'n_words': 1101, 'path': '_notebooks/2021-06-04-Distributions.ipynb', 'repo_name': 'Ankur-singh/learnings', 'size': '9086'}