<jupyter_start><jupyter_text>Distribution doodling<jupyter_code>import os 
import pandas as pd 
import numpy as np
#----- PLOTTING PARAMS ----# 
import matplotlib.pyplot as plt
from matplotlib.pyplot import cm
import seaborn as sns
%config InlineBackend.figure_format = 'retina'
%config InlineBackend.print_figure_kwargs={'facecolor' : "w"}

plot_params = {
'font.size' : 15,
'axes.titlesize' : 15,
'axes.labelsize' : 15,
'xtick.labelsize' : 15,
'ytick.labelsize' : 15,
}
 
plt.rcParams.update(plot_params)
# Functions for PMF and CDF, we will come to those later in the notebook 
def pmf(pandas_series):
    value, counts = np.unique(pandas_series, return_counts=True)
    pmf = np.c_[ value, counts / sum(counts) ]
    return pmf 

def cdf(pandas_series):
    pmf_data = pmf(pandas_series)
    cdf = np.zeros(shape=pmf_data.shape)
    
    for i in range(0, cdf.shape[0]):
        cdf[i] = [ pmf_data[i][0], np.sum(pmf_data[0:i][:,1]) ]
    return cdf<jupyter_output><empty_output><jupyter_text>Warm up<jupyter_code>random_series = pd.Series(np.random.normal(0,1, size=(1000,)), name='normal_random')
random_series.plot()
pmf_d = pmf(random_series)
cdf_d = cdf(random_series)
# Plot PMF
plt.plot(pmf_d[:,0], pmf_d[:,1])
plt.xlabel('Unique entries')
plt.ylabel('Frequency');
# Kernel Density Estimate
random_series.plot.kde();
#CDF
plt.plot(cdf_d[:,0], cdf_d[:,1])
plt.xlabel('Unique entries')
plt.ylabel('Cumulative Frequency');
random_series.hist(cumulative=True, density=1, bins=100)<jupyter_output><empty_output><jupyter_text>Iris Dataset DistributionThe Iris dataset consists of measurements of sepals and petals (in cm) of 3 different plant species:1. Iris setosa2. Iris versicolor3. Iris virginica![](./iris_schematic.png "Image courtesy: codecademy.com")<jupyter_code>from sklearn import datasets
iris = datasets.load_iris()
iris.keys()
iris_column_names = [i[:-5].replace(' ','_') for i in iris.feature_names]
iris_df = pd.DataFrame( iris.data, columns=iris_column_names )
iris_df['Target_num'] = iris.target 
iris_df.sample(4)
target_name_dict = {i : iris.target_names[i] for i in range(len(iris.target_names))}
target_name_dict
iris_df['Target_name'] = [ iris.target_names[i] for i in iris_df['Target_num'] ]
iris_df['Target_name'].value_counts()
for i in iris.target_names:
    iris_df.loc[ iris_df['Target_name'] == i]['sepal_length'].hist(alpha=0.7, label=i)
plt.legend(bbox_to_anchor=(1.04,1), loc="upper left");
sns.displot(iris_df, x="sepal_length", hue="Target_name", stat="probability", alpha=0.3)<jupyter_output><empty_output><jupyter_text>A histogram aims to approximate the underlying probability density function that generated the data by binning and counting observations. Kernel density estimation (KDE) presents a different solution to the same problem. Rather than using discrete bins, a KDE plot smooths the observations with a Gaussian kernel, producing a continuous density estimate:<jupyter_code>sns.displot(iris_df, x="sepal_length", hue="Target_name", kind="kde", fill=True);
sns.displot(iris_df, x="sepal_length", hue="Target_name", kde=True, alpha=0.3, fill=True)<jupyter_output><empty_output><jupyter_text>The ECDF (empirical cumulative distribution function) plot has two key advantages. Unlike the histogram or KDE, it directly represents each datapoint. That means there is no bin size or smoothing parameter to consider. Additionally, because the curve is monotonically increasing, it is well-suited for comparing multiple distributions:<jupyter_code>sns.displot(iris_df, x="sepal_length", hue="Target_name", kind="ecdf");<jupyter_output><empty_output><jupyter_text>The major downside to the ECDF plot is that it represents the shape of the distribution less intuitively than a histogram or density curve. Consider how the bimodality of flipper lengths is immediately apparent in the histogram, but to see it in the ECDF plot, you must look for varying slopes. Nevertheless, with practice, you can learn to answer all of the important questions about a distribution by examining the ECDF, and doing so can be a powerful approach.<jupyter_code>sns.displot(iris_df, x="sepal_length", y="sepal_width", hue="Target_name", kind="kde", fill=True, alpha=0.4)
sns.jointplot(
    data=iris_df,
    x="sepal_length", 
    y="sepal_width", 
    hue="Target_name", 
    kind="kde", fill=True, alpha=0.3
)
iris_df['sepal_length'].describe().round(2).index
iris_df['sepal_length'].describe()['25%']
# Convert to a categorical variable
def convert_to_cat(panda_series):
    first_quarter = panda_series.describe()['25%']
    third_quarter = panda_series.describe()['75%']
    print(first_quarter, third_quarter)
    
    cat_list = ['temp'] * len(panda_series) 
    
    for i, entry in enumerate(panda_series):
        if entry <= first_quarter: 
            cat_list[i] = 'SMALL'
        elif first_quarter < entry <= third_quarter:
            cat_list[i] = 'MED'
        else:
            cat_list[i] = 'LARGE'
    
    return cat_list
_, _, sepal_cat_list = convert_to_cat(iris_df['sepal_length'])
iris_df['sepal_length_cat'] = sepal_cat_list
iris_df['sepal_width'].describe().round(2)
iris_df['sepal_width_cat'] = convert_to_cat(iris_df['sepal_width'])
iris_df['sepal_width_cat'].value_counts()
# One-hot categorical
pd.get_dummies(iris_df['sepal_width_cat'], prefix='sepal_width')
# One-hot discrete variable with more granularity 
def OHE_discreet(point, pandas_series, intervals):
    z = np.linspace(min(pandas_series), max(pandas_series), intervals)
    ohe = np.zeros(len(z))
    ohe[np.argmin(abs(z - point)**2)] = 1
    return ohe
iris_df['sepal_width_OHE'] = iris_df['sepal_width'].apply(OHE_discreet, args=(iris_df['sepal_width'], 11))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12613875262789068, 'fraction_numerical': 0.010686755430974072, 'mean_word_length': 5.870036101083032, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '165703', 'license': 'Apache-2.0', 'n_tokens_mistral': 2097, 'n_tokens_neox': 2007, 'n_words': 470, 'path': '_notebooks/data/.ipynb_checkpoints/Distribution_doodling-checkpoint.ipynb', 'repo_name': 'pgg1610/blog_fastpages', 'size': '5708'}