<jupyter_start><jupyter_text>Statistics basics*** IntroductionIn this tutorial you are going to learn how to use Python tools to visualize and analyze randomly generated data. The purpose is to get acquainted with statistical concepts such as __probability density functions__ and __cumulative distribution functions__.As usual, we start by importing Python packages that we will use later on. In this tutorial, we are using _NumPy_ to create random numbers, the set of statistical tools in _SciPy_, and _Matplotlib_ and _Seaborn_ to plot graphs.<jupyter_code>import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
sns.set_context('notebook')<jupyter_output><empty_output><jupyter_text>Example: normal distributionAs an example, we create an array of random numbers extracted from a normal distribution of mean (`loc`) 0 and standard deviation (`scale`) 1.<jupyter_code># create an array of random numbers taken from a normal distribution<jupyter_output><empty_output><jupyter_text>VisualizationLet's plot the histogram of the random data with the _Seaborn_ function `distplot`.<jupyter_code># histogram<jupyter_output><empty_output><jupyter_text>The histogram visualization is sensitive to the number of bins in which the variable range is divided. `distplot` calculates by default the number of bins that it estimates best, but you can set a fixed number of bins and check how the histagram changes.<jupyter_code># histogram with different number of bins<jupyter_output><empty_output><jupyter_text>To avoid this limitation, `distplot` uses a function called kernel density estimation (KDE) to create a softened line that represents the shape of the data distribution independent of the number of bins.<jupyter_code># histogram + kernel density estimate (default)<jupyter_output><empty_output><jupyter_text>The kernel density estimate resembles what we expect our distribution to look like, a Gaussian bell. Actually, if we know the distribution of our data, we can fit and plot the distribution with `distplot`.<jupyter_code># histogram + fitted normal distribution<jupyter_output><empty_output><jupyter_text>Distribution fittingNow were are going to use thes statistical tools in _SciPy_ (`stats`) to fit a distribution to our data and to learn what are the __probability density function__ and the __cumulative distribution function__.To fit a distribution function in `Scipy.stats` we only need to call the function `fit` to the specific distribution.<jupyter_code># fit a normal distribution<jupyter_output><empty_output><jupyter_text>We check that the fit is not perfect, since we generated the random values from a normal distribution $N(0,1)$, i.e., $loc=0$ and $scale=1$. This is due to a limited amount of data, but the fit is good enough. Probability density function (PDF)The probability density function represents the _relative likelihood_ that the variable takes a specific value. In practical terms, values with a higher PDF are more likely to happen. In the case of the normal distribution, the probability density function is the well-known Gaussian bell. Let's plot it for the distribution we have fitted above.In `scipy.stats`, we will use the function `pdf`.<jupyter_code># linearly spaced values of X

# PDF (probability density function)

# line plot of the PDF<jupyter_output><empty_output><jupyter_text>Cumulative distribution function (CDF)The cumulative distribution function is the probability that the variable $X$ takes values lower or equal than $x$; for that reason it is also called the non-exceedance probability. The cumulative distribution function (CDF) is the integral of the probability density function (PDF).$$CDF = P(x\leq X) = \int PDF$$The CDF is a monotonic incresing function that ranges between 0 and 1.In `scipy.stats`, we will use the function `cdf`.<jupyter_code># CDF (cumulative distribution function)

# line plot of the CDF<jupyter_output><empty_output><jupyter_text>Percent point function (PPF)`Scipy.stats` includes the function `ppf` (percent point function) that inverts the CDF, that is, it calculates the value of $X$ for a given value of the CDF.<jupyter_code># linearly spaced values CDF

# x using 'ppf'

# line plot of the CDF<jupyter_output><empty_output><jupyter_text>Use of the CDFThe fact that the CDF is a monotonic increasing function allows us to connect the values of the variable $X$ with its probability and viceversa. For instance:1. What is the probability that the variable $X$ takes a value lower than $x=1$? → __CDF__2. What is the value $x$ that is not exceeded 99% of the time? → __PPF__<jupyter_code># probability that the variable 𝑋 takes a value lower than 𝑥=1

# plot explaining what 'cdf' does

# value 𝑥 that is not exceeded 99% of the time

# plot explaining what 'ppf' does<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05944272445820433, 'fraction_numerical': 0.003302373581011352, 'mean_word_length': 5.873758865248227, 'pattern_counts': {'":': 0, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '301048', 'license': 'MIT', 'n_tokens_mistral': 1345, 'n_tokens_neox': 1291, 'n_words': 644, 'path': 'G1448-Hydrology/Precipitation/notebooks/Statistics_basics-fitting_distributions(incomplete).ipynb', 'repo_name': 'NorAhmed1/Clases', 'size': '4845'}