<jupyter_start><jupyter_text>Statistics in Python  Central TendencyThe *mean, median, mode* are the three statistical measures for analyzing the __central tendency__ of data. These measures are used to find the central value of the data to summarize the entire data set. There are three additional measures of central tendency: the *trimean*, the *geometric mean*, and the *trimmed mean*. Mean $(\bar x)$The **mean** or **average** is the one number that best represents all these data points.$$\bar x=\frac{x_1+x_2+...+x_n}{n}$$ MedianThe __median__ is the value separating the higher half from the lower half of a data sample (a population or a probability distribution). For a data set, it may be thought of as the "middle" value. The basic advantage of the median in describing data compared to the mean is that it is not skewed so much by extremely large or small values.The mean and the median are the same for symmetric distributions. ModeThe __mode__ of a set of data values is the value that appears most often. It is the value x at which its probability mass function takes its maximum value. In other words, it is the value that is most likely to be sampled.The mean of the dataset would always change if there is a change in any value of the dataset. Median and mode may or may not change with altering a single value in the dataset. TrimeanThe __trimean__ is a weighted average of the 25th percentile, the 50th percentile, and the 75th percentile. Letting $P_{25}$ be the 25th percentile, $P_{50}$ be the 50th and $P_{75}$ be the 75th percentile, the formula for the trimean is:$$Trimean=\frac{P_{25}+2P_{50}+P_{75}}{4}$$The median $P_{50}$ is weighted twice as much as the 25th and 75th percentiles. Geometric meanThe __geometric mean__ is computed by multiplying all the numbers together and then taking the nth root of the product. For example, for the numbers 1, 10, and 100, the product of all the numbers is: 1 x 10 x 100 = 1,000. Since there are three numbers, we take the cubed root of the product (1,000) which is equal to 10. The formula for the geometric mean is:$$Geometric Mean=(\Pi x)^{\frac{1}{N}}$$where the symbol $\Pi$ means to multiply. Trimmed mean__Trimmed mean__ is obtained by removing some of the higher and lower scores and compute the mean of the remaining scores. A mean trimmed 10% is a mean computed with 10% of the scores trimmed off: 5% from the bottom and 5% from the top. A mean trimmed 50% is computed by trimming the upper 25% of the scores and the lower 25% of the scores and computing the mean of the remaining scores. The trimmed mean is similar to the median which, in essence, trims the upper 49+% and the lower 49+% of the scores. Therefore the trimmed mean is a hybrid of the mean and the median.   Variability![line](https://github.com/jamwine/Statistics/blob/master/imgs/line.PNG?raw=true) RangeThe **Range** describes the extent of variability by considering the distance between the biggest and the smallest values. The larger the range, the more noticeable the variation in the data will be. However, the greatest disadvantage is that it ignores the mean, and is swayed by the outliers; that's where variance comes in.$$range=x_{max}-x_{min}$$ Mean DeviationFor Variance, we first calculate __Mean Deviation__ and __Squared Mean Deviation__:$$Mean\ Deviation=x_i-\bar x$$$$Squared\ Mean\ Deviation=(x_i-\bar x)^2$$The sum of mean deviations of the individual will always be 0. Variance $(\sigma^2)$ __Variance__ is the second-most important number to summarize this set of data points. It is the mean squared deviation of a variable from its mean. The higher the variance, the larger the variability of the data.$$Variance=\frac{\sum (x_i-\bar x)^2}{n-1}$$|![wh](https://github.com/jamwine/Statistics/blob/master/imgs/head.png?raw=true)|![wh](https://github.com/jamwine/Statistics/blob/master/imgs/head.png?raw=true)||-|-||![var1](https://github.com/jamwine/Statistics/blob/master/imgs/var1.PNG?raw=true)|![var2](https://github.com/jamwine/Statistics/blob/master/imgs/var2.PNG?raw=true)|Variance is improved by tweaking the denominator - this is called __Bessel's Correction__.Variance Properties:* If a constant is added to each data point, the variance remains the same. Adding a constant shifts the distribution and doesn't affect variability of the data. The deviations from the mean remains the same as well.* If a constant is multiplied to each observation, the standard deviation increases proportionally to the constant and the variance increases to the squared constant. Standard Deviation $(\sigma)$Mean and variance succinctly summarize a set of numbers. __Standard Deviation__ is the square root of Variance.$$Std Dev=\sqrt{\frac{\sum (x_i-\bar x)^2}{n-1}}$$Standard Deviation is the most common way to estimate the uncertainity of a set of outcomes.![std1](https://github.com/jamwine/Statistics/blob/master/imgs/std1.PNG?raw=true) Percentile RankA __percentile rank__ is typically defined as the proportion of scores in a distribution that a specific score is greater than or equal to. For instance, if there is a score of 95 on a math test and this score was greater than or equal to the scores of 88% of the students taking the test, then the percentile rank would be 88 (88th percentile). Alternatively, percentile rank is sometimes defined simply as the proportion of a distribution that a score is greater than.  Frequency DistributionThe distribution of empirical data is called a __frequency distribution__ and consists of a count of the number of occurrences of each value. If the data are continuous, then a grouped frequency distribution is used. Typically, a distribution is portrayed using a frequency polygon or a histogram. Mathematical equations are often used to define distributions. The normal distribution is, perhaps, the best known example.A __grouped frequency distribution__ is a frequency distribution in which frequencies are displayed for ranges of data rather than for individual values. For example, the distribution of heights might be calculated by defining one-inch ranges. The frequency of individuals with various heights rounded off to the nearest inch would then be tabulated.A __histogram__ and a __frequency polygon__ are the graphical representations of a distribution. They partition the variable on the x-axis into various contiguous class intervals of (usually) equal widths. The heights of the _bars_ in a histogram and the _polygon's points_ in a frequency polygon represent the class frequencies.|![wh](https://github.com/jamwine/Statistics/blob/master/imgs/head.png?raw=true)|![wh](https://github.com/jamwine/Statistics/blob/master/imgs/head.png?raw=true)||-|-||![hist](https://github.com/jamwine/Statistics/blob/master/imgs/hist.PNG?raw=true)|![fp](https://github.com/jamwine/Statistics/blob/master/imgs/fp.PNG?raw=true)|  Gaussian DistributionProperties in the real world can be represented by a normal distribution, also known as __Gaussian Distribution__ or the __Bell Curve__. Normal distributions do not necessarily have the same means and standard deviations. A normal distribution with a mean of 0 and a standard deviation of 1 is called a __standard normal distribution__.$$N(\mu,\sigma)=\frac{1}{\sqrt {2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}$$![ga](https://github.com/jamwine/Statistics/blob/master/imgs/ga.PNG?raw=True)|![wh](https://github.com/jamwine/Statistics/blob/master/imgs/head.png?raw=true)|![wh](https://github.com/jamwine/Statistics/blob/master/imgs/head.png?raw=true)|![wh](https://github.com/jamwine/Statistics/blob/master/imgs/head.png?raw=true)||-|-|-||![g1](https://github.com/jamwine/Statistics/blob/master/imgs/g1.PNG?raw=True)|![g2](https://github.com/jamwine/Statistics/blob/master/imgs/g2.PNG?raw=True)|![g3](https://github.com/jamwine/Statistics/blob/master/imgs/g3.PNG?raw=True)|* Normal distributions are symmetric around their mean.* The mean, median, and mode of a normal distribution are equal.* The area under the normal curve is equal to 1.0.* Normal distributions are denser in the center and less dense in the tails. There will be a large number of points close to the average. There will be few extreme values - the number of extreme values at either side of the mean will be the same.  z-ScoreA value from any normal distribution can be transformed into its corresponding value on a standard normal distribution using the following formula:$$z=\frac{x-\mu}{\sigma}$$In other words, __z-score__ is determined in order to normalize the data or standardizing the distribution. The standard normal distribution is sometimes called the __z distribution__. A z score always reflects the number of standard deviations above or below the mean a particular score is.![zs](https://github.com/jamwine/Statistics/blob/master/imgs/zs.PNG?raw=true) ModalityThe __modality__ of a distribution is determined by the number of peaks the data presents.![modality](https://github.com/jamwine/Statistics/blob/master/imgs/modality.PNG?raw=true)* Most distributions are __Unimodal__ which means it has only one frequently occurring score, clustered at the top while a bimodal has two values occurring frequently.* A __Bimodal distribution__ is a continuous probability distribution with two different modes. These appear as distinct peaks (local maxima) in the probability density function.* A __Uniform Distribution__ has a single value which only occurs in a certain range while anything outside that range is just 0. It’s very much an “on or off” distribution. It as an indication of a categorical variable with 2 categories: 0 or the value. The categorical variable might have multiple values other than 0 and it can be visualized in the same way as a piecewise function of multiple uniform distributions.  Example: A random male is selected from the population having 186 cm of height.  * What percentage of the men will be shorter than this individual? * Which height are 83% of men? * What about the likelihood of choosing someone who is 186 cm?<jupyter_code>from scipy import stats
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline

normal_data = stats.norm.rvs(size=10000, loc=174, scale=8.2, random_state=0)
# size is the number of points generated (population), loc is the mean height and scale is the standard deviation in height.

pd.Series(normal_data).plot(kind="hist", bins=50);<jupyter_output><empty_output><jupyter_text>**Cumulative distribution function: cdf()** returns the probability of a certain random observation by taking value as input.<jupyter_code># Cumulative distribution function: cdf()
stats.norm.cdf(x=186,         # value
               loc=174,       # Mean
               scale=8.2)     # Standard Deviation

# Result: Around 93% of men will be shorter.<jupyter_output><empty_output><jupyter_text>**Percent Point Function: ppf()** is the opposite of cdf(), returns a value by taking quantile as input.<jupyter_code># Percent point function: ppf()
stats.norm.ppf(q=0.83,      # quantile
               loc=174,     # Mean
               scale=8.2)   # Standard Deviation

# Result: 83% of men measure less than 181 cm.<jupyter_output><empty_output><jupyter_text>**Probability density function: pdf()** gives the likelihood of a random variable assuming a certain value.<jupyter_code># Probability density function: pdf()
stats.norm.pdf(x=186,         # Value to check
               loc=174,       # Distribution start
               scale=8.2)     # Distribution end<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08129727849752195, 'fraction_numerical': 0.014433527519346144, 'mean_word_length': 4.832657200811359, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 19, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '171477', 'license': 'MIT', 'n_tokens_mistral': 3184, 'n_tokens_neox': 2908, 'n_words': 1457, 'path': '.ipynb_checkpoints/Statistics-checkpoint.ipynb', 'repo_name': 'jamwine/Statistics', 'size': '11501'}