<jupyter_start><jupyter_text>Exploratory Data Analysis<jupyter_code>import pandas as pd
import numpy as np
from scipy.stats import trim_mean
from statistics import *
from matplotlib import pyplot as plt 
import seaborn as sns
medal_count = pd.read_excel("dataset//Medals.xlsx")
medal_count<jupyter_output><empty_output><jupyter_text>Estimate Of Location<jupyter_code># mean
medal_count.Total.mean()
# weighted mean is the same as mean , but except you multiply every value by some (X)i, before adding them up and dividing by the number of instances.
np.average(medal_count.Total,weights=medal_count.Gold)
# trimmed mean is calculcated by removing the top and bottom x values from the dataset and then the mean is calculated. It reduces the impact of outliers.
trim_mean(medal_count.Total,0.1) # the second parameter 0.1 determines the number which has to be removed from the top and bottom which is 10% in this case.
# median , the middle value of sorted values.
medal_count.Total.median()
# Weighted median, multiply every value with some weight (x)i and then pick the middle value such that the sum of values to the left of the median are the same as the sum of values to the rights of the median
def weighted_median(df,median_col,weight_col):
    df_sorted = df.sort_values(median_col)
    cumsum = df_sorted[weight_col].cumsum()
    cutoff = df_sorted[weight_col].sum() / 2
    return df_sorted[cumsum >= cutoff][median_col].iloc[0]

weighted_median(medal_count,"Total","Gold")
# percentile . the value such that P percent of the data lies below it.
# outlier is the value which is different from most of the data.
q3 = np.percentile(medal_count.Total, 75) # the second parameter here defines the percentile number which we want to calculate
q3
medal_count.describe()<jupyter_output><empty_output><jupyter_text>Estimates of Variability (Dispersion Metrics)<jupyter_code># deviations, The difference between the observed values and the estimate of location also called errors or residulas.
# variance, The sum of squared deviations from the mean divided by n – 1 where n is the number of data values. mean-squared-error
variance(medal_count.Gold)
# standard deviation, The square root of the variance.
stdev(medal_count.Gold)
# mean absolute deviation, L-1 Norm, Manhattan Norm,The mean of the absolute values of the deviations from the mean.
np.mean(np.abs((medal_count.Gold)- np.mean(medal_count.Gold)))
# median absolute deviation, The median of the absolute values of the deviations from the median.
np.median(np.abs((medal_count.Gold)-np.median(medal_count.Gold)))
# range, The difference between the largest and the smallest value in a data set.
# order statistics , Metrics based on the data values sorted from smallest to biggest. ranks
# Interquartile range,The difference between the 75th percentile and the 25th percentile.IQR
q3, q1 = np.percentile(medal_count.Gold,[75,25])
iqr = q3-q1
iqr<jupyter_output><empty_output><jupyter_text>Median absolute deviation is robust against outliers and is calculate by taking the median of the absolute value of all values - the median.  Explore the Data Distribution See Plotly library<jupyter_code># boxplots, A plot introduced by Tukey as a quick way to visualize the distribution of data.
medal_count.boxplot(column="Gold")
# frequency tables,A tally of the count of numeric data values that fall into a set of intervals (bins).
frequency_table = medal_count.copy()
frequency_table["freq"] = pd.cut(medal_count.Gold,10)
# frequency_table.groupby("freq")["Rank"].count().reset_index()
frequency_table.value_counts()
# histogram, A plot of the frequency table with the bins on the x-axis and the count (or pro‐ portion) on the y-axis. While visually similar, bar charts should not be confused with histograms.
# np.histogram(medal_count.Gold,bins="auto")
plt.hist(medal_count.Gold, bins = "auto")
plt.show()<jupyter_output><empty_output><jupyter_text>Statistical Moments First Moment is Location (mean) : average of all the data Second Moment is Variability : how closely values are spread around the mean Third Moment is Skewness : Direction of the tail of the data, discovered through visualization. Fourth Moment is Kurtosis : Prepensity for data to have extreme values, discovered through visualization.<jupyter_code># density plot, A smoothed version of the histogram, often based on a kernel density estimate.

ax = medal_count.Gold.plot.hist(density = True, xlim= [0,40], bins = range(1,40))
medal_count.Gold.plot.density(ax=ax)<jupyter_output><empty_output><jupyter_text>Exploring Binary and Categorical Data<jupyter_code># Mode, The most commonly occurring category or value in a data set.
medal_count.Gold.mode()
# expected values, When the categories can be associated with a numeric value, this gives an average value based on a category’s probability of occurrence.

# barplots, The frequency or proportion for each category plotted as bars.
ax = medal_count.plot.bar(x= "Team/NOC",y = "Gold", legend = False)
ax.set_xlabel("Country")
ax.set_ylabel("Gold Medals")
# pie-chart
# medal_count.iloc[:10, :].plot.pie(y = "Gold")
pie_data = medal_count.iloc[:7, :]
plt.pie(pie_data.Gold,labels = pie_data["Team/NOC"],shadow = True,autopct='%1.1f%%', startangle=90)
plt.show()<jupyter_output><empty_output><jupyter_text>Correlation<jupyter_code># correlation coefficient, A metric that measures the extent to which numeric variables are associated with one another (ranges from –1 to +1).
# correlation matrix, A table where the variables are shown on both rows and columns, and the cell values are the correlations between the variables.
medal_count.corr()
# scatterplot, A plot in which the x-axis is the value of one variable, and the y-axis the value of another.
medal_count.plot.scatter(x = "Gold", y= "Silver")
# hexogonal binning, A plot of two numeric variables with the records binned into hexagons.
medal_count.plot.hexbin(x = "Gold", y= "Silver",gridsize=30,sharex = False)
# contour plot, A plot showing the density of two numeric variables like a topographical map.
sns.kdeplot(medal_count["Gold"], medal_count["Silver"])

# heatmap
sns.heatmap(medal_count.corr())
# contigency table
cont_table = medal_count[["Silver","Gold"]].pivot_table(index="Silver", columns="Gold", aggfunc= lambda x: len(x), margins=True)
cont_table
# violin-plot, Similar to a boxplot but showing the density estimate.
sns.violinplot(x = medal_count["Gold"])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08668634457045207, 'fraction_numerical': 0.0069908342395525866, 'mean_word_length': 5.7272727272727275, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '264804', 'license': 'MIT', 'n_tokens_mistral': 2005, 'n_tokens_neox': 1877, 'n_words': 739, 'path': 'Chapter1.ipynb', 'repo_name': 'abishek1233/PracticalStatistics-using-python', 'size': '6437'}