<jupyter_start><jupyter_text>Contents:[EDA](EDA)[DATA](DATA)[Examples for data types](Examples-for-data-types)[Variable](Variable)[Descriptive data measures ](measures)[Measures of central tendency](ct)[Measures of dispersion](dispersion)[Five Number Summary](fivepoint)[Shape of the data](shape)[Boxplot](box)[Co-variance and Co-efficient of correlation](cov)[Let's get our hands dirty with some code ](code)    EDA    Tukey defined data analysis in 1961 as: "Procedures for analyzing data, techniques for interpreting the results of such procedures, ways of planning the gathering of data to make its analysis easier, more precise or more accurate, and all the machinery and results of (mathematical) statistics which apply to analyzing data"       DATA * Apart from the regular understanding of Data, in the context of Machine Learning, Data refers to the data that we use to "train", "test" or "validate" a model.* A very common kind of data we deal with is Structured Data i.e, data in tabular format. It is no different from the regular excel spreadsheet we often encounter. We have rows and columns- rows represent individual observations columns represent attributes/features/variables of the observations.     | Name  | Gender  |  Location |  Education_level | Age | Employement_status  | |---|---|---|---|---|---|| Yogesh  | Male  | Delhi  | PhD  | 40 | employed  || Anisha  | Female  | Mumbai  | Master's | 25 | employed  || Tanul  | Female  | Pune  | Master's  | 37 | Unemployed  |    &nbsp;* Besides the regular tabular data, we also encounter Unstructured data - Text, images, audio, video, analog data etc* Despite the kind of data, we always convert them into arrays of numbers   Examples for data types  Since Python is an object oriented language we have a wide range of specialized datatypes.Some common built-in data types are list, dict, str, int, tuple, set etcRefer: https://docs.python.org/3/library/datatypes.html   Variable   Qualitative Variable * A broad category for any variable that can't be counted (i.e, has no numerical value)* Nominal and ordinal variables fall under this category    Quantitative Variable * A broad category for any variable that can be counted, or has a numerical value associated with it* Examples of variables that fall into this category include discrete and ratio variables    Note:* As we progress in the course, we will refer qualitative variables as 'Categorical variables'* Quantitative variables as 'Continuous variables'* Any attribute whose values cannot be used for arthematic operations, we dub its datatype as a categorical datatype* Any attribute whose values can be used for arthematic operations, we dub its datatype as a continuous datatype   Descriptive data measures    Measure of Central Tendency    Mode     Median     Mean    * When we measure things of same kind, fairly large number of such measurements will tend to cluster around the middle value. Such a value is called a measure of Central Tendency or Measures of location or Statistical Average* Measures of central tendency include mean, median and mode    Mean:  The general mathematical average.    $$\mu  = \frac{1}{n} \sum_{i=0}^{n} a_i $$&nbsp;$$ mean = \frac{Sum\;of\;all\;elements\;in\;set}{number\;of\;elements\;in\;set} $$&nbsp;* Arithematic mean is very susceptible to extreme values or fluctions in data* Not the right measure to consider when the data contains a lot of extreme values    Median:  The central value in an ordered set by position.&nbsp;$$ median = \frac{no.\;of\;observations+1}{2}\;th \;value\;in\;an\;ordered\;set$$ &nbsp;* Median is such that 50% of the observations are above the median and 50% of the observations are below the median* It has better resistance to outliers compared to mean    Mode:  * Mode is that value which occurs most often. In other words mode has the maximum frequency of occurance.* Mode also is highly resistant to outliers   Measure of dispersion   Central tendency describes the central point in a data set whereas dispersion or variability describes the spread of the data.&nbsp;* Measures of dispersion include standard deviation, range, kurtosis and skewness  Variance, Standard Deviation and the relation between the two:  * The following is the formula for variance$$ \sigma^2 = \frac{\displaystyle\sum_{i=1}^{n}(x_i - \mu)^2} {n} $$&nbsp;* The average of the squared differences from the Mean, is Variance* Standard deviation is a quantity expressing by how much each datapoint differs from the average(mean)* Standard deviation is obtained by calculating the square root of variance* Standard deviation is represented by  ' σ '  Range     The difference between the highest and lowest value of a set&nbsp; Range = Max(X) - Min(X)  &nbsp;  where 'X' is a numeric variable  Quartiles* Quartiles split the values into four equal parts* First Quartile divides the smallest 25% of the values from the rest that are larger* Second Quartils, which indeed is the median divides the 50% of the values from the rest that are larger or equal to median* Third Quartile divides the smallest 75% of the values from the rest that are larger.&nbsp;$$ Q1 = \frac{no.\;of\;observations+1}{4}\;th \;value\;in\;an\;ordered\;set$$ &nbsp;$$ Q2 = \frac{no.\;of\;observations+1}{2}\;th \;value\;in\;an\;ordered\;set$$ &nbsp;$$ Q3 = \frac{3*(no.\;of\;observations+1)}{4}\;th \;value\;in\;an\;ordered\;set$$ &nbsp;Note:* Percentiles split a variable into 100 equal parts* First quartile is equivalent to 25th percentile* Second quartile is equivalent to 50th percentile* Third quartile is equivalent to 75th percentile  IQR ( Inter Quartile Range )* It is nothing but Range computed only for the middle 50% of the data IQR = Q3 - Q1   Coefficient of variation&nbsp;Coefficient of variation is a relative measure to compare distributions with respect to their standard deviations&nbsp;$$ CV = \frac{\sigma}{\bar{x}} * 100 $$ * Greater the number, greater the variability in the data irrespective of scale  Z score &nbsp;For a given set, the z score of a value is the difference between a value and the mean of the set, divided by the standard deviation of the set&nbsp;$$ z = \frac{x - \bar{x}}{\sigma} $$ &nbsp;* A Z score of zero means the value IS the mean of the data* Depending on the sign of the Z score (positive or negative) we can say if the value is above or below the mean* The magnitude of the Z score tells us how far away the value is from the mean in terms of standard deviations    Five Number Summary&nbsp;  The five number summary helps describe the center, spread and shape of data. The five summary statistics are as follows,&nbsp;* Minimum* Q1* Q2* Q3* Maximum   Shape of data  Skeweness    Mode     Median     Mean    &nbsp;Extent to which the data values are not symmetrical around the mean![alt text](skeweness.png "Skewed vs normal distribution") $$skeweness = \frac{\sum_{i=1}^{n}(x_i - \bar{x})^3} {(n-1)*\sigma^3}$$&nbsp;&nbsp;* Mean > Median : Positive or right skewed distribution* Mean = Median: Zero skewness or symmetrical distribution* Mean < Median : Negative or left skewed distribution  Box-Plot![alt text](Boxplot.png "Five Number Summary with Boxplot") &nbsp;A visual representation of the five number summary   Distribution of shape and Box-Plot   Determining the skeweness of a data from its box plot![alt text](skeweness_and_boxplots.png "Boxplots")    * The side of the bigger segment tells us how the data is skewed* If both segments are equal, then it is a symmetric distribution   Box-Plot to check the presence of outliers  Detecting the presence of outliers in data![alt text](outlier.png "outlier")* The points on the right are the outliersNote:*     Presence of datapoints beyond the whiskers/fences desn't necessarily mean there are outliers*     The rule that box plot follows to decide what should be an outlier is that "any point greater than Q3 + 1.5IQR or lesser than Q1 - 1.5IQR is an outlier "  Co-variance and Co-efficient of correlation  To measure how one variable varies with respect to the variation of another variable, we compute Co-variance.&nbsp;$$ \text{cov(X, Y)} = \frac{\displaystyle\sum(X_i - \bar{X})*(Y_i - \bar{Y})} {n} $$&nbsp;* The metric only tells us to what extent two variables change together* The metric cannot be the only evidence to conclude that variation in one variable is causing the variation in the other variable* A positive value indicates that the two variables move in the same direction* A negative value indicates that the two variables move in the opposite direction to each other  We can only determine the direction of the relationship between two variables but to determine the strength of the relationship, we compute the Correlation coefficient of the two variables.&nbsp;$$\rho = \frac{\text{cov}(X,Y)}{\sigma_x \sigma_y}$$&nbsp;* Unlike co-variance, correlation coefficient is independant of scale* The range of values correlation coefficient can take is from -1 to +1* A value closer to -1 or +1 means a strong correlation and a value close to zero means weak correlationNote:Correlation does not imply causation ---------------------------------------   Let's get our hands dirty with some code  This dataset is created for prediction of Graduate Admissions from an Indian perspective.    1. GRE Scores ( 290 of 340 )     2. TOEFL Scores ( 92 of 120 )     3. University Rating ( 1 of 5 )    4. Statement of Purpose ( 1 of 5 )      5. Letter of Recommendation Strength ( 1 of 5 )     6. Undergraduate GPA ( 6.8 of 10 )     7. Research Experience ( 0 or 1 )     8. Chance of Admit ( ranging from  0.34 to 0.97)     1.1 Load the libraries:<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt<jupyter_output>In C:\Users\King\anaconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle: 
The text.latex.preview rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
In C:\Users\King\anaconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle: 
The mathtext.fallback_to_cm rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
In C:\Users\King\anaconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle: Support for setting the 'mathtext.fallback_to_cm' rcParam is deprecated since 3.3 and will be removed two minor releases later; use 'mathtext.fallback : 'cm' instead.
In C:\Users\King\anaconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle: 
The validate_bool_maybe_none function was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
In C:\Users\King\anaconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle[...]<jupyter_text>1.2. Import the dataset(Graduate Admissions.csv):<jupyter_code># Data  = pd.read_csv('Admission_Predict.csv') # Import the dataset named 'Admission_predict.csv'
Data  = pd.read_csv('Slump+Test.csv')
Data.head()  # view the first 5 rows of the data<jupyter_output><empty_output><jupyter_text>1.3 Check the Dimension of data?<jupyter_code>Data.shape # see the shape of the data<jupyter_output><empty_output><jupyter_text>**There are 400 Observations / Rows and 9 Attributes / Columns.** 1.4 Check the Information about the data and the datatypes of each respective attributes.<jupyter_code>Data.info() # To see the data type of each of the variable, number of values entered in each of the variable<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 103 entries, 0 to 102
Data columns (total 10 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   Cement    103 non-null    float64
 1   Slag      103 non-null    float64
 2   Fly ash   103 non-null    float64
 3   Water     103 non-null    float64
 4   SP        103 non-null    float64
 5   Coarse    103 non-null    float64
 6   Fine      103 non-null    float64
 7   SLUMP     103 non-null    float64
 8   FLOW      103 non-null    float64
 9   Strength  103 non-null    float64
dtypes: float64(10)
memory usage: 8.2 KB<jupyter_text>EDA Descriptive Statistics  Mean<jupyter_code>print("Data:",Data.mean())  # print the mean of each attribute. Ignore "Serial No. as it is not a continuous variable"<jupyter_output>Data: Cement      229.894175
Slag         77.973786
Fly ash     149.014563
Water       197.167961
SP            8.539806
Coarse      883.978641
Fine        739.604854
SLUMP        18.048544
FLOW         49.610680
Strength     36.039417
dtype: float64<jupyter_text>Mode<jupyter_code>print(Data['GRE Score'].mode())  # Prints the mode of the attribute. There are two values with the same frequency<jupyter_output><empty_output><jupyter_text>Median<jupyter_code>print("Data:",Data.median())  # Print the median values of the data.

# Observe that the values of mean and median are almost the same for most of the attributes<jupyter_output>Data: Cement      248.00
Slag        100.00
Fly ash     164.00
Water       196.00
SP            8.00
Coarse      879.00
Fine        742.70
SLUMP        21.50
FLOW         54.00
Strength     35.52
dtype: float64<jupyter_text>Quantile 25%<jupyter_code>print("Data_quantile(25%):",Data.quantile(q=0.25))  # Prints the value below which 25% of the data lies<jupyter_output>Data_quantile(25%): Cement      152.00
Slag          0.05
Fly ash     115.50
Water       180.00
SP            6.00
Coarse      819.50
Fine        684.50
SLUMP        14.50
FLOW         38.50
Strength     30.90
Name: 0.25, dtype: float64<jupyter_text>Quantile 50%<jupyter_code>print("Data_quantile(50%):",Data.quantile(q=0.50))  # Prints the value below which 50% of the data lies<jupyter_output>Data_quantile(50%): Serial No.           200.50
GRE Score            317.00
TOEFL Score          107.00
University Rating      3.00
SOP                    3.50
LOR                    3.50
CGPA                   8.61
Research               1.00
Chance of Admit        0.73
Name: 0.5, dtype: float64<jupyter_text>Quantile 75%<jupyter_code>print("Data_quantile(75%):",Data.quantile(q=0.75))   # Prints the value below which 75% of the data lies<jupyter_output>Data_quantile(75%): Serial No.           300.2500
GRE Score            325.0000
TOEFL Score          112.0000
University Rating      4.0000
SOP                    4.0000
LOR                    4.0000
CGPA                   9.0625
Research               1.0000
Chance of Admit        0.8300
Name: 0.75, dtype: float64<jupyter_text>1.5a) Plotting the summery mean,mode,median using histogram<jupyter_code>mean=Data['GRE Score'].mean()
median=Data['GRE Score'].median()
mode=Data['GRE Score'].mode()

print('Mean: ',mean,'\nMedian: ',median,'\nMode: ',mode[0])

plt.figure(figsize=(10,5)) # set the figure size
plt.hist(Data['GRE Score'],bins=100,color='lightblue') #Plot the histogram

plt.axvline(mean,color='green',label='Mean')     # Draw lines on the plot for mean median and the two modes we have in GRE Score
plt.axvline(median,color='blue',label='Median')
plt.axvline(mode[0],color='red',label='Mode1')
plt.axvline(mode[1],color='red',label='Mode2')

plt.xlabel('GRE Score')   # label the x-axis
plt.ylabel('Frequency')   # label the y-axis
plt.legend()              # Plot the legend
plt.show()<jupyter_output>Mean:  316.8075 
Median:  317.0 
Mode:  312<jupyter_text>1.6 Check the data.describe()<jupyter_code>Data.describe() # the output results provide the five number summary of the data.<jupyter_output><empty_output><jupyter_text>IQR Value for TOEFL Score<jupyter_code>Data["TOEFL Score"].quantile(0.75) - Data["TOEFL Score"].quantile(0.25)<jupyter_output><empty_output><jupyter_text>1.6a)Undestanding the summary of Statistics plotting  Box plots<jupyter_code># Create boxplot for column="mpg"
Data.boxplot(column="TOEFL Score",return_type='axes',figsize=(8,8))

# create text(x=0.74, y=22.25, s="3rd Quartile")like Median, 1st Quartile,Min,Max,IQR:
plt.text(x=0.74, y=112.00, s="3rd Quartile")
plt.text(x=0.8, y=107.00, s="Median")
plt.text(x=0.75, y=103.00, s="1st Quartile")
plt.text(x=0.9, y=92.00, s="Min")
plt.text(x=0.9, y=120.00, s="Max")
plt.text(x=0.7, y=107.5, s="IQR", rotation=90, size=25)
# we can observe from above graph that there are no extreme values / outliers.the data was equally distributed .<jupyter_output><empty_output><jupyter_text>1.7 Check the Measures of Dispersion:  IQR<jupyter_code>Data.quantile(0.75) - Data.quantile(0.25)
# the above output represents the IQR values for all the attributed.<jupyter_output><empty_output><jupyter_text>Range<jupyter_code>print(Data.max() - Data.min())
# the above represents the difference between the highest value and lowest values for all individual attributes.<jupyter_output><empty_output><jupyter_text>Variance<jupyter_code>print(Data.var())
# the above output brief's regarding how much was the data dispersion .<jupyter_output><empty_output><jupyter_text>Standard Deviation<jupyter_code>print(Data.std())
# the above output brief's regarding how much was the data deviated away from the mean .<jupyter_output><empty_output><jupyter_text>1.8 Check Covariance and Correlation  Covariance<jupyter_code>Data.cov() # Prints the covariance of each attribute against every other attribute<jupyter_output><empty_output><jupyter_text>Correlation<jupyter_code>Data.corr()  # prints the correlation coefficient between every pair of attributes<jupyter_output><empty_output><jupyter_text>1.8a) Pairplot for checking the correlation?<jupyter_code>import seaborn as sns

sns.pairplot(Data, kind="reg")  # plots scatter plots for every pair of attributes and histograms along the diagonal
plt.show()<jupyter_output><empty_output><jupyter_text>* Here we can see that TOEFL, GRE and CGPA Scores highly correlate with the chance of admission* All of the attributes have good correlation with the chance of admission* Apart from the target variable, there is fair amount of correlation amongst the other variables<jupyter_code>Data.drop('Serial No.', axis=1, inplace=True)  # to drop a column from the original dataframe<jupyter_output><empty_output><jupyter_text>1.8b) Heatmap For checking the correlation?<jupyter_code>fig,ax = plt.subplots(figsize=(10, 10))   
sns.heatmap(Data.corr(), ax=ax, annot=True, linewidths=0.05, fmt= '.2f',cmap="magma") # the color intensity is based on 
plt.show()
# we can confirm that TOEFL Score has highest postive co-relation with all the attributes.<jupyter_output><empty_output><jupyter_text>1.9 Check the Skewness and Kurtosis **Positively skewed:** Most frequent values are low and tail is towards high values.**Negatively skewed:** Most frequent values are high and tail is towards low values. If **Mode< Median< Mean** then the distribution is positively skewed.If **Mode> Median> Mean** then the distribution is negatively skewed.<jupyter_code>Data.skew()   # to measure the skeweness of every attribute<jupyter_output><empty_output><jupyter_text>1.10 Histogram for checking the Skewness<jupyter_code>import scipy.stats as stats

#convert pandas DataFrame object to numpy array and sort
h = np.asarray(Data['Fly ash'])
h = sorted(h)
 
#use the scipy stats module to fit a normal distirbution with same mean and standard deviation
fit = stats.norm.pdf(h, np.mean(h), np.std(h)) 
 
#plot both series on the histogram
# plt.plot(h,fit,'-',linewidth = 2,label="Normal distribution with same mean and var")
plt.hist(h,normed=True,bins = 100,label="Actual distribution")      
plt.legend()
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09374196534169794, 'fraction_numerical': 0.03311564765773641, 'mean_word_length': 4.250539956803456, 'pattern_counts': {'":': 0, '<': 97, '<?xml version=': 0, '>': 97, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '627379', 'license': 'MIT', 'n_tokens_mistral': 6271, 'n_tokens_neox': 5603, 'n_words': 2260, 'path': 'M3 Advance Statistics/W2 EDA/EDA -Part1_Basic_Statistics_Exploration (1) (1).ipynb', 'repo_name': 'fborrasumh/greatlearning-pgp-dsba', 'size': '19447'}