<jupyter_start><jupyter_text>Fundamentals of Data Analysis Project ** Introduction to Box-Plots**The schematic plot, box-and-whisker plot, the basic box plot is known by a number of names and was first introduced to the statisitical world by <NAME> in the 1970's. Ultimately box plots allow us to analyse and visually represent the distribution (spread) characteristics of a set or sets of data. The box plot is useful in the identification of distribution parameters such as skewness and symmetry. It also clearly identifies outliers and variability within the data set.The box plot is a powerful graphical technique for comparing samples from two or more different treatments or populations.  ** History of the Box Plot**The box plot has been used since the 1970's when <NAME> introduced it. It did not become widely known until the late 1970's from which is has become a staple of the graphical tools used within statistical analysis.<NAME> was an american statistician who as well as creating the box plot is also credited with coining the terms software and bit. While working for AT&T Bell Labratories he abbreviated the term binary digit to "bit" which was the first use of the now commonly used term today. The box plot was an advancement on the range bar and by identifying appropriate summary statistics that could be applied to developing the usefulness of the range bar for comparing distributions across groups was one of the advancements Tukey applied in the creation of the box plot.  **Constructing a Box-Plot**As previously stated the box plot is useful for revealing the central tendency and variability of a data set, the distribution (particularly symmetry or skewness) of the data, and the presence of outliers.The first step in constructing a box plot, the data within the sample data set has to be ordered (from smallest to larges) and divided into four equal sized groups, called quartile groups.Each group represents 25% of the total data set and is referred to as a quartile, ranging from the lower quartile to the upper quartile. The median is defined as the mid-point of the data set i.e. the middle number from the ordered range of data (smallest to largest) and is represented within the box plot graph as the line that divides the “Inter-Quartile Range” box.Any data value above the median line is in the upper 2 quartiles and any data below the median line is within the lower two quartiles. The upper and lower whiskers represent scores outside the middle 50% of datapoints ** Investigation and Explanation of the Box Plot**The box plot can incorporate seven data components that can be contained within the data to be represented graphically;- Outliers- Lower Extreme (Value)- Lower Whisker (Used to represent the lower quartile)- Lower Quartile- Median- Upper Quartile- Upper Whisker (Used to represent the upper quartile)- Upper Extreme (Value)Box plots can either be represented in vertical or horizontal format as per the graphical representation below.![verticalboxplot.png](attachment:verticalboxplot.png) ![horizontalboxplot.png](attachment:horizontalboxplot.png)The whiskers represent the variability within the data set that is outside the upper and lower quartiles. The whiskers extend from the ends of the lower and upper quartiles. Outliers are determined to be an observation that lies an abnormal distance from other values in a random sample from a population. How the abnormal distance is calculated is what has to be determined as eliminating data based on the theory that it is an outlier may bias the data if the data point is not truely an outlier.  - Outlier; an observation that lies an abnormal distance from other values in a random sample from a population.    - Lower whisker; Extends 1.5 box heights from the bottom of the box to the minimum data point.    - Interquartile range box;  Middle 50% of the data.    ·    Top line - Q3 (third quartile). 75% of the data are less than or equal to this value.    ·    Middle line - Q2 (median). 50% of the data are less than or equal to this value.    ·    Bottom line - Q1 (first quartile). 25% of the data are less than or equal to this value.       - Median; Is the middle value within the data set, 50% of the data is below this line and 50% are above.    - Upper whisker; Extends to the maximum data point within 1.5 box heights from the top of the box.     ** Effects of Sample Size on Box Plots**  A small sample size will shrink the box plot whiskers and give the boxplot the illusion of decreased variability. Another issue with using a boxplot with small samples is that the calculated quartiles can become meaningless.    ** Demonstration of the Boxplot**To demonstrate the use of the box plot I am going to generate two sets of random data. - The first random data set is going to follow the normal distribution.- The second random data set is going to follow the gumbel (SEV) distribution.Both these data sets will give a good discription and illustration of the use of the box plot.<jupyter_code>##Importing numpy library
import numpy as np
##Importing matplotlib library
import matplotlib.pyplot as plt
##Importing seaborn library
import seaborn as sns
## Required to prevent warning in relation to repeat label use in Matplotlib
import warnings
import matplotlib.cbook
warnings.filterwarnings("ignore",category=matplotlib.cbook.mplDeprecation)
## Generation of Random Data following the Normal Distribution
## Location will be represented by the Mean of the random data set which equals 25
## Scale will be represent by the Standard Deviation of the random data set which equals 2.5
## Random Data will include 1000 data points

NormalData = np.random.normal(25, 2.5, 1000)
## Creation of a Subplot Histogram Plot with Random Data following the Normal Distribution
plt.subplot(1,2,1)
plt.hist(NormalData)
plt.title("Normal Random Data")
plt.xlabel("Value")
plt.ylabel("Frequency")
## Creation of a Subplot Box Plot with Random Data following the Normal Distribution
plt.subplot(1,2,2)
plt.boxplot(NormalData)
plt.title("Normal Random Data")
plt.xlabel('Data Points')
plt.ylabel('Scale')
plt.xticks([1], ['Random Normal Data'])
plt.show()<jupyter_output><empty_output><jupyter_text>** Analysis of the Random Normal Data Set Box Plot**As can be seen from the histogram graph the data is centred around the mean value of 25 with approximately 50% of the data skewed to the left and 50% to the right. The graph is typical of the symmetric normal distribution with values ranging from approximately 8 to 36.The boxplot mirrors this distribution i.e. it illustrates a symmetric distribution with two out liers at the lower and upper ends outside the lower and upper extreme whisker values. Again the median value matches the mean value of 25 proving a symmetric distribution.<jupyter_code>## Generation of Random Data following the Gumbel Distribution
## Location will be represented by the Mean of the random data set which equals 25
## beta will be represent by the Standard Deviation of the random data set which equals 2.5
## Random Data will include 1000 data points

SEV=np.random.gumbel(25, 2.5, 1000)
## Creation of a Subplot Histogram Plot with Random Data following the Gumbel Distribution
plt.subplot(1,2,1)
plt.hist(SEV)
plt.title("Gumbel Random Data")
plt.xlabel("Value")
plt.ylabel("Frequency")
## Creation of a Subplot Box Plot with Random Data following the Gumbel Distribution
plt.subplot(1,2,2)
plt.boxplot(SEV)
plt.title("Gumbel Random Data")
plt.xlabel('Data Points')
plt.ylabel('Scale')
plt.xticks([1], ['Random Gumbel Data'])
plt.show()<jupyter_output><empty_output><jupyter_text>** Analysis of the Random Normal Data Set Box Plot**As can be seen from the histogram graph the data is skewed to the right of the mean value of 25. Again the graph is typical of an extreme value distribution with values ranging from approximately 16 to 41 with a greater number of outliers tailing towards the extreme value compared to the normal distribution with the same parameters.The boxplot mirrors this distribution i.e. it illustrates a a right skewed distribution tending towards the upper extreme value of 41 with two out liers at the lower and upper ends outside of the lower and upper extreme whisker values. Again the median value matches the mean value of 25 proving a symmetric distribution. ** Comparison of Normal and Gumbel Distributions **Below I will graph both box plots side by side to illustrate how the box plots can clearly differentiate between two distributions. It will also illustrate that even though two sets of data with the same parameters i.e. mean (location) and standard deviation (scale, beta is the scale factor for the Gumbel Distribution and can approximate the standard deviation) can have very different distributions.<jupyter_code>## Creation of a Box Plot to Compare Normal and Gumbel Distributions
BoxPlot=[NormalData, SEV]
plt.title("Comparison of Normal and Gumbel Distributions")
plt.boxplot(BoxPlot)
plt.xlabel('Distributions')
plt.ylabel('Scale')
plt.xticks([1, 2], ['Normal Distribution', 'Gumbel Distribution'])
plt.show()<jupyter_output><empty_output><jupyter_text>** Analysis of the Box Plots for the Normal and Gumbel Distributions **As outlined earlier the location parameter for both distributions selected was 25. As can be seen from the box plots the normal distribution mean is exactly the same as the median line as the data is ranked. As the Gumbel distribution is right skewed the median line is slightly above the mean value of 25.The box plots clearly display that Gumble distribution is right skewed and the Normal distribution displays symmetry which is a visual que for the acceptance of the Normal distribution.You can also determine that 50% of the Normal distribution values range from approximately 23 to 27 while they range from approximately 24 to 28 for the Gumbel distribution.The first quartile for the Normal distribution ranges from approximately 16 to 23 while it ranges from 17 to 24 for the Gumbel distribution. Finally the fourth quartile for the Normal distribution ranges from 27 to 33 and 28 to 41 for the Gumbel distribution. ** Alternative Graphical Displays to Box Plots**Other graphical options that could be used as alternatives to the standard box plot are;- Violin (Bean) Plots- Interval Plots- Distribution PlotsBased on the random data generated earlier I will generate each of the listed plots to contrast the visual characteristics.<jupyter_code>## Creation of a Subplot Violin Plot with Random Data following the Normal Distribution
plt.subplot(2,2,1)
sns.violinplot(NormalData)
plt.title("Normal Random Data")
plt.xlabel('Points')
plt.ylabel('Scale')
## Creation of a Subplot Point Plot with Random Data following the Normal Distribution
plt.subplot(2,2,2)
sns.pointplot(NormalData,capsize=.2)
plt.title("Normal Random Data")
plt.xlabel('Points')
## Creation of a Subplot Box Plot with Random Data following the Normal Distribution
plt.subplot(2,2,3)
plt.boxplot(NormalData)
plt.title("Normal Random Data")
plt.xlabel('Data Points')
plt.ylabel('Scale')
plt.xticks([1], ['Random Normal Data'])
## Displaying the Sub Plots with the inclusion of a Distribution Plot for the Normal Distribution
plt.subplot(2,2,4)
sns.distplot(NormalData)
plt.title("Normal Random Data")
plt.xlabel('Data Points')
plt.ylabel('Scale')
plt.tight_layout() ## Fix for Over Lapping Axis and Labels
plt.show()
## Creation of a Subplot Violin Plot with Random Data following the Gumbel Distribution
plt.subplot(2,2,1)
sns.violinplot(SEV)
plt.title("Gumbel Random Data")
plt.xlabel('Points')
plt.ylabel('Scale')
## Creation of a Subplot Point Plot with Random Data following the Gumbel Distribution
plt.subplot(2,2,2)
sns.pointplot(SEV,capsize=.2)
plt.title("Gumbel Random Data")
plt.xlabel('Points')
## Creation of a Subplot Box Plot with Random Data following the Gumbel Distribution
plt.subplot(2,2,3)
plt.boxplot(SEV)
plt.title("Gumbel Random Data")
plt.xlabel('Data Points')
plt.ylabel('Scale')
plt.xticks([1], ['Random Gumbel Data'])
## Displaying the Sub Plots with the inclusion of a Distribution Plot for the Gumbel Distribution
plt.subplot(2,2,4)
sns.distplot(SEV)
plt.title("Gumbel Random Data")
plt.xlabel('Data Points')
plt.ylabel('Scale')
plt.tight_layout() ## Fix for Over Lapping Axis and Labels
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.05770321744063538, 'fraction_numerical': 0.013210146689358944, 'mean_word_length': 5.0848126232741615, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '119801', 'license': 'Apache-2.0', 'n_tokens_mistral': 3328, 'n_tokens_neox': 3030, 'n_words': 1758, 'path': 'Fundamentals of Data Analysis Project.ipynb', 'repo_name': 'shanecanny1/Fundamentals-of-Data-Analysis-Project', 'size': '12365'}