<jupyter_start><jupyter_text>Anscombe's QuartetAnscombes Quartet is a series of four datasets that are used as a demonstration of the limitations of using summary statistics alone to analyse a dataset and the benefits of plotting and visualising data.  The four datasets have the same or similar Mean values, Variances, Correlation, Coefficient of determination and line of best fit however when the data is displayed in a plot it can be seen that the individual data points are very different.The data sets were constructed by <NAME> in 1973 to counter the misconception among statisticians that numerical values are exact but graphs are rough. The Pandas library will be used for the numerical analysis of the dataset and the Seaborn libray will be used to visualise the dataset.  The dataset can be loaded from the seaborn library [1]<jupyter_code>import pandas as pd
import seaborn as sns
# Load the example dataset for Anscombe's quartet taken from the seaborn library
df = sns.load_dataset("anscombe")
df<jupyter_output><empty_output><jupyter_text>Summary StatisticsDisplaying the imported dataframe shows that the quartet consists of four datasets I-IV that each contain 11 X,Y values.  The summary statistics for the entire quartet can be calculated using the pandas describe function.<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>We can also view the summary statistics for each individual data set with the quartet using the describe and filtering functionality of pandas.<jupyter_code>print ('Dataset I')
print (df[df['dataset']=='I'].describe())
print ('Dataset II')
print (df[df['dataset']=='II'].describe())
print ('Dataset III')
print (df[df['dataset']=='III'].describe())
print ('Dataset IV')
print (df[df['dataset']=='IV'].describe())<jupyter_output>Dataset I
               x          y
count  11.000000  11.000000
mean    9.000000   7.500909
std     3.316625   2.031568
min     4.000000   4.260000
25%     6.500000   6.315000
50%     9.000000   7.580000
75%    11.500000   8.570000
max    14.000000  10.840000
Dataset II
               x          y
count  11.000000  11.000000
mean    9.000000   7.500909
std     3.316625   2.031657
min     4.000000   3.100000
25%     6.500000   6.695000
50%     9.000000   8.140000
75%    11.500000   8.950000
max    14.000000   9.260000
Dataset III
               x          y
count  11.000000  11.000000
mean    9.000000   7.500000
std     3.316625   2.030424
min     4.000000   5.390000
25%     6.500000   6.250000
50%     9.000000   7.110000
75%    11.500000   7.980000
max    14.000000  12.740000
Dataset IV
               x          y
count  11.000000  11.000000
mean    9.000000   7.500909
std     3.316625   2.030579
min     8.000000   5.250000
25%     8.000000   6.170000
50%     8.000000   7.040000
75% [...]<jupyter_text>The summary statistics show the Mean of the x values to be an exact match for all four datasets and the mean y values match to 2 decimal places.Another characteristic of the quartet is that the variances for x and y match in each datset.  This can be seen by calling the var function in the pandas dataframe.  The results of which show an exact match for the x variance and a match to +/- 0.003 for the y variance.<jupyter_code>print ('Dataset I')
print (df[df['dataset']=='I'].var())
print ('Dataset II')
print (df[df['dataset']=='II'].var())
print ('Dataset III')
print (df[df['dataset']=='III'].var())
print ('Dataset IV')
print (df[df['dataset']=='IV'].var())<jupyter_output>Dataset I
x    11.000000
y     4.127269
dtype: float64
Dataset II
x    11.000000
y     4.127629
dtype: float64
Dataset III
x    11.00000
y     4.12262
dtype: float64
Dataset IV
x    11.000000
y     4.123249
dtype: float64<jupyter_text>The correlation can also be calculated using pandas to show a matching correlation between x and y for each dataset to 3 decimal places.<jupyter_code>print ('Dataset I')
print (df[df['dataset']=='I'].corr())
print ('Dataset II')
print (df[df['dataset']=='II'].corr())
print ('Dataset III')
print (df[df['dataset']=='III'].corr())
print ('Dataset IV')
print (df[df['dataset']=='IV'].corr())<jupyter_output>Dataset I
          x         y
x  1.000000  0.816421
y  0.816421  1.000000
Dataset II
          x         y
x  1.000000  0.816237
y  0.816237  1.000000
Dataset III
          x         y
x  1.000000  0.816287
y  0.816287  1.000000
Dataset IV
          x         y
x  1.000000  0.816521
y  0.816521  1.000000<jupyter_text>Visualising the DataThe seaborn library contains a function called regplot that is used to plot data and a linear regression model fit.  This function has been used to plot each dataset to observe patterns in the data.  Dataset I<jupyter_code>import warnings #removes future warning from regplot
warnings.filterwarnings('ignore')

sns.regplot(x="x", y="y", data=df[df['dataset']=='I'])<jupyter_output><empty_output><jupyter_text>The linear regression model for dataset I shows a good fit.  It can be seen that an increse in x produces a proportional increase in y and this looks like a continuous trend.  There are some outliers but these are generally within 1-2 units of the y value.  Dataset II<jupyter_code>sns.regplot(x="x", y="y", data=df[df['dataset']=='II'])<jupyter_output><empty_output><jupyter_text>The data in dataset II seems to follow a curve and does not fit well to a linear regression model.  The fitted line seems to be lower than all the mid range x values and higher than all the x values at the extremities. The trend suggest that an increse or decrease in x value will result in y values diverging further from the modeled line.    Dataset III<jupyter_code>sns.regplot(x="x", y="y", data=df[df['dataset']=='III'])<jupyter_output><empty_output><jupyter_text>The linear regression model for dataset III seems to be influenced by an outlying value that misrepresents the general trend.  Ten of the eleven values conform to a proportional linear pattern that looks continuous however one outlying data point has rotetd the line in a way that looks as though lower x values will increasingly diverge above the line and higher y values will increasingly diverge below the line.   Dataset IV<jupyter_code>sns.regplot(x="x", y="y", data=df[df['dataset']=='IV'])<jupyter_output><empty_output><jupyter_text>Dataset IV shows an incorrectly modelled dataset.  It can be seen that ten out of eleven datapoints share the same x value (8.0) however one outlying datapoint (19.0) has resulted in a proprotional linear increase between x and y.  A more reliable fit would be a vertical line on or close to the x = 8.0 value though for y values between 5.25 and 12.5. There does not seem to be any correlation between x and y so it is unlikely the relationship can be modelled beyond the observed range of values.  This dataset shows how one outlier can produce a high correlation coefficient.  Seaborn.implot The seaborn.implot function combines the regplot function and adds it to factegrid, this enables multiple conditional data subsets and linear regression models to be plot on a grid.  Plotting these datasets side by side it can be seen that they all have the same linear regression line.The below line of code is taken from the seaborn [anscombe_quartet](https://seaborn.pydata.org/examples/anscombes_quartet.html) example.<jupyter_code># Show the results of a linear regression within each dataset
sns.lmplot(x="x", y="y", col="dataset", hue="dataset", data=df,
           col_wrap=2, ci=None, palette="muted", height=4,
           scatter_kws={"s": 50, "alpha": 1})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08279328131199577, 'fraction_numerical': 0.08874487501653221, 'mean_word_length': 3.9135802469135803, 'pattern_counts': {'":': 2, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '480860', 'license': 'Apache-2.0', 'n_tokens_mistral': 2706, 'n_tokens_neox': 2246, 'n_words': 970, 'path': 'Anscombes-Quartet.ipynb', 'repo_name': 'tommirrington/Anscombes-Quartet', 'size': '7571'}