<jupyter_start><jupyter_text>An Investigation of Anscombe's Quartet<jupyter_code>#Fundamentals of Data Analyis Project GMIT "<NAME>" 14-10-18
#https://en.wikipedia.org/wiki/Anscombe%27s_quartet<jupyter_output><empty_output><jupyter_text>Backround: Anscombe's Quartet was constructed by <NAME> in 1973. Anscombe created 4 datasets that exhibited the same summary statisitics. The purpose of Anscombes Quartet was to show that standard summary statistics can lead to incorrect conclusions[1]. When you use the standard decriptive methods to describe the data like mean, variance, standard deviation, correlation and linear regression the results are almost indentical. When the dataset's are plotted visually it tells a different story. The graphic plots reveal the data and this can be more precise and and tell a clearer story than conventional statistical methods[2]. This will be investigated in this notebook.Anscombe thought that computer programs, textbooks and statisical analysis did not pay enough attention to graphs that can help us see if there is something unique in the data[3]. Anscombe advocated that graphs aided in enhancing these methods. If data is not visualised it may lead to different data that gives same conclusions which is not optimal for data analysis[4].<jupyter_code>#import libraries
import matplotlib.pyplot as plt
import numpy as np
import pylab
Polynomial = np.polynomial.Polynomial
import matplotlib.patches as mpatches
#numpy array Anscombes data
x1 = np.array([10, 8, 13, 9, 11, 14, 6, 4, 12, 7, 5])
y1 = np.array([8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68])
x2 = np.array([10, 8, 13, 9, 11, 14, 6, 4, 12, 7, 5])
y2 = np.array([9.14, 8.14, 8.74, 8.77, 9.26, 8.10, 6.13, 3.10, 9.13, 7.26, 4.74])
x3 = np.array([10, 8, 13, 9, 11, 14, 6, 4, 12, 7, 5])
y3 = np.array([7.46, 6.77, 12.74, 7.11, 7.81, 8.84, 6.08, 5.39, 8.15, 6.42, 5.73])
x4 = np.array([8, 8, 8, 8, 8, 8, 8, 19, 8, 8, 8])
y4 = np.array([6.58, 5.76, 7.71, 8.84, 8.47, 7.04, 5.25, 12.50, 5.56, 7.91, 6.89])<jupyter_output><empty_output><jupyter_text>A Simple Quartet of Plots of Anscombe's Data Shows Variation and Outliers.<jupyter_code>#plot x andy y data

plt.plot(x1, y1, 'go')
plt.title("I")
plt.ylabel("Y1")
plt.xlabel("X1")
plt.axis([2, 21, 2, 14])
plt.show()


plt.plot(x2, y2, 'go')
plt.title('II')
plt.ylabel("Y2")
plt.xlabel("X2")
plt.axis([2, 21, 2, 14])
plt.show()

plt.plot(x3, y3, 'go')
plt.title('III')
plt.ylabel("Y3")
plt.xlabel("X3")
plt.axis([2, 21, 2, 14])
plt.show()

plt.plot(x4, y4, 'go')
plt.title('IV')
plt.ylabel("Y4")
plt.xlabel("X4")
plt.axis([2, 21, 2, 14])
plt.show()<jupyter_output><empty_output><jupyter_text>The Descriptive Data Gives the Impression the Datasets are Indentical When Looking at Central Tendency.<jupyter_code>#Calculate the mean of x data 
meanx1 = np.mean(x1)
print("Mean of X1 is", meanx1)
meanx2 = np.mean(x2)
print("Mean of X2 is", meanx2)
meanx3 = np.mean(x3)
print("Mean of X3 is", meanx3)
meanx4 = np.mean(x4)
print("Mean of X4 is", meanx4)
#Calculate mean of all y data
meany1 = np.mean(y1)
print("Mean of Y1 is", meany1)
meany2 = np.mean(y2)
print("Mean of Y2 is", meany2)
meany3 = np.mean(y3)
print("Mean of Y3 is", meany3)
meany4 = np.mean(y4)
print("Mean of Y4 is", meany4)
#calculate standard deviation of x data
standardx1 = np.std(x1)
print("Standard Deviation of X1 is", standardx1)
standardx2 = np.std(x2)
print("Standard Deviation of X2 is", standardx2)
standardx3 = np.std(x3)
print("Standard Deviation of X3 is", standardx3)
standardx4 = np.std(x4)
print("Standard Deviation of X4 is", standardx4)
#calculate standard deviation of y data
standardy1 = np.std(y1)
print("Standard Deviation of Y1 is", standardy1)
standardy2 = np.std(y2)
print("Standard Deviation of Y2 is", standardy2)
standardy3 = np.std(y3)
print("Standard Deviation of Y3 is", standardy3)
standardy4 = np.std(y4)
print("Standard Deviation of Y4 is", standardy4)
#Show Pearson Correlation Coefficients of x and y data
coex1 = np.corrcoef(x1, y1)
print("Correlation between X1 and Y1 is", coex1)
coex2 = np.corrcoef(x2, y2)
print("Correlation between X1 and Y1 is", coex2)
coex3 = np.corrcoef(x3, y3)
print("Correlation between X3 and Y3 is", coex3)
coex4 = np.corrcoef(x4, y4)
print("Correlation between X4 and Y4 is", coex4)
#show the variance of both x and y data
varx1 = np.var(x1)
print("Variance of x1 is", varx1)
varx2 = np.var(x2)
print("Variance of x2 is", varx2)
varx3 = np.var(x3)
print("Variance of x3 is", varx3)
varx4 = np.var(x4)
print("Variance of x4 is", varx4)
vary1 = np.var(y1)
print("Variance of y1 is", vary1)
vary2 = np.var(y2)
print("Variance of y2 is", vary2)
vary3 = np.var(y3)
print("Variance of y3 is", vary3)
vary4 = np.var(y4)
print("Variance of y4 is", vary4)<jupyter_output>Variance of x1 is 10.0
Variance of x2 is 10.0
Variance of x3 is 10.0
Variance of x4 is 10.0
Variance of y1 is 3.75206280992
Variance of y2 is 3.75239008264
Variance of y3 is 3.74783636364
Variance of y4 is 3.74840826446<jupyter_text>Plot x and y data with best fit line. This gives a clear picture of the difference between visualising the data over the decriptive results. This will show if relationship maybe linear. Below is edited from https://scipython.com/book/chapter-6-numpy/examples/finding-a-best-fit-straight-line/<jupyter_code>#x1 and y1 straight line fit , root mean squared and show residual left over


cmin, cmax = min(x1), max(x1)
pfit, stats = Polynomial.fit(x1, y1, 1, full=True, window=(cmin, cmax),
                                                    domain=(cmin, cmax))

print('Raw fit results:', pfit, stats, sep='\n')
#use numpy root mean squared and polyfit
x10, m = pfit
resid, rank, sing_val, rcond = stats
rms = np.sqrt(resid[0]/len(y1))

 
print('Fit: A = {:.2f}[P] + {:.2f}'.format(m, x10),
      '(rms residual = {:.2f})'.format(rms))

#plot x1 and y1 with best fit line
pylab.plot(x1, y1, 'o', color='m')
pylab.plot(x1, pfit(x1), color='navy')
pylab.title('I')
pylab.ylabel('y1')
pylab.xlabel('x1')
navy_line = mpatches.Patch(color='navy', label='Best Fit Line')
pylab.legend(handles=[navy_line])
pylab.show()


#x2 and y2 straight line fit , root mean squared and show residual left over 
cmin, cmax = min(x2), max(x2)
pfit, stats = Polynomial.fit(x2, y2, 1, full=True, window=(cmin, cmax),
                                                    domain=(cmin, cmax))

print('Raw fit results:', pfit, stats, sep='\n')

x10, m = pfit
resid, rank, sing_val, rcond = stats
rms = np.sqrt(resid[0]/len(y2))

print('Fit: A = {:.2f}[P] + {:.2f}'.format(m, x10),
      '(rms residual = {:.2f})'.format(rms))

#plot x2 and y2 with best fit line
pylab.plot(x2, y2, 'o', color='m')
pylab.plot(x2, pfit(x2), color='navy')
pylab.title("II")
pylab.ylabel("Y2")
pylab.xlabel("X2")
navy_line = mpatches.Patch(color='navy', label='Best Fit Line')
pylab.legend(handles=[navy_line])
pylab.show()

#x3 and y3 straight line fit , root mean squared and show residual left over 
cmin, cmax = min(x3), max(x3)
pfit, stats = Polynomial.fit(x3, y3, 1, full=True, window=(cmin, cmax),
                                                    domain=(cmin, cmax))

print('Raw fit results:', pfit, stats, sep='\n')

x10, m = pfit
resid, rank, sing_val, rcond = stats
rms = np.sqrt(resid[0]/len(y3))

print('Fit: A = {:.2f}[P] + {:.2f}'.format(m, x10),
      '(rms residual = {:.2f})'.format(rms))

#plot x3 and y3 with best fit line
pylab.plot(x3, y3, 'o', color='m')
pylab.plot(x3, pfit(x3), color='navy')
pylab.title("II")
pylab.ylabel("Y3")
pylab.xlabel("X3")
navy_line = mpatches.Patch(color='navy', label='Best Fit Line')
pylab.legend(handles=[navy_line])
pylab.show()

#x4 and and y4 straight line fit , root mean squared and show residual left over 
cmin, cmax = min(x4), max(x4)
pfit, stats = Polynomial.fit(x4, y4, 1, full=True, window=(cmin, cmax),
                                                    domain=(cmin, cmax))

print('Raw fit results:', pfit, stats, sep='\n')

x10, m = pfit
resid, rank, sing_val, rcond = stats
rms = np.sqrt(resid[0]/len(y4))

print('Fit: A = {:.2f}[P] + {:.2f}'.format(m, x10),
      '(rms residual = {:.2f})'.format(rms))

#plot x4 and y4 with best fit line
pylab.plot(x4, y4, 'o', color='m')
pylab.plot(x4, pfit(x4), color='navy')
pylab.title("IV")
pylab.ylabel("Y4")
pylab.xlabel("X4")
navy_line = mpatches.Patch(color='navy', label='Best Fit Line')
pylab.legend(handles=[navy_line])
pylab.show()<jupyter_output>Raw fit results:
poly([ 3.00009091  0.50009091])
[array([ 13.76269]), 2, array([ 1.39407903,  0.23778908]), 2.4424906541753444e-15]
Fit: A = 0.50[P] + 3.00 (rms residual = 1.12)
{'fraction_non_alphanumeric': 0.15227008149010476, 'fraction_numerical': 0.07182770663562282, 'mean_word_length': 4.3493150684931505, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '624172', 'license': 'Apache-2.0', 'n_tokens_mistral': 3768, 'n_tokens_neox': 3385, 'n_words': 1028, 'path': 'Fundamentals of Data Analysis Assignment.ipynb', 'repo_name': 'GregFee/Anscombes-Data', 'size': '8605'}