<jupyter_start><jupyter_text>Machine Learning Assignment Tasks 2020This notebook will present solutions to four tasks which were asigned with Machine Learning and Statistics in this semester.  * Student Name: **<NAME>*** Student ID: G00376332* Lecturer: Dr. <NAME>   Table of Contents* [Goals and objectives](goals)* [Task 1 - Calculate the square root of 2](task1)* [Task 2 - Chi-Squared Test](task2)* [Task 3 - Excel Standard Deviation](task3)* [Task 4 - Scikit-learn Iris Data Set - K-Means Clustering ](task4)* [References](references)  Goals and objectives * Research and investigations of each task * Introduce approach and solution for all tasks  * Implementation and presentation results of each task  Task 1 - Calculate the square root of 2***  (Wikipedia, 2016)  Description Write a Python function called sqrt2 that calculates and prints to the screen the square root of 2 to 100 decimal places. Your code should not depend on any module from the standard library or otherwise. You should research the task first and include references and a description of your algorithm.  Research The square root of 2, or the one-half power of 2, written in mathematics as $\sqrt{2}$, is the positive algebraic number that, when multiplied by itself, equals the number 2. Technically, it must be called the principal square root of 2, to distinguish it from the negative number with the same property.[1]Methods of computing square roots are numerical analysis algorithms for finding the principal, or non-negative, square root (usually denoted $\sqrt(S)$, $2\sqrt(S)$, or $S1/2$) of a real number. Arithmetically, it means given S, a procedure for finding a number which when multiplied by itself, yields S; algebraically, it means a procedure for finding the non-negative root of the equation $x2 - S = 0$; geometrically, it means given the area of a square, a procedure for constructing a side of the square.Every real number has two square roots. The principal square root of most numbers is an irrational number with an infinite decimal expansion. As a result, the decimal expansion of any such square root can only be computed to some finite-precision approximation. However, even if we are taking the square root of a perfect square integer, so that the result does have an exact finite representation, the procedure used to compute it may only return a series of increasingly accurate approximations.[2]To start calculating and implement Python code for $\sqrt{2}$ it is good to review available tools and most suitible methods of calculation.The most common analytical methods are iterative and consist of two steps: finding a suitable starting value, followed by iterative refinement until some termination criteria is met. The starting value can be any number, but fewer iterations will be required the closer it is to the final result. The most familiar such method, most suited for programmatic calculation, is **Newton's method**, which is based on a property of the derivative in the calculus. A few methods like **paper-and-pencil, synthetic division and series expansion, do not require a starting value**. In some applications, an integer square root is required, which is the square root rounded or truncated to the nearest integer (a modified procedure may be employed in this case).The method employed depends on what the result is to be used for (i.e. **how accurate it has to be**), how much effort one is willing to put into the procedure, and what tools are at hand. The methods may be roughly classified as those suitable for mental calculation, those usually requiring at least paper and pencil, and those which are implemented as programs to be executed on a digital electronic computer or other computing device. Algorithms may take into account convergence (how many iterations are required to achieve a specified precision), computational complexity of individual operations (i.e. division) or iterations, and error propagation (the accuracy of the final result).Procedures for finding square roots (particularly the square root of 2) have been known since at least the period of ancient Babylon in the 17th century BCE. Heron's method from first century Egypt was the first ascertainable algorithm for computing square root. Modern analytic methods began to be developed after introduction of the Arabic numeral system to western Europe in the early Renaissance. Today, nearly all computing devices have a fast and accurate square root function, either as a programming language construct, a compiler intrinsic or library function, or as a hardware operator, based on one of the described procedures.[2] Base on the above reviews it seems to be a good approach to use Newton's method for the begining. Newton’s method, also known as Newton-Raphson method is a root-finding algorithm that produces successively better approximations of the roots of a real-valued function. The approximations of the root go as:$x_(n+1) = x_n - f(x_n) / f’(x_n)$$x_0$ is the rough approximation of the root done at the first and the successive approximations go as $x_1, x_2, ….$$f(x_n)$ is the function whose root is to be determined and f’(x_n) is the derivative of the function. **Newton’s method for square root**If there have to be find the square root of a number $n$, the function would be $f(x) = x² - N$ and the root of the function, $f(x)$ would have to be found.The value $f(x_n)$ at $x = x_n$ is:$f(x_n) = x_n² - N$And, the derivative at the point is:$f’(x_n) = 2 * x_n$Now, the better approximation can be found using (1).$x_(n+1) = x_n - (x_n² - N) / (2 * x_n)$$x_(n+1) = x_n - x_n² / (2 * x_n) + N/ (2 * x_n)$$x_(n+1) = x_n - x_n / 2+ N/ (2 * x_n)$$x_(n+1) = x_n / 2+ N/ (2 * x_n)$$x_(n+1) = (x_n + N/ x_n) / 2$This is how the algorithm for finding square root of a number comes.  Employing python to calculate the square root of 2  To start writing a function in Python is good to know first what tools are offered and what are their limitations.The task doesn't allow it to use any standard python librarys however it is worth to check what results can be produced by those modules as a reference point.The math library contains **sqrt** function that calculates the square root of a number in a very convenient way with limited precision. For better understanding of squere root is also good to plot graph that will show how that kind function will look like.<jupyter_code># Import all required librarys
import numpy as np 
import matplotlib.pyplot as plt
import math 

# Use magic function to render the figure in a notebook
%matplotlib inline

# Returns 1000 evenly spaced samples for numbers in range 0-10
x = np.linspace(0, 10, 1000)
# Use python built-in function for calculation of square root
y = np.sqrt(x)

# Define chart size
plt.figure(figsize=(10,5))
# Add grid lines
plt.grid(True)
# Add title to the chart
plt.title("Visualization of Square Root Function", fontweight="bold", fontsize=20)
# # Add x lable to the chart
plt.xlabel('x', size=12)
# Add y lable to the chart
plt.ylabel('$y=\sqrt{x}$', size=12)


plt.plot(x,y)
plt.show()<jupyter_output><empty_output><jupyter_text>Looking at the figure on the above graph is visible that square root of 2 is around 1.4.  Now is worth to check what result can be obtained by using python built-in function to calculate $\sqrt2$.<jupyter_code># Calculate square root of 2 using python function
math.sqrt(2)<jupyter_output><empty_output><jupyter_text>The next step is to see how precise the result is.<jupyter_code># Display result with precition of 100 decimal places 
print (format(math.sqrt(2),'.100f'))<jupyter_output>1.4142135623730951454746218587388284504413604736328125000000000000000000000000000000000000000000000000<jupyter_text>The result has its limitation and does not give a required precision even as a reference point for this task. Python buil-in **sqrt()** function has issue with floating point representation. Representation error refers to the fact that some (most, actually) decimal fractions cannot be represented exactly as binary (base 2) fractions. This is the chief reason why Python (or Perl, C, C++, Java, Fortran, and many others) often won’t display the exact decimal number that is expected.[3]This can be very easy shown by multiplying $\sqrt2$ * $\sqrt2$. The expected result should be obviously equal to **2**.<jupyter_code># Checking the result of multiplication of sqrt(2) * sqrt(2)
math.sqrt(2)*math.sqrt(2)<jupyter_output><empty_output><jupyter_text>Task Solution - Newton's method The issue with decimal point representation requires different approach to resolve this task.Function sqrt2 will be created base on the Newton’s method for square root which was choosen for this task in research section.To implement this method to the code and create function it is necessary write a simple translation of the Newtons method to python. Computers typically compute the square root of $x$ using a loop. Starting with some guess $z$, the z based can be adjusted on how close $z^2$ is to $x$, producing a better guess:$$ z_{next} = z - \frac{z^2-x}{2z} $$Repeating this adjustment makes the guess better and better until an answer that is as close to the actual square root as can be is reached.<jupyter_code># Adapted from: https://tour.golang.org/flowcontrol/8 [4]
 
def n_sqrt2(x):
  # Initial guess.
  z = 1.0
  # Keep getting a better estimate for the square root
  # of x, until you are within two decimal places.
  while abs(z*z - x) >= 0.0001:
    # Get a better approximation for the square root.
    z -= (z*z - x) / (2*z)
  # Return z.
  return z
 
# Calculate the square root of 2.
z = n_sqrt2(2)
# Print z.
print(format (z,'.100f'))
# Print the square of the square of z.
print(z*z)<jupyter_output>1.4142156862745098866440685014822520315647125244140625000000000000000000000000000000000000000000000000
2.000006007304883<jupyter_text>The results in this case also don't satisfy requirement for this task. The function n_sqrt2 which uses Newton's method have the same issue with decimal point representation.To overcome comon problem with floating point decimal representation different approach has to be considered.The easiest way is to use integer calculations rather than floating point.Below code works by finding the integer square root of $a$ after multiplying by $10$ raised to the $2 x digits$.At the end the result needs to be converted to a string and add the decimal point at the correct location. The downside of this method is that converting a very large integer to a string isn't very fast but in this task case (100 digits) doen't cause any issues.<jupyter_code># Adapted from:https://stackoverflow.com/questions/5187664/generating-digits-of-square-root-of-2 [5]

# Create sqrt2 function
def sqrt2(a, digits):
    '''This function will calculate square root of a given number with defined decimal digit precision '''
    # Define required precition as a integer number
    a = a * (10**(2*digits))
    # Define start point 
    x_prev = 0
    # Define next point of approximation
    x_next = 1 * (10**digits)
    # Using while loop iterate steps to get closer approximation
    while x_prev != x_next:
        x_prev = x_next
        # Claculate next step and use bit shift by 1
        x_next = (x_prev + (a // x_prev)) >> 1
    # The loop stops when x_prev is equal t x_next     
    return x_next
# Final result before converting to string
sq2=(sqrt2(2, 100))
# Format string with appropriete . place for final result and display result
print(f'{sq2 // 10**100}.{r % 10**100:0100d}')<jupyter_output><empty_output><jupyter_text>To compare any calculation of square root of two with 100 decimal places precition, it is important to have very acurate value as a reference point. The good source that can be used is computation done by <NAME> in NASA.[6]<jupyter_code>nasa="1.414213562373095048801688724209698078569671875376948073176679737990732478462107038850387534327641572735013846230912"
1
print(nasa[:102])<jupyter_output><empty_output><jupyter_text>Python has built-in module called Decimal. The decimal module provides support for fast correctly-rounded decimal floating point arithmetic. It offers several advantages over the float datatype.[7] It is a good idea to check what result it produces for square root of 2 and use for validation against this task calculations as well.<jupyter_code># Import Decimal 
from decimal import *
# Set required precition
getcontext().prec = 101
# Display result
print(Decimal(2).sqrt())<jupyter_output><empty_output><jupyter_text>Summary and Conclusions Standard methods of calculation of square root of two have a precision limitation due to to the fact that number is irrational and computers have an issue with floating-point representation. Fractions cannot be represented exactly as binary fractions, base 2 system. If precision is the key point in calculations this needs to be considered when floating-point numbers are engaged. In this task, the remedy to representation issue was using integers for calculation and convert result to string with proper decimal point shift. This is not an ideal solution, and it has its downsides, however, for purpose of this task is acceptable.  [**<<<**](table)  Task 2 - Chi-Squared Test***  Description The Chi-squared test for independence is a statistical hypothesis test like a t-test. It is used to analyse whether two categorical variables are independent. The [Wikipedia article](https://en.wikipedia.org/w/index.php?title=Chi-squared_test&oldid=983024096) gives the table below as an example, stating the Chi-squared value based on it is approximately 24.6. Use scipy.stats to verify this value and calculate the associated p value. You should include a short note with references justifying your analysis in a markdown cell. |              | A   | B   | C   | D   | Total ||--------------|-----|-----|-----|-----|-------|| White Collar | 90  | 60  | 104 | 95  | 349   || Blue Collar  | 30  | 50  | 51  | 20  | 151   || No Collar    | 30  | 40  | 45  | 35  | 150   || Total        | 150 | 150 | 200 | 150 | 650   |  Research The Pearson’s Chi-Squared test, or just Chi-Squared test for short, is named for <NAME>, although there are variations on the test.The Chi-Squared test is a statistical hypothesis test that assumes (the null hypothesis) that the observed frequencies for a categorical variable match the expected frequencies for the categorical variable. The test calculates a statistic that has a chi-squared distribution, named for the Greek capital letter Chi (X) pronounced “ki” as in kite.[8]The chi-square statistical test is used to determine whether there’s a significant difference between an expected distribution and an actual distribution. It’s typically used with categorical data such as educational attainment, colors, or gender.[9]The Chi-Squared test uses a contingency table, first calculating the expected frequencies for the groups, then determining whether the division of the groups, called the observed frequencies, matches the expected frequencies.  The result of the test is a test statistic that has a chi-squared distribution and can be interpreted to reject or fail to reject the assumption or null hypothesis that the observed and expected frequencies are the same.[10]  $\chi^2$ -Test in Python Contingency table was provided for the purpose of this task.When analysis of categorical data is concerned with more than one variable, two-way tables (also known as contingency tables) are employed. These tables provide a foundation for statistical inference, where statistical tests question the relationship between the variables on the basis of the data observed.[11]<jupyter_code># Import all required librarys
import numpy as np
import pandas as pd
import statsmodels.stats.weightstats as stat
import scipy.stats as ss
import matplotlib.pyplot as plt


# Create contingency table that was provided for the task
table = pd.DataFrame({"A":[90,30,30],"B":[60,50,40],"C":[104,51,45],
                      "D":[95,20,35]}, index=["White Collar","Blue Collar","No Collar"])
print("Contingency Table")
table<jupyter_output>Contingency Table<jupyter_text>Is good to visualize this data.<jupyter_code>table.plot(kind='bar', title='Residence by Occupation')
plt.xlabel("Collar") 
plt.ylabel("Number of Recidence") 
plt.show()<jupyter_output><empty_output><jupyter_text>Having a contingency table ready the next step is to create code and run the chi-squared test. This test should verify if two categorical variables are independent of each other.<jupyter_code># Run Chi-square test of independence of variables in a contingency table

# chi2 = the test statistic, p = probability, dof = degrees of freedom, ex = the expected frequencies
chi2, p, dof, ex = ss.chi2_contingency(table)
# Print results 
print("The Chi2 is: " + format(chi2,'.4f')) 
print("The p value is: "+ format(p,'.4f'))
print("Degrees of freedom: ", dof)
#print("The expected frequencies: ", ex)<jupyter_output>The Chi2 is: 24.5712
The p value is: 0.0004
Degrees of freedom:  6<jupyter_text>Summary and Conclusions The obtained results show that null hypothesis can be rejected in this case. The null hypothesis is that each person's neighborhood of residence is independent of the person's occupational classification.[12] The p value is below threshold of 0.05 what means that the variables are not independent of each other. [**<<<**](table)  Task 3 -  Excel Standard Deviation***  Description The standard deviation of an array of numbers x is calculated using numpy as $np.sqrt(np.sum((x - np.mean(x))**2)/len(x))$.However, Microsoft Excel has two different versions of the standard deviation calculation, STDEV.P and STDEV.S . The STDEV.P function performs the above calculation but in the STDEV.S calculation the division is by $len(x)-1$ ratherthan $len(x)$ . Research these Excel functions, writing a note in a Markdown cell about the difference between them. Then use numpy to perform a simulation demonstrating that the STDEV.S calculation is a better estimate for the standarddeviation of a population when performed on a sample. Note that part of this task is to figure out the terminology in the previous sentence.  Research As a starting point the general terminology for this task have to be understanded and explained.**Standard Deviation SD**The standard deviation is a statistic that measures the dispersion of a dataset relative to its mean and is calculated as the square root of the variance. The standard deviation is calculated as the square root of variance by determining each data point's deviation relative to the mean. If the data points are further from the mean, there is a higher deviation within the data set; thus, the more spread out the data, the higher the standard deviation.[13]In another word Standard deviation is a number that describes how spread out the values are. A low standard deviation means that most of the numbers are close to the mean (average) value.A high standard deviation means that the values are spread out over a wider range.[14]In the example below it is visible how that works in practice in pythoon.<jupyter_code># Adapted fromhttps://www.w3schools.com/python/python_ml_standard_deviation.asp[15]
import numpy

speed = [86,87,88,86,87,85,86]

x = numpy.std(speed)

print(round(x,2))<jupyter_output>0.9<jupyter_text>In this case the standard deviation of speeds in dataset is 0.9 what means that most of the values are within the range of 0.9 from the mean value, which is 86.4. **Excel STDEV.P** The STDEV.P function calculates the standard deviation for a sample set of data. Standard deviation is a measure of how much variance there is in a set of numbers compared to the average (mean) of the numbers. The STDEV.P function is meant to estimate standard deviation for an entire population.[16]The **population standard deviation** formula is:[16]  The population standard deviation is a parameter, which is a fixed value calculated from every individual in the population. **Excel STDEV.S** The STDEV.S function calculates the standard deviation in a sample set of data. Standard deviation is a measure of how much variance there is in a set of numbers compared to the average (mean) of the numbers. The STDEV.S function is meant to estimate standard deviation in a sample.[17] The **sample standard deviation** formula is:[17]  The sample standard deviation is a statistic. This means that it is calculated from only some of the individuals in a population.  Python Simulation By undestanding the differences between each of the function is possible to perform calculation and simulation in Pyton.<jupyter_code># Import all required librarys
import numpy as np
import pandas as pd
from random import sample,choice
import matplotlib.pyplot as plt
import seaborn as sns
import scipy

# Use seed for repeating results in this notebook
np.random.seed(3)
# Create a population data by using randit from nympy which returns 100 random integers 
# from low 0 (inclusive) to high 101 (exclusive).
data = np.random.randint(0,101,size=100)
# Display data
print(data)
# Create function to calculate standard deviation for a population. Equivalent of STDEV.P in Excel.

def stdev_p(data):
    """Manually created function to calculate standard deviation for a population.  """
    return np.sqrt(np.sum((data - np.mean(data))**2)/(len(data)))

print(f"Population standard deviation stdev_p: {stdev_p(data):.2f}")  
# Compare results with built-in function in numpy
print(f"Population standard deviation numpy: {stdev_p(data):.2f}")<jupyter_output>Population standard deviation numpy: 28.33<jupyter_text>The above results confirm that as a standard deviation function in numpy the population standard deviation is used. Now the samples from entire population will be taken and sample standard deviation function will be created.<jupyter_code># Assign first 30 values from data to sample array
samples=data[0:30]
# Create function to calculate sample standard deviation. Equivalent of STDEV.S in Excel.

def stdev_s(data):
    """Function to calculate sample standard deviation."""
    return np.sqrt(np.sum((data - np.mean(data))**2)/(len(data)-1))

print(f"Sample standard deviation stdev_s: {stdev_s(samples) :.2f}")<jupyter_output>Sample standard deviation stdev_s: 30.58<jupyter_text>The next step is to use stdev_p created function on the samples instead of on entire data.<jupyter_code># Check the results with stdev_p function

print(f"Population standard deviation stdev_p used with samples: {stdev_p(samples) :.2f}")<jupyter_output>Population standard deviation stdev_p used with samples: 30.07<jupyter_text>Having all results together it is possible to calculate the accuracy of those two functions.<jupyter_code># Calculate difference for population and sample standard deviation

data_pdev = stdev_p(data)
sam_sdev = stdev_s(samples)
sam_pdev = stdev_p(samples)

print(f" The difference between population standard deviation on entire data and 30 samples: {abs(data_pdev - sam_pdev):.2f}")
print(f" The difference between population standard deviation on entire data and sample standard deviation on 30 samples: {abs(data_pdev - sam_sdev):.2f}")<jupyter_output>The difference between population standard deviation on entire data and 30 samples: 1.74
 The difference between population standard deviation on entire data and sample standard deviation on 30 samples: 2.26<jupyter_text>It is clearly visible that sample standard deviation produced more accurate results for sample dataset.  **How does it work in practice?** Let's say that there is a problem where somebody want to analyze the heights of the oak trees in a forest. The man can of course, measure every single tree of the forest and so have collected statistics about the entire forest, but this could be very expensive and would take a very long time. Instead it is better to obtain a sample of , let’s say, 20 trees and try to relate sample statistics and population statistics.Using the formula with $N-1$, a sample variance is given, which on average, is equal to the unknown population variance. So, also with few samples, a reasonable estimate of the actual but unknown parameters of the population distribution can be  obtained.So, when the $N$ is used instead of $N-1$, an error called statistical bias occuring, which means that the sample variance (the estimator) is systematically different from the true population parameter (in this case, the variance).The correcting factor $N/(N-1)$ is called the Bessel factor and allow to get the average of unbiased variance s² as a function of biased one:  If N, the number of observations, is small, the Bessel factor has to be applied to have a better estimate of the real variance. If $N$ is big, Bessel can be omited because $N/(N-1)$ is approximately equal to 1.[19] * In the simulation below the forest of 10000 oak trees will be considered. This is the entire population. * The distribution of heights needs to be estimated.* Let's assume that the heights are normally distributed with an average of 10m and a standard deviation (square root of variance) of 2m.* These are the statistical parameters of the entire population.<jupyter_code># Adapted from https://towardsdatascience.com/variance-sample-vs-population-3ddbd29e498a[19]

# Use seed for repeating results in this notebook
np.random.seed(1234)
# define mean, in this case the heights are normally distributed with an average of 10m 
mu = 10
# define standard deviation, in this case the standard deviation is 2m 
sigma = 2
# Generate normally distributed population of 10000 oak trees
pop = np.random.normal(mu, sigma, 10000)
# Define chart size
plt.figure(figsize=(16,8))
# Add grid lines
plt.grid(True)
# Add title to the chart
plt.title("Normally Distributed Population of Oak Trees", fontweight="bold", fontsize=15)
# # Add x lable to the chart
plt.xlabel('Trees Height', fontweight="bold", size=12)
# Add y lable to the chart
plt.ylabel('Density', fontweight="bold", size=12)
# Plot a histogram with population
count, bins, ignored = plt.hist(pop, 100, density=True, color = 'green')<jupyter_output><empty_output><jupyter_text>* The simulation tries to estimate these values through a sample of 20 random oak trees. * The experiment will be repeated 100 times.<jupyter_code># Sets a plot style for seaborn
sns.set_style('whitegrid')
# The simulation will be repeated 100 times
tests = 100
# Create empty array for 
sam = []
# Create empty array for mean
mean =[]
# Create empty array for biased standard deviation N
std_b = []
# Create empty array for unbiased standard deviation N-1
std_u = []
# Show two plots beside each other
fig, axs = plt.subplots(ncols=2,figsize=(14,7))
# Show results using a kernel density estimate (KDE) plot
sns.kdeplot(pop, bw_adjust=2, ax=axs[0])
# Define space between subplots
fig.tight_layout(pad=5.0)
# Create loop to repeat test 100 times
for i in range(tests):
    # Select 20 random samples from population using choice 
    sam_20 = np.random.choice(pop, 20)
    # Plot selected samples 
    sns.kdeplot(sam_20, bw_adjust=0.3, ax=axs[1])
    # Add selected samples to array
    sam.append(sam_20)
    # Add mean of selected samples to array
    mean.append(np.mean(sam_20))
    # Calculate and add biased standard deviation(STDEV.P) of selected samples to array
    std_b.append(np.std(sam_20))
    # Calculate and add biased standard deviation(STDEV.S) of selected samples to array
    # In numpy.std can perform (STDEV.S) calculation by adding ddof parameter. Means Delta Degrees of Freedom "ddof"
    # The divisor used in calculations is N - ddof, where N represents the number of elements. By default ddof is zero.
    # https://numpy.org/doc/stable/reference/generated/numpy.std.html
    std_u.append(np.std(sam_20, ddof=1))
# Put all results to dataframe
frame = {'mean':mean, 'std_b': std_b, 'std_u': std_u}
table = pd.DataFrame(frame)
plt.show()<jupyter_output><empty_output><jupyter_text>This way, a sample mean “mean” standard deviation (biased) “std_b” and unbiased (ddof=1 is the same as (N-1) instead of N) standard deviation “std_u” for each of the 100 experiments was generated. Given that 20 samples are a tiny subset of 10000 items of population, every time the test was ran, different distributions was obtained. However on average, a reasonable estimate of the real mean and standard deviation was received.<jupyter_code># Show standard statistical parameters obtained in simulation
table.describe()<jupyter_output><empty_output><jupyter_text>The unbiased sample standard deviation (STDEV.S) in average is nearest to the value of the population parameter than the biased one (STDEV.P).  Summary and Conclusions The sample variance is affected by statistical bias, due to the distortion of very few observed data comparing to the cardinality of the entire population. The simulation shows method to mitigate this error by the Bessel factor.In general, it is crucial to understand what method and conditions are used to calculate the standard deviation.As a curiosity, an example of computing standard deviation in pandas and NumPy can be used. By using default options in those two modules different results will be produced. The main reason for that is because NumPy by default uses the **population standard deviation** contrary to the **sample standard deviation** the one that pandas uses by default. [**<<<**](table)  Task - 4 Scikit-learn Iris Data Set - K-Means Clustering***  Description Use scikit-learn to apply k-means clustering to Fisher’s famous Iris data set. You will easily obtain a copy of the data set online. Explain in a Markdown cell how your code works and how accurate it might be, and then explain how your model could be used to make predictions of species of iris.  Research Iris data set is a multivariate data set introduced by the British statistician and biologist <NAME> in his 1936 paper.The use of multiple measurements in taxonomic problems as an example of linear discriminant analysis. It is sometimes called Anderson's Iris data set because <NAME> collected the data to quantify the morphologic variation of Iris flowers of three related species. The data set consists of 50 samples from each of three species of Iris:  Four features were measured from each sample:* Sepal length(cm)* Sepal Width(cm)* Petal Length(cm)* Petal Width(cm)  Based on the combination of the above four features, Fisher developed a linear discriminant model to distinguish the species from each other. Interesting and challenging in the relation of Iris flowers is the fact that sepal and petal look quite similar contrary to other flowers where sepal is usually green and easy to distinguish from the petal. **Data set review** The data set used for this project was downloaded from the UCI Machine Learning Repository. This repository contains iris dataset widely used in many publications and research projects across the world. The Iris flower data set is a classic, well-known data set example for data mining and data exploration and is traditionally used for classification and prediction* Number of Instances: 150 (50 in each of three classes)* Number of Attributes: 4 numeric, predictive attributes and the classAttribute Information:1. sepal length in cm1. sepal width in cm1. petal length in cm1. petal width in cm1. class:    * Iris Setosa    * Iris Versicolour    * Iris Virginica Fisher developed a linear discriminant model to distinguish the species from each other using above combinations. This data set became a typical test case for many statistical classification techniques in machine learning.Investigating data set using Python  **Evaluating data sets** In the first step, a quick evaluation of the data set will be carried out.<jupyter_code># Import all necessary libraries
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd

# Load Iris data from seaborn database
dataset = sns.load_dataset("iris")<jupyter_output><empty_output><jupyter_text>Having data loaded in to pandas it was easy to obtain the required information.Using print(dataset.shape) it displays the number of instances and attributes: (150, 5).To visualize first ten rows data with appropriate column names print(dataset.head(10)) was used.<jupyter_code>print(dataset.shape)
print(dataset.head(10))<jupyter_output>sepal_length  sepal_width  petal_length  petal_width species
0           5.1          3.5           1.4          0.2  setosa
1           4.9          3.0           1.4          0.2  setosa
2           4.7          3.2           1.3          0.2  setosa
3           4.6          3.1           1.5          0.2  setosa
4           5.0          3.6           1.4          0.2  setosa
5           5.4          3.9           1.7          0.4  setosa
6           4.6          3.4           1.4          0.3  setosa
7           5.0          3.4           1.5          0.2  setosa
8           4.4          2.9           1.4          0.2  setosa
9           4.9          3.1           1.5          0.1  setosa<jupyter_text>Finally using print(dataset.describe()), it was possible to do some statistical summary on data set.<jupyter_code>print(dataset.describe())<jupyter_output>sepal_length  sepal_width  petal_length  petal_width
count    150.000000   150.000000    150.000000   150.000000
mean       5.843333     3.057333      3.758000     1.199333
std        0.828066     0.435866      1.765298     0.762238
min        4.300000     2.000000      1.000000     0.100000
25%        5.100000     2.800000      1.600000     0.300000
50%        5.800000     3.000000      4.350000     1.300000
75%        6.400000     3.300000      5.100000     1.800000
max        7.900000     4.400000      6.900000     2.500000<jupyter_text>Looking at the table above , it is clear that all values have the same scale and ranges between 0 and 8 centimetres. It means all data in data set do not require normalization to produce valuable results. **Visualization of data** A quick visualization of data set is presented by using functionsns.pairplot() below.It shows a relationship between different data across multiple dimensions.<jupyter_code>sns.pairplot(dataset, hue="species", palette="husl", markers=["o", "s", "D"])<jupyter_output><empty_output><jupyter_text>Machine Learning The Scikit-learn software machine learning library for the Python was incorporated for this task. This library has a number of useful features like various classification, regression and clustering algorithms including support vector machines. Scikit has also built-in Iris data set that can be loaded straight in to the python program but for this project existing data set dataset from seaborn was used. The following steps was carried out:* Investigate K-means algorithm* Compare K-means vs. KNN * Select the best prediction model* Predict the answer **K-means** One of the most common clustering methods is K-means algorithm. The goal of this algorithm isto partition the data into set such that the total sum of squared distances from each point to the mean point of the cluster is minimized.K means works through the following iterative process[22]:- Pick a value for k (the number of clusters to create)- Initialize k ‘centroids’ (starting points) in your data- Create your clusters. Assign each point to the nearest centroid.- Make your clusters better. Move each centroid to the center of its cluster.- Repeat steps 3–4 until your centroids converge.    **How does it work in practice?**<jupyter_code># Code adapted from[22]
from sklearn import datasets
from sklearn.cluster import KMeans
# Load the iris data set from sklearn
iris = datasets.load_iris()
# Define chart size
plt.figure(figsize=(12,8))
# Define your target and predictors
X = iris.data[:, :2]
y = iris.target
# look at our data through a scatter plot
plt.scatter(X[:,0], X[:,1], c=y, cmap='gist_rainbow')
plt.xlabel('Spea1 Length', fontsize=16)
plt.ylabel('Sepal Width', fontsize=16)
plt.show()<jupyter_output><empty_output><jupyter_text>Next, the k-Means machine learning algorithm will be applied on this data set. It will compute and plot the sum of squared distances against the different values of k starting from 1. The optimal number of clusters is given by the sharp turn in the Elbow graph.<jupyter_code># Remove labels of secies from data set
iris_no_sp = dataset[['sepal_length', 'sepal_width', 'petal_length', 'petal_width']]
# Determine optimal nuber of clusters
# Compute the squared of distances of data points to centroids
SumSquareDist = []
K = range (1,10)
optimalK = 1
for k in K:
    km = KMeans(n_clusters=k)
    km = km.fit(iris_no_sp)
    SumSquareDist.append(km.inertia_)
    if k > 1:
        ratio = SumSquareDist[k-1]/SumSquareDist[k-2]
        if ratio < 0.55:
            optimalK = k
# Plot Elbow graph

# Define chart size
plt.figure(figsize=(16,8))
plt.plot(K,SumSquareDist, 'bx-')
plt.xlabel('k',fontsize=16)
plt.ylabel('Sum of Square Distances',fontsize=16)
plt.title('Elbow Method For Optimal Clusters k',fontsize=16)
plt.show()<jupyter_output><empty_output><jupyter_text>Based on the above graph, the optimal number of k is 3.To verify that the optimal number is 3, the Principal of Component Analysis (PCA) technique can be used with reduce number of features to 2. It will be called two reduced features PCA1 and PCA2.<jupyter_code># Code Adapted from [23]
# PCA
from sklearn.decomposition import PCA
model = PCA(n_components = 2)
model.fit(iris_no_sp)
X_2D = model.transform(iris_no_sp)
iris_no_sp['PCA1'] = X_2D[:, 0]
iris_no_sp['PCA2'] = X_2D[:, 1]
iris_no_sp.head()<jupyter_output><empty_output><jupyter_text>Plotting the PCA2 against PCA1 for the 3 clusters show distinct plots, suggesting that there are 3 Iris species in the data set.<jupyter_code># Plot PCA2 against PCA1 for optimal K Clusters
from sklearn.mixture import GaussianMixture
model = GaussianMixture(n_components=optimalK, covariance_type='full')
model = model.fit(iris_no_sp)
y_gmm = model.predict(iris_no_sp)
iris_no_sp['cluster'] = y_gmm
# Sets a chart style
sns.set(style="darkgrid", color_codes=True)
sns.lmplot(x='PCA1', y='PCA2', data=iris_no_sp, col='cluster', fit_reg=False)
plt.show()
# instantiate and fit our K means cluster model. 
# use 3 clusters and a random state of 21
km = KMeans(n_clusters = 3, random_state=21)
km.fit(X)
# following code identify the center points of the data
centers = km.cluster_centers_
print(centers)
#this will tell us to which cluster does the data observations belong.
new_labels = km.labels_
# Plot the identified clusters and compare with the answers
fig, axes = plt.subplots(1, 2, figsize=(16,6))
axes[0].scatter(X[:, 0], X[:, 1], c=y, cmap='gist_rainbow',
edgecolor='k', s=150)
axes[1].scatter(X[:, 0], X[:, 1], c=new_labels, cmap='jet',
edgecolor='k', s=150)
axes[0].set_xlabel('Sepal length', fontsize=16)
axes[0].set_ylabel('Sepal width', fontsize=16)
axes[1].set_xlabel('Sepal length', fontsize=16)
axes[1].set_ylabel('Sepal width', fontsize=16)
axes[0].tick_params(direction='in', length=10, width=5, colors='k', labelsize=20)
axes[1].tick_params(direction='in', length=10, width=5, colors='k', labelsize=20)
axes[0].set_title('Actual', fontsize=16)
axes[1].set_title('Predicted', fontsize=16)
plt.show()<jupyter_output><empty_output><jupyter_text>Here is a list of the main advantages and disadvantages of this algorithm.**Advantages:**- K-Means is simple and computationally efficient.- It is very intuitive and their results are easy to visualize.**Disadvantages:**- K-Means is highly scale dependent and is not suitable for data of varying shapes and densities.- Evaluating results is more subjective. It requires much more human evaluation than trusted metrics.  Additional Part - Compare K-means vs. KNN Having all knowledge about the K-means algorithm it will be a good exercise to compare this algorithm with KNN which was used by the author in the first semester of Data Analytics course. Backing time then the KNN was chosen as the best solution to implement in prediction model that can be used as a part of machine learning to validate iris data set. **KNN Algorithm** K-Means Clustering is an unsupervised learning algorithm that is used for clustering whereas KNN is a supervised learning algorithm used for classification.The k-nearest neighbors algorithm is a **supervised** classification algorithm. It takes a bunch of labeled points and uses them to learn how to label other points. To label a new point, it looks at the labeled points closest to that new point which are its nearest neighbors, and has those neighbors vote. So whichever label, the most of the neighbors have is the label for the new point. Here “k” in K-Nearest Neighbors is the number of neighbors it checks. It is supervised because you are trying to classify a point based on the known classification of other points. **How it works?** - Determine the value for K- Calculate the distances between the new input (test data) and all the training data- Sort the distance and determine k nearest neighbors based on minimum distance values- Analyze the category of those neighbors and assign the category for the test data based on majority vote- Return the predicted class Based on the same iris data set it can be shown how to implement KNN algorithm. Dataset was split into two groups where one group was used as a validation dataset. Using KNN algorithm directly on the validation dataset, below results were produced (accuracy score, confusion matrix, classification report).<jupyter_code># Code adapted from[20] and from my project in first semester.
# Import libraries for scikit-learn for Machine Learning
from sklearn import model_selection
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
# Split the loaded dataset
array = dataset.values
X = array[:,0:4]
Y = array[:,4]
# Use 20% of data as validation dataset 
validation_size = 0.20
seed = 7
X_train, X_validation, Y_train, Y_validation = model_selection.train_test_split(X, Y, test_size=validation_size, random_state=seed)
# Make predictions on validation dataset
knn = KNeighborsClassifier()
knn.fit(X_train, Y_train)
predictions = knn.predict(X_validation)
print(accuracy_score(Y_validation, predictions))
print(confusion_matrix(Y_validation, predictions))
print(classification_report(Y_validation, predictions))<jupyter_output>0.9
[[ 7  0  0]
 [ 0 11  1]
 [ 0  2  9]]
              precision    recall  f1-score   support

      setosa       1.00      1.00      1.00         7
  versicolor       0.85      0.92      0.88        12
   virginica       0.90      0.82      0.86        11

    accuracy                           0.90        30
   macro avg       0.92      0.91      0.91        30
weighted avg       0.90      0.90      0.90        30<jupyter_text>The results reveal that the accuracy of this prediction model is 0.9 (90%). Precision, recall, f1-score and support also showing satisfying feedback for each class.In the end, this model was tested on the user entered data. Data for three different species were used.<jupyter_code># Test prediction model by entering Iris sepal and petal user values
s_len = float(input("Please enter a sepal length in cm: "))
s_wid = float(input("Please enter a sepal width in cm: "))
p_len = float(input("Please enter a petal length in cm: "))
p_wid = float(input("Please enter a petal width in cm: "))
result = knn.predict([[s_len,s_wid,p_len,p_wid]])
print("\n")
print("The tested species is: ", result)<jupyter_output>Please enter a sepal length in cm: 4.9
Please enter a sepal width in cm: 3.0
Please enter a petal length in cm: 1.4
Please enter a petal width in cm: 0.2


The tested species is:  ['setosa']
{'fraction_non_alphanumeric': 0.06882443858532594, 'fraction_numerical': 0.032366578779889084, 'mean_word_length': 4.451920693928129, 'pattern_counts': {'":': 4, '<': 120, '<?xml version=': 0, '>': 114, 'https://': 6, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '206135', 'license': 'MIT', 'n_tokens_mistral': 12875, 'n_tokens_neox': 11576, 'n_words': 5883, 'path': 'Tasks.ipynb', 'repo_name': 'G00376332/52954-MachineLearning', 'size': '44040'}