<jupyter_start><jupyter_text>Machine Learning and Statistics - Task 1 These are my solution to the Task 1 Assessment The author is <NAME> email : <EMAIL>****  Task 1: Calculate Square Root of 2We can calculate the square root of two using Newton's method [1], [2],[3], [4],[5],[6]To find the square root 2, we can iterate using the following equation$$ z = z - \frac{z^2 - x} {2z} $$[1] A Tour of Go; Exercise: Loops and Functions https://tour.golang.org/flowcontrol/8[2] Floating Point Limitations https://docs.python.org/3.4/tutorial/floatingpoint.html [3] How to use Python exponents with ** operator https://kodify.net/python/math/exponents/[4] Wikipedia,"Different methods of Finding Square root of 2"  https://en.wikipedia.org/wiki/Square_root_of_2[5] "Finding square root of 2 with 100 decimal places" https://stackoverflow.com/questions/8505385/finding-square-root-of-2-upto-more-than-100-decimal-places[6] Newton's Method to find Square Root https://en.wikipedia.org/wiki/Newton%27s_method<jupyter_code>def sqrt2(num):
    
    
    """ A function to calculate the square root of number 2
    """
    
    # Intial value of the square root of z1
    z1 = (num * 1.0) / 2
    
    # Inital value of the square root of z2
    z2 = (z1 + (num / z1) / 2)
    
    # Loop for to run until the value of z1 - z2 is less than 0.00001
    while abs(z1 - z2) > 0.00001:
        
        z1 = z2
        
        z2 = (z1 + (num / z1)) / 2
  
        # format given value with 100 decimal place
        y = format(z2, ',.100f')
        
    # Return the square root of 2        
    return y
sqrt2(2)<jupyter_output><empty_output><jupyter_text>Tests of the functionWe are testing the function with some known values<jupyter_code># Test the function on 25
sqrt2(25)
# Test the function on 2.
sqrt2(2)<jupyter_output><empty_output><jupyter_text>The floating value has limitation with decimal, due to the fact after I formatted the value into 100 decimal it gives zeros after 50th decimal.<jupyter_code>import math
math.sqrt(2)<jupyter_output><empty_output><jupyter_text>Since the float decimal limitations we are converting the float value into string and returning the results.<jupyter_code>def sqrtnew():
     
    """ A function to calculate the square root of number 2
    """
    # Assign the value of num with 2
    num = 2
    
    # Multiplying num by 100 and finding exponential(power) of that value 

    S = num * 100 ** 100
    
    # Finding Floor Division of S 
    new = S//2
    
    #Loop for to run until the value of S minus exponential of 2 is less than 0
    while S - (new**2) < 0:
        # assigning value of new plus Floor Division of S which is again floor divison of 2
        new = (new + S//new)//2

    # changing the float into string by doing this we are able to get the square root of 2 mutiplied by 100**100
    new = (str(new))
    
    # Return square root of 2 by concatenating 2 string values
    # Return zero will bring the 1 character of the result and return 1: will bring the rest of the result from position 1 till the end
      
    
    return(new[0] + "." + new[1:])
    
sqrtnew()<jupyter_output><empty_output><jupyter_text>Tests of the functionWe are testing the function(the number 2 is the value which is given already on the code) so its generate the answer for the same<jupyter_code>sqrtnew()
# Using the library function following square root with 100 decimal places to match the above value square root of 2
import decimal
decimal.getcontext().prec=101
decimal.Decimal("2").sqrt()<jupyter_output><empty_output><jupyter_text>ConclusionThe result of the above can be compared against the approximation of $\sqrt{2}$ value at https://apod.nasa.gov/htmltest/gifcity/sqrt2.1mil1.4142135623730950488016887242096980785696718753769480731766797379907324784621070388503875343276415727This task represent the problem with the certanity of the values of the decimal places and not able to produce the  way you expect.  The values has irrational numbers which cannot be saved into the momory of computer.  The python floating decimal limitation which restrict the decimal into 50.  Machine Learning and Statistics - Task 2 Task 2: Chi-squared test for independence[1],[2],[3],[4]Chi-Square Test for Independence https://stattrek.com/chi-square-test/independence.aspxCalculate Chi Square Statistic by Hand https://www.statisticssolutions.com/non-parametric-analysis-chi-square/Chi Square Statistic Video https://www.youtube.com/watch?v=LE3AIyY_cn8Chi Squared Test Wikipedia https://en.wikipedia.org/wiki/Chi-squared_testChi Square test SciPy.org https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chisquare.htmlTwo-Way Tables and the Chi-Square Test http://www.stat.yale.edu/Courses/1997-98/101/chisq.htmScipy.stats.chi2_contingency https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.chi2_contingency.htmlContingency table (also known as a cross tabulation or crosstab) https://en.wikipedia.org/wiki/Contingency_tableChi-square statistic for hypothesis testing https://www.khanacademy.org/math/ap-statistics/chi-square-tests/chi-square-goodness-fit/v/chi-square-statistic?modal=1Statistics : Chi-suared test https://pythonhealthcare.org/2018/04/13/58-statistics-chi-squared-test/By using chi2_contingency we are able to find the Chi-squared value is 24.6 and the p value is 0.0004098425861096696 $$ x^2 = \sum_ \frac{f(o) - f(e)} {f(e)} $$ The sample data has 3 rows and 4 coloumns<jupyter_code>from scipy.stats import chisquare
chisquare([16, 18, 16, 14, 12, 12])
chisquare([16, 18, 16, 14, 12, 12], f_exp=[16, 16, 16, 16, 16, 8])
import numpy as np
obs = np.array([[16, 18, 16, 14, 12, 12], [32, 24, 16, 28, 20, 24]]).T
obs.shape

chisquare(obs)
#By setting axis=None, the test is applied to all data in the array, which is equivalent to applying the test to the flattened array.
chisquare(obs, axis=None)


chisquare(obs.ravel())<jupyter_output><empty_output><jupyter_text>We can use the same and change the array to our data<jupyter_code>SData = np.array([[90, 60, 104, 95], [30, 50, 51, 20], [30, 40, 45, 35]]).T
SData.shape

chisquare(SData)
# A two-way example (2x3)

from scipy.stats import chi2_contingency
obs = np.array([[10, 10, 20], [20, 20, 20]])
chi2_contingency(obs)

g, p, dof, expctd = chi2_contingency(obs, lambda_="log-likelihood")
g, p, dof, expctd
# A four-way example (2 x 2 x 2 x 2):

# chi2 : float The test statistic
# p : float The p-value of the test
# dof : int Degrees of freedom
# expected : ndarray, same shape as observed (The expected frequencies, based on the mariginal sums of the table)
obs = np.array(
    [[[[12, 17],
       [11, 16]],
      [[11, 12],
       [15, 16]]],
     [[[23, 15],
       [30, 22]],
      [[14, 17],
       [15, 16]]]])
chi2_contingency(obs)

g, p, dof, expctd = chi2_contingency(obs, lambda_="log-likelihood")
g, p
import numpy as np
import pandas as pd
import scipy.stats as stats

cols = ['A', 'B', 'C', 'D']
data = pd.DataFrame(columns=cols)

data.loc['White Collar'] = [90, 60, 104, 95]
data.loc['Blue Collar'] = [30, 50, 51, 20]
data.loc['No collar'] = [30, 40, 45, 35]



ChiS, pVal, dof, expected = stats.chi2_contingency(data) 
# add correction=False for uncorrected Chi-square


print("Chi Square Value is:", round(ChiS,1))
print ("P value of the Data is:", round(pVal,4))
print ("The Expected Value of each rows:\n", (pd.DataFrame(expected,index = ["White Collar", "Blue Collar", "No Collar"], columns =["A", "B", "C", "D"])).round(2))
print("Degree of Freedom is:",dof)<jupyter_output>Chi Square Value is: 24.6
P value of the Data is: 0.0004
The Expected Value of each rows:
                   A      B       C      D
White Collar  80.54  80.54  107.38  80.54
Blue Collar   34.85  34.85   46.46  34.85
No Collar     34.62  34.62   46.15  34.62
Degree of Freedom is: 6<jupyter_text>ConclusionThe Value of Chi-squared is 24.6 is accurate and I have identified this by doing the same manually with the following 2 steps.1) Define Null and Alternative Hypotheses2) State Alph alpha = 0.053) Calculate Degrees of Freedomdf = (rows-1)(columns-2)df=(3-1)(4-1)df=(2)(3)=64) State Decision RuleUsing Chi Square Table we can get the value is 12.5916If ${\chi}^2$ is greater than 12.5916, reject $h_o$5) Calculate Test Statistic$$f_e = \frac{f_cf_r} {n}$$$$ {\chi}^2=\sum\frac{f_o - f_e} {f_e} $$Using first the equation above I did the Chi Square value manually.i.e. White Collar and A 150 * 349 (150 is the total of A column and 349 is the total of White collar) divided by 650 (Total of all 4 columns) and I arrived 80.53 rounded into 81.I did the same for All the rows and and arrived the value.Using 2nd equation above I calcualted the chi square as followsWhite Collar and A column90 minus 81 and find square of the answer 9 * 9 = 81 divied by 81 = 1I did the same for all the rows and arrived value for each of them.Adding all these will give ${\chi}^2$ which is 24.6 Using the Scipy.Stats I have generated the value of Chi Square and arrived 24.6 and its accuarate and I am able to arrive the same value by doing it manully by using the equation.  The p value is 0.0004.  Using the Chi Suare table and alpha value of 0.05 the value is 12.5916 and the result is greater than 12.5916 which proves the null hypothesis.   Machine Learning and Statistics - Task 3[1],[2],[3],[4],[5],[6],[7],[8]What is the STDEV.P Function? https://corporatefinanceinstitute.com/resources/excel/functions/stdevp-function/Using Excel function STDEV.P & STDEV.S https://www.exceltip.com/statistical-formulas/how-to-use-excel-stdev-p-function.htmlSTDEV.S Function in Excel https://support.microsoft.com/en-us/office/stdev-s-function-7d69cf97-0c1f-4acf-be27-f3e83904cc23Why use n-1 when calculating a standard deviation? https://www.graphpad.com/support/faqid/1383/Compute the standard deviation using numpy.std https://numpy.org/doc/stable/reference/generated/numpy.std.htmlDraw random samples from a normal (Gaussian) distribution https://numpy.org/doc/stable/reference/random/generated/numpy.random.normal.htmlNormal Distribution https://en.wikipedia.org/wiki/Normal_distributionPopulation vs Sample Data http://mathbitsnotebook.com/Algebra1/StatisticsData/STPopSample.html Task 3: The standard deviation of an array of numbers<jupyter_code>a = np.array([[1, 2], [3, 4]])
np.std(a)

np.std(a, axis=0)

np.std(a, axis=1)
a = np.zeros((2, 512*512), dtype=np.float32)
a[0, :] = 1.0
a[1, :] = 0.1
np.std(a)
np.std(a, dtype=np.float64)<jupyter_output><empty_output><jupyter_text>Functions to calculate sample standard deviation in ExcelTo calculate standard deviation based on a sample.Excel STDEV functionSTDEV(number1,[number2],…) is the oldest Excel function to estimates standard deviation based on a sample.Logical values and text representations of numbers supplied directly in the list of arguments are counted. In arrays and references, only numbers are counted; empty cells, logical values of TRUE and FALSE, text and error values are ignored.Excel STDEV.S functionSTDEV.S(number1,[number2],…) is an improved version of STDEV.  Like STDEV, the STDEV.S function calculates the sample standard deviation of a set of values based on the classic sample standard deviation formula discussed in the previous section. Calcuating the standard divation of numbers:The Standard Deviation is a measure of how spread out numbers are. I made the following blood test reading and time with 2 columns and calculated the standard divation using STDEV.P and STDEV.S.  The result of this is shown at the bottom.![image.png](attachment:image.png)The STDEV.P Function is categorized under Excel Statistical functions. STDEV.P will calculate the standard deviation that is based on an entire population given as arguments. It will ignore logical values and text.As a financial analyst, the STDEV.P function can be useful in, for example, calculating deviations in revenue. Portfolio managers often use standard deviation to measure and keep track of their portfolios’ risks.The STDEV.P function was introduced in MS Excel 2010 and, hence, is not available in earlier versions. However, the function is simply an updated version of the older STDEVP function.The standard deviation of the population is evaluated when there is minimal risk to apply the result on aggregate data. Here we take the whole population for calculating the standard deviation. The sum of squared deviation is divided by N (number of arguments).STDEV.P vs STDEV.SSo, what are the difference in STDEV.P and STDEV.S function? Let’s Explore.The STDEV.P function is used when your data represents the entire population.The STDEV.S function is used when your data is a sample of the entire population.In STDEV.P function, the squared deviation is divided by the total number of arguments, mostly represented as N.In STDEV.S or STDEV, the squared deviation is divided by the total number of sample -1. It is represented as N-1.Since STDEV.P function considers entire data, and some factors may dominate the result standard deviation. And since it will be taken as the standard deviation for everyone in data, even for minorities, this is called Biased Analysis. This is why this standard deviation is recommended to use only when an analysis is non-destructive.The STDEV.S or STDEV function is used on a small sample of the entire population, and we subtract one from the denominator (number of samples arguments). This is called non-biassed analysis of standard deviation. This is used when an analysis is destructive. How ito calculate the standard deviation1. Compute the square of the difference between each value and the sample mean.2. Add those values up.3. Divide the sum by n-1. This is called the variance.4. Take the square root to obtain the Standard Deviation. Why n-1?Why divide by n-1 rather than n in the third step above? In step 1, you compute the difference between each value and the mean of those values. You don't know the true mean of the population; all you know is the mean of your sample. Except for the rare cases where the sample mean happens to equal the population mean, the data will be closer to the sample mean than it will be to the true population mean. So the value you compute in step 2 will probably be a bit smaller (and can't be larger) than what it would be if you used the true population mean in step 1. To make up for this, divide by n-1 rather than n.v This is called Bessel's correction.But why n-1? If you knew the sample mean, and all but one of the values, you could calculate what that last value must be. Statisticians say there are n-1 degrees of freedom. When should the SD be computed with a denominator of n?Statistics books often show two equations to compute the SD, one using n, and the other using n-1, in the denominator. Some calculators have two buttons. The n-1 equation is used in the common situation where you are analyzing a sample of data and wish to make more general conclusions. The SD computed this way (with n-1 in the denominator) is your best guess for the value of the SD in the overall population.If you simply want to quantify the variation in a particular set of data, and don't plan to extrapolate to make wider conclusions, then you can compute the SD using n in the denominator. The resulting SD is the SD of those particular values. It makes no sense to compute the SD this way if you want to estimate the SD of the population from which those points were drawn. It only makes sense to use n in the denominator when there is no sampling from a population, there is no desire to make general conclusions. The goal of science is always to generalize, so the equation with n in the denominator should not be used. The only example I can think of where it might make sense is in quantifying the variation among exam scores. But much better would be to show a scatterplot of every score, or a frequency distribution histogram. GraphPad Prism and InStat always compute the SD with the n-1 denominator.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
mu, sigma1, sigma2 = 0, 0.1, 0.05 # mean and standard deviation


# calculating sd for sigma1
s_1 = np.random.normal(mu, sigma1, 100)

ssd1= np.sqrt(np.sum((s_1 - np.mean(s_1))**2)/(len(s_1) -1))

se1 = ((ssd1 - sigma1)/sigma1) * 100
print(f'sample stdev = {ssd1:.4f} for population with sigma = {sigma1}')
print(f'difference = {se1:.2f}%')

count, bins, ignored  = plt.hist(s_1, 30, density=True, color='g', alpha=0.5, label='n = 100 from sigma = 0.1')
plt.plot(bins, 1/(sigma1 * np.sqrt(2 * np.pi)) * np.exp( - (bins - mu)**2 / (2 * sigma1**2)), linewidth=3, color='g')


# calculating sd for sigma2

s_2 = np.random.normal(mu, sigma2, 1000)

ssd2= np.sqrt(np.sum((s_2 - np.mean(s_2))**2)/(len(s_2) -1))

se2 = ((ssd2 - sigma2)/sigma2) * 1000
print(f'sample stdev = {ssd2:.4f} for population with sigma = {sigma2}')
print(f'difference = {se2:.2f}%')

count, bins, ignored  = plt.hist(s_2, 30, density=True, color='m', alpha=0.3, label='n = 1000 from sigma = 0.05')
plt.plot(bins, 1/(sigma2 * np.sqrt(2 * np.pi)) * np.exp( - (bins - mu)**2 / (2 * sigma2**2)), linewidth=3, color='m')
plt.xlabel('x', fontsize='18')
plt.ylabel('Probability Density', fontsize ='14')
plt.title('Probability Density Function', fontsize='14')
plt.grid()
plt.legend()
plt.show()


#abs(mu - np.mean(s))
#abs(sigma - np.std(s, ddof=1))
import matplotlib.pyplot as plt
mu, sigma = 0, 0.1 # mean and standard deviation
s = np.random.normal(mu, sigma, 1000)

count, bins, ignored = plt.hist(s, 30, density=True)
plt.plot(bins, 1/(sigma * np.sqrt(2 * np.pi)) *
               np.exp( - (bins - mu)**2 / (2 * sigma**2) ),
         linewidth=2, color='r')
plt.show()
np.random.normal(3, 2.5, size=(2, 4))
from numpy import random
import matplotlib.pyplot as plt
import seaborn as sns

sns.distplot(random.normal(size=1000), hist=False)

plt.show()<jupyter_output><empty_output><jupyter_text>Excel STDV.S functionSTDEV.S assumes that its arguments are a sample of the population. If your data represents the entire population, then compute the standard deviation using STDEV.P.The standard deviation is calculated using the "n-1" method.Arguments can either be numbers or names, arrays, or references that contain numbers.Logical values and text representations of numbers that you type directly into the list of arguments are counted.If an argument is an array or reference, only numbers in that array or reference are counted. Empty cells, logical values, text, or error values in the array or reference are ignored.Arguments that are error values or text that cannot be translated into numbers cause errors.If you want to include logical values and text representations of numbers in a reference as part of the calculation, use the STDEVA function.STDEV.S and STDEV.P uses the following formulas:Formula for STDEV.P$$ STDEV.P =  \sqrt{\sum\frac{(x - \overline{x})^2} {n}} $$Formula for STDEV.P$$ STDEV.S =  \sqrt{\sum\frac{(x - \overline{x})^2} {(n-1)}} $$where x is the sample mean AVERAGE(number1,number2,…) and n is the sample size.<jupyter_code># Function for producing the standard diviation using above formulas for STDEV.P AND STDEV.S

def stdev_p_s(x):
    
    # initialize the numerator
    numerator = 0
    # convering the input in to numpy array
    x= np.array(x)
    # calculating 
    for i in range(len(x)):
        numerator += (x[i]-np.mean(x))**2
        
    stdev_p = np.sqrt(numerator/(len(x)))
        
    stdev_s = np.sqrt(numerator/(len(x)-1))
        
    print("STDEV.P SD of the given value:", stdev_p)
    print("STDEV.S SD of the given value:", stdev_s)


test_results = np.array([13.4,6.5,10.6,9,8.2,15.2,9.4,11.2])
        
    
stdev_p_s(test_results)<jupyter_output>STDEV.P SD of the given value: 2.6390990413396764
STDEV.S SD of the given value: 2.821315549273322
{'fraction_non_alphanumeric': 0.09549567767049189, 'fraction_numerical': 0.04549820534856681, 'mean_word_length': 4.114270941054809, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 23, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '460165', 'license': 'Apache-2.0', 'n_tokens_mistral': 6716, 'n_tokens_neox': 5955, 'n_words': 2558, 'path': 'ML AND S TASK1.ipynb', 'repo_name': 'babubaskaran/Machine-Learning-and-Statistics-Tasks', 'size': '19798'}