<jupyter_start><jupyter_text>Fundamentals of Data AnalysisTasks notebook for Fundamentals of Data Analysis @ GMIT - 2020Author: <NAME> (<EMAIL>) Github: mizydorek***  Task 1 *Write a Python function called counts that takes a list asinput and returns a dictionary of unique items in the list as keys and the number of times each item appears as values.*<jupyter_code># Version 1.0 
# Really simple approach to solve the Task 1 
# It takes a list, iterate through it and count items in a list and return as a dictionary
def counts():
    # Initial values 
    d = {}
    l = []
    
    l = ['A', 'A', 'B', 'C', 'A']
    
    # iterate through list
    for item in l:
        # check if item is on the list
        if item not in d:
            d[item] = 1
        else:
            d[item] += 1
    print(d)

counts()
# Version 1.1
# More complex approach to solve the task 1 with nested lists
l = ['A', ['A', 'A', 'B'], 'A', 'B', ['B', 'B']]

# function that counts elements of list and returns as a dictionary 
def counts(l):
    # different approach to count the occurrence by using dict comprehension and built-in count function
    # https://stackoverflow.com/questions/2600191/how-can-i-count-the-occurrences-of-a-list-item
    return { i:l.count(i) for i in l }

# Main function that checks if there is a nested list, append all elements to temporary list and applies counts function to it
def reduceall(f, l):
    # temporary list 
    ans = []
    for el in l:
        # checks if element is a list
        # https://stackoverflow.com/questions/26544091/checking-if-type-list-in-python
        if isinstance(el, list):
            # loop through it and append to list
            for subel in el:
                ans.append(subel)
        # if element is not a list, just rewrite to list
        else:
            ans.append(el)
    # applies counts function to list
    ans = f(ans)
    return ans

reduceall(counts, l)<jupyter_output><empty_output><jupyter_text>*** Task 2*Write a Python function called dicerolls that simulatesrolling dice. Your function should take two parameters: the number of dice k andthe number of times to roll the dice n. The function should simulate randomlyrolling k dice n times, keeping track of each total face value. It should then returna dictionary with the number of times each possible total face value occurred.*<jupyter_code># Version 1.0 
# import random library to generate random numbers
# https://docs.python.org/3/library/random.html
import random 

# main function to simulate randomly rolling k dice n times and return a sorted dictionary with the total number of times each possible total face value occurred

def diceroll(k, n):

    d = {}
    l = []

    # iterate n number of times 
    for i in range(n):
        # temp variable 
        num = 0
        # iterate k number of dice
        for j in range(k):
            # generate random number between 1 and 6
            # https://docs.python.org/3/library/random.html
            p = random.randint(1,6)
            # sum up result 
            num += p
            
        # append to list 
        l.append(num)
    
    # iterate through a list
    for item in l:
            # check if item is on the list and add one
            if item not in d:
                d[item] = 1
            # if on the list increment by one 
            else:
                d[item] += 1

    # finding on how to sort the dictionaries in python
    # https://stackoverflow.com/questions/9001509/how-can-i-sort-a-dictionary-by-key
    # return sorted dictionary by items
    return dict(sorted(d.items()))

diceroll(2,1000)

# Version 1.1
# dict comprehension used to count the occurrence
# https://realpython.com/list-comprehension-python/
import random 

# main function to simulate randomly rolling k dice n times and return a sorted dictionary with the total number of times each possible total face value occurred

def diceroll(k, n):
    l = []
    # iterate n number of times 
    for i in range(n):
        # temp variable 
        totalFace = 0
        # iterate k number of dice
        for j in range(k):
            # generate random number between 1 and 6 and sum up the result 
            totalFace += random.randint(1,6)   
        # append to list 
        l.append(totalFace)
    # different approach to count the occurrence by using dict comprehension and built-in count function
    d = { i:l.count(i) for i in l }
    return dict(sorted(d.items()))

diceroll(2,1000)<jupyter_output><empty_output><jupyter_text>*** Task 3*The numpy.random.binomial function can be used tosimulate flipping a coin with a 50/50 chance of heads or tails. Interestingly, if acoin is flipped many times then the number of heads is well approximated by abell-shaped curve. For instance, if we flip a coin 100 times in a row the chance ofgetting 50 heads is relatively high, the chances of getting 0 or 100 heads is relativelylow, and the chances of getting any other number of heads decreases as you moveaway from 50 in either direction towards 0 or 100. Write some python code thatsimulates flipping a coin 100 times. Then run this code 1,000 times, keeping trackof the number of heads in each of the 1,000 simulations. Select an appropriateplot to depict the resulting list of 1,000 numbers, showing that it roughly followsa bell-shaped curve. You should explain your work in a Markdown cell above thecode.*  — BinomialThe binomial distribution is a common discrete probability distribution that there is only one outcome for each independent trial of the same probability of success from two independent values under a given set. The Binomial distribution is a discrete version of normal distribution.— [1] — [Binomial distribution](https://www.investopedia.com/terms/b/binomialdistribution.asp)[2] — [Numpy: Binomial](https://numpy.org/doc/stable/reference/random/generated/numpy.random.binomial.html)<jupyter_code># version 1.0
# Research: There are two ways of testing data for normal distribution either statistical or visual and it's recommended to do both as a statistical tests have the limitations due to sample size (V70 observations).
# https://en.wikipedia.org/wiki/Normality_test
# https://www.graphpad.com/guides/prism/8/statistics/stat_choosing_a_normality_test.htm
# https://www.graphpad.com/support/faq/testing-data-for-normal-distrbution/
# https://numpy.org/doc/stable/reference/random/generated/numpy.random.binomial.html

# import libraries 
import numpy as np
import matplotlib.pyplot as plt 
import seaborn as sns

# plot settings
%matplotlib inline
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = [12,7]

rng = np.random.default_rng(5)

# simulating tossing a coin using numpy binomial function

# Input variables
# n: number of independent experiments in each trial
# p: probability of success for each experiment
# size: number of trials
n, p, size = 100, .5, 1000

# numpy binomial function that runs our Coin tossing simulation
samples = rng.binomial(n, p, size)<jupyter_output><empty_output><jupyter_text>— *Preview of data*<jupyter_code># preview data of first 100 samples
samples[:100]<jupyter_output><empty_output><jupyter_text>— *Histogram*<jupyter_code># seaborn distplot function that visually shows the distribution of the variables and the frequency of occurrence per value in the dataset.
sns.histplot(samples, stat="count", bins=15, kde=True, color='#ADB5BD').set_title("Coin tossing Distributions");
# based on 1000 trials we can observed that distribution reminds bell curve with the mean value around 50 
samples.mean()<jupyter_output><empty_output><jupyter_text>— *Boxplot*<jupyter_code># plotting the boxplot give better overview to visualize distributions of multiple variables at the same time (minimum, first quartile, median, third quartile and maximum.)
sns.boxplot(samples, orient="h", color='#ADB5BD').set_title("Coin tossing Distributions");
# the probability of success
samples.sum() / size<jupyter_output>c:\users\izydom\documents\gmit\venv\lib\site-packages\seaborn\_decorators.py:36: FutureWarning: Pass the following variable as a keyword arg: x. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  warnings.warn(<jupyter_text>— *QQ plot*`Quantile-Quantile Plot is another useful plot for checking the distribution of dataset whether follows a normal distribution. Checks the quantiles of our variable with idealized variable of Gaussian distribution and allows to identify the deviation of a distribution. Implementing a QQ plot can be done by using statsmodels package.`<jupyter_code># import qqplot from statsmodels
# https://stackoverflow.com/questions/48009614/quantile-quantile-plot-using-python-statsmodels-api
from statsmodels.graphics.gofplots import qqplot

qqplot(samples, line='s');<jupyter_output><empty_output><jupyter_text>`Looks more likely that our simulation falls into normal distribution with small deviation at both tails that cause doubts about normality, in which case statistical test will be performed.`  — *Statistical Normality Tests*<jupyter_code># There are many test to evalute normality but four of them are favored: D'Agostino-Pearson, Anderson-Darling, Shapiro-Wilk and Kolmogorov-Smirnov. Each test produces the p-value that determine the result. Either p-value is not significant and fails to reject H0 (p > alpha) or p-value is significant and rejects H0.

# Try to evalute dataset with Shapiro-Wilk normality test. 
# According to ncss Shapiro-Wilk test for normality has been found to be the most powerful test in most situations.
# https://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/NCSS/Normality_Tests.pdf

# import Scipy.stats package
import scipy.stats as ss

# sets a treshold for a correctly assumed hypothesis
alpha = 0.05

stat, p = ss.shapiro(samples)
print('Statistics={:.3f}, p={:.3f}'.format(stat, p))

# Evaluation 
if p > alpha:
    print('Samples fall into normal distribution (fail to reject H0)')
else:
    print('Samples do not fall into normal distribution (reject H0)')
# Let's have a look if we get a confirmation of rejection H0 of Shapiro-Wilk test by using D'Agostino's normality test.

stat, p = ss.normaltest(samples)
print('Statistics={:.3f}, p={:.3f}'.format(stat, p))

# sets a treshold for a correctly assumed hypothesis
alpha = 0.05

# Evaluation
if p > alpha:
    print('Samples fall into normal distribution (fail to reject H0)')
else:
    print('Samples do not fall into normal distribution (reject H0)')<jupyter_output>Statistics=0.357, p=0.837
Samples fall into normal distribution (fail to reject H0)<jupyter_text>— ConclusionFor an assumption of 100 independent coin flip with 50% probability of success and 1000 of trials Shapiro-Wilk statistical test showed that samples do not fall into normal distribution and rejected H0. By looking for a confirmation of rejection H0 of Shapiro-Wilk test, D'Agostino's normality test disclosed that samples fall into Gaussian distribution and failed to reject H0. If some of the normality tests suggest that the samples are Gaussian and some not, then it can be assumed that data is Gaussian-like. ***  Task 4*Simpson’s paradox is a well-known statistical paradox where a trend evident in a number of groups reverses when the groups are combined into one big data set. Use numpy to create four data sets, each with an x array and a corresponding y array, to demonstrate Simpson’s paradox. You might create your $x$ arrays using numpy.linspace and create the $y$ array for each $x$ using notation like $y = a * x + b$ where you choose the a and b for each $x$, $y$ pair to demonstrate the paradox. You might see the Wikipedia page for Simpson’s paradox for inspiration.*  — *Simpson's paradox***Simpson’s paradox**, also called **Yule-Simpson effect**,  is a phenomenon in probability and statistics, and occurs when groups of data show one particular trend, but this trend is reversed when the groups are combined together. Identifying and understanding this conception is crucial for correctly interpreting data.  — DatasetsIn order to simulate Simpson's paradox four subsets will be created that contains x and y values which corresponds to x and y coordinates on plot. X values will be generated by using NumPy linspace function within given parameters whereas Y values will be generated by using linear equation: $$y = a * x + b$$ where $a$ is the slope of the regression line and $b$ is a constant also known $y$ intercept (the value of $y$ when $x$ = 0). Within each subset, linear model will be fitted using NumPy polyfit function to observe trend and compare slopes. At the very last stage all subsets will be aggregated into one master set and linear model will be fitted  to check on how the trend looks like for a whole set. —[1] — [Britannica: Smpson's Paradox](https://www.britannica.com/topic/Simpsons-paradox)[2] — [Wikipedia: Smpson's Paradox](https://en.wikipedia.org/wiki/Simpson%27s_paradox:~:text=Simpson's%20paradox%2C%20which%20also%20goes,when%20these%20groups%20are%20combined.)<jupyter_code># import packages 
import numpy as np
import matplotlib.pyplot as plt
import matplotlib
import seaborn as sns

# plot settings
%matplotlib inline
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = [16,12]

# Set your custom color palette
colors = ["#212529", "#495057", "#6C757D", "#CED4DA", "#ADB5BD"]
sns.set_palette(sns.color_palette(colors))
cmap = matplotlib.colors.ListedColormap(colors)
# Version 1 of task 4.
# Create four dummy data sets each with x and corresponding y array using numpy.linspace for x arrays and notation y = a * x + b
# Function that creates n datasets, counts coefficients for them using numpy polyfit function and plot the data.

def simpson(n):
    # create empty arrays to combine all datasets into one large set to simulate the simpson's paradox
    x = np.empty(0)
    y = np.empty(0) 
    # For loop that creates n datasets, counts coefficients for them using numpy polyfit function and plot the data.
    for i in range(n):
        # Create x values using numpy linspace function 
        xi = np.linspace(7.5 * i, 7.5 * i + 7.5, 100)
        # and y values using notation y = a * x + b (in this case b =  10 * i + some noise created using numpy random)
        yi = -1.0 * xi + 10 * i + np.random.normal(-1.0, 1.0, len(xi))
        
        # counts coefficients for dataset using numpy polyfit function
        # https://numpy.org/doc/stable/reference/generated/numpy.polyfit.html
        ai, bi = np.polyfit(xi, yi, 1)

        # plot the data
        plt.plot(xi, yi, '.');
        plt.plot(xi, ai * xi + bi, '-' )
        # good practise of the string formatters found on https://pyformat.info/
        plt.text(7.5 * i + i, 2 * i, 'y = {:.2f}x + {:.2f}'.format(ai, bi))
        
        # append data to master set 
        # https://stackoverflow.com/questions/22392497/how-to-add-a-new-row-to-an-empty-numpy-array
        x = np.append(x, xi)
        y = np.append(y, yi)
        
    # counts coefficients for master dataset 
    a, b = np.polyfit(x, y, 1)
    # and plot the data.
    plt.plot(x, a * x + b, 'r-' ) 
    
    plt.text(13.5, 1, 'y = {:.2f}x {:.2f}'.format(a, b), color='red')

simpson(4)
# Version 1.1 of task 4.
# Dataset 1.

# Create x values using numpy linspace function 
x1 = np.linspace(0.0, 2.5, 50)

# and y values using notation y = a * x + b + some noise created using numpy random
y1 = -.45 * x1 + 1.25 + np.random.normal(-0.1, 0.1, len(x1))
# Dataset 2.

# Create x values using numpy linspace function 
x2 = np.linspace(2.5, 5.0, 50)

# and y values using notation y = a * x + b + some noise created using numpy random
y2 = -.50 * x2 + 3.75 + np.random.normal(-0.1, 0.1, len(x2))
# Dataset 3.

# Create x values using numpy linspace function 
x3 = np.linspace(5.0, 7.5, 50)

# and y values using notation y = a * x + b + some noise created using numpy random
y3 = -.55 * x3 + 6.25 + np.random.normal(-0.1, 0.1, len(x3))
# Dataset 4.

# Create x values using numpy linspace function 
x4 = np.linspace(7.5, 10.0, 50)

# and y values using notation y = a * x + b + some noise created using numpy random
y4 = -.60 * x4 + 8.75 + np.random.normal(-0.1, 0.1, len(x4))
# Count coefficients that minimizes the squared error by using NumPy polyfit function.
# https://numpy.org/doc/stable/reference/generated/numpy.polyfit.html

# Good practise of the string formatters found on https://pyformat.info/

# Dataset 1.
a1, b1 = np.polyfit(x1, y1, 1)
print('\nDataset 1 regression line\ny = {:.2f} * x + {:.2f}\n'.format(a1,b1))

# Dataset 2.
a2, b2 = np.polyfit(x2, y2, 1)
print('Dataset 2 regression line\ny = {:.2f} * x + {:.2f}\n'.format(a2,b2))

# Dataset 3.
a3, b3 = np.polyfit(x3, y3, 1)
print('Dataset 3 regression line\ny = {:.2f} * x + {:.2f}\n'.format(a3,b3))

# Dataset 4.
a4, b4 = np.polyfit(x4, y4, 1)
print('Dataset 4 regression line\ny = {:.2f} * x + {:.2f}\n'.format(a4,b4))
# Plot the subsets with the linear model fit.

# Dataset 1.
plt.plot(x1, y1, '.');
plt.plot(x1, a1 * x1 + b1, '-', label='Dataset 1 regression line')
plt.text(0, 0, 'y = {:.2f} * x + {:.2f}'.format(a1, b1), size=14);

# Dataset 2.
plt.plot(x2, y2, '.');
plt.plot(x2, a2 * x2 + b2, '-', label='Dataset 2 regression line')
plt.text(2, 2.6, 'y = {:.2f} * x + {:.2f}'.format(a2, b2), size=14);

# Dataset 3.
plt.plot(x3, y3, '.');
plt.plot(x3, a3 * x3 + b3, '-', label='Dataset 3 regression line')
plt.text(4, 3.6, 'y = {:.2f} * x + {:.2f}'.format(a3, b3), size=14);

# Dataset 4.
plt.plot(x4, y4, '.');
plt.plot(x4, a4 * x4 + b4, '-', label='Dataset 4 regression line')
plt.text(8, 2.4, 'y = {:.2f} * x + {:.2f}'.format(a4, b4), size=14);

plt.title("Subsets with Regression Line");
plt.legend();
# Creates whole set containing all subsets by using NumPy concatenate function.
x = np.concatenate((x1, x2, x3, x4))
y = np.concatenate((y1, y2, y3, y4))

# Main dataset.
a, b = np.polyfit(x, y, 1)
print('\nDataset regression line\ny = {:.2f} * x + {:.2f}\n'.format(a,b))
# Plot main dataset with regression line.
plt.plot(x, y, '.');
plt.plot(x, a * x + b, 'r-', label='Dataset Regression line');
plt.text(4.2, 2.5, 'y = {:.2f} * x + {:.2f}'.format(a, b), color='red', size=14);
plt.title("Dataset with Regression Line");<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10251798561151079, 'fraction_numerical': 0.026651406147809024, 'mean_word_length': 3.6265758951084215, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 40, 'https://': 21, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '449593', 'license': 'MIT', 'n_tokens_mistral': 6023, 'n_tokens_neox': 5573, 'n_words': 2272, 'path': 'tasks.ipynb', 'repo_name': 'mizydorek/Fundamentals-of-Data-Analysis-Tasks-2020', 'size': '18367'}