<jupyter_start><jupyter_text>TASKSThis notebook is for solutions to the various task assignments part of the Fundamentals of Data Analysis Module at GMIT.**The author is me, <NAME> - <EMAIL>**  Task 1 Brief - Write a Python function called CountsThis function takes a list as input and returns a dictionary of unique items in the list as keys and the number of times each item appears as values.  So, with an input of ['A','A','B','C','A'], should return the output {'A': 3,'B': 1,'C': 1}.Your code should not depend on any module from the standard library or otherwise. You should research the task first and include a description with references of your algorithm in the notebook.  Research and thoughts on the problemAt face value to count the number of occurences of each value in a list will first need to loop through every item in list. As it loops through item by item check has this item already been seen, if so increment some counter by 1, if not set a count of 1 for the current item.In python terms:- Need a list as an input - A dict to store the items of the list (key) - A count of the number of occurences of that specific item (value)- And finally the input list will be iterated through using a for loop.The list and for loop are pretty self-explanatory but how to add items to a dictionary and check if one has already been added is the interesting part.<jupyter_code>#creating a dict object and assigning a value to a key
#dictionaries operate as a key-pair or key-value, where when you supply a key to a dict object it 
# returns the corresponding value
test_dict = {}
print(type(test_dict))
test_dict['a'] = 1
print(f"Printing the entire dict - {test_dict}")
print(f"Printing the dict's key-value - {test_dict['a']}")<jupyter_output><class 'dict'>
Printing the entire dict - {'a': 1}
Printing the dict's key-value - 1<jupyter_text>To check whether key exists already or not, dict.get() seems like a good approach. [dict.get() documentation](https://docs.python.org/3/library/stdtypes.html?highlight=dict.get)The first argument passed to dict.get() represents the key we are trying to get, the second optional argument is what to return if the key cannot be found, if supplying no value here *None* is returnedThe next cell is testing this idea out in a for loop and then turning that bit of logic into a function.<jupyter_code>#first I need a list to test with
test_list = ['A', 'B', 'X', 'Y', 'X', 'A', 'A']
#next I will need a dictionary for storing the key-value pairs
test_dict = {}

#simple for loop to iterate through the list
for i in test_list:
    print(i, end=' ')
    #second arg for .get is 0, .get returns this value should the key not exist
    test_dict[i] = test_dict.get(i,0) + 1
    '''
    Older solution, can be done in single line instead taking advantage of get return value if nothing found
    if test_dict.get(i, False) == False:
        test_dict[i] = 1 #add key and set value to one
    else: 
        test_dict[i] += 1 #increment key value
        
    '''
print() #introduce a new line between previous print and next print
print(test_dict)<jupyter_output>A B X Y X A A 
{'A': 3, 'B': 1, 'X': 2, 'Y': 1}<jupyter_text>Taking tested solution and enclosing with function<jupyter_code>def counts(input_list):
    #initialise blank dict
    output_dict = {}
    #loop through provided list
    for item in input_list:
        #taking advantage of dict.get return, if item not found in dict return 0 then add 1
        output_dict[item] = output_dict.get(item, 0) + 1
    
    return output_dict #return the dictionary
print(f"Provided list - {test_list}")
returned_dict = counts(test_list) #store the result from function call in variable my_dict, then print it
print(f"Count of items in list - {returned_dict}")
from IPython.display import clear_output #Clearing the notebook cell https://stackoverflow.com/questions/24816237/ipython-notebook-clear-cell-output-in-code
'''
Here for a bit of fun I wanted to refamiliarise myself with taking a user input from 
one of the previous modules.
'''
user_list = []
user_in = str(input("Enter some values and stop with blank input:"))

while user_in != '':
    clear_output() #after every input clear the output cell
    user_list.append(user_in) #after every input append it to list of user inputs
    user_in = str(input("Enter again if you would like, stop with a blank input: "))

print("You have entered the following: ", user_list)
counts(user_list)<jupyter_output>Enter again if you would like, stop with a blank input:<jupyter_text>Alternate way of getting same countInstead of using dict.get to verify whether key already exists or not, it's simpler to just use list.count() to return a count of the specified item within the list. - [see list.count() in python3 datastructures documentation](https://docs.python.org/3/tutorial/datastructures.html)Interestingly the count function likewise works for tuples.<jupyter_code>#initially did this without a set, but can make it more efficient if using a set to first convert the list to just it's unique items
#in practice this will probably not generally be an issue, but in cases of massive lists, would probably be a very efficient approach
unique_items_set = set(test_list) 
print(unique_items_set)
my_dict = {}
for i in unique_items_set:
    my_dict[i] = test_list.count(i)
print(my_dict)
def list_count_to_dict(input_list):
    #convert list into set to get unique values of the list
    input_set = set(input_list)
    #initialise blank dictionary
    output_dict = {}
    #iterate through each item in set (e.g. each unique value from input_list)
    for item in input_set:
        output_dict[item] = input_list.count(item) #output_dict keys are equal to the count of that item from the list
        
    return output_dict
#Returns the counted dict object
print(f"Provided list - {test_list}")
returned_dict = list_count_to_dict(test_list)
print(f"Count of items in list - {returned_dict}")
'''To test the functions against one another, thought it would be neat to make a large list of random strings'''
import numpy as np #importing numpy for it's random package
import string #importing string to return a string containing letters of the alphabet upper and lowercase

alphabet_list = list(string.ascii_letters) #https://stackoverflow.com/questions/2823316/generate-a-random-letter-in-python
random_array = np.random.choice(alphabet_list, size=10000)

#annoyingly but maybe obviously, list_count_to_dict function can't handle numpy arrays as it relies on list.count
#https://numpy.org/doc/stable/reference/generated/numpy.ndarray.tolist.html
big_test_list = random_array.tolist()
%%timeit
counts(big_test_list)
%%timeit
list_count_to_dict(big_test_list)<jupyter_output>6.15 ms ± 158 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)<jupyter_text>Summary and Conclusion of Task 1I found 2 different ways of getting the number of times an item appears in a list.The first was the more obvious solution to me, knowing that as we iterate through a list we can simply add to the dictionary and increment as we go but also accounting for when the item does not already exist in the dictionary.- First was simply to iterate through the list, checking whether the item was already present in the dictionary object, if it was not present add the item to the dictionary and set its value to 1, if it is present increment its value by 1- Alternative solution is to create a set out of the given list to get all of the unique values, then iterating through the set add each item to the dictionary object and set its value equal to the count of that item from the list. This solution has a quirk by changing the order the list items appear in, so if the order of items appearing in the list were to matter this would not work as a solution.In practice I would gravitate more towards the second function which uses a Set to get just the unique items in the list and then iterating through the Set assign each item as a key in the dictionary and put its value as the count of that item from the list - the advantage of this approach is you are not looping through for every item in a list, instead you loop through each unique item in the list by taking advantage of how Sets work and inbuilt list functions.However despite my preference, as we can see with the above %%timeit magic commands, the list_to_dict function (which relies on inbuilt dict functions) runs far quicker than it's list_count_to_dict function counterpart (which makes use of sets and list inbuilt functions).***  Task 2 Brief - Write a Python function called dicerollsThe task this time is to create a function that simulates rolling dice, it should take two parameters, the number of dice and the number of times to roll them. The function will simulate random rolls of the dice and keep track of the total number of occurences of the sum of the dice rolls.After the function has finished it returns a dictionary object holding the number of occurrences the different dice face totals occurred.`{2:19,3:50,4:82,5:112,6:135,7:174,8:133,9:114,10:75,11:70,12:36}`<jupyter_code>#With this function I want to show what kind of size of dictionary will eventually be returned by the dicerolls function
#the more dice you add, the more face totals there will be
def exploring_dicerolls(n_dice, n_rolls):
    print(f"Number of dice selected {n_dice} to roll {n_rolls} times")
    print(f"Minimum face total = {n_dice}")
    print(f"Maximum face total = {n_dice*6}")
    
#try different numbers of dice to show min and max face totals
exploring_dicerolls(2,1000)
exploring_dicerolls(4,10)<jupyter_output>Number of dice selected 2 to roll 1000 times
Minimum face total = 2
Maximum face total = 12
Number of dice selected 4 to roll 10 times
Minimum face total = 4
Maximum face total = 24<jupyter_text>Research and attempts***Exploring the task in earnest now, I can see that we will need to use a library to produce random numbers, unfortunately given the task brief I can only make use of the Standard Python Library, which in this case rules out numpy.random.Handily though the Standard Python Library does have a package that should do exactly what is needed for the dice rolls https://docs.python.org/3/library/random.htmlSpecifically the module random.randint is relevant here as I only want to return an integer value and want to specify 2 numbers as the limits.Initially I thought of a shortcut, where instead of doing individual dice rolls I could instead just get a random number between the min and max of the face totals of the dice rolled, however in practice I saw the issue with this pretty quickly and have imported matplotlib below to illustrate.<jupyter_code>import random
import matplotlib.pyplot as plt

n_dice = 2
n_rolls = 100000
my_list = []
for i in range(n_rolls):
    my_list.append(random.randint(n_dice, n_dice*6))
plt.hist(my_list);<jupyter_output><empty_output><jupyter_text>The above histogram plot was very surprising, I expected to see a uniform (practically flat) distribution but instead see a peak around 12.After a bit of playing around with the histogram plot I realised it was down to the number of bins used, as you can see above the last bar around 12 is about twice the height of all the other bars so the bar at 12 is actually covering 11 and 12, see corrected histogram plot below.<jupyter_code># want the number of bins to be 1 greater than the max minus the min 
# as we want our bins to be inclusive of every point on the range.
num_bins = ((n_dice*6) - n_dice) + 1 
plt.hist(my_list, bins=num_bins);<jupyter_output><empty_output><jupyter_text>The issue with the above is that it is not representative of what we would expect to see when rolling dice, when rolling dice (and as illustrated with example Ian provided in the Assignment) we should see a normal distribution, where values to the min and max of the range appear less frequently than those in the middle.If we rolled two dice a number of times, values like 2 and 12 occur less frequently than more central values like and the reason for this is very simple. You can only get 2 and 12 by rolling 1, 1 and 6, 6 respectively, but for other values like say 5, can instead be made up by a number of potential dice rolls (5 could be rolled with a combination of a 1 and 4, 2 and 3 and those combinations in reverse too as the order they occured could change).So for 2 and 12 there is only 1 scenario where we can get either face total, but for 5 we have 4 possible scenarios, therefore 5 is more likely to be seen more frequently.What this means for the function is we need to do independent dice rolls, e.g. a random number between 1 and 6 2 times as opposed to a random number between 2 and 12.<jupyter_code>#keeping same structure as before
n_dice = 2
n_rolls = 100000
my_list = []
#but changing how the for loop works
for i in range(n_rolls):
    face_total = 0
    for j in range(n_dice):
        face_total += random.randint(1,6)
    
    my_list.append(face_total)
plt.hist(my_list, bins=num_bins);<jupyter_output><empty_output><jupyter_text>The above histogram plot looks much more like what I initially expected to see for the dice rolls.<jupyter_code>my_dict = list_count_to_dict(my_list) #using the previous function from Task 1 to count the list
print(my_dict)<jupyter_output>{2: 2776, 3: 5476, 4: 8278, 5: 11126, 6: 13946, 7: 16733, 8: 13833, 9: 11208, 10: 8211, 11: 5632, 12: 2781}<jupyter_text>*** Function - diceRolls***From the last few cells I think I now have a pretty good way of getting the face totals and their count into a dictionary, so now to convert that into a function that takes inputs<jupyter_code>def diceRolls(n_dice, n_rolls):
    face_total_list = []
    #loop by number of rolls
    for i in range(n_rolls):
        face_total = 0 #set face_total to 0 on every new roll
        for j in range(n_dice): #for every dice to be rolled n_dice 
            face_total += random.randint(1,6) #pick a random number from 1 to 6 and add it to face_total
        face_total_list.append(face_total) #after finishing loop to tally face_total, continue roll loop
    
    #finally return a dictionary object in the format {face total : number of occurence} using solution from first task
    return counts(face_total_list)
    
print(diceRolls(1,1000))
print(diceRolls(100,1))
print(diceRolls(10, 10000))<jupyter_output>{5: 166, 3: 175, 6: 159, 2: 168, 1: 161, 4: 171}
{345: 1}
{28: 338, 34: 702, 35: 711, 39: 554, 32: 614, 40: 453, 30: 479, 31: 577, 43: 252, 36: 706, 38: 648, 33: 701, 44: 191, 46: 90, 27: 244, 25: 155, 37: 674, 26: 188, 47: 61, 29: 429, 41: 402, 21: 23, 50: 9, 45: 159, 22: 34, 42: 346, 20: 14, 23: 66, 48: 38, 24: 102, 51: 9, 18: 2, 49: 15, 52: 4, 19: 4, 16: 1, 54: 2, 17: 1, 13: 1, 53: 1}<jupyter_text>Task 2 Summary and ConclusionFor generating the above dice rolls the only packages imported were python's inbuilt random package for generating random numbers and matplotlib for showing the distribution of dice rolls.To simulate the dice rolls I created a function that takes two input arguments, n_dice for the number of dice to use and n_rolls for the number of rolls, esentially the amount of dice to throw per roll and the number of rolls total.Within the function it loops through the number of rolls and per roll it has an inner loop for each of the dice thrown, the for loops simulate the dice rolls to be performed (using random.randint(1,6)) and the outcome of each set of dice rolls is kept track of in face_total_list variable.As the face_total_list variable contains a list of the sum of all dice rolls, we can pass it as an argument to the list_to_dict function from Task 1, which will return a dictionary object counting the amount of times each sum of dice roll - which the above function diceRolls returns. *** Task 3 Distribution of coin flipsThe third task is to simulate coin flips, specifically it is to simulate the number of heads occurring after 100 coin flips and then simulate that 1000 times, visualising the resulting list/array of 1000 numbers (this assumes that the coin is fair, so either heads or tails is equally likely).First I will set up a random number generator to provide random values.<jupyter_code>import numpy as np
rng = np.random.default_rng() #using a default_rng object instead of numpy.random functions directly https://numpy.org/doc/stable/reference/random/generator.html
print("quick test of random number generator -", rng.integers(10))<jupyter_output>quick test of random number generator - 2<jupyter_text>Brief bit of theory around Binomial Distributions[Wikipedia Entry on Binomial Distribution](https://en.wikipedia.org/wiki/Binomial_distribution)[Statistics How To Link](https://www.statisticshowto.com/probability-and-statistics/binomial-theorem/binomial-distribution-formula/)A binomial distribution in simple terms is a distribution that shows the amount of times some event occurs when there are two possible outcomes and given it has a fixed probability of occurring and all occurrences are independent (an example of this for coin flip being yes/no for heads over 100 coin flips).I've skipped ahead a bit and produced a binomial distribution which can be seen in the below image, here the y axis shows the probability for each of the bins, the bins are along the x-axis and this represents the number of heads from coin flips (the bin size in this case is in steps of 5 starting from 20 ending at 80), and in this case the probability of getting between 50 and 55 is around 0.35.![image.png](images/binomial_example.png) Binomial Distribution function in numpy packageFrom the numpy.random documentation the binomial distribution function takes 3 arguments, n for number of trials, p for probability and size for the size of the array to return.https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.binomial.htmlnumpy.random.Generator.binomialFrom the task definition from Ian, we know that our size (n) is 100 as we want to flip a coin 100 times and observe the result, secondly we know that our probability (p) is 0.5 as there's a 50% chance for either heads or tails and lastly or size is 1000 as we are simulating this 1000 times overall.In the case of this example whether we are counting the occurrences of heads or tails with the distribution is subjective (as either event is just as likely), in fact with the binomial function we are not explicitly asking it to tell us the number of occurences of heads, instead what we are asking of the function is to show us the number of occurrences for an event with a 50% chance of happening, happens over 100 tries and then repeating that 1000 times.This then gives us our distribution showing the total number of 'wins' or 'heads' that occurred for each set of 100 tries over the 1000 repeats.<jupyter_code>coinflip_binomial = rng.binomial(n=100,p=0.5,size=1000)
print("What the output of the binomial function returns for distribution of 100 coinflips simulated 1000 times\n", coinflip_binomial)
print("Minimum number of heads for 100 coinflips -", min(coinflip_binomial))
print("Minimum number of heads for 100 coinflips -", max(coinflip_binomial))
print("Mean number of heads for 100 coinflips -", np.mean(coinflip_binomial))
print("Standard Deviation of heads per 100 coinflips -", np.std(coinflip_binomial))<jupyter_output>What the output of the binomial function returns for distribution of 100 coinflips simulated 1000 times
 [54 42 55 55 57 50 52 55 50 48 53 43 49 58 46 48 48 49 48 51 62 50 60 61
 50 55 51 57 51 46 45 46 50 51 46 55 44 47 57 48 46 56 50 52 45 52 50 46
 49 46 52 53 49 48 58 55 47 55 50 47 48 55 48 46 61 39 55 44 52 52 44 58
 53 49 55 48 42 44 52 55 55 59 36 47 43 51 51 57 50 48 56 48 50 43 52 48
 48 54 41 51 53 56 51 52 45 52 55 51 54 56 50 44 54 43 62 50 38 48 53 44
 52 48 45 52 39 50 49 48 58 53 48 49 42 50 58 48 49 44 43 51 50 45 46 44
 46 49 50 49 45 50 54 51 55 49 57 43 44 57 48 54 47 48 52 44 46 44 56 54
 54 48 54 60 58 47 53 56 49 46 51 46 44 60 56 41 43 43 52 50 48 51 55 42
 54 46 46 46 54 56 41 47 48 50 42 53 53 44 49 45 50 44 44 41 49 56 55 51
 45 53 48 51 47 55 58 52 46 53 50 50 51 46 36 48 42 46 53 54 49 52 58 40
 47 57 58 46 53 51 49 46 61 46 51 52 42 51 50 59 53 51 48 56 46 52 51 53
 42 50 52 45 51 48 45 46 54 47 50 57 54 50 53 50 60 58 53 60 48 52 48 52
 47 46 52 44 50 58 [...]<jupyter_text>Visualising the binomial distribution for coin flipsFrom looking at seaborn documentation it appears that distplot is most appropriate way to visualise the binomial distribution as it combines both a histogram and [kernel density estimation](https://en.wikipedia.org/wiki/Kernel_density_estimation) into a single plot.https://seaborn.pydata.org/generated/seaborn.distplot.html<jupyter_code>import seaborn as sns
sns.distplot(coinflip_binomial, color='orange').set_title('binomial distribution for 1000 games of 100 coinflips')
plt.xlabel('Heads');<jupyter_output>c:\python38\lib\site-packages\seaborn\distributions.py:2551: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>We can see from the above distplot that the values do indeed center around 50 with values out towards 40 and 60 being less probable and beyond that very unlikely.If a larger dataset were to be generated where size argument was 10,000 instead of 1,000, then you would get a smoother distribution. Saving Binomial Distribution as image<jupyter_code>'''
This cell is what I did to get an image to use in the markdown text above, this proved an interesting enough exercise
so I decided to leave it in it's own standalone cell.
On the face of it, it was simple to do but as soon as I went about describing it I realised I needed to determine what the 
'''
bin_steps=np.linspace(start=20,stop=80,num=13) #steps to use for bins I want steps of 5 between (and inclusive of) 20 to 80
plt.tick_params(which='minor', length=5, color='black')
plt.tick_params(which='minor', length=4, color='r')
plt.hist(coinflip_binomial, weights=np.ones(len(coinflip_binomial))/len(coinflip_binomial), color='orange', bins=bin_steps) #in order to get y axis as percentage not sum of total occurences I have, this argument for weights
plt.title('Fair Coin Flip Simulation')
plt.ylabel('Probability')
plt.xlabel('Flips')
plt.xticks(bin_steps); #this one was surprisingly tricky as mostly guides show examples using subplot not plot, only myself to blame for sticking with plot - https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.tick_params.html
#plt.savefig('images/binomial_example.png') #Uncomment this to save your own version
coinflip_binomial = rng.binomial(n=100,p=0.5,size=10000)
sns.distplot(coinflip_binomial, color='orange').set_title('binomial distribution for 1000 games of 100 coinflips')
plt.xlabel('Heads');<jupyter_output>c:\python38\lib\site-packages\seaborn\distributions.py:2551: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>*** Tangent to the original taskIn the next cell I thought it would be interesting to show the binomial distribution of an unfair coin flip, where instead of it being p=0.5 for heads (or tails) I would instead specify p=0.75 to represent a coin that was weighted so as to return heads 75% of the time.By shifting the value for p, so to does the center of the distribution which makes sense, if we are doing 100 coin flips with 75% probability of getting heads then after 1000 repetitions of this we should see that on average we are mostly getting 75 heads.Similarly if we set p=1 we would get 100 heads for every single repetition as there's 100% chance of getting heads<jupyter_code>#little tweak to the binomial distribution showing an uneven weighting for the coin
unfair_coinflip_binomial = rng.binomial(n=100,p=0.75,size=1000)
#print("What the output of the binomial function returns for distribution of 100 coinflips simulated 1000 times\n", unfair_coinflip_binomial)
print("Minimum number of heads for 100 coinflips -", min(unfair_coinflip_binomial))
print("Minimum number of heads for 100 coinflips -", max(unfair_coinflip_binomial))
sns.distplot(unfair_coinflip_binomial, color='orange').set_title('binomial distribution for 1000 games of 100 coinflips')
plt.xlabel('Heads');
#little tweak to the binomial distribution showing an uneven weighting for the coin
reallyunfair_coinflip_binomial = rng.binomial(n=100,p=0.99,size=1000)
print("Minimum number of heads for 100 coinflips -", min(reallyunfair_coinflip_binomial))
print("Minimum number of heads for 100 coinflips -", max(reallyunfair_coinflip_binomial))
sns.distplot(reallyunfair_coinflip_binomial, color='orange').set_title('binomial distribution for 1000 games of 100 coinflips')
plt.xlabel('Heads');<jupyter_output>Minimum number of heads for 100 coinflips - 95
Minimum number of heads for 100 coinflips - 100<jupyter_text>Interestingly when we give a really close to 1 value for p we get what multiple peaks for our Kernel Density Estimate, what I think is happening here is due to the proximity to 100 which is the max possible wins it removes what would be the right side of the distribution which would even it out. *** Recreating the Binomial Distribution without using binomial functionBased on the following line of the task brief I am attempting to recreate the binomial distribution using other numpy.random functions- "Write some python code that simulates flipping a coin 100 times.  Then run this code 1,000 times, keeping track of  the  number  of  heads  in  each  of  the  1,000  simulations."To do so I use numpy.random.choice to recreate the 100 coin flips and will then loop through that 1000 times to simulate 1000 games of 100 coin flips while keeping track of the total heads for each game.https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.choice.htmlAlternately I could use np.random.integers to represent 1/0 for head/tails but I like the idea that with choice we can explicitly state the probability for each option. I could also use a list of ['Heads','Tails'] similarly but that would then make summing the total heads more involved.<jupyter_code>coinface_list = [1,0] #here 1 is representing heads and 0 represents tails
coinflip_choice = rng.choice(a=coinface_list,p=[0.5,0.5]) #p=[0.5,0.5] as need to supply the probabilities for both 1 and 0 for argument a
print(coinflip_choice)<jupyter_output>0<jupyter_text>Summing an array - https://numpy.org/doc/stable/reference/generated/numpy.sum.html<jupyter_code>coinflip_choice = rng.choice(a=coinface_list,p=[0.5,0.5], size=100) #size=100 as want to get an array returned representing 100 coinflips
heads = sum(coinflip_choice) #as we have 1's representing heads and 0's representing tails can get the total heads by simply summing the array
print(heads)
heads_list = [] #declare empty list to be appended to
for i in range(1000):
    coinflip_choice = rng.choice(a=coinface_list,p=[0.5,0.5], size=100) #per loop get 100 results 
    heads = sum(coinflip_choice) #get sum of current total heads
    heads_list.append(heads) #append to heads list
sns.distplot(heads_list, color='orange')
plt.title('Number of heads over 100 coinflips, repeated 1000 times')
plt.xlabel('Heads');<jupyter_output>c:\python38\lib\site-packages\seaborn\distributions.py:2551: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
  warnings.warn(msg, FutureWarning)<jupyter_text>Task 3 Summary and ConclusionWith this task I took two different approaches to generate a distribution for coinflips, first I generated the 1000 repeats of 100 coinflips using the numpy binomial function and second using numpy's choice function within a for loop.While the binomial distribution option is by far the more simple and direct version it was interesting exploring how to reproduce it without using the binomial function explicitly.I took a tangent to the task to see what unfair coin flips would look like where the probabilities aren't equal.Finally I also used the seaborn libraries distplot function to visualise both a histogram and a kernel density estimate line to show the distribution. *** Task 4 Simpson's ParadoxThe objective for this task is to demonstrate the Simpson's Paradox.The Simpson's Paradox is a phenomenon in probability and statistics where a general trend that can be seen across various groups when viewed independently is reversed when they are grouped together, Simpson's Paradox highlights how misleading statistics can be when presented without being properly considered what's being shown.A simple example of this would be to imagine some figures are being published based on crime rates across different demographics, when viewed independently by their demographic group the trend may show a decline in crime rate, however if someone naievly (or as is often the case - if the outlet reporting this has an agenda and does so maliciously) groups all this data together it would show a trend upwards, suggesting crime rates are increasing.<jupyter_code>#x1 goes from 0 to 10 in 10 steps
x1 = np.linspace(1,10, 10)
y1 = 4*x1+20 
#x2 goes from 5 to 15 in 10 steps
x2 = np.linspace(5,15,10)
y2 = 1*x2
#in an effort to create a single combined dataset in order to show the line of fit for x1 and x2 and y1 and y2
x1x2 = np.concatenate((x1,x2), axis=None) #x1x2 gives array of x2 concatenated on top of x1
y1y2 = np.concatenate((y1,y2), axis=None)
plt.scatter(x1,y1, color='r')
plt.scatter(x2,y2, color='g')
m,c = np.polyfit(x1x2,y1y2,1) #as shown by Ian in video on 'Fitting Lines'
plt.plot(np.unique(x1x2), m*np.unique(x1x2)+c);<jupyter_output><empty_output><jupyter_text>With the simple example above we have an illustration of the simpsons paradox, whereby two datasets each show a clear upward linear trend. However when line of fit is acquired for both datasets together it shows a downward linear trend Adding some randomness to the data<jupyter_code>#introducing some randomness to the line's so they are no longer perfectly linear
rng = np.random.default_rng() #first declare a random num generator
x3 = np.linspace(1,10,100)
y3 = 5*x3+40+rng.normal(0.0,3,len(x3))
m3,c3 = np.polyfit(x3,y3,1)

x4 = np.linspace(10, 20, 100)
y4 = 5*x4-50+rng.normal(0.0,5,len(x4))

m4,c4 = np.polyfit(x4,y4, 1)

#in an effort to create a single combined dataset in order to show the line of fit for x1 and x2 and y1 and y2
x3x4 = np.concatenate((x3,x4), axis=None)
y3y4 = np.concatenate((y3,y4), axis=None)

comb_m, comb_c = np.polyfit(x3x4,y3y4,1)
plt.figure(figsize=(10,6))
plt.plot(np.unique(x3), m3*np.unique(x3)+c3, 'orange', linewidth=2) #want the line underneath the transparent dot's
plt.scatter(x3,y3, marker='o', alpha=0.75, label='x3 and y3') #set the scatter dots to be slightly transparent
plt.scatter(x4,y4, label='x4 and y4')
plt.plot(np.unique(x4), m4*np.unique(x4)+c4, 'green', linewidth=2) #want the line underneath the transparent dot's
plt.plot(np.unique(x3x4), comb_m*np.unique(x3x4)+comb_c, color='black', label='Simpsons Paradox Line')
plt.legend()
plt.title('x3 and x4 vs. y3 and y4')<jupyter_output><empty_output><jupyter_text>By doing the previous example I can now see a general feel for how to generate complimentary datasets that will show the Simpson's Paradox, and that is when you have a fairly consistent slope (m) by increasing the intercept (b) fairly significantly it leads to a downwards trend when grouping the two together. Demonstrating Simpson's Paradox with 4 datasets on single trend<jupyter_code>x1 = np.linspace(1,10,100)
y1 = -5*x1+50+rng.normal(0.0,3,len(x1))
m1,c1 = np.polyfit(x1,y1,1)

x2 = np.linspace(1, 20, 100)
y2 = -5*x2+100+rng.normal(0.0,5,len(x2))
m2,c2 = np.polyfit(x2,y2, 1)

x3 = np.linspace(5,30,100)
y3 = -5*x3+200+rng.normal(0.0,3,len(x3))
m3,c3 = np.polyfit(x3,y3,1)

x4 = np.linspace(20, 40, 100)
y4 = -5*x4+300+rng.normal(0.0,5,len(x4))
m4,c4 = np.polyfit(x4,y4, 1)

#in an effort to create a single combined dataset in order to show the line of fit for x1 and x2 and y1 and y2
x_all = np.concatenate((x1,x2,x3,x4), axis=None)
y_all = np.concatenate((y1,y2,y3,y4), axis=None)
m_all,c_all = np.polyfit(x_all,y_all, 1)
#comb_a, comb_b = np.polyfit(x3x4,y3y4,1)
plt.figure(figsize=(10,6))
plt.plot(np.unique(x1), m1*np.unique(x1)+c1, 'orange', linewidth=2) #want the line underneath the transparent dot's
plt.scatter(x1,y1, marker='o', alpha=0.75, label='x1 and y1') #set the scatter dots to be slightly transparent
plt.scatter(x2,y2, label='x2 and y2')
plt.plot(np.unique(x2), m2*np.unique(x2)+c2, 'green', linewidth=2) #want the line underneath the transparent dot's
#plt.plot(np.unique(x3x4), comb_a*np.unique(x3x4)+comb_b, color='black', label='Simpsons Paradox Line')
plt.plot(np.unique(x3), m3*np.unique(x3)+c3, 'purple', linewidth=2) #want the line underneath the transparent dot's
plt.scatter(x3,y3, marker='o', alpha=0.75, label='x3 and y3') #set the scatter dots to be slightly transparent
plt.scatter(x4,y4, label='x4 and y4')
plt.plot(np.unique(x4), m4*np.unique(x4)+c4, 'green', linewidth=2) #want the line underneath the transparent dot's
plt.plot(np.unique(x_all), m_all*np.unique(x_all)+c_all, color='black', label='Simpsons Paradox Line')
plt.legend()
plt.title('x1,x2,x3,x4 vs. y1,y2,y3,y4');<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07149624015631476, 'fraction_numerical': 0.05133518858428563, 'mean_word_length': 4.821957945536022, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 70, 'https://': 15, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '492793', 'license': 'MIT', 'n_tokens_mistral': 11430, 'n_tokens_neox': 9709, 'n_words': 4801, 'path': 'Tasks.ipynb', 'repo_name': 'itskeithstudent/Fundamentals-Of-Data-Analysis-52446-Tasks', 'size': '33792'}