<jupyter_start><jupyter_text>Programming for Data Analysis Assignment November 2018  <NAME>  H.Dip in Data Analytics  Problem StatementThe following assignment concerns the numpy.random package in Python. I am required to create a Jupyter notebook explaining the use of the package, including detailed explanation of at least five of the distributions provided for in the package.There are four distinct tasks which must be carried out in the Jupyter notebook:* Explain the overall purpose of the package* Explain the use of the "Simple Random Data" and "Permutations" functions* Explain the use and purpose of at least five "Distributions" functions* Explain the use of seeds in generating pseudorandom numbers  Purpose of NumPy.random package NumPy is a popular data analysis package that is used with through the Python programming language. When we use numpy we can speed up workflow by a significant percentage and also, use alongside other popular python packages such as scikit-learn.What NumPy is, is a Python library which provides a multidimensional array object, various derived objects (like matrices and masked arrays), and also an assortment of routines for extremely fast operations on arrays, which includes mathematical, logical, shape manipulation, random simulation sorting and basic linear algebra.NumPy was released in 2006 by a number of contributors such as <NAME>, <NAME> and <NAME>, <NAME>, <NAME> and <NAME> and was an updated version of a previous package known as “Numeric”. Almost all machine learning or data analysis packages that are available in Python leverage NumPy is one way or another.  My AnalysisIn order for me to compete this assignment, there were a number of resources I needed. These included:* Jupyter Notebook* Github* Python version 3.6 downloaded through Anaconda3After a lot of research I realised that I would also need to ensure that there was a number of other libraries installed on Python so that I would be able to complete this assignment. These included matplotlib, numpy and pandas.  The use of "Simple Random Data" and "Permutations" Functions  Simple Random DataSimple Random Data generates random values depending on which specific function you choose to select. There are ten functions within Simple Random Data that can be used the generate random data. These include:* rand(d0, d1, ..., dn)  which returns random values in a given shape.* randint(low[, high, size, dtype]) which returns random integers from low (inclusive) to high (exclusive).* random([size]) [0.0, 1.0) which returns random floats in the half-open interval.* randn(d0, d1, ..., dn)  which returns a sample (or samples) from the “standard normal” distribution.  PermutationsThere are two functions which can be used under Permutations - Permutations and the Shuffle Function.  PermutationsWhat permutations does is, it adjusts a series of numbers in a completely random manner. There are three main ways in which it can do this. The first being, if the order of numbers selected is a multi-dimensional array, it will display an array that was shuffled onward its first index. Secondly, if the order of numbers is an array, the function will display a shuffled version of the array. Finally if the order of numbers used is an integer, it will display the np.arange(X) in a random way.<jupyter_code>import pandas as pd  
import matplotlib.pyplot as plt  
import seaborn as sns  
import numpy as np  
#Example of Permutation Function

np.random.permutation(37) #Display all numbers between 0 and 36 in a random manner
np.random.permutation ([1, 4, 7, 88, 104, 256, 567, 810]) #Display all numbers selected in a random manner<jupyter_output><empty_output><jupyter_text>Shuffle FunctionThe shuffle function within permutations changes the order in which the numbers appear, it does not change the numbers within the array itself. It shuffles its contents – Hence the name.<jupyter_code>#Example of the Shuffle Function

R = np.random.random((12, 2))

R
np.random.shuffle (R)

R<jupyter_output><empty_output><jupyter_text>The use and purpose of at least five "Distributions" functions There are over 30 different so-called distributions functions available in the NumPy.random package. All of these are in some way related to different probability distributions. For the purpose of this assignment the 5 which I will focus on are:1.	Normal Distribution2.	Standard Normal Distribution3.	Uniform Distribution4.	Binomial Distribution5.	Poisson Distribution  Normal DistributionA normal distribution, which is sometimes called the bell curve, is a distribution that can occur naturally in a number of situations. For example, the bell curve is seen in tests like the Leaving Certificate. The majority of students who undertake these exams will score the average  grade which is a C, while a smaller percentage of students will achieve a grade of a D or B. Then an even smaller percentage of students achieve a grade of an A or an F. As a result, this leads to the creation of a distribution that looks like a bell.As we all know a bell curve is symmetrical. This means that 50 per cent of the results will fall to the left of the mean; 50 per cent will fall to the right.Normal Distributions is widely used by government and statistics bodies. It can be used to measure Points on a test, Salaries. Blood pressure and IQ scores.The empirical rule which states that for a normal distribution, nearly all of the data will fall within three standard deviations of the mean. tells you what percentage of your data falls within a certain number of standard deviations from the mean:• 68% of the data falls within one standard deviation of the mean.• 95% of the data falls within two standard deviations of the mean.• 99.7% of the data falls within three standard deviations of the mean. The conditions that are normally used in Normal Distribution are loc (which is the mean/average of the distribution), scale (which is the Standard Deviation of the distribution) and size. Loc and scale will normally be a float or an array of floats, while size should be integer or a series of integers.<jupyter_code>loc = 0
sc = 0.25
nd = np.random.normal (loc, sc, 25000)

#Plot the Normal Distribution
count, bins, ignored = plt.hist(nd, 50, density=True)
plt.plot(bins, 1/(sc * np.sqrt(2 * np.pi)) * np.exp( - (bins - loc)**2 / (2 * sc**2) ), linewidth=2, color='g')
plt.show()<jupyter_output><empty_output><jupyter_text>Standard Normal DistributionAnother way of assessing how data is distributed is to plot if on visual aid. If we can see that the data which we are working with is easily distributed, then we will be presented with a bell curve as discussed previously in the Normal Distribution section of this assignment. As we know a bell curve has very small percentage of the points on both ends of the curve and then the bigger percentage of the data is represented in the middle part of the curve. In the standard normal distribution in and around 5 per cent of the data which we would be working with falls into the ends of the bell curve and around 90 per cent will be displayed in the middle section of the bell-shaped curve.In a real life example, we can look at a student in secondary school studying for their Leaving Certificate – the standard normal distribution can help students to comprehend what subject they are performing well in and what subject they may have to put more effort into as they may not be performing as well in that subject when we compare to the other. If students get a mark in one subject that is higher than their mark in another subject, they naturally might think that they are better in the subject where you got the higher mark in. This isn’t necassarily always the case. Secondary school students (and anyone in general) can only say that they are better in a particular subject, if they get a mark which has a certain number of standard deviations above the mean. The standard deviation tells them how tightly their grades are clustered around the mean; It can allow them to compare different distributions that have different types of data — including different means.For example, if a student gets a mark of 80 in Irish and 90 in Economics, they might think that they are better in Economics than in Irish. However, in Irish, their mark is 2 standard deviations above the mean. In Economics it’s only one standard deviation above the mean. This can tell us that in Irish, the students mark is exceptionally higher than most of the students who are taking this subject (their mark falls into the tail section (the ends of the bell curve)).  The condition that is normally used in Standard Normal Distribution is size - however it is not a necessity. Similar to Normal Distribution, size should be integer or a series of integers.<jupyter_code>snd = np.random.standard_normal (27500)

snd<jupyter_output><empty_output><jupyter_text>Uniform DistributionUniform Distribution is also known as rectangular distribution and is a probability distribution which has what is known as constant probability. Uniform distribution is defined by two requirements, a and b where a is the minimum and b is the maximum.Uniform distribution has two distinct classifications. Continuous uniform Distribution being the more popular of the two. The second classification which is known as Discrete Uniform Distribution. Similar to the continuous, discrete looks like a rectangle but a series of dots (which are referred to as “known, finite number of outcomes” are used to represent the points instead of the line which is used in Continuous.A real-life example of discrete uniform distribution would be rolling a dice. When you roll a dice there are six possible outcomes of that roll – 1, 2, 3, 4, 5 or 6 – each number has a 1/6 chance of being rolled.<jupyter_code>u = np.random.uniform(-1,0,2750)
#All values are within the given interval:
np.all(u >= -1)

np.all(u < 0)

#Display the histogram of the samples, along with the probability density function:

count, bins, ignored = plt.hist(u, 15, density=True)
plt.plot(bins, np.ones_like(bins), linewidth=5, color='g')
plt.show()<jupyter_output><empty_output><jupyter_text>Binomial DistributionThe easiest way to understand binomial distribution is to think of it as the probability of an outcome being a SUCCESS or a FAILURE in an experiment that can be repeated on numerous occasions. Binomial is a strain of distribution that has only two possible outcomes. A real life example would be taking your driving test, as it can only have two possible outcomes: pass or fail.Binomial consists of two variables. The first variable in the binomial formula, n, and this represents the number of times an experiment will run for. The second variable, p, represents the probability of one specific outcome. An example of this would be that you wanted to know the probability of getting a 1 when rolling a dice. If you rolled a dice 20 times, the probability of rolling a one on any throw is 1/6. If you roll the dice twenty times one after another, you have a binomial distribution of (n=20, p=1/6). SUCCESS would be “roll a one” and FAILURE would be “roll everything else except a one.” If you wanted to know what the probability was of the dice landing on 2, 4 or 6, then the binomial distribution would be (n=20, p=1/2). That’s because the probability of throwing an even number is one half and the even numbers account for 3 of the 6 numbers on a dice.Conditions that are necessary for the BN are n (number of trials), p (probability) and size (number of times that the experiment will happen).<jupyter_code># Binomial example - coin toss at the start of a hurling match

import pandas as pd 
import matplotlib.pyplot as plt  
import seaborn as sns 
import numpy as np

# Coin toss at the start of a hurling match
#n, p, k = 20, 0.5, 500 ( n= number of trials, p= probability of each trial, k = tested 500 times)
n = 20
p = 0.5
k = 500
binomial_coin = np.random.binomial (n,p,k) 

print (binomial_coin)

# Plot the Distribution
plt.figure()
plt.subplot(1,2,1)
plt.plot(binomial_coin, 'go--', alpha=0.5)
plt.ylim(0,np.max(binomial_coin)+1)
plt.xlabel('Number of times tested')
plt.ylabel('r successes')
plt.grid()
plt.subplot(1,2,2)
nB, dum, dum = plt.hist(binomial_coin, bins=50,color='g',alpha=0.5)

#plt.xlim(0,n)
plt.xlabel('r successes')
plt.ylabel('Frequency')
plt.title('n=%i, p=%.2f, N=%i experiments' % (n,p,k))
plt.grid()<jupyter_output>[ 8 10  8  3 12 10  5 11  6 14 11 11 10 11  9 10 10 11  9  9  8 13 13 10
  8 12  8 10  9  8  9  8 10  6 10  9  7  7  8 11  9  6  8 11  9  9 10 14
 13  9 10  8 12 11  5 10 12 11 11 11  9 10 11 13 13  6  9 11 12  6 10  8
 12 10 11 10 10 11 10 10 14  8  7 10 10  8  9 11 12  9  9 10  9 10 11 10
  7  8 10 13 10  9 11 12 10  6 12 14 11  8  7 12 10  6 12 10 13  7  8  7
  6 13  9 13 11 11 13 14 10  9  7  9 14 15  9 12 12 12  8  7  9  5 11  8
  5 10  9 12 13  9 11 11  9  6 13 12 10 10 13 12 11  9 12 15 11 12  8 12
  8 10 13  6 14 13 10 12  9 11 10 11  9 13  9 10  9 10  9 12  9  9 11 10
  9 10 10  9 12 10  9  9 12 11 10 10 12 13 14 11  7  7 11 10  9  9  9  5
 10  9 10 12 12 16  6  7 10  9 11  7 13  7 12  9 15 12  9 11 13  8 11 10
  9 10 13 12  6 11 17 11 10  9  9  9 16 10  8  8  8 12  9 11 12 13 12  8
 10 10 12 10 14 11  8 11  9 10 12  8 15  9 14  9 10  8 12  8  7  7 13 10
 13  9  9 14 10 12  9  9 11 14  8 10  8 10  9  9 11 13  9 10 12 10  8  9
  7  6 12 13 15 10  7  9 10 10 10 11 12  9 10 12 11[...]<jupyter_text>Poisson DistributionPoisson Distribution enables us to anticipate the probability specific events from taking place when you have a good idea of how frequent these events would take place. What it does is it gives us the probability of a certain number of events occurring in a set period of time.A pub sells an average of around 300 drinks every Friday evening. From this data we can predict that they will sell more drinks (possibly 350/400) the following Friday evening. We can also use the example of the number of customers in a café. If for example the average number of customers spread out over the week is 350, we can anticipate the probability of one specific day of the week having more custom than another.Poisson is also used for forecasting customers and sales in business – for example for race week in Galway, there is an influx of racegoers to the city. Pubs and hotels would hire extra staff for the week to facilitate the extra strain on their existing resources. They would also purchase extra stock in advance to prevent them from running out. Poisson helps these types of businesses as it allows them to adjust their existing supply to meet anticipated demand and also works in the opposite way to ensure that resources do not go to waste.<jupyter_code>mu = 6
size = 14500

# Create an array of 14500 elements each drawn from a poissonian probability density function for a mean value of 0.6.
poisson = np.random.poisson(mu,size)

#Plot the Distribution
ax = sns.distplot(poisson, bins = 20, kde=False, color='r', hist_kws={"linewidth": 20,'alpha':1})
ax.set(xlabel='Poisson', ylabel='Frequency')<jupyter_output><empty_output><jupyter_text>Use of seeds in generating pseudorandom numbersThe random function in returns a super-fast pseudorandom number generator that derives from the algorithm known as “Mersenne Twister”. This was initially created to create data for the Monte Carlo simulations. It produces numbers with uniform distribution and a large period which make it suitable to be used across an extensive range of applications.If you run this program multiple times it will always produce a different sequence of numbers.To generate numbers that fall in under a specific range, I would use the uniform() function to do this. random() tends to generate different numbers each time it is used and it takes a lot of generating and time before it will start to repeat values it has already produced. This function would be widely used for producing unique numbers.The seed function enables us to generate random numbers and save that dataset. Seeds control the first number that is created by the formula which is used to create these pseudo random numbers which also saves that data set after we change the seed.   How to seed the pseudorandom number generator<jupyter_code>from random import seed
from random import random
# seed random number generator
seed(3)
# generate some random numbers
print(random(), random(), random())
# reset the seed
seed(5)
# generate some random numbers
print(random(), random(), random())

# seed random number generator
seed(3)
# generate some random numbers
print(rand(5))
# reset the seed
seed(3)
# generate some random numbers
print(rand(5))<jupyter_output>[0.5507979  0.70814782 0.29090474 0.51082761 0.89294695]
[0.5507979  0.70814782 0.29090474 0.51082761 0.89294695]<jupyter_text>How to reshuffle a specific sequence<jupyter_code># randomly shuffle a sequence
from random import seed
from random import shuffle
# seed random number generator
seed(1)
# prepare a sequence
sequence = [i for i in range(50)]
print(sequence)
# randomly shuffle the sequence
shuffle(sequence)
print(sequence)<jupyter_output>[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
[2, 15, 47, 9, 21, 29, 48, 5, 46, 14, 11, 32, 34, 26, 49, 19, 39, 38, 12, 22, 37, 20, 42, 35, 33, 10, 3, 18, 45, 25, 23, 17, 43, 0, 27, 40, 1, 44, 6, 13, 24, 41, 30, 28, 31, 7, 16, 4, 36, 8]
{'fraction_non_alphanumeric': 0.05146806809094025, 'fraction_numerical': 0.0569519022049583, 'mean_word_length': 4.238479952124476, 'pattern_counts': {'":': 1, '<': 42, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '312432', 'license': 'MIT', 'n_tokens_mistral': 5457, 'n_tokens_neox': 4552, 'n_words': 2888, 'path': 'numpy.random Assignment November 2018.ipynb', 'repo_name': 'richellaod/numpy.random', 'size': '17555'}