<jupyter_start><jupyter_text>The numpy-random package ![title](images/python-numpy-logo.png) In this assignment I will be looking at the numpy.random package. I hope to:1) Explain the overall purpose of the package.2) Explain the use of the "Simple random data" and "Permutations" functions.3) Explain the use and purpose of at least five "Distributions" functions.4) Explain the use of seeds in generating pseudorandom numbers.  Why generate random numbers?Randomness is used as a tool or feature in preparing data and in learning algorithims that map input data to output data in order to make predictions. The online casino industry is a good example of how this is used on a daily basis.   What is the NumPy Random Package? NumPy is the fundamental package for scientific computing in Python. The name is an acronym for "Numeric Python" It is a library that provides a multidimensional array object, various derived objects and an assortment of routines for fast operations on arrays, including mathematical, logical, shape manipulation, sorting, selecting, I/O, discrete Fournier transforms, basic linear algebra, basic statistical operations, random simulation and much more.The Numpy.Random package is a package within the overall Numpy package.It's unsually used in conjunction with other Python packages.  This package allows for generating random data that can be analysed. This package contains numerous functions that fall under four seperate headings, random sampling, permutations, distributions and random generator. I will be looking into these further in this assignment.<jupyter_code># Need to import the numpy package and matplotlib to investihgate the functions.
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
sns.set(color_codes=True) # settings for seaborn plotting style
sns.set(rc={'figure.figsize':(4.5,3)}) # settings for seaborn plot sizes<jupyter_output><empty_output><jupyter_text>Simple Random Sampling The simple random data function in the numpy.random package provides a number of different functions that allow for pseudorandom random data to be generated. A look at the main Rand function:  Rand FunctionThis function generates random numbers between 0 and 1 but not including 1.<jupyter_code>np.random.rand()<jupyter_output><empty_output><jupyter_text>It's also possible to generate more than one random number at a time by adding any additional number of rows and columns.<jupyter_code>np.random.rand(5,10)<jupyter_output><empty_output><jupyter_text>It's also possible to go further and add new arrays.<jupyter_code>np.random.rand(3, 4, 5)<jupyter_output><empty_output><jupyter_text>Plotting a histogram to have a closer look at these randomly generated numbers where numpy will take all of the random values generated and plot them accordingly.<jupyter_code>x = np.random.rand (10000)
plt.hist(x)<jupyter_output><empty_output><jupyter_text>On the histogram it's possible to see that out of the 10000 values generated, the breakdown betweem 0.0 and 1 has been broken into 10's (0.0, 0.1, 0.2, 0.3 etc) and there random numbers generated within these subsets is "roughly" the same. The values are all near 1000 so it shows that there was a relatively even spread of random numbers selected between 0.0 and 1.0.   Permutations In mathematics and statistics, a permutation is an arrangement of the members of a non-empty set. This is different to a combination which focuses on the selection of objects without regard to the order in which they are selected. In numpy.random there are two permutation functions that allow for a distinct set of numbers to be generated in permutations. These are the Permutation Function and the Shuffle Function.  Permutation FunctionThis function allows a sequence of numbers to be generated in no specific order<jupyter_code>np.random.permutation(10)<jupyter_output><empty_output><jupyter_text>It's also possible to break the sequence of numbers into seperate sets.<jupyter_code>arr = np.arange(12).reshape((4, 3))
np.random.permutation(arr)<jupyter_output><empty_output><jupyter_text>It's also possible to provide a specific set of numbers which can be returned in a permutated range.<jupyter_code>np.random.permutation([1, 5, 8, 11, 15 ])<jupyter_output><empty_output><jupyter_text>Shuffle FunctionThis function when used like this is very similar to the permutations function.<jupyter_code>arr = np.arange(10)
np.random.shuffle(arr)
arr<jupyter_output><empty_output><jupyter_text>With multi-dimensional arrays,these are only shuffled along the first axis so it is the array rows that are shuffled but not the numbers in each set.<jupyter_code>arr = np.arange(9).reshape((3, 3))
np.random.shuffle(arr)
arr<jupyter_output><empty_output><jupyter_text>Distributions The distribution of a statistical data set is a listing or function showing all the possible values of the data and how often they occur. When a distribution of categorial data is organized, it's possible to see the number or percentage in each group. In numpy there are multiple functions that allow data samples to be drawn from different types of distributions. I will take a closer look at five of these different Distribution Functions.   1. Binomial DistributionIn statistics, the binomial distribution is a frequency distribution of the possible number of successful outcomes in a given number of trials in each of which there is the same probability of success. The criteria for a binomial distribution are:1) The experiment consists of "n" repeated trials.2) Each trial can only result in two possible outcomes, a "success" or a "failure".3) The probability of success, denoted by "p", is the same on every trial.4) The trials are interdependent where the outcome of one trial does not affect the outcome of the other trials.The best example of a binomial distribution is a simple coin flip experiment.A coin is flipped 10(n) times.Each flip can result only in two possible outcomes, heads or tails. Lets say heads = success and tails = failureThe probability of success (heads) is constant - 0.5(p) on every trial.Each flip has no impact on the previous flip.Let's have a look using the np.random.bionomial function:<jupyter_code>n, p = 11, .5  # number of trials, probability of each trial
s = np.random.binomial(n, p, 100) # s is the number of times the coin lands showing heads (success)
plt.hist(s)
plt.show()
# result of flipping a coin 10 times (and if this trial of 10 flips was conducted 100 times)<jupyter_output><empty_output><jupyter_text>Looking at binomial distribution on an additional graph available online see references<jupyter_code>from scipy.stats import binom # import binom module from scipy.stats to generate random variables 
binom.rvs(n=10,p=0.5) # as with the coin example lets look at the number of success from 10 trials with a 0.5 probability of success
data_binom = binom.rvs(n=10,p=0.5,size=1000) # use binom.rvs to repeat the trials with size arguement so 1000 times the trails are repeated
ax = sns.distplot(data_binom,
                  kde=False,
                  color='skyblue',
                  hist_kws={"linewidth": 15,'alpha':1})
ax.set(xlabel='Binomial', ylabel='Frequency')<jupyter_output>C:\Users\I304302\AppData\Local\Continuum\anaconda3\lib\site-packages\matplotlib\axes\_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.
  warnings.warn("The 'normed' kwarg is deprecated, and has been "<jupyter_text>We can see the graph produces similar results to the example, if a coin is flipped 10 times there is more of a chance that the probability of success "landing on heads" will be closer to the central probability chance of 5 in 10 then on the outer findges of a 9 in 10 or 1 in 10 chance of success as indicated in the graph. In actual fact it is clear to see that the outcome of 5 heads and 5 tails in 10 flips occured in the higest number of trials about 260 times. We can also see a similar structure to the first graph I drafted.  2. Uniform DistributionIn statistics, uniform distribution also known as rectangle distribution, is a type of probability in which all outcomes are equally likely, where each variable has the same probability that it will be the outcome. There are two types of uniform distribution, discrete and continuous. The possible result of rolling a die is an example of discrete uniform distribution. It's posssible to roll a 1,2,3,4,5,6 so a roll of the die generates a discrete distribution with  a probability (p)= 1/6 for each outcome. A uniform distribution with only two possible outcomes is a special case of the binomial distribution which was discussed above.A variable X is said to be uniformly distributed if the density function is f(x) = 1/(b-a) where a and b are the parameters.Lets have a look using the np.random.uniform function:<jupyter_code>s = np.random.uniform(1,7,1000)
np.all(s >= 1)
np.all(s < 7)
count, bins, ignored = plt.hist(s, 15, density=True)
plt.plot(bins, np.ones_like(bins))
plt.show()<jupyter_output><empty_output><jupyter_text>Looking at uniform distribution on an additional graph available online see references<jupyter_code>from scipy.stats import uniform   # import uniform distribution module
n = 10000                         # random numbers from uniform distribution
a = 0                             # Generate  numbers from 0 to 10
b = 10
data_uniform = uniform.rvs(size=n, loc = a, scale=b)
ax = sns.distplot(data_uniform,
                  bins=100,
                  kde=False,
                  color='skyblue',
                  hist_kws={"linewidth": 15,'alpha':1})
ax.set(xlabel='Uniform ', ylabel='Frequency')<jupyter_output>C:\Users\I304302\AppData\Local\Continuum\anaconda3\lib\site-packages\matplotlib\axes\_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.
  warnings.warn("The 'normed' kwarg is deprecated, and has been "<jupyter_text>The np.random.uniform function is very similar to the np.random.rand function the main difference being that the random.uniform function will return values of any interger.Looking at the above graph I drafted it looks almost rectangle as the probability of rolling all six sides of the die is equal so no one side has a greater probability hence there is no one section with a much larger bar on the graph.The second graph also indicates this type of uniform relationship but spread out over a larrger data set.  3. Normal DistributionNormal distribution, sometimes referred to as the Gaussian distribution or bell curve, is a distribution that occurs naturally in many situations. An example of this would be if a class of students completed an exam, the bulk of students will score a C, while a smaller numbers of students will score a B or a D and an even smaller percentage again will score an A or an F. This creates a distribution that represents a bell where half of the data will fall to the left of the mean and half of the data will fall to the right.The properties of a normal distribution aee:1) The mean, mode and median are all equal.2) The curve is symmetric at the centre, i.e. around the mean.3) Exactly half of the values are to the left of the centre and half of the values are to the right.4) The total area under the curve is one.Lets have a look at the np.random.normal function:<jupyter_code>mu, sigma = 0, 1                       # mean and standard deviation
s = np.random.normal(mu, sigma, 1000)
plt.hist(s)
plt.show<jupyter_output><empty_output><jupyter_text>Looking at normal distribution on an additional graph available online see references<jupyter_code>from scipy.stats import norm # import norm module from scipy.stats to generate random variables
data_normal = norm.rvs(size=10000,loc=0,scale=1) # generate random numbers from N(0,1). loc is the mean and scale is the standard deviation
ax = sns.distplot(data_normal,
                  bins=100,
                  kde=False,
                  color='skyblue',
                  hist_kws={"linewidth": 15,'alpha':1})
ax.set(xlabel='Normal', ylabel='Frequency')<jupyter_output>C:\Users\I304302\AppData\Local\Continuum\anaconda3\lib\site-packages\matplotlib\axes\_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.
  warnings.warn("The 'normed' kwarg is deprecated, and has been "<jupyter_text>Looking at the first graph I drafted based on the example in the numpy documentation, if we tie it back to the example of a class of students getting exam results. If we assume the 0 = a score of C, 2 = a score of B and -2 = a score of D, we can see that the majority of students would fall into the middle of the graph, getting not too high of a score and not too low of a score, and then a much lower number of students falling in the outer fringes of the curve either scoring very low (on the left) or very high (on the right). The second graph also shows how this type of distribution results in a bell shaped curve when graphed.  4. Gamma DistributionIn statistics gamma distribution is a two-parametre family of continuous probability distributions. Gamma distributions occur frequently in models used in engineering for checking time to failure of equipment, in meterology to measure rain fall and in business areas such as insurance claims and loan defaults for which the variables are always positive and the results are skewed. A really good example is that of a call centre. We may know fairly precisely the average voulme of calls that is received but we cannot effectively calculate the probability that any one call will arrive at a specific time. The gamma distribution can be useful when dealing with rates, such as call arrival times, or wait times in a queue.Lets have a look using the np.random.gamma function:<jupyter_code>shape, scale = 2., 2.  # mean=4, std=2*sqrt(2)   
s = np.random.gamma(shape, scale, 1000)
plt.hist(s)
plt.show
from scipy.stats import gamma            # import the gamma distribution module
data_gamma = gamma.rvs(a=5, size=10000)
ax = sns.distplot(data_gamma,
                  kde=False,
                  bins=100,
                  color='skyblue',
                  hist_kws={"linewidth": 15,'alpha':1})
ax.set(xlabel='Gamma', ylabel='Frequency')<jupyter_output>C:\Users\I304302\AppData\Local\Continuum\anaconda3\lib\site-packages\matplotlib\axes\_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.
  warnings.warn("The 'normed' kwarg is deprecated, and has been "<jupyter_text>Looking at the above graphs, we see that the gamma function is typically right skewed which means that the peak of the probabiltiy distribution function will generally be found on the left-hand side of the plot. The graph is skwed like this because a natural limit prevents outcomes on one side, for example call waiting times cannot be less than zero.   5. Poisson DistributionIn statistics the poissson distribution is a discrete probability distribution of a given number of events occurring in a fixed interval of time or space if these events occur with a known constant rate and independently of the time since the last event. The conditions for poisson distribution are:1) An event can occur any number of times during a time period.2) The rate of occurance is constant.3) The probability of an event occuring is proportional to the length of the time period, for example it should be twice as likely for an event to occur in a two hour time period than it is for an event to occur in a one hour period.A call centre is again a good exapmple of this distribution. It can be used to model the number ofcalls an office would receive during a given hour, let's say lunch hour, if they know they average 10 calls per hour during this time period. Although the average is 10 calls, the centre could get any number of calls during this time. The events are independent as one call is not expected to trigger another call. The occurence rate may be assumed to be constant. Therefore it is reasonable that it is as likely to receive a call during the first half hour as it is in the last.Lets have a look using the np.random.poisson function:<jupyter_code>plt.subplot(1, 2, 1)
s = np.random.poisson(5, 10000)
count, bins, ignored = plt.hist(s, 14, density=True)
plt.hist(s)


plt.subplot(1, 2, 2)
s = np.random.poisson(20, 10000)      # changed the mu to 20
count, bins, ignored = plt.hist(s, 14, density=True)
plt.hist(s)

plt.show()<jupyter_output><empty_output><jupyter_text>The basic shape of a poission distribution graph is subject to change. A graph with a low mean density is highly skewed, see above left, but if I increase this mean the distribution will spread out and become more symmetric as in above right.This can be seen even more clearly on the graphs below.  Looking at normal distribution on an additional graph available online see references<jupyter_code>from scipy.stats import poisson               # import the gamma distribution module
data_poisson = poisson.rvs(mu=3, size=10000)
ax = sns.distplot(data_poisson,
                  kde=False,
                  color='skyblue',
                  hist_kws={"linewidth": 15,'alpha':1})
ax.set(xlabel='Poisson', ylabel='Frequency')
from scipy.stats import poisson                  # import the gamma distribution module
data_poisson = poisson.rvs(mu=10, size=10000)    # changed the mu to 10
ax = sns.distplot(data_poisson,
                  kde=False,
                  color='skyblue',
                  hist_kws={"linewidth": 15,'alpha':1})
ax.set(xlabel='Poisson', ylabel='Frequency')<jupyter_output>C:\Users\I304302\AppData\Local\Continuum\anaconda3\lib\site-packages\matplotlib\axes\_axes.py:6462: UserWarning: The 'normed' kwarg is deprecated, and has been replaced by the 'density' kwarg.
  warnings.warn("The 'normed' kwarg is deprecated, and has been "<jupyter_text>The use of Seeds in Generating Pseudorandom Numbers.Most random numbers that are generated in numpy often look random but they are actually determined by some algorithims. So these are actually pseudo random numbers rather than an out and out random number. During this random number generation, the seed is actually the starting point in the sequence. The idea behind using seeds is for debugging programs.If the same seed is used when generating random numbers, the result will be the same random numbers.Lets have a look at this in numpy:<jupyter_code>np.random.rand(5) # Generate 5 random numbers
np.random.rand(5) # Generate 5 more random numbers. We see they are all different to the first 5 numbers generated.
np.random.seed(20) # Define a seed for the rand function
np.random.rand(5)  # After setting a seed we can see the 5 numbers generated are the same as the initial 5 generated.<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06026205506339186, 'fraction_numerical': 0.017770940533658693, 'mean_word_length': 4.439122908251587, 'pattern_counts': {'":': 6, '<': 77, '<?xml version=': 0, '>': 77, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '665372', 'license': 'Apache-2.0', 'n_tokens_mistral': 5264, 'n_tokens_neox': 4870, 'n_words': 2538, 'path': 'numpy-random.ipynb', 'repo_name': 'TommyVaughan/numpy-random', 'size': '18851'}