<jupyter_start><jupyter_text>![NumPy](images/numpy.png) NumPy Random ---GMIT Programming for Data Analytics Assignment 2021 - <NAME> (G00387878)--- Introduction NumPy (which stands for Numerical Python) was created in 2005 by <NAME>, and is a Python library which is used for working with arrays. It is an open source project which can be used freely [1].  InstallationTo install NumPy using the PIP installer, run the following command:```pip install numpy``` Importing NumPyTo use NumPy in our program, we need to import the module - "np" is commonly used an an abbreviation, but you can name it as you wish. ```pythonimport numpy as np``` Task 1. Explain the overall purpose of the NumPy.Random packageNumPy is the universal standard for working with numerical data in Python, and it's users include everyone from coding students to experienced computer scientists. The NumPy API is used in Pandas, SciPy, Matplotlib, scikit-learn, scikit-images and many other Python packages. It can be used to perform a wide variety of mathematical operations on arrays [2].The NumPy "random" module is a suite of functions based on pseudo random number generation [3]. It's important to note that the term "random number" here does not mean that it will be a different number each time. "Random" means something that can not be predicted logically. As a computer program uses an algorithm to generate a random number, it can be predicated, and therefore it is not truly random. Therefore we call a "random" number that has been generated using an algorithm, "pseudo random" [4].It is possible to generate a truly random number on a computer using random data from an outside source (keystrokes, moust movements etc.), however, for the purpose of this notebook we do not need truly random numbers and are content that "pseudo random" numbers are acceptable.NumPy's random number routines produce pseudo random numbers using combinations of a BitGenerator to create sequences and a Generator to use those sequences to sample from different statistical distributions. These are defined in the the NumPy documentation, as follows [5]:* BitGenerators: Objects that generate random numbers. Typically unsigned integer words filled with sequences of either 32 or 64 random bits.* Generators: Objects that transform sequences of random bits from a BitGenerator into sequences of numbers that follows a specific probability distributution within a specified interval.The Generator provides access to a wide range of distributions, which we'll look at in more detail later. It is a replacement for NumPy RandomState. The main difference between the two is that Generator relies on an additional BitGenerator to manage state sand generate random bits, which are then transformed into random values from useful distributions [6]. The default BitGenerator used is PCG64, which is a 128-bit implementation of O'Neill's permutation congruential generator [7].To construct a new Generator with the default BitGenerator (PCG64) we use:```numpy.random.default_rng()```  Task 2: Explain the use of the "Simple random data" and "Permutations" functions. 2.1 Simple Random DataThere are four methods under the NumPy.random simple random data function which we'll look into in more detail below:* integers: returns random integers from low (inclusive) to high (exclusive), or if endpoint=True, low (inclusive) to high (inclusive).* random: returns random floats in the half-open interval.* choice: generates a random sample from a given array.* bytes: returns random bytes.We'll explore each of these further. 2.1.1 Integers```random.Generator.integers(low, high=None, size=None, dtype=np.int64, endpoint=False)```This method returns random integers from low (inclusive) to high (exclusive), or if endpoint=True, low (inclusive) to high (inclusive) [8]. In the example below, we use this method to generate an array of 5 integers between 0 and 10 (inclusive as we have specified that endpoint=True).<jupyter_code>import numpy as np
rng = np.random.default_rng()
rng.integers(0, 10, size=5, dtype=np.int64, endpoint=True)<jupyter_output><empty_output><jupyter_text>2.1.2 Random```random.Generator.random(size=None, dtype=np.float64, out=None)```This method returns random floats in the half-open interval (0.0, 1.0) [9]. In the example below, we use this method to generate an array of 5 floating point numbers between 0 and 1.0.<jupyter_code>#import numpy as np
rng = np.random.default_rng()
rng.random(size = 5)<jupyter_output><empty_output><jupyter_text>We can also use this method to generate an array of random numbers using different parameters. In the example below, we generate a three-by-two array of random numbers from (-5, 0) [9]:<jupyter_code>5 * rng.random((3, 2)) - 5
# Code taken from https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.random.html#numpy.random.Generator.random<jupyter_output><empty_output><jupyter_text>2.1.3 Choice```random.Generator.random(size=None, dtype=np.float64, out=None)```This method generates a random sample from a given array [10]. In the example below, we generate a uniform random sample of three numbers from a range of 0 to 10 (inclusive):<jupyter_code>rng = np.random.default_rng()
rng.choice(10, 3)<jupyter_output><empty_output><jupyter_text>2.1.4 Bytes```random.Generator.bytes(length)```This method returns random bytes of a defined length [11].<jupyter_code>np.random.default_rng().bytes(5)<jupyter_output><empty_output><jupyter_text>2.2 Random PermutationsA permutation refers to an arrangement of elements, e.g. (3,2,1) is a permutation of (1,2,3) and vice-versa [12]. The NumPy Random Module provides three methods for this: shuffle, permutation and permuted [13]. 2.2.1 Shuffle```random.Generator.shuffle(x, axis=0)```This method modifys an array or sequence in-place by shuffling its contents [14]. In the example below, we shuffle an array of ten integers from 0 to 9 inclusive:<jupyter_code>rng = np.random.default_rng()
arr = np.arange(10)
rng.shuffle(arr)
arr<jupyter_output><empty_output><jupyter_text>2.2.2 Permutation```random.Generator.permutation(x, axis=0)```This method randomly permutes a sequence, or returns a permuted range [15]. In the example below, we permute a range from 0 to 9 inclusive:<jupyter_code>rng = np.random.default_rng()
rng.permutation(10)<jupyter_output><empty_output><jupyter_text>2.2.3 Permuted```random.Generator.permuted(x, axis=None, out=None)```This method randomly permutes x along axis axis. Unlike shuffle, each slice along the given axis is shuffled independently of the others [16].  Task 3: Explain the use and purpose of at least five "Distributions" functions".A probability distribution shows the probabilities of getting different outcomes. Take a coin toss as a simple example, the distribution of landing on heads or tails is 0.5 and 0.5.A discrete distribution is one in which the values that the data can take on are countable. A continuous distribution is one in which the values that the data can take on are not countable [17].  3.1 Normal (Gaussian) distribution![Normal](images/normal.png)A normal distribution is "an arrangement of a data set in which most values cluster in the middle of the range and the rest taper off symmetrically toward either extreme". A good example of this is height, as most people are of average height, the number of people that are shorter or taller than average are similar and a small number of people are either very tall or very short [18]. The NumPy.Random method for drawing a random sample from a normal (Gaussian) distribution is:```random.Generator.normal(loc=0.0, scale=1.0, size=None)```<jupyter_code># Code taken from: https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.normal.html#numpy.random.Generator.normal
mu, sigma = 0, 0.1 # mean and standard deviation
s = np.random.default_rng().normal(mu, sigma, 1000)
#Verify the mean and the variance:
abs(mu - np.mean(s))
abs(sigma - np.std(s, ddof=1))
#Display the histogram of the samples, along with the probability density function:

import matplotlib.pyplot as plt
count, bins, ignored = plt.hist(s, 30, density=True)
plt.plot(bins, 1/(sigma * np.sqrt(2 * np.pi)) *
               np.exp( - (bins - mu)**2 / (2 * sigma**2) ),
         linewidth=2, color='r')
plt.show()<jupyter_output><empty_output><jupyter_text>3.2 Uniform distribution Uniform distribution "is a form of probability distribution in which an event is equally likely to occur within a certain interval"[19]. It has a symetrical bell shape which means that mean and median are equal and are both located at the centre of the distribution. A simple example of this is rolling a singular dice. The probability that any given number will appear is equal to that of every other number, i.e. the probability of one appearing is the same as two appearing, or any other number up to six. The NumPy.Random method for drawing a random sample from a uniform distribution is:```random.Generator.uniform(low=0.0, high=1.0, size=None)```<jupyter_code>import numpy as np
s = np.random.default_rng().uniform(0.0, 1, 100000)
plt.hist(s)<jupyter_output><empty_output><jupyter_text>As we can see, the above plot illustrates a uniform distribution of 100,000 random samples between 0 and 1, proving that the probability that any given number will appear is equal to that of every other number.  3.3 Binomial distribution![binomial](images/binomial.png)A binomial distribution can be described as simply the probability of a success or failure outcome in an action that is repeated multiple times. For example, a coin toss has two possible outcomes (heads or tails). It has three parameters:* n = number of trials* p = probability of occurence of each trial (0.5 for a coin toss)* size = shape of the returned arrayThe NumPy.Random method for drawing a random sample from a binomial distribution is:```random.Generator.binomial(n, p, size=None)```The below code takes 10 trials for a coin toss, generates 1000 data points and visualizes the binomial distribution [20].<jupyter_code>rng = np.random.default_rng()
n, p = 10, .5  # number of trials, probability of each trial
s = rng.binomial(n, p, 1000)

plt.hist(s)<jupyter_output><empty_output><jupyter_text>3.4 Weibull distribution![Weibull](images/weibull.png)Named after Swedish mathematician <NAME>, the Weibull distribution is a "continuous probability distribution". It is used to describe "various types of observed failures of components and phenomena" [21]. It is essentially the failure rate over time. In terms of machines like car parts, this is called "Time to Failure", and these are provided by manufacturers for planning purposes [22].A Weibull distibution has a shape and scale parameter. The shape is how quickly the componenet is likely to fail over time and is shown by the steepness of the curve. NumPy does not require the scale distribution - instead we simply multiply the Weibull value by the scale to determine the scale distribution [22]. The NumPy.Random method for drawing a random sample from a weibull distribution is:```random.Generator.weibull(a, size=None)```<jupyter_code>import numpy as np
import matplotlib.pyplot as plt

# Draw samples from the distribution. 
rng = np.random.default_rng()
a = 5. # shape
s = rng.weibull(a, 1000)
#Display the histogram of the samples, along with the probability density function:
x = np.arange(1,100.)/50.
def weib(x,n,a):
    return (a / n) * (x / n)**(a - 1) * np.exp(-(x / n)**a)

count, bins, ignored = plt.hist(rng.weibull(5.,1000))
x = np.arange(1,100.)/50.
scale = count.max()/weib(x, 1., 5.).max()
plt.plot(x, weib(x, 1., 5.)*scale)
plt.show()

# Code taken from: https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.weibull.html#numpy.random.Generator.weibull<jupyter_output><empty_output><jupyter_text>3.5 Poisson Distribution![po](images/poisson.png)The Poisson distribution is "a discrete distribution that represents how many times an event is likely to occur within a specific time period" [17]. It is most commonly used in queuing theory, such as to predict the number of customers who are likely to join a queue within a given time period.The NumPy.Random method for drawing a random sample from a Pisson distribution is [23]:```random.Generator.poisson(lam=1.0, size=None)```<jupyter_code># Draw samples from the distribution:
rng = np.random.default_rng()
s = rng.poisson(5, 10000)

#Display histogram of the sample:
count, bins, ignored = plt.hist(s, 14, density=True)
plt.show()

# Code taken from https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.poisson.html#numpy.random.Generator.poisson<jupyter_output><empty_output><jupyter_text>Task 4: Explain the use of seeds in generating pseudorandom numbersWolfram MathWorld defines a pseudorandom number simply as a "computer-generated random number" [24]. "Random" numbers generated by computer algorithms appear to be random, however, they are determined by the algorithm.Let's generate a list of 5 pseudorandom integers between 0 and 9 using numpy.random.int:<jupyter_code>import numpy as np
np.random.seed(0)
np.random.randint(10, size = 5)<jupyter_output><empty_output><jupyter_text>This creates an array of 5 numbers: 5, 0, 3, 3, 7. While this array appears to be random, if we run the same code again, we get the same output:<jupyter_code>import numpy as np
np.random.seed(0)
np.random.randint(10, size = 5)<jupyter_output><empty_output><jupyter_text>This proves that pseudorandom number generators produce numbers that look random, but are in fact determined [25]. You can try this piece of code on any computer, anywhere in the world, and you will get the same output: [5, 0, 3, 3, 7].A random seed specifies the start point when a computer generates a random number. This can be any number, but usually comes from seconds on a computer system's clock, which counts seconds from 01/01/1970 - a system called Unix time. This provides a large number of seeds (o to 1+ billion) which ensures that it is highly unlikely that the same random seed will be used twice [26]. NumPy.random.seed is a function that sets the random seed of the NumPy pseudorandom number generator. It is used in conjunction with other NumPy.random functions.<jupyter_code>import numpy as np
np.random.seed(74)
np.random.randint(low = 0, high = 100, size = 5)
import numpy as np
np.random.seed(74)
np.random.randint(low = 0, high = 100, size = 5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0829995856925839, 'fraction_numerical': 0.02140588316530866, 'mean_word_length': 5.218548733361958, 'pattern_counts': {'":': 0, '<': 64, '<?xml version=': 0, '>': 64, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '182850', 'license': 'MIT', 'n_tokens_mistral': 4260, 'n_tokens_neox': 3936, 'n_words': 1870, 'path': 'numpy-random.ipynb', 'repo_name': 'jbrad88/PDA-Assignment', 'size': '14507'}