<jupyter_start><jupyter_text>Programing for Data Analysis Assignment This assignment includes information about **numpy.random** package and detailed explanation of five distributions provided for in the package. Table of Contents* [About the numpy.random package](about-the-numpy)* [Explanation of the use of "Simple random data"](simple-random-data)* [Explanation of the use of "Permutation function"](permutation-function)* [Explanation of the use of "Distributions functions"](distributions-functions)    * [Binomial Distribution](binomial-distribution)    * [Pareto Distribusion](pareto-distribusion)    * [Laplace Distribusion](laplace-distribusion)    * [Uniform Distribusion](uniform-distribution)    * [Expotential Distribusion](expotential-distribusion)    * [Normal Distribusion](normal-distribusion)    * [Explanation of the use of seeds in generating pseudorandom numbers](seeds)* [Summary](summary)* [References](references)  About the numpy.random packageThe **numpy.random** is a part of NumPy (Numerical Python) package. NumPy is widely used in all sort of mathematical and numerical operation. It is a fundamental package for scientific computing in Python. To generate pseudorandom numbers **numpy.random** subpackage was developed. It uses the PRNG algorithm as its core generator and is separate from old Python random. The package itself includes many functions and some of them will be discussed in details in this assignment.The package numpy.random contains following [main sections](https://docs.scipy.org/doc/numpy-1.15.0/reference/routines.random.html):   * Simple Random Data   * Permutations   * Distributions   * Random Generator     [**<<<**](table)   Explanation of the use of "Simple random data"The Simple random data generates random values in output based on the selected function. There are numbers of functions to generate random data. Below examples demonstrate some functions to simply show how they can be used in practice.  numpy.random.randThis function creates an array of the given shape and populates it with random samples from a uniform distribution with number values between 0 and 1. The code below uses random.ran to create an array of 100 numbers and plot a graph to visualize how often those numbers were picked.<jupyter_code># Import all necessary libraries
import numpy as np 
import matplotlib.pyplot as plt
# Use magic function to render the figure in a notebook
%matplotlib inline
# Create an array of 100 numbers using the rand function
x = np.random.rand(100)
# Show the frequency of number occurrence on the bar chart
plt.hist(x)
plt.show()<jupyter_output><empty_output><jupyter_text>numpy.random.choiceThe "choice" is a simple function and includes four parameters that can be defined as shown below.  The **"a"** parameter allows specifying the array of input values. This is the set of input elements from which function will generate the random sample.The **"size"** parameter determines the total number of elements in the array.The **"replace"** specifies whether or not the sample is with replacement. It will control that chosen element gets back or not into the pool for next choices. The last parameter **"p"** controls the probabilities associated with each entry. By default, each item in the input array has an equal probability of being selected.  The below short example creates a simple array and shows how straightforward "choice" function is to pick the random number.<jupyter_code># Import NumPy library. 
# There is no need to import the library or repeat some code in Jupyter Notebook in every cell with new code.
# For this assignment some part of code or operations might be repeated for clarity.
import numpy as np

# Create the simple integer numbers 0 -14 array
numbers = np.arange(0,15)

# Display the array
numbers

# Select a random number from the "numbers" array
np.random.choice(numbers,1,replace = "True")<jupyter_output><empty_output><jupyter_text>Colour Dice Roll    A practical example where "choice" function was hired is a dice roll. To make this exercise more interesting Rubik's cube colours were used instead of numbers.<jupyter_code># Import all libraries
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Define function that will be call
def roll_dice():
    # Define colours of dice sides
    dice_sides = ["red","blue","white","orange","green","yellow"]
    # Pick random colour and return selection to the place of the function
    return np.random.choice(dice_sides)  

# Create samples array with randomly selected colours in defined numbers of tries - 100
samples100 = [roll_dice() for i in range(100)]
# Increasing range numbers makes a big impact on the distribution - 10000 tries
samples10000 = [roll_dice() for i in range(10000)]

# Defines chart size
plt.figure(figsize=(16,8))
# Creates colours array which will be assigned on the chart
colors=["red","blue","white","orange","green","yellow"]
# Sets a chart style
sns.set(style="darkgrid", color_codes=True)

# Creates a  chart with samples100 data
plt.subplot(1,2,1)
plt.title("100 Tries", fontweight="bold", fontsize=15)
sns.countplot(samples100, palette=sns.xkcd_palette(colors), edgecolor=(0,0,0), 
              order=("red","blue","white","orange","green","yellow") )

# Creates a  chart with samples10000 data
plt.subplot(1,2,2)
plt.title("10000 Tries", fontweight="bold", fontsize=15)
sns.countplot(samples10000, palette=sns.xkcd_palette(colors), edgecolor=(0,0,0), 
              order=("red","blue","white","orange","green","yellow") )
plt.show()<jupyter_output><empty_output><jupyter_text>Looking on results from the graphs above side by side for the different range of samples it becomes clear that higher range corresponds with less significant result in an entire statistical distribution.  [**<<<**](table)   Explanation of the use of "Permutation function" The meaning of "Permutation" in mathematics is the act of arranging the members of a set into a sequence or order, or, if the set is already ordered, rearranging (reordering) its elements—a process called permuting. Permutations are studied in almost every branch of mathematics, and in many other fields of science. In computer science, they are used for analyzing sorting algorithms; in quantum physics, for describing states of particles; and in biology, for describing RNA sequences. Permutations may act on structured objects by rearranging their components, or by certain replacements (substitutions) of symbols.One of the most popular example of permutation is Rubik's cube invented 1974 by <NAME>, each turn of the puzzle faces creates a permutation of the surface colors.   The **numpy.random** package includes two permutation function: shuffle and permutation.  ShuffleThis function doesn't change the content of the data array rather shuffled data. It can be very useful if i.e. the order of the given list or numbers need to be randomly altered.<jupyter_code># Create name list with a specific order
names = ["John", "Mark", "Thomas", "Matthew", "Barry", "Pat"]
# Use the shuffle function to randomly change the list order
np.random.shuffle(names)

names<jupyter_output><empty_output><jupyter_text>PermutationThe same as the above function the permutation doesn't change the content of the data in the array but rearrange the order of the array. The main difference between those two functions is the way of handling the original array. Permutation keeps data in the original array intact and creates a newly rearranged array. Inversely, the **Shuffle** function messing around with original array.<jupyter_code># Create name list with specific order
names = ["John", "Mark", "Thomas", "Matthew", "Barry", "Pat"]

np.random.permutation(names)
# Display original array
names<jupyter_output><empty_output><jupyter_text>[**<<<**](table)   Explanation of the use of "Distributions functions" The distribution function is the mathematical expression that describes the probability that a system will take on a specific value or set of values. The numpy.random package contains over 30 different distributions that can be used. Below there are a few different distributions which were explained in details.  Binomial Distribution   The tossing coin is a classic example of how to use binomial distribution in practice. Looking at binomial distribution from the mathematical perspective, the probability density is:**P(N) = \binom{n}{N}p^N(1-p)^{n-N} *** ***n*** is the number of trials* ***p*** is the probability of success* ***N*** is the number of successesIn this case, getting head of the coin will be success ***p*** in a given number of trials ***n***.<jupyter_code># Import all libraries
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# One time coin toss with fair 50% chancess of success
n = 1
p = 0.5

np.random.binomial(n,p)
# 100 time coin toss with fair 50% chancess of success
n = 100
p = 0.5

np.random.binomial(n,p)<jupyter_output><empty_output><jupyter_text>By changing the probability of success ***p*** it is possible to make unfair coin toss.<jupyter_code># 100 time coin toss with unfair chances of success
n = 100
# increasing chances of success
p = 0.7

np.random.binomial(n,p)<jupyter_output><empty_output><jupyter_text>Increasing p parameter to 1 it makes totally biased coin toss.<jupyter_code># 100 time totaly biased coin toss 
n = 100
# 100% chances of success
p = 1

np.random.binomial(n,p)<jupyter_output><empty_output><jupyter_text>Adding the ***N*** parameter to the function allows repeating the experiment required number of times.<jupyter_code># 10 time coin toss
n = 10
# fair 50% chancess of success
p = 0.5
# Repeat 100 times
N = 1000

result = np.random.binomial(n,p,N)
# Defines chart size
plt.figure(figsize=(12,6))
# Defines chart colour and plot chart
plt.hist(result, color='green')
plt.xlabel('Number of Heads',fontsize=14)
plt.ylabel('Number of Trials',fontsize=14)
plt.show()<jupyter_output><empty_output><jupyter_text>Using the probability p of success less than 0.5 the distribution is **positively skewed binomial**.The peak of the distribution will not be in the centre of the distribution, it will be on the left, with relatively fewer observations on the right of the distribution.Using the probability p of success grater than 0.5 the distribution is **negatively skewed binomial**.The peak will be on the right side of the distribution, with relatively fewer observations on the left.<jupyter_code># 10 time coin toss
n = 10
# chances less than 50% - positively skewed binomial
p1 = 0.2
# chances greater than 50% - negatively skewed binomial
p2 = 0.8
# Repeat 100 times
N = 1000

# Defines chart size
plt.figure(figsize=(16,8))
# Creates a  chart with negatively skewed binomial
plt.subplot(1,2,1)
plt.title("Positively skewed binomial", fontweight="bold", fontsize=15)
result = np.random.binomial(n,p1,N)
plt.hist(result, color='red')
plt.xlabel('Number of Heads',fontsize=14)
plt.ylabel('Number of Trials',fontsize=14)

# Creates a  chart with positively skewed binomial
plt.subplot(1,2,2)
plt.title("Negatively skewed binomial", fontweight="bold", fontsize=15)
result = np.random.binomial(n,p2,N)
plt.hist(result, color='blue')
plt.xlabel('Number of Heads',fontsize=14)
plt.ylabel('Number of Trials',fontsize=14)

plt.show()<jupyter_output><empty_output><jupyter_text>Pareto Distribusion The Pareto distribution refers to the observation that 80% of Italy's wealth belonged to only 20% of the population.It was named after the Italian economist <NAME> and is also called **80/20 Rule**.  Pareto observation was that most things in life were not distributed evenly. It can mean i.e. folowwing things:* 20% of the input creates 80% of the result* 20% of the workers produce 80% of the result* 20% of the customers create 80% of the revenue* 20% of the bugs cause 80% of the crashes* 20% of the features cause 80% of the usageThe very practical observation is that employees are most productive and complete 80% of their most valuable work using only 20% of their time.The probability density for the Pareto distribution is** p(x) = \frac{am^a}{x^{a+1}} **where **a** is the shape and **m** the scale.<jupyter_code># Import all libraries
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# define shape
shape = 3. 
# define scale 
scale = 2.
# define the size of samples
size = 1000
s = (np.random.pareto(shape, size) + 1) * scale
# defines chart size
plt.figure(figsize=(12,6))
# Plot the Pareto Distribution
count, bins, _ = plt.hist(s, 100, density=True)
fit = shape*scale**shape / bins**(shape+1)
plt.plot(bins, max(count)*fit/max(fit), linewidth=2, color='r')
plt.show()<jupyter_output><empty_output><jupyter_text>[**<<<**](table)   Laplace Distribusion The Laplace distribution, one of the earliest known probability distributions, is a continuous probability distribution named after the French mathematician <NAME>.   It is similar to the Gaussian/normal distribution, but is sharper at the peak and has fatter tails. It represents the difference between two independent, identically distributed exponential random variables. This distribution is the result of two exponential distributions, one positive and one negative; It is sometimes called the double exponential distribution, because it looks like two exponential distributions spliced together back-to-back.It has the probability density function**f(x; \mu, \lambda) = \frac{1}{2\lambda} \exp\left(-\frac{|x - \mu|}{\lambda}\right).**The first law of Laplace, from 1774, states that the frequency of an error can be expressed as an exponential function of the absolute magnitude of the error, which leads to the Laplace distribution. For many problems in economics and health sciences, this distribution seems to model the data better than the standard Gaussian distribution.<jupyter_code># Import all libraries
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib.pyplot as plt

# defines chart size
plt.figure(figsize=(12,6))
# define location and diversity
loc, scale = 0., 1.
# generate samples from distribution
s = np.random.laplace(loc, scale, 1000)

count, bins, ignored = plt.hist(s, 30, density=True)
# create an array of values
x = np.arange(-8., 8., .01)
# create pdf(probability density function)
pdf = np.exp(-abs(x-loc)/scale)/(2.*scale)
plt.plot(x, pdf,label="PDF")
# plot Gaussian distribution for comparison
g = (1/(scale * np.sqrt(2 * np.pi)) *
     np.exp(-(x - loc)**2 / (2 * scale**2)))
plt.plot(x,g,label="Gaussian Distribution")
plt.legend(loc="upper right")
plt.title("Laplace Distribusion", fontweight="bold", fontsize=15)
plt.show()<jupyter_output><empty_output><jupyter_text>[**<<<**](table)   Uniform Distribusion Uniform Distribution is a type of probability distibution in which all outcomes are equally likely; each variable has the same probability that it will be the outcome. A deck of cards has within it uniform distributions because the likelihood of drawing a heart, a club, a diamond or a spade is equally likely. A coin also has a uniform distribution because the probability of getting either heads or tails in a coin toss is the same.The uniform distribution can be visualized as a straight horizontal line.<jupyter_code># define low band of the output
low = -1
# define high band of the output excluding defined number
high = 0
# define size of samples
size = 1000
# generate uniform distribution
x = np.random.uniform(low,high,size)

# keep all values within defined intervals
np.all(x >= -1)
np.all(x < 0)

# defines chart size
plt.figure(figsize=(12,6))

# Plot the Uniform Distribution 
count, bins, ignored = plt.hist(x, 15, density=True)
plt.plot(bins, np.ones_like(bins), linewidth=2, color='r')
plt.title("Uniform Distribusion", fontweight="bold", fontsize=15)
plt.show()<jupyter_output><empty_output><jupyter_text>[**<<<**](table)   Expotential Distribusion The exponential distribution is often concerned with the amount of time until some specific event occurs. For example, the amount of time (beginning now) until an earthquake occurs has an exponential distribution. Other examples include the length, in minutes, of long distance business telephone calls, and the amount of time, in months, a car battery lasts. It can be shown, too, that the value of the change that you have in your pocket or purse approximately follows an exponential distribution.Values for an exponential random variable occur in the following way. There are fewer large values and more small values. For example, the amount of money customers spend in one trip to the supermarket follows an exponential distribution. There are more people who spend small amounts of money and fewer people who spend large amounts of money.The exponential distribution is widely used in the field of reliability. Reliability deals with the amount of time a product lasts.Its probability density function is**f(x; \frac{1}{\beta}) = \frac{1}{\beta} \exp(-\frac{x}{\beta})**for x > 0 and 0 elsewhere. \beta is the scale parameter, which is the inverse of the rate parameter \lambda = 1/\beta. The rate parameter is an alternative, widely used parameterization of the exponential distribution<jupyter_code># Import all libraries
import numpy as np
import pylab as plt

# defines chart size
plt.figure(figsize=(12,6))

# define the scale and size of the distribution
target = 250
beta = 1.0/target
size = 5000

# Plot the Expotential Distribution
Y = np.random.exponential(beta, size)
plt.hist(Y, density=True, bins=200,lw=0,alpha=.8)
plt.plot([0,max(Y)],[target,target],'red')
plt.ylim(0,target*1.1)
plt.title("Expotential Distribusion", fontweight="bold", fontsize=15)
plt.show()<jupyter_output><empty_output><jupyter_text>[**<<<**](table)   Normal Distribusion The Normal distribution, also known as the Gaussian distribution, is a probability distribution that is symmetric about the mean, showing that data near the mean are more frequent in occurrence than data far from the mean. In graph form, normal distribution will appear as a bell curve.  In a normal distribution, 68% (34%+34%) of the results fall within one standard deviation, and 95% (68%+13.5%+13.5%) fall within two standard deviations. At the center (the 0 point in the image above) the median (the middle value in the set), the mode (the value that occurs most often), and the mean (arithmetic average) are all the same.<jupyter_code># Import all libraries
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib.pyplot as plt
from scipy import stats

# define mean and standard deviation
mu, sigma = 0, 0.1 
s = np.random.normal(mu, sigma, 1000)
abs(mu - np.mean(s)) < 0.01
abs(sigma - np.std(s, ddof=1)) < 0.01
# defines chart size
plt.figure(figsize=(12,6))
# plot Normal Distribution
count, bins, ignored = plt.hist(s, 30, density=True)
plt.plot(bins, 1/(sigma * np.sqrt(2 * np.pi)) *
               np.exp( - (bins - mu)**2 / (2 * sigma**2) ),
         linewidth=2, color='r')
plt.title("Normal Distribusion", fontweight="bold", fontsize=15)
plt.show()
# plot Normal Distribution with Seaborn - much shorter code version
plt.figure(figsize=(12,6))
plt.title("Normal Distribusion Seaborn", fontweight="bold", fontsize=15)
sns.distplot(s, color="g");<jupyter_output><empty_output><jupyter_text>[**<<<**](table)   Explanation of use seeds in generating pseudorandom numbers The numpy.random.seed function works in conjunction with other functions from numpy.random. The important thing about using a seed for a pseudo-random number generator is that it makes the code repeatable. The Pseudo-random number generators operate by a deterministic process. If the same input is given then the pseudo-random number generator will produce the same output.In examples below it was shown the difference of using functions with and without defined **seed**.<jupyter_code># Generate random outputs with defined seed
np.random.seed(3)
np.random.randint(50, size = 10)
# Generate random outputs without defined seed
np.random.randint(50, size = 10)
# Generate random outputs with defined seed
np.random.seed(3)
np.random.randint(50, size = 10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08797421274485495, 'fraction_numerical': 0.01904289610711629, 'mean_word_length': 5.278331257783313, 'pattern_counts': {'":': 0, '<': 95, '<?xml version=': 0, '>': 70, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '662924', 'license': 'MIT', 'n_tokens_mistral': 5915, 'n_tokens_neox': 5400, 'n_words': 2521, 'path': 'Numpy_Assignment.ipynb', 'repo_name': 'G00376332/52465_Assignment', 'size': '20192'}