<jupyter_start><jupyter_text>NumPy.Random Package   Overall Purpose: The purpose of the package is to build upon the existing random module in the python standard library. The existing random module has functionality for creating single random numbers and some simple distributions. The numpy random package offers all of this functionality with the addition of generating random numbers inside numpy arrays.It also supports a larger amount of distributions with further options within those distributions.NumPy.random is also significantly faster than the standard random module for generating very large samples. Both NumPy and the standard library use the Mersenne Twister pseudo-random number generator.    Why Random? Random data (or in this case pseudorandom since computers are deterministic) is useful for simulating real world phenomena which can often appear very complex or appear random. By simultating these events we can learn more about what is likely to occur and when / how often etc. Examples of this would be weather forcasting (meteorology), measurement uncertainty (metrology), economics (trading and finance)and manufacturing (FEA).  Simple Random Data You can visit https://docs.scipy.org/doc/numpy-1.15.1/reference/routines.random.html for a list of simple random data numpy docstrings.<jupyter_code># an easier way to view information on all functions. It includes great examples. 
# type help(np.random)
#import librarys and functions
import numpy as np
from numpy import random<jupyter_output><empty_output><jupyter_text>The .rand function allows you to create an array of a specified shape. That can be 1D by passing one number as an argument, 2D by passing two and 3D by passing three. The array will contain data from a uniform distribution over the range 0-1. For example, in the next line of code it shows a 5 row x 3 column array with all positive values.<jupyter_code>measurements = np.random.rand(3,3)
print(measurements)<jupyter_output>[[0.67764231 0.81081983 0.54293319]
 [0.74378546 0.99214046 0.76024875]
 [0.6284149  0.67467189 0.09959446]]<jupyter_text>When no argument is passed, i.e. nothing in the parenthesis (), then a single floating point number is returned.<jupyter_code>np.random.rand()<jupyter_output><empty_output><jupyter_text>The .randn function will return an array (1D-3D) of samples or just one sample (depending on whether an argument is passed or not) from a normal distribution of mean = 0 and variance =1. Similiar to the .rand function only data is taken from a normal and not uniform distribution.<jupyter_code>temp_read = np.random.randn(2,3)
temp_read<jupyter_output><empty_output><jupyter_text>Randomly generated data can be manipulated using simple maths.<jupyter_code># multiply the values in the array by 2
temp_read = np.random.randn(2,3)*2
temp_read
# multiplication and addition
temp_read = np.random.randn(2,3)*3 + 2
temp_read<jupyter_output><empty_output><jupyter_text>The .randint function returns random integers from a discrete uniform distribution from a range of low to high, including the low but excluding the high. The size of the array that is returned can be set i.e. give me five random numbers from the list of 1-8 (including one but excluding eight)<jupyter_code># random number between 0 and 3 (inclusive of 0, exclusive of 3. The high is not set) 
np.random.randint(3)
# inclusive of 1 but excluding 5
np.random.randint(1,5)
#return 10 numbers in the array, from 3-14)
np.random.randint(low=3,high=15, size=10)<jupyter_output><empty_output><jupyter_text>It might be useful to plot your data to ensure you have the right distribution<jupyter_code># to show that rand, randn and randint are distributed as they should be. should return uniform/normal/uniform 

# import library - pyplot needed for subplot
import matplotlib.pyplot as plt

fig, axes = plt.subplots (1,3, figsize=(15,3))
axes[0].hist(np.random.rand(15000))
axes[0].set_title ("rand dist")
axes[1].hist(np.random.randn(15000))
axes[1].set_title("randn dist")
axes[2].hist(np.random.randint(low=1, high=1000, size=15000), bins=12, align = 'left')
axes[2].set_title ("randint dist")

plt.show

print("Data distributions from functions within the numPy.random package")<jupyter_output>Data distributions from functions within the numPy.random package<jupyter_text>Create a random list of computer games and make a dataframe.<jupyter_code>##import librarys and functions
import numpy as np
from numpy import random
import pandas as pd

#define the games
games = ['GTA5','HALO 5','CIVILISATION VI','BATTLEFIELD V',
         'CALL OF DUTY', 'SUPER MARIO BROS','TOMB RAIDER',
         'GEARS OF WAR', 'DOOM', 'FIFA 19', 'CRYSIS']
randgames = []
# i just want 50 games for this example. loop until i = 50
for i in range (50): 
    # random ints from 0 to length of the games list
    game = games[np.random.randint(low=0, high=len(games))]
    randgames.append(game)

# creating my own dataset 
gamedataset = list(zip(randgames))
df=pd.DataFrame(data = gamedataset, 
                columns=['games'])

df<jupyter_output><empty_output><jupyter_text>The choice function allows you to create a unique set of integers from a population. You won't have duplication, i.e. it will ensure that you do not / you cannot select the same integer twice just as you would if you were randomly sampling from a population or sample population.<jupyter_code># return uniform sample from 0-11 (not including 11) of size 8 (number of samples)
np.random.choice(11, 8)
# replace keyword used. False means that the ints are selected without replacement.
np.random.choice(11, 8, replace = False)<jupyter_output><empty_output><jupyter_text>Since it's integers, we can use this for a list of items also and also include probability of those items being selected. If the probability is not given it wil assume a uniform distribution over the entries.<jupyter_code>np.random.choice(5, 3, p = [0.1,0.5,0.1,0.1,0.2,])

# return 5 cars from 'cars' and call that a dataset

#import librarys and functions
import numpy as np
from numpy import random
cars = ['vw','ford','audi','skoda','seat',]
dataset = np.random.choice(cars,5, p = [0.1,0.5,0.1,0.1,0.2,])
print(cars)
print(dataset)
# pick a random gift from a list for winners of the GMIT christmas draw!

#import librarys and functions
import numpy as np
from numpy import random
gifts = ['tv', 'playstation', 'xbox', 'vodka', '1yr of free dinners' ]
prize = np.random.choice(gifts)
print('Drumroll please......The winner has won a', prize)<jupyter_output>Drumroll please......The winner has won a vodka<jupyter_text>Permutations The shuffle function modifies a sequence in-place by shuffling it's contents. You can do this for 1D arrays and 2D or more.<jupyter_code># pick a winner from 'winners', then shuffle and pick a winner again
import numpy as np
from numpy import random

winners = ['pat', 'tom', 'izzie', 'mike', 'mary'] 
print (np.random.choice(winners))

# change the sequence and choose again
np.random.shuffle(winners) 
print(np.random.choice(winners))
#import librarys and functions
import numpy as np
from numpy import random

lotto_no = np.array([27,6,19,8,5,32])

print(lotto_no)

#now shuffle
np.random.shuffle(lotto_no)
print(lotto_no)<jupyter_output>[27  6 19  8  5 32]
[ 8  5  6 19 32 27]<jupyter_text>The Permutations function randomly permutes a sequence or returns a permuted range. This could be useful for returning a random sample from a dataframe.<jupyter_code>#import librarys and functions
import numpy as np
from numpy import random
import seaborn as sns

#load anscombe dataset from seaborn
df = sns.load_dataset("anscombe")

# display a random permutation/sample of 15 rows from anscombe df 
df.take(np.random.permutation(len(df))[:15])<jupyter_output><empty_output><jupyter_text>Distributions:  Normal Distribution The normal distribution is sometimes called the bell curve due to its shape. It has two parameters, X bar (mean) and sigma (standard deviation). It occurs quite a lot in the real world. If we had a real world application that we wanted to model where most of the values fall around a value Xbar with a spread of sigma then this distribution would suit. Examples might be (1) the  height of adult males in ireland where average is 180cm and standard dev is 20cm or (2) speed of cars where 80km/h is average and 10km is the standard deviation. If i have a large sigma, i'm more likely to pick a value further away from Xbar.<jupyter_code>import numpy as np
from numpy import random
# one sample drawn from the norm dist of mean 80 and stdev 10
print(np.random.normal(80,10, size = None))

#10 drawn samples in an array from a normal distribution of mean 80, stdev 10
print(np.random.normal(80,10, 10))
#plot the histogram of the 5000 data point from a normal distribution 
import numpy as np
from numpy import random
import matplotlib.pyplot as plt
plt.hist(np.random.normal(80,10, 5000))

plt.title(' Driver speed in Ireland')

plt.show()<jupyter_output><empty_output><jupyter_text>This could be used for comparing datasets or set up a script for outlier detection. Here we can try to see how two variables of 5000 points each from normal distributions correlate by inducing an outlier into one variable.<jupyter_code>import numpy as np
from numpy import random
#from the scipy library
import scipy
#from the scipy library import the pearsonr function
from scipy.stats import pearsonr



# dist1 variable with 5000 points
dist1 = np.random.normal(80,10,5000)

#create a copy of the dist1 variable and call it outlierdist
outlierdist = dist1.copy()
# induce an outlier at index 0 with a value of 200
outlierdist[0] = 200

#note - I probably should do some formatting here
#ouput some summary stats
print('Dist1 Summary')
print (np.mean (dist1))
print (np.var (dist1))
print (np.median (dist1))

print('Dist2 Summary')
print (np.mean (outlierdist))
print (np.var (outlierdist))
print (np.median (outlierdist))

#check for correlation  
print('Correlation Coefficients')
print (pearsonr(dist1,outlierdist))

#graphing both drawn variables 
fig, axes = plt.subplots (1,2, figsize=(15,3))
axes[0].hist(dist1)
axes[0].set_title ("Sample 1")
axes[1].hist(outlierdist)
axes[1].set_title("Sample 2")

#plt.hist((dist1,outlierdist))
plt.show()

#filter data or data of interest and do something with it as if real world! 
index = outlierdist > 110
data = outlierdist[index]
print(data)
data2 = data/4*100
print (data2)<jupyter_output>Dist1 Summary
80.06536482423213
98.50725460853823
80.09160371029824
Dist2 Summary
80.09160226002406
101.3581056001285
80.09348888010929
Correlation Coefficients
(0.9828817672829242, 0.0)<jupyter_text>Binomial Distribution The binomial distribution is a discrete probability distribution used when there are only two possible outcomes for a random variable: success and failure. Success and failure are mutually exclusive; they cannot occur at the same time. The binomial distribution assumes a finite number of trials, n. NumPy requires the probability set from 0-1 and an integer for number of trials.<jupyter_code>import numpy as np
from numpy import random

# n = 100, p = 0.6

n, p = 19, 0.6 
# get a variable from a random binomial distribution, call it s
s = np.random.binomial(n,p,10) # repeat 10 times

#show s
s
# a shop makes red bikes and blue bikes. the mix is 50% blue and 50% red
# you choose 10 bikes at random and picking blue is success. you do this 20000 times.

import numpy as np
from numpy import random
import matplotlib.pyplot as plt

n, p = 10, 0.5
N = 20000
binom = np.random.binomial(n,p,N)

# it should be about 50/50, roughly peaking around 5 as i have a 50/50 chance
plt.hist(binom, bins =15)
plt.show()
# a company makes 10 concept prototypes per year. Each prototype has a success probability of 0.2
# all 10 fail, what are the chances of that? Trial it 10000 times

import numpy as np
from numpy import random


n,p = 10,0.2
N= 10000

ans = sum(np.random.binomial(n,p,10000) == 0)/10000

ans

ans = ans*100

print('The probability of that happening is in percent',ans)<jupyter_output>The probability of that happening is in percent 10.7<jupyter_text>Poisson Distribution Poisson distributions are usually used to model the number of times an event happened in a time interval. This could be how many goals were scored in the 3rd round of the FA Cup or how much defects will there be in every 1000 products I make. It's similiar to binomial except we are not counting a success of a number of trials.<jupyter_code>#import librarys and functions
import numpy as np
from numpy import random
import matplotlib.pyplot as plt 

# lamda is set to 5, size is set to 10000
#lamda is the rate that event occurs i.e. average or variance
s = np.random.poisson(5,10000)
count, bins, ignored = plt.hist(s, 14, density=True)
plt.show()
#import librarys and functions
import numpy as np
from numpy import random
import matplotlib.pyplot as plt 

#try different values for Iam and n, then look at the drawn sample 
lam = 2
n = 10000
x = np.random.poisson(lam,n)


print ('Mean is =',np.mean(x))
print ('StDev is =',np.std(x))

#graphing
plt.hist(x, bins=25)
plt.title("10000 Poisson Random Variables")
plt.xlabel("No of Goals Scored")
plt.show()<jupyter_output>Mean is = 1.9952
StDev is = 1.4016336753945375<jupyter_text>Above, we can see that the number of goals (rate of occurence or lamda)are largest around the mean. You could guess that you are likely to see 2 goals on average.Below, we can attempt to figure out which of two call centres takes more calls per hour<jupyter_code>#two call centres, 1 takes 5000 calls in 1000 hrs, the other 8000 calls in 5000 hrs
#import librarys and functions
import numpy as np
from numpy import random
import matplotlib.pyplot as plt

call_centre1 = np.random.poisson(5000,1000)
call_centre2 = np.random.poisson(8000,5000)

#graphing both drawn variables 
fig, axes = plt.subplots (1,2, figsize=(15,4))
axes[0].hist(call_centre1)
axes[0].set_title ("Call Centre 1")
axes[1].hist(call_centre2)
axes[1].set_title("Call Centre 2")

#plt.hist((call_centre1,call_centre2))
plt.show()

# now change the occurence to the same overvation space
# Call Centre 1 = 5000/1000 = 5 per hour
# Call Centre 2 = 8000/5000 = 1.6 per hour

#call centres with the same observation space 
call_centre1x = np.random.poisson(5,1)
call_centre2x = np.random.poisson(1.6,1)

#graphing both drawn variables 
fig, axes = plt.subplots (1,2, figsize=(15,4))
axes[0].hist(call_centre1x)
axes[0].set_title ("Call Centre 1")
axes[1].hist(call_centre2x)
axes[1].set_title("Call Centre 2")

#plt.hist((call_centre1x,call_centre2x))
plt.show()<jupyter_output><empty_output><jupyter_text>Exponential Distribution The exponential distribution is usually used to represent a process that has events that occur continuously and independently at a constant average rate. As an example, it is commonly used in manufacturing to determine how long it takes for some equipment to fail/ what is that equipments likely lifetime.<jupyter_code>#import librarys and functions
import numpy as np
from numpy import random
import matplotlib.pyplot as plt

# The scale is the inverse of lambda

# size is None or default, it will return a single value
y = np.random.exponential(scale=1, size=None)
print (y)
#size is set to 10000, it will return 10000 drawn samples in an array
x = np.random.exponential(scale=2, size=10000)

#to check if the drawn variables are exponential
plt.hist(x, bins =200)
plt.title(" 10000 samples from an Exponential distribution")
plt.show()<jupyter_output>0.765718371532524<jupyter_text>F Distribution The f distribution is typically used to estimate the difference in variances of groups and output a ratio. If the ratio is close to one, then the variances are likely the same and it is assumed that the population means don't differ. The 'F' comes from <NAME>, who developed the distribution.<jupyter_code>#import librarys and functions
import numpy as np
from numpy import random
import matplotlib.pyplot as plt

#to show 1000 samples drawn from the random f distribution
#5 = Numerator degrees of freedom, 50 is the Denominator degrees of freedom 
f = np.random.f(5,50,1000)
plt.hist(f, bins =20)
plt.show()
# the ratio of variance of two random variable containing 40 samples
#import librarys and functions
import numpy as np
from numpy import random
#import the scipy library for stats
import scipy
from scipy import stats 

x = np.random.normal(80,10,40)
y = np.random.normal(178,25,40)

#define variances of both drawn samples
var_1 = np.var(x,ddof=1)
var_2 = np.var(y,ddof=1)

# divide one variance by the other 
f_value = (var_1/var_2)

#is the f value close to 1 or not? it shouldn't be as means and stdev set differently above 
print(f_value)<jupyter_output>0.15964060077630526<jupyter_text>Seeds when generating PRN Using the seed when generating (pseudo)random numbers ensures that the same random numbers are generated every time. The seed is essentially the position you start from and by setting it with the seed function you initialise the generator to a set state. If for example you designed code to test something,where the results is the output of a comparison between two datasets, you might want to check that this code is repeatable or reproduceable and robust. You might also want to be able to make some predictions in which case you would have to know the seed.<jupyter_code># some code to demonstrate the seed function
#import librarys and functions
import numpy as np
from numpy import random

# a random number is returned
a = np.random.rand()

#run this cell multiple times to see random values returned 
a
#import librarys and functions
import numpy as np
from numpy import random

# set the seed to 12345
np.random.seed(12345)

# give me a random number
b = np.random.rand()

# output it - run this cell multiple times, does it repeat? It should!
b<jupyter_output><empty_output><jupyter_text>Here's a real world example of useful seeding the prn. It's the use of a t-Test but using random variables as i don't have real data yet. I just want to ensure my code is correct<jupyter_code>#import librarys and functions
import numpy as np
import scipy
from scipy import stats

#set the seed
np.random.seed(555)

#generate two random variables from normal distributions. assign to x and y
x = np.random.normal(80,10,40)
y = np.random.normal(178,25,40)

#perform a t-Test
(t,pVal) = stats.ttest_rel (x,y)

#print the p-value - it should repeat every time
#if it repeats everytime i run this cell, my code is good(ish) 

#i dont care about the result as such in this case, only that it is repeatable 
print (pVal)<jupyter_output>1.1555476315114144e-25
{'fraction_non_alphanumeric': 0.07647752869248096, 'fraction_numerical': 0.04462083020486968, 'mean_word_length': 4.986195826645265, 'pattern_counts': {'":': 0, '<': 76, '<?xml version=': 0, '>': 77, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '260298', 'license': 'Apache-2.0', 'n_tokens_mistral': 6254, 'n_tokens_neox': 5554, 'n_words': 2491, 'path': 'numpy.random exploration.ipynb', 'repo_name': 'andreroche/numpy-random', 'size': '18653'}