<jupyter_start><jupyter_text>numpy.random Package Investigation¶ This is a Solution to the first assessment on Programming for Data Analysis¶ Author: <NAME> (G00364694)¶ using my github account dewaledr Start Date: October 12, 2018 Tasks in this assignment:1. Explain the overall purpose of the package.2. Explain the use of the “Simple random data” and “Permutations” functions.3. Explain the use and purpose of at least five “Distributions” functions.4. Explain the use of seeds in generating pseudorandom numbers.  Installation Instructions:¶- It is highly recommended you install Python using the Anaconda distribution to make sure all underlying dependencies (such as Linear Algebra libraries) all sync up. It can also be installed onto existing suystem with: conda install numpy Once you've installed NumPy you can import it as a library:  Import numpy as np  1. NumPy¶NumPy (or Numpy) is a Linear Algebra Library for Python. Numpy is incredibly fast, as it has bindings to C libraries. NumPy's arrays are more compact than Python lists -- a list of lists as you describe, in Python, would take at least 20 MB or so, while a NumPy 3D array with single-precision floats in the cells would fit in 4 MB. Access in reading and writing items is also faster with NumPy [1].NumPy is the fundamental package for scientific computing with Python. It contains among other things: a powerful N-dimensional array object sophisticated (broadcasting) functions tools for integrating C/C++ and Fortran code useful linear algebra, Fourier transform, and random number capabilities Besides its obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can also be defined.[2].Broadcasting is the term used to describe the implicit element-by-element behavior of operations; generally speaking, in NumPy all operations, not just arithmetic operations, but logical, bit-wise, functional, etc., behave in this implicit element-by-element fashion, i.e., they broadcast. For example  a and b could be multidimensional arrays of the same shape, or a scalar and an array, or even two arrays of with different shapes, provided that the smaller array is “expandable” to the shape of the larger in such a way that the resulting broadcast is unambiguous. For detailed “rules” of broadcasting see numpy.doc.broadcasting.NumPy fully supports an object-oriented approach, starting, once again, with ndarray. For example, ndarray is a class, possessing numerous methods and attributes. Many of its methods mirror functions in the outer-most NumPy namespace, giving the programmer complete freedom to code in whichever paradigm she prefers and/or which seems most appropriate to the task at hand. [3].  Numpy has many built-in functions and capabilities. In this notebook, I will focus on some of the most important aspects of Numpy: vectors, arrays, matrices, and number generation.At the core of the NumPy package, is the ndarray object. This encapsulates n-dimensional arrays of homogeneous data types, with many operations being performed in compiled code for performance. There are several important differences between NumPy arrays and the standard Python sequences:1. NumPy arrays have a fixed size at creation, unlike Python lists (which can grow dynamically). Changing the size of an ndarray will create a new array and delete the original.2. The elements in a NumPy array are all required to be of the same data type, and thus will be the same size in memory. The exception: one can have arrays of objects, thereby allowing for arrays of different sized elements.3. NumPy arrays facilitate advanced mathematical and other types of operations on large numbers of data. Typically, such operations are executed more efficiently and with less code than is possible using Python’s built-in sequences.4. A growing plethora of scientific and mathematical Python-based packages are using NumPy arrays; though these typically support Python-sequence input, they convert such input to NumPy arrays prior to processing, and they often output NumPy arrays. In other words, in order to efficiently use much (perhaps even most) of today’s scientific/mathematical Python-based software, just knowing how to use Python’s built-in sequence types is insufficient - one also needs to know how to use NumPy arrays.  Numpy Arrays¶Numpy arrays essentially come in two flavors: vectors and matrices. Vectors are strictly 1-d arrays and matrices are 2-d (note: a matrix can still have only one row or one column). Creating NumPy Arrays¶1.	From a Python List: We can create an array by directly converting a list or list of lists:<jupyter_code>import numpy as np
my_list = [1,2,3]
my_list<jupyter_output><empty_output><jupyter_text>2.	By casting a list to numpy array<jupyter_code>np.array(my_list)
my_matrix = [[3,2,1],[4,6,5],[7,8,9]]
np.array(my_matrix)<jupyter_output><empty_output><jupyter_text>There are lots of built-in ways to generate Arrays1. arange: Return evenly spaced values within a given interval.2. zeros and 3. ones: Generate arrays of zeros or ones4. linspace: Return evenly spaced numbers over a specified interval.5. eye: Creates an identity matrix. Usually a square matrix useful in Linear algebra<jupyter_code>#arange - Return evenly spaced values within a given interval 1 default when not specified.
np.arange(0,10)
np.arange(0,11,2) #Return evenly spaced values within a given interval, 2 specified.
#2. zeros - Generate arrays of zeros, shape given
np.zeros(3)
np.zeros((5,5)) #Shape given 5x5
np.ones(3)
np.ones((3,3))
#4. linspace: Return evenly spaced numbers over a specified interval.
np.linspace(0,10,3)
np.linspace(0,10,50) #Returns 50 1D array of evenly spaced numbers from  0 - 10
#5. eye: Creates an identity matrix. Usually a square matrix useful in Linear algebra
np.eye(4)<jupyter_output><empty_output><jupyter_text>Array Attributes and Methods1. reshape - One of the most useful methods that could be used on an array is the reshape method. Reshape will return an array containing the same data type in new shape.2. max - These are all useful methods for finding max or min values or to find their index locations using argmin or argmax3. min4. argmax5. argmin6. shape - Shape is an attribute that arrays have (not a method):7. dtype - We can also grab the data type of the object in the array:<jupyter_code>arr = np.arange(25) # Return a 1D array values between 0 and 24
arr
ranarr = np.random.randint(0,50,10)  #Return an array of 10 random integers between 0 and 50
ranarr
# Will return a 5x5 2D array from the original array (arr). 
# Note that arr must have 25 members (5x5), else reshape will fail.
arr.reshape(5,5) 
ranarr.max() #return the maximum value in ranarr
ranarr.min() #return the minimum value in ranarr
ranarr.argmin() #return the index location of min value in ranarr
ranarr.argmax() # return the index location of the max value in ranarr
# Will return a 5x5 2D array from the original array (arr). 
# Note that arr must have 25 members (i.e. rows of 5 x columns of 5 in this case), else reshape will fail.
arr.reshape(5,5)
print('The code shown next will generate an error if run because the shape size does not match the array size...')
# arr.reshape(5,3) #--- This will FAIL
#Shape: Shape is an attribute that arrays have (not a method):
# Vector: 
arr.shape
# Notice here I get 25 comma and that's indicating that arr array was just a one dimensional vector.
# Notice the two sets of brackets
arr.reshape(1,25)
arr.reshape(1,25).shape   # (1, 25) returns a 1x25 array
# arr.reshape(25,1)
arr.reshape(25,1).shape   # (25, 1) returns a - 25x1 array
#arr.reshape(5,5)
arr.reshape(5,5).shape   # (5, 5) returns a - 5x5 array
# You can also grab the data type of the object in the array:
arr.dtype<jupyter_output><empty_output><jupyter_text>... Please NOTE: Rather than saying import    =====> numpy.random as np  We can use a specific import option as shown here, i.e.:  =====> from numpy.random import rand, randint, randn- So, you may just use for example, randint(4,50) rather than the longer np.random.randint(4,50).[4]  2.    Simple random data 2.1.	RandomNumpy also has lots of ways to create random number arrays, for example:1. rand: numpy.random.rand(d0, d1, ..., dn)2. numpy.random.randn(d0, d1, ..., dn)3. numpy.random.randint(low, high=None, size=None, dtype='l')  1: rand(d0, d1, ..., dn)Create an array of the given shape and populate it with random samples from a uniform distribution over "0, 1" Mathematically, this convention [ [0, 1) ]means first element (0) is inclusive while the second element (1) is not. NOTE: np.random + TAB shows the existing methods for np.random<jupyter_code># np.random.rand(2) will create an array of shape 2 and populate the array with a random sample 
# from a uniform distribution over 0 to 1
np.random.rand(2)
# And that means if I want just five one dimensional array of random numbers uniformly distributed from 0 to 1 
# I can pass in a single digit and I'll get back that much one-dimensional array, i.e 5 as shown next 
np.random.rand(5)
# And if I want a 3 by 4 matrix of random numbers I just pass and 3,4 
np.random.rand(3,4)<jupyter_output><empty_output><jupyter_text>2: randn(d0, d1, …, dn).If we want to return a sample or many samples from the standard normal distribution or a Gaussian distribution instead of using rand we can use randn<jupyter_code># Return a sample (or samples) from the "standard normal" distribution. Unlike rand which is uniform:
# and this will return numbers not from a uniform distribution from 0 to 1 but instead from a 
# standard normal distribution center around zero. 
# And that means I can go ahead and pass in 9 and I get nine random numbers from a standard normal distribution.
np.random.randn(9)

# Again if I want this to be two dimensional I can just pass in two dimensions. 
# So for a four by five I'll pass in 4,5.
# Note: we’re not passing a tuple here separate argument separated by a comma. 
# And we can see by the set of two brackets is a two dimensional matrix as an output.
np.random.randn(4,5)
# A plot of a normal distribution curve or a Gaussian distribution curve is shown next: 
# REF: https://stackoverflow.com/questions/14873203/plotting-of-1-dimensional-gaussian-distribution-function
from matplotlib import pyplot as mp
# import numpy as np

def gaussian(x, mu, sig):
    return np.exp(-np.power(x - mu, 2.) / (2 * np.power(sig, 2.)))

for mu, sig in [(0, 1)]:
    mp.plot(gaussian(np.linspace(-5, 5, 100), mu, sig))

mp.show()<jupyter_output><empty_output><jupyter_text>3: randint(low, high=None, size=None, dtype='l')randint(low[, high, size, dtype]) Returns random integers from low (inclusive) to high (exclusive).- np.random.randint(SHIFT + TAB … will show the description of expected arguments…<jupyter_code># This will produce just one random integer between 1 and 100
np.random.randint(1,100) 
# To return a particular number of random integers you can pass them in as a third argument. 
# So to return 10 random integers from 1 to 100. But not including 100.
# I can just pass 10 and there are 10 random integers inclusive on the low and exclusive on the high end.
np.random.randint(1,100,10)<jupyter_output><empty_output><jupyter_text>2.2.	PermutationsA permutation of a set of objects is an arrangement of the objects in a certain order.For example:The possible permutations of letters a, b and c is: abc/ acb/ bac/ bca/ cab/ cba/Numpy also has two functions to handle elements under permutations:1. numpy.random.shuffle(x)2. numpy.random.permutation(x)  shuffle:<jupyter_code># numpy.random.shuffle(x) modify a sequence in-place by shuffling its contents.
# This function only shuffles the array along the first axis of a multi-dimensional array. 
# The order of sub-arrays is changed but their contents remains the same.
# Example:
arr = np.arange(10)
arr
# This array can be shuffled around as shown here
np.random.shuffle(arr)
arr
# Create an array of 9 elements, then reshape it to a 3x3 matrix, then shuffle...
arr = np.arange(9).reshape((3, 3))
arr
# after shuffling
np.random.shuffle(arr)
arr<jupyter_output><empty_output><jupyter_text>permutation:<jupyter_code># numpy.random.permutation(x)¶
# Randomly permute a sequence, or return a permuted range.
# Example:
np.random.permutation(10)
# If x is a multi-dimensional array, it is only shuffled along its first index
# Example:
arr = np.arange(9).reshape((3, 3))
np.random.permutation(arr)<jupyter_output><empty_output><jupyter_text>3. Distributions   Discrete Distributions: Discrete probability distributions are also called probability mass functions. Examples are:1.	Uniform Distribution2.	Binomial Distribution3.	Poisson Distribution Continuous Distributions: Continuous probability distributions are also called probability density functions. Examples are:1.	Normal Distribution2.	Exponential Distribution3.	Beta Distribution  3.1.	uniform([low, high, size])<jupyter_code># Draw samples from a uniform distribution.
# Samples are uniformly distributed over the half-open interval [low, high) (includes low, but excludes high). 
# In other words, any value within the given interval is equally likely to be drawn by uniform.

np.random.uniform(low=0.0, high=1.0, size=None)
s = np.random.uniform(-1,0,1000)
# Display the histogram of the samples, along with the probability density function:
import matplotlib.pyplot as plt
count, bins, ignored = plt.hist(s, 10, density=True)
plt.plot(bins, np.ones_like(bins), linewidth=2, color='r')
plt.show()<jupyter_output><empty_output><jupyter_text>3.2.	binomial(n, p, size=None) “Binomial” means there are two discrete, mutually exclusive outcomes of a trial. heads or tails/ on or off/ success or failureA Bernoulli Trial is a random experiment in which there are only two possible outcomes - success or failureA series of trials n will follow a binary distribution so long as:   a) the probability of success p is constant   b) trials are independent of one another<jupyter_code># numpy.random.binomial(n, p, size=None)...Draw samples from a binomial distribution.
# Samples are drawn from a binomial distribution with specified parameters, 
# n trials and p probability of success where n an integer >= 0 and p is in the interval [0,1]. 
# (n may be input as a float, but it is truncated to an integer in use)

n, p = 10, 0.5      # number of trials, probability of each trial
# result of flipping a coin 10 times, tested 1000 times.
np.random.binomial(n, p, 1000)<jupyter_output><empty_output><jupyter_text>3.3.	poisson(lam=1.0, size=None) A binomial distribution considers the number of successes out of n trialsA Poisson Distribution considers the number of successes per unit of time*over the course of many units or any other continuous unit, e.g. distance<jupyter_code># Poisson distribution draw samples from a Poisson distribution.
# The Poisson distribution is the limit of the binomial distribution for large N.
# import numpy as np
s = np.random.poisson(10, 10000)  #1000 element values center around 10, for example... 2,3, 6,10,18,...21..
# Display histogram of the sample:
import matplotlib.pyplot as plt
count, bins, ignored = plt.hist(s, 14, density=True)
plt.show()<jupyter_output><empty_output><jupyter_text>3.4 normal(loc=0.0, scale=1.0, size=None) Draw random samples from a normal (Gaussian) distribution.The probability density function of the normal distribution, first derived by De Moivre and 200 years later by both Gauss and Laplace independently [2], is often called the bell curve because of its characteristic shape (see the example below).The normal distributions occurs often in nature. For example, it describes the commonly occurring distribution of samples influenced by a large number of tiny, random disturbances, each with its own unique distribution [5]<jupyter_code># Example: Draw samples from the distribution:
mu, sigma = 0, 0.1     # mean and standard deviation
s = np.random.normal(mu, sigma, 1000)
# Verify the mean and the variance:
abs(mu - np.mean(s)) < 0.01
abs(sigma - np.std(s, ddof=1)) < 0.01

# Display the histogram of the samples, along with the probability density function:
# import matplotlib.pyplot as plt
count, bins, ignored = plt.hist(s, 30, density=True)
plt.plot(bins, 1/(sigma * np.sqrt(2 * np.pi)) * np.exp( - (bins - mu)**2 / (2 * sigma**2) ), linewidth=2, color='r')
plt.show()<jupyter_output><empty_output><jupyter_text>3.4 laplace([loc, scale, size]) numpy.random.laplace(loc=0.0, scale=1.0, size=None)¶Draw samples from the Laplace or double exponential distribution with specified location (or mean) and scale (decay).The Laplace distribution is similar to the Gaussian/normal distribution, but is sharper at the peak and has fatter tails. It represents the difference between two independent, identically distributed exponential random variablesThe first law of Laplace, from 1774, states that the frequency of an error can be expressed as an exponential function of the absolute magnitude of the error, which leads to the Laplace distribution. For many problems in economics and health sciences, this distribution seems to model the data better than the standard Gaussian distribution<jupyter_code># Example: Draw samples from the distribution
loc, scale = 0., 1.
s = np.random.laplace(loc, scale, 1000)
# Display the histogram of the samples, along with the probability density function:
# import matplotlib.pyplot as plt
count, bins, ignored = plt.hist(s, 30, density=True)
x = np.arange(-8., 8., .01)
pdf = np.exp(-abs(x-loc)/scale)/(2.*scale)
plt.plot(x, pdf, linewidth=3, color='r')
# Plot Gaussian for comparison:
g = (1/(scale * np.sqrt(2 * np.pi)) * np.exp(-(x - loc)**2 / (2 * scale**2)))
plt.plot(x,g, linewidth=3, color='y')<jupyter_output><empty_output><jupyter_text>4… The use of seeds in generating pseudorandom numbers  Pseudo-random number generators work by performing some operation on a value. Generally this value is the previous number generated by the generator. However, the first time you use the generator, there is no previous value.Seeding a pseudo-random number generator gives it its first "previous" value. Each seed value will correspond to a sequence of generated values for a given random number generator. That is, if you provide the same seed twice, you get the same sequence of numbers twice. Generally, you want to seed your random number generator with some value that will change each execution of the program. For instance, the current time is a frequently-used seed. The reason why this doesn't happen automatically is so that if you want, you can provide a specific seed to get a known sequence of numbers, for example by using the decimal portion of the irrational number pi.[6]  Seeding will enable me get the same result each time I run this dataframe<jupyter_code>import pandas as pd
#import numpy as np 
from numpy.random import randn
np.random.seed(101)  #Seeding will enable me get the same result each time I run this dataframe
df = pd.DataFrame(randn(5,4), index='A B C D E'.split(), columns='W X Y Z'.split())
print("DataFrame from above")
print(df)
print("\nGet a column 'X' from above dataframe. This is actually a Series")
print (df['X'])<jupyter_output>DataFrame from above
          W         X         Y         Z
A  2.706850  0.628133  0.907969  0.503826
B  0.651118 -0.319318 -0.848077  0.605965
C -2.018168  0.740122  0.528813 -0.589001
D  0.188695 -0.758872 -0.933237  0.955057
E  0.190794  1.978757  2.605967  0.683509

Get a column 'X' from above dataframe. This is actually a Series
A    0.628133
B   -0.319318
C    0.740122
D   -0.758872
E    1.978757
Name: X, dtype: float64
{'fraction_non_alphanumeric': 0.07614394055072021, 'fraction_numerical': 0.03033542016592864, 'mean_word_length': 4.652474108170311, 'pattern_counts': {'":': 0, '<': 63, '<?xml version=': 0, '>': 64, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '496188', 'license': 'Apache-2.0', 'n_tokens_mistral': 5917, 'n_tokens_neox': 5450, 'n_words': 2737, 'path': 'numpyRandom.ipynb', 'repo_name': 'dewaledr/PDAnalysis', 'size': '19656'}