<jupyter_start><jupyter_text>The numpy.random package in PythonAn important part of any simulation is the ability to generate random numbers. For the same we use NumPy. NumPy provides various routines in the submodule random. It uses a particular algorithm, called the Mersenne Twister, to generate pseudorandom numbers.Randomization is very important in data science. just think about experimenting and A/B testing! If you import the random module, you can generate random numbers by various rules.<jupyter_code># let us see in practice

import numpy as np

# The below is to generate random number from 1 to 10

np.random.randint(0,10)

np.random.random()<jupyter_output><empty_output><jupyter_text>The rand function is used to generate two dimensional arrays<jupyter_code>np.random.rand(3,2)<jupyter_output><empty_output><jupyter_text>The purpose of the Numpy random package is to generate random numbers in a variety of ways. Useful to create a random matrix of any shape.  Simple Random DataSimple Random Data sampling is a technique in which every item in the population has an equal possiblility of being selected witin the totality of the sample. This completely depends on tgoes the concepts of probability and therefore known as the method of chances.An important part of any simulation is the ability to generate random numbers. For this purpose, NumPy provides various routines in the submodule random. It uses a particular algorithm, called the Mersenne Twister, to generate pseudorandom numbers.<jupyter_code>np.random.random()<jupyter_output><empty_output><jupyter_text>The output of the above code gives the random numbers generated between 0 and 1.<jupyter_code>ran_number = np.random.random()
print(ran_number)<jupyter_output>0.5950321774074693<jupyter_text>PermutationsIn mathematics, the notion of permutation relates to the act of arranging all the members of a set into some sequence or order, or if the set is already ordered, rearranging (reordering) its elements, a process called permuting. These differ from combinations, which are selections of some members of a set where order is disregarded.In the following image each of the six rows is a different permutation of three distinct balls. Pictorial Presentation![image.png](attachment:image.png)<jupyter_code>import itertools
print(list(itertools.permutations([1,2,3])))<jupyter_output>[(1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2), (3, 2, 1)]<jupyter_text>The above code gives you the different permutations.  DistributionsProbability measures how likely it is for an event to occur on a scale from 0 (the event never occurs) to 1 (the event always occurs.). When working with data, variables in the columns of the data set can be thought of as random variables: variables that vary due to chance. A probability distribution describes how a random variable is distributed; it tells us which values a random variable is most likely to take on and which values are less likely.In statistics, there are a range of precisely defined probability distributions that have different shapes and can be used to model different types of random events. In this lesson we'll discuss some common probability distributions and how to work with them in Python. The Binomial DistributionThe binomial distribution is a discrete probability distribution that models the outcomes of a given number of random trails of some experiment or event. The binomial is defined by two parameters: the probability of success in any given trial and the number of trials. The binomial distribution tells you how likely it is to achieve a given number of successes in n trials of the experiment. For example, we could model flipping a fair coin 10 times with a binomial distribution where the number of trials is set to 10 and the probability of success is set to 0.5. In this case the distribution would tell us how likely it is to get zero heads, 1 head, 2 heads and so on.The scipy name for the binomial is binom. Let's generate and investigate some binomial data:<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import scipy.stats  as stats
%matplotlib inline
fair_coin_flips = stats.binom.rvs(n=10,        # Number of flips per trial
                                  p=0.5,       # Success probability
                                  size=10000)  # Number of trials

print( pd.crosstab(index="counts", columns= fair_coin_flips))

pd.DataFrame(fair_coin_flips).hist(range=(-0.5,10.5), bins=11)<jupyter_output>col_0   0   1    2     3     4     5     6     7    8   9   10
row_0                                                         
counts   8  89  428  1101  2094  2480  2048  1207  434  99  12<jupyter_text>Note that since the binomial distribution is discrete, it only takes on integer values so we can summarize binomial data with a frequency table and its distribution with a histogram. The histogram shows us that a binomial distribution with a 50% probability of success is roughly symmetric, with the most likely outcomes lying at the center. This is reminiscent of the normal distribution, but if we alter the success probability, the distribution won't be symmetric:<jupyter_code>biased_coin_flips = stats.binom.rvs(n=10,      # Number of flips per trial
                                  p=0.8,       # Success probability
                                  size=10000)  # Number of trials

# Print table of counts
print( pd.crosstab(index="counts", columns= biased_coin_flips))

# Plot histogram
pd.DataFrame(biased_coin_flips).hist(range=(-0.5,10.5), bins=11)<jupyter_output>col_0   2   3   4    5    6     7     8     9     10
row_0                                               
counts   3   8  63  244  863  2016  3038  2698  1067<jupyter_text>For continuous probability density functions, you use pdf() to check the probability density at a given x value. For discrete distributions like the binomial, use stats.distribution.pmf() (probability mass function) to check the mass (proportion of observations) at given number of successes k:<jupyter_code>stats.binom.pmf(k=6,        # Probability of k = 6 successes
                n=10,       # With 10 flips
                p=0.6)      # And success probability 0.5

stats.binom.pmf(k=9,        # Probability of k = 9 successes
                n=10,       # With 10 flips
                p=0.9)      # And success probability 0.9<jupyter_output><empty_output><jupyter_text>The Normal DistributionThe normal or Gaussian distribution is a continuous probability distribution characterized by a symmetric bell-shaped curve. A normal distribution is defined by its center (mean) and spread (standard deviation.). The bulk of the observations generated from a normal distribution lie near the mean, which lies at the exact center of the distribution: as a rule of thumb, about 68% of the data lies within 1 standard deviation of the mean, 95% lies within 2 standard deviations and 99.7% lies within 3 standard deviations.The normal distribution is perhaps the most important distribution in all of statistics. It turns out that many real world phenomena, like IQ test scores and human heights, roughly follow a normal distribution, so it is often used to model random variables. Many common statistical tests assume distributions are normal.The scipy nickname for the normal distribution is norm. Let's investigate the normal distribution:<jupyter_code>prob_under_minus1 = stats.norm.cdf(x= -1,  
                                loc = 0,               
                                scale= 1)     

prob_over_1 = 1 - stats.norm.cdf(x= 1,  
                                loc = 0,               
                                scale= 1) 

between_prob = 1-(prob_under_minus1+prob_over_1)

print(prob_under_minus1, prob_over_1, between_prob)<jupyter_output>0.158655253931 0.158655253931 0.682689492137<jupyter_text>The output shows that roughly 16% of the data generated by a normal distribution with mean 0 and standard deviation 1 is below -1, 16% is above 1 and 68% lies between -1 and 1, which agrees with the 68, 95, 99.7 rule. Let's plot the normal distribution and inspect areas we calculated:<jupyter_code># Plot normal distribution areas*

plt.fill_between(x=np.arange(-4,-1,0.01), 
                 y1= stats.norm.pdf(np.arange(-4,-1,0.01)) ,
                 facecolor='red',
                 alpha=0.35)

plt.fill_between(x=np.arange(1,4,0.01), 
                 y1= stats.norm.pdf(np.arange(1,4,0.01)) ,
                 facecolor='yellow',
                 alpha=0.35)

plt.fill_between(x=np.arange(-1,1,0.01), 
                 y1= stats.norm.pdf(np.arange(-1,1,0.01)) ,
                 facecolor='green',
                 alpha=0.35)

plt.text(x=-1.8, y=0.03, s= round(prob_under_minus1,3))
plt.text(x=-0.2, y=0.1, s= round(between_prob,3))
plt.text(x=1.4, y=0.03, s= round(prob_over_1,3))<jupyter_output><empty_output><jupyter_text>The plot above shows the bell shape of the normal distribution, the area below and above one standard deviation and the area within 1 standard deviation of the mean.Finding quantiles of the normal distribution is a common task when performing statistical tests. You can check normal distribution quantiles with stats.norm.ppf():<jupyter_code>print( stats.norm.ppf(q=0.025) ) # Find the quantile for the 2.5% cutoff

print( stats.norm.ppf(q=0.975) ) # Find the quantile for the 97.5% cutoff<jupyter_output>-1.95996398454
1.95996398454<jupyter_text>The quantile output above confirms that roughly 5% of the data lies more than 2 standard deviations from the mean.  The Uniform DistributionThe uniform distribution is a probability distribution where each value within a certain range is equally likely to occur and values outside of the range never occur. If we make a density plot of a uniform distribution, it appears flat because no value is any more likely (and hence has any more density) than another.Many useful functions for working with probability distributions in Python are contained in the scipy.stats library. Let's load in some libraries, generate some uniform data and plot a density curve:<jupyter_code>uniform_data = stats.uniform.rvs(size=10000,  # Generate 100000 numbers
                                 loc = 0,      # From 0 
                                 scale=10)     # To 10
pd.DataFrame(uniform_data).plot(kind="density",  # Plot the distribution
                               figsize=(9,9),
                               xlim=(-1,11))<jupyter_output><empty_output><jupyter_text>In the code above, we generated 10,000 data points from a uniform distribution spanning the range 0 to 10. In the density plot, we see that the density of our uniform data is essentially level meaning any given value has the same probability of occurring. The area under a probability density curve is always equal to 1.Probability distributions in scipy come with several useful functions for generating random data and extracting values of interest:-stats.distribution.rvs() generates random numbers from the specified distribution. The arguments to rvs() will vary depending on the type of distribution you're working with; in the case of the uniform distribution, we have to specify the starting and ending points and the size (number of random points to generate.).-stats.distribution.cdf() is used to determine the probability that an observation drawn from a distribution falls below a specified value (known as the cumulative distribution function.). In essence, cdf() gives you the area under the distribution's density curve to the left of a certain value on the x axis. For example, in the uniform distribution above, there is a 25% chance that an observation will be in the range 0 to 2.5 and a 75% chance it will fall in the range 2.5 to 10. We can confirm this with cdf():<jupyter_code>stats.uniform.cdf(x=2.5,         # Cutoff value (quantile) to check
                  loc=0,         # Distribution start
                  scale=10)      # Distribution end<jupyter_output><empty_output><jupyter_text>-stats.distribution.ppf() is the inverse of cdf(): it returns the x axis cutoff value (quantile) associated with a given probability. For instance, if we want to know the cutoff value for which we have a 40% chance of drawing an observation below that value, we can use ppf():<jupyter_code>stats.uniform.ppf(q=0.4,         # Probability cutoff
                  loc=0,         # Distribution start
                  scale=10)      # Distribution end<jupyter_output><empty_output><jupyter_text>-stats.distribution.pdf() gives you the probability density (height of the distribution) at a given x value. Since the uniform distribution is flat, all x values within its range will have the same probability density and x values outside the range have a probability density of 0:<jupyter_code>for x in range(-1,15,3):
    print("Density at x value " + str(x))
    print( stats.uniform.pdf(x, loc=0, scale=10) )<jupyter_output>Density at x value -1
0.0
Density at x value 2
0.1
Density at x value 5
0.1
Density at x value 8
0.1
Density at x value 11
0.0
Density at x value 14
0.0<jupyter_text>The Geometric and Exponential DistributionsThe geometric and exponential distributions model the time it takes for an event to occur. The geometric distribution is discrete and models the number of trials it takes to achieve a success in repeated experiments with a given probability of success. The exponential distribution is a continuous analog of the geometric distribution and models the amount of time you have to wait before an event occurs given a certain occurrence rate.The scipy nickname for the geometric distribution is "geom". Let's use the geom functions to model the number of trials it takes to get a success (heads) when flipping a fair coin:<jupyter_code>np.random.seed(12)

flips_till_heads = stats.geom.rvs(size=1000,  # Generate geometric data
                                  p=0.5)       # With success prob 0.5

# Print table of counts
print( pd.crosstab(index="counts", columns= flips_till_heads))

# Plot histogram
pd.DataFrame(flips_till_heads).hist(range=(-0.5,max(flips_till_heads)+0.5)
                                    , bins=max(flips_till_heads)+1)<jupyter_output>col_0    1    2    3   4   5   6   7   8   9   10  11  12
row_0                                                    
counts  485  251  123  76  35  17   4   3   1   2   2   1<jupyter_text>The distribution looks similar to what we'd expect: it is very likely to get a heads in 1 or 2 flips, while it is very unlikely for it to take more than 5 flips to get a heads. In the 1,000 trails we generated, the longest it took to get a heads was 13 flips.Let's use cdf() to check the probability of needing 6 flips or more to get a success:<jupyter_code>first_four = stats.geom.cdf(k=4,   # Prob of success in first 4 flips
                           p=0.4)

1 - first_four<jupyter_output><empty_output><jupyter_text>Use pmf() to check the probability of seeing a specific number of flips before a successes:<jupyter_code>stats.geom.pmf(k=2,   # Prob of needing exactly 2 flips to get first success
              p=0.5)<jupyter_output><empty_output><jupyter_text>The scipy name for the exponential distribution is "expon". Let's investigate the exponential distribution:<jupyter_code># Get the probability of waiting more than 1 time unit before a success

prob_1 = stats.expon.cdf(x=1,        
                         scale=1)  # Arrival rate

1 - prob_1<jupyter_output><empty_output><jupyter_text>Let's plot this exponential distribution to get an idea of its shape:<jupyter_code>plt.fill_between(x=np.arange(0,1,0.01), 
                 y1= stats.expon.pdf(np.arange(0,1,0.01)) ,
                 facecolor='blue',
                 alpha=0.35)

plt.fill_between(x=np.arange(1,7,0.01), 
                 y1= stats.expon.pdf(np.arange(1,7,0.01)) ,
                 facecolor='red',
                 alpha=0.35)


plt.text(x=0.3, y=0.2, s= round(prob_1,3))
plt.text(x=1.5, y=0.08, s= round(1 - prob_1,3))<jupyter_output><empty_output><jupyter_text>Similar to the geometric distribution, the exponential starts high and has a long tail that trails off to the right that contains rare cases where you have to wait much longer than average for an arrival.  The Poisson DistributionThe Poisson distribution models the probability of seeing a certain number of successes within a time interval, where the time it takes for the next success is modeled by an exponential distribution. The Poisson distribution can be used to model traffic, such as the number of arrivals a hospital can expect in a hour's time or the number of emails you'd expect to receive in a week.The scipy name for the Poisson distribution is "poisson". Let's generate and plot some data from a Poisson distribution with an arrival rate of 1 per time unit:<jupyter_code>np.random.seed(12)

arrival_rate_1 = stats.poisson.rvs(size=10000,  # Generate Poisson data
                                   mu=1 )       # Average arrival time 1

# Print table of counts
print( pd.crosstab(index="counts", columns= arrival_rate_1))

# Plot histogram
pd.DataFrame(arrival_rate_1).hist(range=(-0.5,max(arrival_rate_1)+0.5)
                                    , bins=max(arrival_rate_1)+1)<jupyter_output>col_0      0     1     2    3    4   5  6
row_0                                    
counts  3705  3666  1816  632  147  29  5<jupyter_text>The histogram shows that when arrivals are relatively infrequent, it is rare to see more than a couple of arrivals in each time period. When the arrival rate is high, it becomes increasingly rare to see a low number of arrivals and the distribution starts to look more symmetric:<jupyter_code>np.random.seed(12)

arrival_rate_10 = stats.poisson.rvs(size=10000,  # Generate Poisson data
                                   mu=10 )       # Average arrival time 10

# Print table of counts
print( pd.crosstab(index="counts", columns= arrival_rate_10))

# Plot histogram
pd.DataFrame(arrival_rate_10).hist(range=(-0.5,max(arrival_rate_10)+0.5)
                                    , bins=max(arrival_rate_10)+1)<jupyter_output>col_0   0   1   2   3    4    5    6    7     8     9  ...   15   16   17  18  \
row_0                                                  ...                      
counts   1   2  17  56  157  445  607  979  1088  1228 ...  358  198  122  77   

col_0   19  20  21  22  23  24  
row_0                           
counts  42  17   8   2   3   1  

[1 rows x 25 columns]<jupyter_text>As with other discrete probability distributions, we can use cdf() to check the probability of achieving more or less than a certain number of successes and pmf() to check the probability of obtaining a specific number of successes:<jupyter_code>stats.poisson.cdf(k=5,     # Check the probability of 5 arrivals or less
                  mu=10)   # With arrival rate 10
stats.poisson.pmf(k=10,     # Check the prob f exactly 10 arrivals
                  mu=10)    # With arrival rate 10<jupyter_output><empty_output><jupyter_text>Use of Seeds in generating pseudorandom numbers.seed() method sets the integer starting value used in generating random numbers. Call this function before calling any other random module function. The below code is an example on how the seed function helps in generating the pseudorandom numbers.<jupyter_code>import numpy as np

np.random.seed( 10 )
print ("Random number with seed 10 : ", np.random.random())
 
# It will generate same random number
np.random.seed( 10 )
print ("Random number with seed 10 : ", np.random.random())
 
# It will generate same random number
np.random.seed( 10 )
print ("Random number with seed 10 : ", np.random.random())<jupyter_output>Random number with seed 10 :  0.771320643266746
Random number with seed 10 :  0.771320643266746
Random number with seed 10 :  0.771320643266746
{'fraction_non_alphanumeric': 0.06376869994459276, 'fraction_numerical': 0.04130358132272201, 'mean_word_length': 3.2054649438678244, 'pattern_counts': {'":': 0, '<': 86, '<?xml version=': 0, '>': 86, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 34, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '611250', 'license': 'Apache-2.0', 'n_tokens_mistral': 5973, 'n_tokens_neox': 5361, 'n_words': 2416, 'path': 'Numpy_Random_Assignment.ipynb', 'repo_name': 'cdigap/ProgDA_Assignment', 'size': '19853'}