<jupyter_start><jupyter_text>Studies Notes  Lesson 2 Concepts in Experiment DesignThese are some personal studies notes about Udacity course of Data Science. They don´t represent the content of the course, and can present a lot of inconsistencies. It is only to follow the thinking line in order to create useful functions for future projects.--- Between-subjects design - 1 participant = 1 condition- A/B test - compare performance between A and B Within-subjects design - 1 participant = n conditions (like a degustation)- only one rate: personal inclination + a good formula (a lot of variance - someone have the inclination to give allways good rates - e.g. over 8)- other rates: individual full preferences $\rightarrow$ reduces variance on data- you can also have +1 conditions (e.g. 1 control group + 2 experiment groups)- and... you can collect data in different ways, compare their meaning to turn your model more robust  Types of sampling Strategy1: sample over the population - considers that it is homogeneous- **Simple Random Sampling** $\rightarrow$  all members = equal chance of selection Strategy2: break it in subgoups - e.g. in one district, rural zone people have one kind of life, and people living in city zone have another colpletely kind of life (probably different prefferences)- **Stratified Random Sampling** $\rightarrow$ each subgroup, have an equal number of relative to its population- the variability now was not left to chance!  Measuring outcomesSepparate: goal x measure of successGoal: "improve the recommendations $\rightarrow$ get better performance"Evaluation metrics: - watch time- ranking- number of searches (they are **concrete**, measurable)Take care that your metrics are:- alligned with the goals that we set- changing of a metric is NOT the main poin of the study!- implications of this change is what is important- goal $\rightarrow$ center purpose of your study  Creating metricsYou have one, or more hypothesis, as "inserting a picture, increases the chance of a purchase"Your so your **purchase sequence** $\rightarrow$ **User Funnel** (steps or path, finishing to destination state)And you can have **alternative** User Funnels! (Unit of Diversion - observational unit to split users)1. **Evaluation Metrics** $\rightarrow$ features that provide $\rightarrow$ Objective Measurements - measure the success of a experimental manipulation- purchases x view, etc..2. **Invariant Metrics** $\rightarrow$ objectively check the equivalence of the groups  Controlling variablesOur **goal**: hability to say $\rightarrow$  changes to variable 1 $\rightarrow$  cause changes in variable 2 - I need to control the effects of other variables!  Analysis TrapsConfounding Variable $\rightarrow$ hidden variable(s) that modulate both the (apparent) **"cause"** and **"consequence"**  Checking validityWhat they say = what they mean **Experimental Validity** is:     - degree of experiments $\rightarrow$ accomplishes the stated conclusions The 3 ways for a experiment to be valid**Construct Validity**: - objective metric results $\rightarrow$ goals of the study (degree) **Internal Validity**: - claims of causality $\rightarrow$ supported by the analysis (ensure) **External Validity**: - experiments results $\rightarrow$ generalizes to cases outside the experiment (degree)  Checking bias*Biologically*:     - bias = quick efficient decisionsIn an **Experiment**:     - represents systematic errors $\rightarrow$ bad interpretations of the results**Sampling Bias** $\leftarrow$ the **Sample** don't represent your **Population**- geography, subgroups...- self-selection ("OK, we, all the motoboys present in this bar voluntarize to make the test!") - sampling bias- error on randomization- novelty bias- order bias (primacy bias, recency bias)  Statistical Significance - SMART- Specific- Measurable- Achievable- Relevant- Timely fashion --- Lesson 3 Statistical Considerations into testing First Notebook - L2 - Statistical SignificanceI have a randomizer cookie $\rightarrow$ I want to know if it is really randomizing well (50%-50%)- perhaps you have more than one **Webservice**, and one is providing only one version for their clients- perhaps my **randomizer algorithm** is not working so well In Udacity text:"Let's say that we've collected data for a web-based experiment. In the experiment, we're testing the change in layout of a product information page to see if this affects the proportion of people who click on a button to go to the download page. This experiment has been designed to have a cookie-based diversion, and we record two things from each user: which page version they received, and whether or not they accessed the download page during the data recording period. (We aren't keeping track of any other factors in this example, such as number of pageviews, or time between accessing the page and making the download, that might be of further interest.)" TaskOur objective in this notebook is to perform a statistical test on both recorded metrics to see if there is a statistical difference between the two groups.<jupyter_code>#import libraries
import numpy as np
import pandas as pd
import scipy.stats as stats
import udacourse3 #my library

#graphs
#import matplotlib.pyplot as plt
#import matplotlib.patches as mpatches
#import matplotlib.style as mstyles
#import matplotlib.pyplot as mpyplots
#from matplotlib.pyplot import hist
#from matplotlib.figure import Figure
#import seaborn as sns

#from statsmodels.stats import proportion as proptests
from time import time
#% matplotlib inline<jupyter_output><empty_output><jupyter_text>Import data- this data emulates a server distribution of two different websites, 1 is Group A and 2 is Group B- the idea is to check if they were fair distributed or not<jupyter_code>df = udacourse3.fn_read_data('data/statistical_significance_data.csv')
df.head(2)<jupyter_output><empty_output><jupyter_text>I. Check de **Invariant Metric**:- if they are imbalanced $\rightarrow$ subgroups may exist! $\rightarrow$ data is **biased**!- in this case, one of your webservices could be providing only one of the options- for 2-sided tests $\rightarrow$ test the **proportion** of visitors for each group in our case The number of accesses for g1 and g2 is similar?It is a **invariant metric** (a ground prerequisite for our inference):- if the distribution is not 50-50, this will be a **unbalanced group**- so verify with care for **bias**, as the existence of subgroups on our population (server 1 that caused the problem is placed on an area that is more accessed by older people, etc..)- in the worst case, you will need to **remake** your experiment! Using 2-sided hypothesis testIn this case:- don´t matter Who is the **Control** and who is the **Experiment** groupI am using Standard Deviation for **Binomial Distribution**. I can find some material [here](https://www.statisticshowto.com/probability-and-statistics/standard-deviation/) and the formula is:![stdeviationBinomial](graphs/stdevBinomial.gif) In Udacity text:"In the dataset, the 'condition' column takes a 0 for the control group, and 1 for the experimental group. The 'click' column takes a values of 0 for no click, and 1 for a click. Checking the Invariant Metric In Udacity text:"First of all, we should check that the number of visitors assigned to each group is similar. It's important to check the invariant metrics as a prerequisite so that our inferences on the evaluation metrics are founded on solid ground. If we find that the two groups are imbalanced on the invariant metric, then this will require us to look carefully at how the visitors were split so that any sources of bias are accounted for. It's possible that a statistically significant difference in an invariant metric will require us to revise random assignment procedures and re-do data collection.In this case, we want to do a two-sided hypothesis test on the proportion of visitors assigned to one of our conditions. Choosing the control or the experimental condition doesn't matter: you'll get the same result either way. Feel free to use whatever method you'd like: we'll highlight two main avenues below.You can check your results by completing the following the workspace and the solution on the following page. You could also try using multiple approaches and seeing if they come up with similar outcomes!"  Analytic approacha. **exact Binomial Distribution** $\rightarrow$ **p-value** of the test orb. **Normal Distribution** approximation (it works due to **Central Limit Theorem**) $\leftarrow$ for **large** sample size**b** is the more usual approachand then... for a precise **p-value**:- perform a **Continuity Correction**, adding or subtraction 0.5 from the total count before computing the area underneath de curve  In Udacity text:"If you want to take an analytic approach, you could use the exact binomial distribution to compute a p-value for the test. The more usual approach, however, is to use the normal distribution approximation. Recall that this is possible thanks to our large sample size and the central limit theorem. To get a precise p-value, you should also perform a continuity correction, either adding or subtracting 0.5 to the total count before computing the area underneath the curve. (e.g. If we had 415 / 850 assigned to the control group, then the normal approximation would take the area to the left of  $(415 + 0.5) / 850 = 0.489$  and to the right of  $(435 − 0.5 ) / 850 = 0.511$ .)" function `fn_invariant_analytic` created Standard Deviation: Khan Academy [here](https://pt.khanacademy.org/math/statistics-probability/summarizing-quantitative-data/variance-standard-deviation-sample/a/population-and-sample-standard-deviation-review)    - in this case, we are measuring for all the **Population**, so we will divide for **n** (NOT n-1, as in Sample)It is a Standard Deviation for a **Binomial Distribution**, so the formula is [here](https://www.statisticshowto.com/probability-and-statistics/standard-deviation/)![Binomial](graphs/sdBinomial.gif) z Score: ![z score](graphs/zScore.jfif)Description [here](https://www.simplypsychology.org/z-score.html)"A z-score describes the position of a raw score in terms of its distance from the mean, when measured in standard deviation units. The z-score is positive if the value lies above the mean, and negative if it lies below the mean.""It is also known as a standard score, because it allows comparison of scores on different kinds of variables by standardizing the distribution. A standard normal distribution (SND) is a normally shaped distribution with a mean of 0 and a standard deviation (SD) of 1." Why are z-scores important?"It is useful to standardized the values (raw scores) of a normal distribution by converting them into z-scores because:(a) it allows researchers to calculate the probability of a score occurring within a standard normal distribution;(b) and enables us to compare two scores that are from different samples (which may have different means and standard deviations)." "A normal **Cumulative Distribution Function (CDF)** will return the percentage of the **Normal Distribution Function** that is less than or equal to the random variable specified." (Asked Google)The central point is at $x=0$, and the area must be $0.5$ at this point: `stats.norm.cdf(0)` "In **Null Hypothesis** significance testing, the **p-value** is the probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is **correct**" [wiki](https://en.wikipedia.org/wiki/P-value)"A very **small** p-value means that such an extreme observed outcome would be very unlikely under the null hypothesis"As it is a **2-tails** experiment, so the value is multiplied by $2$:- 61.3% of **explanation** is not sufficient to **reject** the null hypothesis!<jupyter_code>udacourse3.fn_invariant_analytic(df, 
                                 verbose=True)<jupyter_output><empty_output><jupyter_text>--- Simulation approach In Udacity text:"If you want to take a simulation-based approach, you can simulate the number of visitors that would be assigned to each group for the number of total observations, assuming that we have an expected 50/50 split. Do this many times (200 000 repetitions should provide a good speed-variability balance in this case) and then see in how many simulated cases we get as extreme or more extreme a deviation from 50/50 that we actually observed. Don't forget that, since we have a two-sided test, an extreme case also includes values on the opposite side of 50/50. (e.g. Since simulated outcomes of .48 and lower are considered as being more extreme than an actual observation of 0.48, so too will simulated outcomes of .52 and higher.) The proportion of flagged simulation outcomes gives us a p-value on which to assess our observed proportion. We hope to see a larger p-value, insufficient evidence to reject the null hypothesis." Function `fn_invariant_simulated` created! Random Binomial: `np.random.binomial(n, p, 1000)`    - result of flipping a coin $10$ times, tested $1000$ times `sum(samples >= (n_obs - n_control))` $\leftarrow$ samples that extrapolates the boundaries<jupyter_code>udacourse3.fn_invariant_simulated(df, 
                                  verbose=True)<jupyter_output><empty_output><jupyter_text>Checking the Evaluation MetricNow, we want to know if **clicks** changes with the choice of the server with our new or old webpage. In Udacity text:"After performing our checks on the invariant metric, we can move on to performing a hypothesis test on the evaluation metric: the click-through rate. In this case, we want to see that the experimental group has a significantly larger click-through rate than the control group, a one-tailed test.There's a few analytic approaches possible here, but you'll probably make use of the normal approximation again in these cases. In addition to the pooled click-through rate, you'll need a pooled standard deviation in order to compute a z-score. While there is a continuity correction possible in this case as well, it's much more conservative than the p-value that a simulation will usually imply. Computing the z-score and resulting p-value without a continuity correction should be closer to the simulation's outcomes, though slightly more optimistic about there being a statistical difference between groups."  Analytic Approachfunction `fn_variant_analytic`created!<jupyter_code>udacourse3.fn_variant_analytic(df, 
                               verbose=True)<jupyter_output><empty_output><jupyter_text>Simulation Approach`np.random.binomial(n, p, size)` at numpy documentation [here](https://numpy.org/doc/stable/reference/random/generated/numpy.random.binomial.html)- n $\rightarrow$ number of trials (e.g. 6 coin flips)- p $\rightarrow$ probability of success (e.g. 0.5 on a fair coin)- size $\rightarrow$ shape of the output (e.g. 1 number, 2x2 Array, 3x4x2 Array) in Udacity text"The simulation approach for this metric isn't too different from the approach for the invariant metric. You'll need the overall click-through rate as the common proportion to draw simulated values from for each group. You may also want to perform more simulations since there's higher variance for this test."<jupyter_code>n=6
p=0.5
size=(3, 4)
np.random.binomial(n, p, size)<jupyter_output><empty_output><jupyter_text>function `fn_variant_simulated` created!<jupyter_code>udacourse3.fn_variant_simulated(df, 
                                verbose=True)<jupyter_output><empty_output><jupyter_text>--- Practical SignificanceYou have **secondary effects** on doing some kind of promotional event:- costs of coupons- reveneue loss- effort/cost to create/maintain your new website In Udacity text:"Even if an experiment result shows a statistically significant difference in an evaluation metric between control and experimental groups, that does not necessarily mean that the experiment was a success. If there are any costs associated with deploying a change, those costs might outweigh the benefits expected based on the experiment results. **Practical significance** refers to the level of effect that you need to observe in order for the experiment to be called a true success and implemented in truth. Not all experiments imply a practical significance boundary, but it's an important factor in the interpretation of outcomes where it is relevant.If you consider the confidence interval for an evaluation metric statistic against the null baseline and practical significance bound, there are a few cases that can come about." Confidence interval is fully in practical significance region- $m_0 \rightarrow$ **null** statistic value- $d_{min} \rightarrow$ **practical significance** bound,- the blue line $\rightarrow$ **confidence interval** for the observed statistic.  In Udacity text:"We assume that we're looking for a positive change, ignoring the negative equivalent for $d_{min}$.![significance1](graphs/sig1.png)If the confidence interval for the statistic does not include the null or the practical significance level, then the experimental manipulation can be concluded to have a statistically and practically significant effect. It is clearest in this case that the manipulation should be implemented as a success. Confidence interval completely excludes any part of practical significance region![significance2](graphs/sig2.png)If the confidence interval does not include any values that would be considered practically significant, this is a clear case for us to not implement the experimental change. This includes the case where the metric is statistically significant, but whose interval does not extend past the practical significance bounds. With such a low chance of practical significance being achieved on the metric, we should be wary of implementing the change. Confidence interval includes points both inside and outside practical significance bounds![significance3](graphs/sig3.png)This leaves the trickiest cases to consider, where the confidence interval straddles the practical significance bound. In each of these cases, there is an uncertain possibility of practical significance being achieved. In an ideal world, you would be able to collect more data to reduce our uncertainty, reducing the scenario to one of the previous cases. Outside of this, you'll need to consider the risks carefully in order to make a recommendation on whether or not to follow through with a tested change. Your analysis might also reveal subsets of the population or aspects of the manipulation that do work, in order to refine further studies or experiments."--- Second Notebook - L5 - Experiment Size in Udacity text:"We can use the knowledge of our desired practical significance boundary to plan out our experiment. By knowing how many observations we need in order to detect our desired effect to our desired level of reliability, we can see how long we would need to run our experiment and whether or not it is feasible." QuestionNow where we have a baseline click-through rate of **10%** and want to see a manipulation increase this baseline to **12%**. How many observations would we need in each group in order to detect this change with power  **1−𝛽=.80**  (i.e. detect the **2%** absolute increase **80%** of the time), at a Type I error rate of  **𝛼=.05** ?"  Basic principle of StatisticsYou need to estimate your Sample Size **before** the starting of the experiment!*G* Power* (software) $\rightarrow$ Statistical Power, Accuracy- sample size- effect size- significant level $\rightarrow$ p-value smaller, power higher*ANOVA* $\rightarrow$ Student t-Test*Factors influencing power**SPSS*  By Trial & Error in Udacity text:"One way we could solve this is through trial and error. Every sample size will have a level of power associated with it; testing multiple sample sizes will gradually allow us to narrow down the minimum sample size required to obtain our desired power level. This isn't a particularly efficient method, but it can provide an intuition for how experiment sizing works." function from Udacourse.py:<jupyter_code>#gra_style='dark_background',
#mstyles.use(gra_style)    
#fig_zgen = mpyplots.figure() #creating the object
#axis_zgen = fig_zgen.add_axes([0,0,1,1]) #creating an axis
#axis_zgen.bar(language, summation, color=bar_colors) #generating the barplot
#fig_zgen.autofmt_xdate() #later use autoformat for auto-rotating axis legends
#legends, commands = fn_create_patches(bar_colors=bar_colors, verbose=verbose) #creating patches for legends
#blue_patch = mpatches.Patch(color='b', label='web languages')
#for command in commands:
#    exec(command)
#axis_zgen.set_title(txt_title, fontsize=14)
#axis_zgen.legend(handles=eval(legends))<jupyter_output><empty_output><jupyter_text>Percentage Point Function - PPF"The method `norm.ppf()` takes a percentage and returns a standard deviation multiplier for what value that percentage occurs at." (google Search) Numpy Linspace Function"Return evenly spaced numbers over a specified interval." [numpy documentation](https://numpy.org/doc/stable/reference/generated/numpy.linspace.html) Probability Distribution Function PDF"In probability theory and statistics, a probability distribution is the mathematical function that gives the probabilities of occurrence of different possible outcomes for an experiment. It is a mathematical description of a **random phenomenon** in terms of its sample space and the probabilities of events (subsets of the sample space)." [wikipedia](https://en.wikipedia.org/wiki/Probability_distribution) **Variance** for the difference in proportions under the null hypothesis for our two groupsRemember the formulae for **Standard Deviation**:https://www.statisticshowto.com/probability-and-statistics/standard-deviation/)![Standard Deviation](graphs/StDev.png) Actually we are using the formula for Standard Deviation of a **Population**: ([here](https://stackoverflow.com/questions/47570903/confidence-interval-for-the-difference-between-two-proportions-in-python) some intuitions about)- the base probability is given by **p_null**- the variance of the difference distribution is the sum of the variances for the individual distributions- each group is assigned n observations.remembering that **t-Student** is:$\frac{signal}{noise} = \frac{\sqrt{\bar{x}_1-\bar{x}_2}}{\sqrt{\frac{S_1^{2}}{n_1} + \frac{S_2^{2}}{n_2}}}$<jupyter_code>p_null = 0.10
num= 1000

(p_null * (1-p_null) + p_null * (1-p_null)) / num
(0.1 * 0.9 + 0.1 * 0.9) / 1000<jupyter_output><empty_output><jupyter_text>Function `fn_power` created!Function `fn_plot` created!<jupyter_code>udacourse3.fn_power(p_null=0.10, 
                    p_alt=0.12, 
                    num=1000, 
                    plot=True,
                    verbose=True)<jupyter_output><empty_output><jupyter_text>In Udacity text"Fill in the **power()** function below following these steps:- Under the null hypothesis, we should have a critical value for which the Type I error rate is at our desired alpha level.- **se_null**: Compute the standard deviation for the difference in proportions under the null hypothesis for our two groups. The base probability is given by p_null. Remember that the variance of the difference distribution is the sum of the variances for the individual distributions, and that each group is assigned n observations.- **null_dist**: To assist in re-use, this should be a scipy norm object. Specify the center and standard deviation of the normal distribution using the "loc" and "scale" arguments, respectively.- **p_crit**: Compute the critical value of the distribution that would cause us to reject the null hypothesis. One of the methods of the null_dist object will help you obtain this value (passing in some function of our desired error rate alpha). The power is the proportion of the distribution under the alternative hypothesis that is past that previously-obtained critical value.- **se_alt**: Now it's time to make computations in the other direction. This will be standard deviation of differences under the desired detectable difference. Note that the individual distributions will have different variances now: one with p_null probability of success, and the other with p_alt probability of success.- **alt_dist**: This will be a scipy norm object like above. Be careful of the "loc" argument in this one. The way the power function is set up, it expects p_alt to be greater than p_null, for a positive difference.- **beta**: Beta is the probability of a Type-II error, or the probability of failing to reject the null for a particular non-null state. That means you should make use of alt_dist and p_crit here!<jupyter_code>assert np.isclose(udacourse3.fn_power(.1, .12, 1000, plot = False), 0.4412, atol = 1e-4)
assert np.isclose(udacourse3.fn_power(.1, .12, 3000, plot = False), 0.8157, atol = 1e-4)
assert np.isclose(udacourse3.fn_power(.1, .12, 5000, plot = False), 0.9474, atol = 1e-4)
print('You should see this message if all the assertions passed!')<jupyter_output><empty_output><jupyter_text>By Analytic Solution Udacity text:"Now that we've got some intuition for power by using trial and error, we can now approach a closed-form solution for computing a minimum experiment size. The key point to notice is that, for an $\alpha$ and $\beta$ both < .5, the critical value for determining statistical significance will fall between our null click-through rate and our alternative, desired click-through rate. So, the difference between $p_0$ and $p_1$ can be subdivided into the distance from $p_0$ to the critical value $p^*$ and the distance from $p^*$ to $p_1$.![fig 1](graphs/ExpSize_Power.png)Those subdivisions can be expressed in terms of the standard error and the z-scores:$$p^* - p_0 = z_{1-\alpha} SE_{0},$$$$p_1 - p^* = -z_{\beta} SE_{1};$$$$p_1 - p_0 = z_{1-\alpha} SE_{0} - z_{\beta} SE_{1}$$In turn, the standard errors can be expressed in terms of the standard deviations of the distributions, divided by the square root of the number of samples in each group:$$SE_{0} = \frac{s_{0}}{\sqrt{n}},$$$$SE_{1} = \frac{s_{1}}{\sqrt{n}}$$Substituting these values in and solving for $n$ will give us a formula for computing a minimum sample size to detect a specified difference, at the desired level of power:$$n = \lceil \big(\frac{z_{\alpha} s_{0} - z_{\beta} s_{1}}{p_1 - p_0}\big)^2 \rceil$$where $\lceil ... \rceil$ represents the ceiling function, rounding up decimal values to the next-higher integer. Implement the necessary variables in the function below, and test them with the cells that follow." Function `fn_experiment_size`created!<jupyter_code>udacourse3.fn_experiment_size(p_null=0.1, 
                              p_alt=0.12,
                              verbose=True)
assert np.isclose(udacourse3.fn_experiment_size(.1, .12), 2863)
print('You should see this message if the assertion passed!')<jupyter_output><empty_output><jupyter_text>Notes on Interpretation Udacity text"The example explored above is a one-tailed test, with the alternative value greater than the null. The power computations performed in the first part will _not_ work if the alternative proportion is greater than the null, e.g. detecting a proportion parameter of 0.88 against a null of 0.9. You might want to try to rewrite the code to handle that case! The same issue should not show up for the second approach, where we directly compute the sample size.If you find that you need to do a two-tailed test, you should pay attention to two main things. First of all, the "alpha" parameter needs to account for the fact that the rejection region is divided into two areas. Secondly, you should perform the computation based on the worst-case scenario, the alternative case with the highest variability. Since, for the binomial, variance is highest when $p = .5$, decreasing as $p$ approaches 0 or 1, you should choose the alternative value that is closest to .5 as your reference when computing the necessary sample size.Note as well that the above methods only perform sizing for _statistical significance_, and do not take into account _practical significance_. One thing that should be realized is that if the true size of the experimental effect is the same as the desired practical significance level, then it's a coin flip whether the mean will be above or below the practical significance bound. This also doesn't even consider how a confidence interval might interact with that bound. In a way, experiment sizing is a way of checking about whether or not you'll be able to get what you _want_ from running an experiment, rather than checking if you'll get what you _need_."  Alternative Approaches Udacity text"There are also tools and Python packages that can also help with sample sizing decisions, so you don't need to solve for every case on your own. The sample size calculator [here](http://www.evanmiller.org/ab-testing/sample-size.html) is applicable for proportions, and provides the same results as the methods explored above. (Note that the calculator assumes a two-tailed test, however.) Python package "statsmodels" has a number of functions in its [`power` module](https://www.statsmodels.org/stable/stats.htmlpower-and-sample-size-calculations) that perform power and sample size calculations. Unlike previously shown methods, differences between null and alternative are parameterized as an effect size (standardized difference between group means divided by the standard deviation). Thus, we can use these functions for more than just tests of proportions. If we want to do the same tests as before, the [`proportion_effectsize`](http://www.statsmodels.org/stable/generated/statsmodels.stats.proportion.proportion_effectsize.html) function computes [Cohen's h](https://en.wikipedia.org/wiki/Cohen%27s_h) as a measure of effect size. As a result, the output of the statsmodel functions will be different from the result expected above. This shouldn't be a major concern since in most cases, you're not going to be stopping based on an exact number of observations. You'll just use the value to make general design decisions."<jupyter_code>from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# leave out the "nobs" parameter to solve for it
size = NormalIndPower().solve_power(effect_size=proportion_effectsize(0.12, 0.1), 
                                    alpha=0.05, 
                                    power=0.8,
                                    alternative='larger')
print('experiment size: {:.1f}'.format(size))<jupyter_output><empty_output><jupyter_text>--- Using Dummy TestsDesign experiments that deals with **changes** and **manipulations**Is also known as an **AA-Test** $\right$ you create an experiment that compares a group with a similar group, by the same conditionsIt seems a bit strange, but making it you can:- assert that eveything is OK in your environment **before** doing a real test  - as your distribution token is working fairly    - it the paths are OK  - know better the steps that your custommer will follow during the test- know some data, as variability- you will be able to predict how long a experiment will need to run in order to collect your needed dataThere is some problems too:- takes time, that you could use in a **real test** In Udacity text:"When it comes to designing an experiment, it might be useful to run a dummy test as a predecessor to or as part of that process. In a dummy test, you will implement the same steps that you would in an actual experiment to assign the experimental units into groups. However, the experimental manipulation won't actually be implemented, and the groups will be treated equivalently.There are multiple reasons to run a dummy test. First, a dummy test can expose if there are any errors in the randomization or assignment procedures. A short dummy test can be worth the investment if an invariant metric is found to have a statistically significant difference, or if some other systematic bias is identified, because it can help avoid larger problems down the line. A second reason to run a dummy test is to collect data on metrics' behaviors. If historic data is not enough to predict the outcome of recorded metrics or allow for experiment duration to be computed, then a dummy test can be useful for getting baselines.Of course, performing a dummy test requires an investment of resources, the most important of which is time. If time is of the essence, then you may need to just go ahead with the experiment, keeping an eye on invariant metrics for any trouble. An alternative approach is to perform a hybrid test. In the A/B testing paradigm, this can take the form of an A/A/B test. That is, we split the data into three groups: two control and one experimental. A comparison between control groups can be used to learn about null-environment properties before making inferences on the effect of the experimental manipulation."--- Third Notebook - L8 - Non-Parametric Tests In Udacity text:"Up until now, you've been using standard hypothesis tests on means of normal distributions to design and analyze experiments. However, it's possible that you will encounter scenarios where you can't rely on only standard tests. This might be due to uncertainty about the true variability of a metric's distribution, a lack of data to assume normality, or wanting to do inference on a statistic that lacks a standard test. It's useful to know about some **non-parametric tests** not just as a workaround for cases like this, but also as a second check on your experimental results. The main benefit of a non-parametric test is that they don't rely on many assumptions of the underlying population, and so can be used in a wider range of circumstances compared to standard tests. In this notebook, you'll cover two non-parametric approaches that use resampling of the data to make inferences about distributions and differences."  Bootstrapping In Udacity text:"Bootstrapping is used to estimate sampling distributions by using the actually collected data to generate new samples that could have been hypothetically collected. In a standard bootstrap, a bootstrapped sample means drawing points from the original data _with replacement_ until we get as many points as there were in the original data. Essentially, we're treating the original data as the population: without making assumptions about the original population distribution, using the original data as a model of the population is the best that we can do.Taking a lot of bootstrapped samples allows us to estimate the sampling distribution for various statistics on our original data. For example, let's say that we wanted to create a 95% confidence interval for the 90th percentile from a dataset of 5000 data points. (Perhaps we're looking at website load times and want to reduce the worst cases.) Bootstrapping makes this easy to estimate. First of all, we take a bootstrap sample (i.e. draw 5000 points with replacement from the original data) and record the 90th percentile and repeat this a large number of times, let's say 100 000. From this bunch of bootstrapped 90th percentile estimates, we form our confidence interval by finding the values that capture the central 95% of the estimates (cutting off 2.5% on each tail). Implement this operation in the cells below, using the following steps:- Initialize some useful variables by storing the number of data points in `n_points` and setting up an empty list for the bootstrapped quantile values in `sample_qs`.- Create a loop for each trial where:  - First generate a bootstrap sample by sampling from our data with replacement. ([`random.choice`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.choice.html) will be useful here.)    - Then, compute the `q`th quantile of the sample and add it to the `sample_qs` list. If you're using numpy v0.15 or later, you can use the [`quantile`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.quantile.html) function to get the quantile directly with `q`; on v0.14 or earlier, you'll need to put `q` in terms of a percentile and use [`percentile`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.percentile.html) instead.  - After gathering the bootstrapped quantiles, find the limits that capture the central `c` proportion of quantiles to form the estimated confidence interval." Quantile"In statistics and probability, quantiles are cut points dividing the range of a probability distribution into continuous intervals with equal probabilities, or dividing the observations in a sample in the same way" [wiki](https://en.wikipedia.org/wiki/Quantile) Function `fn_quantile` created!<jupyter_code>data = udacourse3.fn_read_data('data/bootstrapping_data.csv')
data.head(5)
lims = udacourse3.fn_quantile_ci(data=data['time'], 
                                 q=0.9,
                                 plot=True,
                                 verbose=True)
print(lims)<jupyter_output><empty_output><jupyter_text>Bootstrapping Notes In Udacity notes"Confidence intervals coming from the bootstrap procedure will be optimistic compared to the true state of the world. This is because there will be things that we don't know about the real world that we can't account for, due to not having a parametric model of the world's state. Consider the extreme case of trying to understand the distribution of the maximum value: our confidence interval would never be able to include any value greater than the largest observed value and it makes no sense to have any lower bound below the maximum observation. Intuitively, however, there's a pretty clear possibility for there to be unobserved values that are larger than the one we've observed, especially for skewed data like shown in the example.This doesn't override the bootstrap method's advantages, however. The bootstrap procedure is fairly simple and straightforward. Since you don't make assumptions about the distribution of data, it can be applicable for any case you encounter. The results should also be fairly comparable to standard tests. But it does take computational effort, and its output does depend on the data put in. For reference, for the 95% CI on the 90th percentile example explored above, the inferred interval would only capture about 83% of 90th percentiles from the original generating distribution. But a more intricate procedure using a binomial assumption to index on the observed data only does about one percentage point better (84%). And both of these depend on the specific data generated: a different set of 5000 points will produce different intervals, with different accuracies."Binomial solution for percentile CIs reference: [1](https://www-users.york.ac.uk/~mb55/intro/cicent.htm), [2](https://stats.stackexchange.com/questions/99829/how-to-obtain-a-confidence-interval-for-a-percentile)  Permutation Tests In Udacity notes:"The permutation test is a resampling-type test used to compare the values on an outcome variable between two or more groups. In the case of the permutation test, resampling is done on the group labels. The idea here is that, under the null hypothesis, the outcome distribution should be the same for all groups, whether control or experimental. Thus, we can emulate the null by taking all of the data values as a single large group. Applying labels randomly to the data points (while maintaining the original group membership ratios) gives us one simulated outcome from the null.The rest follows similar to the sampling approach to a standard hypothesis test, except that we haven't specified a reference distribution to sample from – we're sampling directly from the data we've collected. After applying the labels randomly to all the data and recording the outcome statistic many times, we compare our actual, observed statistic against the simulated statistics. A p-value is obtained by seeing how many simulated statistic values are as or more extreme as the one actually observed, and a conclusion is then drawn."Try implementing a permutation test in the cells below to test if the 90th percentile of times is staistically significantly smaller for the experimental group, as compared to the control group: Task- Initialize an empty list to store the difference in sample quantiles as `sample_diffs`.- Create a loop for each trial where:  - First generate a permutation sample by randomly shuffling the data point labels. ([`random.permutation`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.permutation.html) will be useful here.)  - Then, compute the `q`th quantile of the data points that have been assigned to each group based on the permuted labels. Append the difference in quantiles to the `sample_diffs` list.- After gathering the quantile differences for permuted samples, compute the observed difference for the actual data. Then, compute a p-value from the number of permuted sample differences that are less than or greater than the observed difference, depending on the desired alternative hypothesis.<jupyter_code>data = udacourse3.fn_read_data('data/permutation_data.csv')
data.head(5)<jupyter_output><empty_output><jupyter_text>Function `fn_quantile_permutation_test` created!<jupyter_code># Just how different are the two distributions' 90th percentiles?
print(np.percentile(data[data['condition'] == 0]['time'], 90),
      np.percentile(data[data['condition'] == 1]['time'], 90))
udacourse3.fn_quantile_permutation_test(x=data['time'], 
                                        y=data['condition'], 
                                        q=0.9,
                                        alternative='less',
                                        plot=True,
                                        verbose=True)<jupyter_output><empty_output><jupyter_text>--- Forth Notebook - L 10 - More Non-Parametric Tests In Udacity notes:"Up until now, you've been using standard hypothesis tests on means of normal distributions to design and analyze experiments. However, it's possible that you might encounter scenarios where you can't rely on only standard tests. This might be due to uncertainty about the true variability of a metric's distribution, a lack of data to assume normality, or wanting to do inference on a statistic that lacks a standard test. It's useful to know about some **non-parametric tests** not just as a workaround for cases like this, but also as a second check on your experimental results."  Rank-Sum Test (Mann-Whitney) In Udacity notes:"The rank-sum test is fairly different from the two previous approaches. There's no resamplng involved; the test is performed only on the data present. The rank-sum test, also known as the Mann-Whitney U test, is not a test of any particular statistic, like the mean or median. Instead, it's a test of distributions: let's say we draw one value at random from the populations behind each group. The null hypothesis says that there's an equal chance that the larger value is from the first group as the second group; the alternative hypothesis says that there's an unequal chance, which can be specified as one- or two-tailed.In order to test this hypothesis, we should look at the data we've collected and see in how many cases values from one group win compared to values in the second. That is, for each data point in the first group, we count how many values in the second group that are smaller than it. (If both values are equal, we count that as a tie, worth +0.5 to the tally.) This number of wins for the first group gives us a value $U$.It turns out that $U$ is approximately normally-distributed, given a large enough sample size. If we have $n_1$ data points in the first group and $n_2$ points in the second, then we have a total of $n_1 n_2$ matchups and an equivalent number of victory points to hand out. Under the null hypothesis, we should expect the number of wins to be evenly distributed between groups, and so the expected wins are $\mu_U = \frac{n_1 n_2}{2}$. The variability in the number of wins can be found to be the following equation (assuming no or few ties):$$ \sigma_U = \sqrt{\frac{n_1n_2(n_1+n_2+1)}{12}}$$These $\mu_U$ and $\sigma_U$ values can then be used to compute a standard normal z-score, which generates a p-value. Implement this method of performing the rank-sum test in the cells below!"- HINT: scipy stats' [`norm`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html) class can be used to obtain p-values after computing a z-score. Function `fn_ranked_sum` created!<jupyter_code>data = udacourse3.fn_read_data('data/permutation_data.csv')
data.head(5)
udacourse3.fn_ranked_sum(x=data[data['condition'] == 0]['time'],
                         y=data[data['condition'] == 1]['time'],
                         z=data['time'],
                         alternative='greater',
                         plot=True,
                         verbose=True)<jupyter_output><empty_output><jupyter_text>Rank-Sum Test Notes in Udacity notes:"For smaller sample sizes, something like the permutation test can be performed. After exhaustively checking the distribution of victories for every possible assignment of group labels to value, a p-value can be computed for how unusual the actually-observed $U$ was.Also, there already exists a function in the scipy stats package [`mannwhitneyu`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.mannwhitneyu.html) that performs the Mann Whitney U test. This function considers more factors than the implementation above, including a correction on the standard deviation for ties and a continuity correction (since we're approximating a discretely-valued distribution with a continuous one). In addition, the approach they take is computationally more efficient, based on the sum of value ranks (hence the rank-sum test name) rather than the matchups explanation provided above."Reference: [Wikipedia](https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test)<jupyter_code>stats.mannwhitneyu(data[data['condition'] == 0]['time'],
                   data[data['condition'] == 1]['time'],
                   alternative='greater')<jupyter_output><empty_output><jupyter_text>Sign Test in Udacity notes:"The sign test also only uses the collected data to compute a test result. It only requires that there be paired values between two groups to compare, and tests whether one group's values tend to be higher than the other's.In the sign test, we don't care how large differences are between groups, only which group takes a larger value. So a comparison of 0.21 vs. 0.22 and 0.21 vs. 0.31 are both counted equally as a point in favor of the second group. This makes the sign test a fairly weak test, though also a test that can be applied fairly broadly. It's most useful when we have very few observations to draw from and can't make a good assumption of underlying distribution characteristics. For example, you might use a sign test as an additional check on click rates that have been aggregated on a daily basis.The count of victories for a particular group can be modeled with the binomial distribution. Under the null hypothesis, it is equally likely that either group has a larger value (in the case of a tie, we ignore the comparison): the binomial distribution's success parameter is $p = 0.5$. Implement the sign test in the function below!"- HINT: scipy stats' [`binom`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binom.html) class can be used to obtain p-values after computing the number of matchups and victories. Function `fn_sign_test` created!<jupyter_code>data = udacourse3.fn_read_data('data/signtest_data.csv')
data.head(5)
udacourse3.fn_sign_test(x=data['control'], 
                        y=data['exp'],
                        z=data['day'],
                        alternative='less',
                        plot=True,
                        verbose=True)<jupyter_output><empty_output><jupyter_text>--- Missing Data**NaN** is not the same thing as an **zero**:- for zero, you have one value for the data, NaN is the absence of data- so you can **exclude** all rows that have NaN values $\rightarrow$ this **shrinks** your data to train/test Machine Learning- you can insert the **mean** of the colum on NaN $\rightarrow$ this **dissolves** the error on the feature for training Machine Learning- you can just insert a **Zero** $\rightarrow$ except in specific cases, that is **not** a good strategy- you can **train** a Machine Learning for data completion $\rightarrow$ if you stops here, fair enough. But if then you use this feature for training another Machine Learning, you are inducing too much **noise** to the model- you can **randomize** and inserting some data, preserving both **mean** and **standard deviation** $\rightarrow$ sometimes this approach is better than just inserting the **mean**- you can **look** for more data $\rightarrow$ this has high **human cost** Analysing Multiple MetricsTake care of the **criteria** to call your experiment as a success!- assume **independence/dependence** of some of the measured variables?- an experiment, with 2 variables, with 5% general percentage of error, what is the probability to call a fail experiment, a success?  - 10% $\rightarrow$ both have tails on the percentage graph, so 2 sides of tails **duplicate** the uncertainty!   In Udacity notes:"If you're tracking multiple evaluation metrics, make sure that you're aware of how the Type I error rates on individual metrics can affect the overall chance of making some kind of Type I error. The simplest case we can consider is if we have _n_ independent evaluation metrics, and that seeing one with a statistically significant result would be enough to call the manipulation a success. In this case, the probability of making at least one Type I error is given by $\alpha_{over} = 1 - (1-\alpha_{ind})^n$ , illustrated in the below image for individual $\alpha_{ind} = .05$ and $\alpha_{ind} = .01: ![multimetrics](graphs/multimetrics1.png)To protect against this, we need to introduce a correction factor on the individual test error rate so that the overall error rate is at most the desired level. A conservative approach is to divide the overall error rate by the number of metrics tested:$\alpha_{ind} = \alpha_{over}/n$This is known as the Bonferroni correction. If we assume independence between metrics, we can do a little bit better with the Šidák correction:$\alpha_{ind} = 1-(1-\alpha_{over})^{1/n}![multimetrics](graphs/multimetrics2.png)The Šidák correction is only slightly higher than the line drawn by the Bonferroni correction.In real life, evaluation scenarios are rarely so straightforward. Metrics will likely be correlated in some way, rather than being independent. If a positive correlation exists, then knowing the outcome of one metric will make it more likely for a correlated metric to also point in the same way. In this case, the corrections above will be more conservative than necessary, resulting in an overall error rate smaller than the desired level. (In cases of negative correlation, the true error rate could go either way, depending on the types of tests performed.)In addition, we might need multiple metrics to show statistical significance to call an experiment a success, or there may be different degrees of success depending on which metrics appear to be moved by the manipulation. One metric may not be enough to make it worth deploying a change tested in an experiment. Reducing the individual error rate will make it harder for a truly significant effect to show up as statistically significant. That is, reducing the Type I error rate will also increase the Type II error rate – another conservative shift.Ultimately, there is a small balancing act when it comes to selecting an error-controlling scheme. Being fully conservative with one of the simple corrections above means that you increase the risk of failing to roll out changes that actually have an impact on metrics. Consider the level of dependence between metrics and what results are needed to declare a success to calibrate the right balance in error rates. If you need to see a significant change in all of your metrics to proceed with it, you might not need a correction factor at all. You can also use dummy test results, bootstrapping, and permutation approaches to plan significance thresholds. Finally, don't forget that practical significance can be an all-important quality that overrides other statistical significance findings.While the main focus of this page has been on interpretation of evaluation metrics, it's worth noting that these cautions also apply to invariant metrics. The more invariant metrics you test, the more likely it will be that some test will show a statistically significant difference even if the groups tested are drawn from equivalent populations. However, it might not be a good idea to apply a correction factor to individual tests since we want to avoid larger issues with interpretation later on. As mentioned previously, a single invariant metric showing a statistically significant difference is not necessarily cause for alarm, but it is something that merits follow-up in case it does have an effect on our analysis."--- Fifth Notebook - L14 - Early Stopping In Udacity notes:"If you peek at the results of an experiment before data collection is complete, and choose to stop early because the test is showing statistical significance, you run the risk of a significant increase in your Type I error rate: believing that your experiment had an effect, when in fact it did not. In this notebook, you'll duplicate the assertions made in the video: namely that for an experiment based off of a single traditional statistical test, doing a single peek halfway through the run-time will increase a base Type I error rate from 5% to about 8.6%."  In Udacity notes:"The simulation function below uses a bernoulli / binomial success model for the outcome metric, measuring against a historic baseline. That is, each observation is a single coin flip with success probability "p". If we see a number of successes that is unusual for our baseline value of "p", then we declare a statistically significant result. We will divide the experiment length into multiple 'blocks', checking the status of the experiment after each block is complete. Our outputs of interest are the proportion of trials that are statistically significant in _any_ test, and the proportion of trials that are statistically significant after _each_ individual block." TaskThere are three main steps to filling out the `peeking_sim()` function.1. Simulate some data - Compute the number of trials per block. For simplicity, just round up any fractions so that each block has the same number of trials: we might end up with slightly more trials per block than the corresponding function parameter. - Generate a data matrix with the number of successes observed in each block: the number of rows should be the number of simulations and the number of columns the number of blocks. You can do this with a single call to numpy's [`random.binomial`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.binomial.html) function.2. Compute z-scores at each 'peek' - For each row, compute the cumulative number of successes after each 'block' of the experiment using numpy's [`cumsum`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.cumsum.html) function. The result should be a matrix with the same dimensions as the data, but each column cumulatively adds up the values in each row up to that point. - Compute the expected mean and standard deviation for the number of successes after each 'block' of the experiment. Remember that this will be based on the [binomial distribution](https://en.wikipedia.org/wiki/Binomial_distribution) and is centered on the raw counts, rather than proportion of successes. It'll be useful to create a vector with the cumulative sum of trials after each block to facilitate these calculations. - Use the cumulative counts, the expected counts, and the standard deviations, to compute the z-scores for each peek at the experiment.3. Aggregate test outcomes - Compute a critical z-value using the supposed Type I error rate. Use this critical value to flag which of the z-scores would be counted as statistically significant, and which would not. - The proportion of trials that are significant at _any_ test will be the proportion of rows that have at least one flagged value. The proportion of trials that are significant at _each_ block will be the mean number of flagged values in each column; this will be a 1-d array. Return both of these values as the output of the function.  In Udacity notes:"Running the function on the default parameters as given should return a tuple of results where the probability of any significant test outcome across the two blocks is around 8.6% and the probability of a significant test outcome at each individual block checkpoint is around 5%. Increase the number of trials and number of simulations to get more accurate estimates. You should also see how the overall Type I error rate increases with additional peeks!"<jupyter_code>z = udacourse3.fn_peeking_simulation(num_trials=10_000, 
                                     num_sims=100_000,
                                     verbose=True)
z<jupyter_output><empty_output><jupyter_text>A Multiple Comparisons Approach to Early Peeking In Udacity notes:"The safest way we could deal with performing multiple checks and making poor early stopping decisions is to simply not do it. Once an experiment has been planned and all assignment procedures checked, you should just let the experiment run to completion and just assess the results at the very end. That's not to say that you can't perform early stopping, but it does require additional planning.One way in which you could solve for multiple peeking is to adjust the significance level of individual tests so that the overall error rate is at its desired level. But applying the Bonferroni or Šidák corrections as shown earlier in the lesson will definitely be too conservative, since we know that there is a correlation in test results between peeks. If we see some simulated run with z-score above the threshold at the halfway point, it's more likely to be above that threshold at the end point, compared to some other simulated run that is not statistically significant at the halfway point. One way in which we can obtain a better significance threshold is through the power of simulation. After performing the same steps 1 and 2 above, we want to find a significance level that would call our desired proportion of simulated tests as statistically significant: Task1. Simulate some data (as above)2. Compute z-scores at each 'peek' (as above)3. Obtain required individual-test error rate  - A run is considered statistically significant if it exceeds the critical bounds at _any_ peek. Obtain the maximum z-score from each row as a worst-case scenario for a null run to be falsely rejected.  - Find the z-score threshold that would reject our desired overall Type I error rate.  - Convert that z-score into an equivalent individual-test error rate.  In Udacity notes:"Running the function on the default parameters should give a required individual error rate of about .029. Note how this is somewhat higher than the .025 or .0253 that would have been generated from the Bonferroni and Šidák corrections, respectively. Test with a higher number of simulations and trials to get more accurate estimates, and try out different numbers of blocks to see how it changes the individual error rate needed. The results should approximately match up with the numbers given in the table in the middle of [this article](https://www.evanmiller.org/how-not-to-run-an-ab-test.html); note that peeking $n$ times means splitting the experiment into $n + 1$ blocks."<jupyter_code>udacourse3.fn_peeking_correction(num_trials=10_000, 
                                 num_sims=100_000,
                                 verbose=True)<jupyter_output><empty_output><jupyter_text>--- Test Area<jupyter_code>raise Exception('test area reached')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06076020678685048, 'fraction_numerical': 0.008433854718981972, 'mean_word_length': 4.5691612069761005, 'pattern_counts': {'":': 0, '<': 90, '<?xml version=': 0, '>': 90, 'https://': 27, 'lorem ipsum': 0, 'www.': 9, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '292395', 'license': 'MIT', 'n_tokens_mistral': 14918, 'n_tokens_neox': 13939, 'n_words': 8319, 'path': 'L2L3StudiesNotese.ipynb', 'repo_name': 'epasseto/ThirdProjectStudies', 'size': '60352'}