<jupyter_start><jupyter_text>Statictics Fundamentals  Fitting distributions to get parameters  Fitting normal distribution on solar cell efficiency data<jupyter_code>!wget -q --show-progress https://github.com/PacktPublishing/Practical-Data-Science-with-Python/raw/main/Chapter8/data/solar_cell_efficiencies.csv
import pandas as pd

df = pd.read_csv('solar_cell_efficiencies.csv')
df.describe()
df.hist(bins=40);
df['efficiency'].skew()
df['efficiency'].kurt()
import scipy.stats

scipy.stats.norm.fit(df['efficiency'])<jupyter_output><empty_output><jupyter_text>Fitting weibull on MISO wind data<jupyter_code>!wget -q --show-progress https://github.com/PacktPublishing/Practical-Data-Science-with-Python/raw/main/Chapter8/test_your_knowledge/data/miso_wind_data.csv
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from scipy.stats import weibull_min
df = pd.read_csv('miso_wind_data.csv')
df.head()
df.describe()
df.info()
df['MWh'] = df['MWh'].astype('float')
sns.histplot(df['MWh'], kde=True);<jupyter_output><empty_output><jupyter_text>Recall from the chapter that Weibull can often be used to model windspeed-related data. The distribution doesn't look perfect here, we should probably break it up by season. But we will still try fitting with a Weibull and see how it compares.<jupyter_code># this gives us c, loc, and scale
wb_fit = weibull_min.fit(df['MWh'])
wb_fit
wb = weibull_min(c=wb_fit[0], loc=wb_fit[1], scale=wb_fit[2])
x = np.linspace(0, 20000, 1000)
plt.plot(x, wb.pdf(x))
plt.show()<jupyter_output><empty_output><jupyter_text>That doesn't look right at all. We need to give starting values for our parameters so it has a better chance of fitting. Let's play around with the parameters to figure out what it should be closer to. The key here was the scale parameter - it needs to be on the order of the spread of the data.<jupyter_code>wb = weibull_min(c=5, loc=0, scale=10000)
plt.plot(x, wb.pdf(x))
plt.show()
params = weibull_min.fit(df['MWh'].values, scale=20000)
params
wb = weibull_min(c=params[0], loc=params[1], scale=params[2])
f, ax = plt.subplots()
# the density=1 argument makes the integral of the histogram equal 1, so it's on the same scale as the PDF
df['MWh'].hist(density=1, ax=ax, bins=50)
ax.plot(x, wb.pdf(x))
plt.show()<jupyter_output><empty_output><jupyter_text>Hey, not a bad fit! It looks like it is like a bi-modal distribution, composed of two or three Weibulls from the different seasons. We chose the Weibull, again, because it's known to represent this sort of data. Also, we can eyeball the histogram and pick a distribution that seems to fit.<jupyter_code>means = []
for i in range(10000):
    sample = np.random.choice(df['MWh'], 1000, replace=True)
    means.append(sample.mean())
sns.histplot(means)
plt.show()<jupyter_output><empty_output><jupyter_text>Yes, looks like it's approaching a normal distribution.  Statistical Tests  1-sample 2-sided T-test<jupyter_code>import pandas as pd

solar_data = pd.read_csv('solar_cell_efficiencies.csv')
from scipy.stats import ttest_1samp

print(solar_data['efficiency'].mean())
ttest_1samp(solar_data['efficiency'], 14, alternative='two-sided')
sample = solar_data['efficiency'].sample(30, random_state=1)
print(sample.mean())
ttest_1samp(sample, 14)<jupyter_output>14.100587953561668<jupyter_text>When we are considering full sample, p-value is less than significance level ($\alpha$) of 0.05, so we reject the null hypothesis. This means the solar efficiency is more than 14%. On the other hand, we failed to reject the null hypothesis on a sample data of 30 records. The proper test to use for larger sample sizes is the z-test. This ends up being about the same as a t-test, however. We can use this from the statsmodels package.<jupyter_code>from statsmodels.stats.weightstats import ztest
ztest(solar_data['efficiency'], value=14)<jupyter_output><empty_output><jupyter_text>Let's say we want to make sure the average efficiency of our latest batch of solar cells is greater than 14%. The sample we used that was measured from a recent production run is in our solar data we've already loaded. We can formulate our null hypothesis as this: the sample mean is less than or equal to the expected mean of 14%. The alternative hypothesis is then: the sample mean is greater than the expected mean of 14%. We can perform this test with scipy like so:<jupyter_code>ttest_1samp(solar_data['efficiency'], 14, alternative='greater')<jupyter_output><empty_output><jupyter_text>The alternative argument is set to 'greater', meaning the alternative hypothesis is that the sample mean is greater than the expected mean. Our results show the null hypothesis is rejected, and it looks like our sample mean is greater than 14% with statistical significance.<jupyter_code>ttest_1samp(solar_data['efficiency'], 14, alternative='less')<jupyter_output><empty_output><jupyter_text>A/B testing<jupyter_code>!wget -q --show-progress https://github.com/PacktPublishing/Practical-Data-Science-with-Python/raw/main/Chapter9/data/ab_sales_data.csv<jupyter_output>ab_sales_data.csv     0%[                    ]       0  --.-KB/s               
ab_sales_data.csv   100%[===================>]  39.08K  --.-KB/s    in 0.002s<jupyter_text>Let's say we have a website selling t-shirts and want to experiment with the design to try and drive more sales. We're going to change the layout in a B version of the site and compare our sales rates to the A version.<jupyter_code>ab_df = pd.read_csv('ab_sales_data.csv')
ab_df.head()<jupyter_output><empty_output><jupyter_text>We have a column for the A design, and each row is a website visitor. A value of 1 represents a sale, while 0 represents no sale. The B design column is the same, and the samples are not paired up (each sample from A and B is individual and independent). We can look at the mean sales rates easily:<jupyter_code>ab_df.mean()<jupyter_output><empty_output><jupyter_text>This shows us B has a slightly higher sales rate. To test if B is really better than A, we can first try a two-sample, two-sided t-test. The null hypothesis is that the means of the two groups are the same; the alternative is that they are not the same (for a two-sided test).<jupyter_code>ztest(ab_df['a_sale'], ab_df['b_sale'])<jupyter_output><empty_output><jupyter_text>Remember that the first value is the z-statistic, and the second value in the tuple is the p-value. In this case, it looks like there is a significant difference in the means, since the p-value of 0.024 is less than our significance threshold of 0.05. We already know from examination that the B sales rate was a little higher, so it appears the B design is better. To be a little more precise, we can also specify the direction of the test. With statsmodels, the options for the alternative argument are two-sided, larger, and smaller. Specifying larger means the alternative hypothesis is that A's mean is larger than B's. The null hypothesis in that case is that A's mean is less than or equal to B's mean. We'll use smaller to carry out our one-sided z-test to see if B's average sales value is greater than A's:<jupyter_code>ztest(ab_df['a_sale'], ab_df['b_sale'], alternative='smaller')
ztest(ab_df['a_sale'], ab_df['b_sale'], value=-0.01, alternative='smaller')
ztest(ab_df['b_sale'], ab_df['a_sale'], value=0.01, alternative='larger')<jupyter_output><empty_output><jupyter_text>Bootstrap A/B<jupyter_code>!pip install -q bootstrapped<jupyter_output>Building wheel for bootstrapped (setup.py) ... [?25l[?25hdone<jupyter_text>Bootstrapping is another method for A/B testing. With this, we can use sampling with replacement (bootstrapping) to calculate many means of our A and B datasets, then get the confidence intervals of the difference in mean values between A and B. If the confidence interval for the difference in means doesn't pass through 0, we can say with a certain percent confidence that the means are different. For example, we can use the bootstrapped package (which you will need to install with pip install bootstrapped) to do this:<jupyter_code>import bootstrapped.bootstrap as bs
import bootstrapped.compare_functions as bs_compare
import bootstrapped.stats_functions as bs_stats

bs.bootstrap_ab(test=ab_df['b_sale'].values,
                ctrl=ab_df['a_sale'].values,
                stat_func=bs_stats.mean,
                compare_func=bs_compare.difference,
                alpha=0.05)<jupyter_output><empty_output><jupyter_text>The values are small, but we can see the 95% confidence interval doesn't quite pass through 0, so we can say with 95% confidence B is better than A. However, it could be that B is only better than A by 0.0008 in absolute value, which wouldn't be much of an improvement on A.  Testing between several groups with ANOVATesting one or two samples is useful in many situations, but we can also find ourselves needing to test the means between several groups. We can use multiple t-tests with the Bonferroni correction as one method, but another way is to use ANOVA and post hoc tests.Let's say we want to test more than one design at a time and compare them all to see which is best: A, B, and C designs. For comparing the means of three or more groups, we can use an ANOVA test. There is also a way to compare several groups with t-tests using what's called the Bonferroni correction; this is available in the scikit_posthocs.posthoc_ttest() function from the scikit-posthocs package (you will need to install this package with conda or pip). This would tell us the difference between all the pairs from our groups of data – we will come back to other ways to do this shortly.However, ANOVA can be first used to see if there is any difference between any of the groups. Instead of a t-test, it uses an F-test. Again, this method provides a p-value, which we compare to a significant value we choose (usually 0.05).<jupyter_code>!wget -q --show-progress https://github.com/PacktPublishing/Practical-Data-Science-with-Python/raw/main/Chapter9/data/abc_sales_data.csv<jupyter_output>abc_sales_data.csv    0%[                    ]       0  --.-KB/s               
abc_sales_data.csv  100%[===================>]   6.11K  --.-KB/s    in 0s<jupyter_text>Because one assumption for ANOVA is that the data comes from normal distributions, we are using data from binomial distributions. This is chunks of 100 website visitors, with a count of how many visitors made a purchase.Each row is a number between 0 and 100. As we learned in the previous chapter, sampling data from distributions many times tends toward a normal distribution, so if we structure our data in this way, we can approach a normal distribution instead of a binomial distribution like with our other set of A/B sales data.In this case, a binomial distribution is based on Bernoulli trials (like coin flips), and a collection of binomial distribution samples tends toward a normal distribution. We can load the data with pandas, then conduct an ANOVA test:<jupyter_code>from scipy.stats import f_oneway

abc_df = pd.read_csv('abc_sales_data.csv')
abc_df.head()
f_oneway(abc_df['a_sale'], abc_df['b_sale'], abc_df['c_sale'])
abc_df.mean()<jupyter_output><empty_output><jupyter_text>Here, we provide as many datasets as we want to our f_oneway() function, which performs an ANOVA test. We get an F-statistic and p-value. As usual, we compare the p-value to our significance level to determine if we can reject the null hypothesis. The null hypothesis here is that the means are all the same; the alternative is that the means are different. Since p < 0.05, we can reject the null hypothesis, and our test shows the means to be different. Looking at the means with abc_df.mean(), we can see they are 4.9, 5.5, and 6.9 for A, B, and C, which look quite different. However, it would be nice to know which differences between the groups are significant. For this, we can use a post hoc test. There are several post hoc tests, but we will use one common post hoc test: the Tukey test. This is named after <NAME>, the legendary statistician who created boxplots and pioneered EDA. Different ANOVA post hoc tests have different subtleties that make them useful in different situations, but Tukey is a decent general test to use as a default.<jupyter_code>!pip install -q scikit_posthocs
from scikit_posthocs import posthoc_tukey

melted_abc = abc_df.melt(var_name='groups', value_name='values')
melted_abc.head()
posthoc_tukey(melted_abc, group_col='groups', val_col='values')<jupyter_output><empty_output><jupyter_text>These are p-values for the hypothesis we are testing that the means are not different between pairs. Since the p-values are small between all the pairs (0.001, much less than 0.05) we can say the differences between the means of all the groups are significant. It is possible with the test that some differences between groups may be significant while others may not.  Comparing Winds in different seasons with ANOVA<jupyter_code>!wget -q --show-progress https://github.com/PacktPublishing/Practical-Data-Science-with-Python/raw/main/Chapter9/test_your_knowledge/data/miso_wind_data.csv
import pandas as pd
from scikit_posthocs import posthoc_tukey
df = pd.read_csv('miso_wind_data.csv')
df.head()
df.info()
df['MWh'] = df['MWh'].astype('float')
df['Market Day'] = pd.to_datetime(df['Market Day'])
df.set_index('Market Day', inplace=True)

spring = df['3-1-2020': '5-31-2020'][['MWh']]
summer = df['6-1-2020': '8-31-2020'][['MWh']]
fall = df['9-1-2020': '11-30-2020'][['MWh']]
winter = df['12-1-2020':][['MWh']]

spring.columns = ['spring']
summer.columns = ['summer']
fall.columns = ['fall']
winter.columns = ['winter']

spring.reset_index(inplace=True, drop=True)
summer.reset_index(inplace=True, drop=True)
fall.reset_index(inplace=True, drop=True)
winter.reset_index(inplace=True, drop=True)

tukey_df = pd.concat([spring.iloc[:744], summer.iloc[:744], fall.iloc[:744], winter.iloc[:744]], axis=1)
tukey_df.head()
melted = tukey_df.melt(var_name='groups', value_name='values')
melted.head()
posthoc_tukey(melted, group_col='groups', val_col='values')
tukey_df.mean()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09186629920146987, 'fraction_numerical': 0.01710126492827362, 'mean_word_length': 5.134373645426962, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 80, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '105593', 'license': 'Apache-2.0', 'n_tokens_mistral': 4568, 'n_tokens_neox': 4224, 'n_words': 1717, 'path': '_docs/nbs/T873451-Statistics-fundamentals.ipynb', 'repo_name': 'sparsh-ai/recohut', 'size': '14155'}