<jupyter_start><jupyter_text>**A/B Testing:**Statistical analysis is our best tool for predicting outcomes we don’t know, using the information we know.Picture this scenario – You have made certain changes to your website recently. Unfortunately, you have no way of knowing with full accuracy how the next 100,000 people who visit your website will behave. That is the information we cannot know today, and if we were to wait until those 100,000 people visited our site, it would be too late to optimize their experience.This seems to be a classic Catch-22 situation!This is where a data scientist can take control. A data scientist collects and studies the data available to help optimize the website for a better consumer experience. And for this, it is imperative to know how to use various statistical tools, especially the concept of A/B Testing.  **What is A/B Testing**A/B testing is a basic randomized control experiment. It is a way to compare the two versions of a variable to find out which performs better in a controlled environment.For instance, let’s say you own a company and want to increase the sales of your product. Here, either you can use random experiments, or you can apply scientific and statistical methods. A/B testing is one of the most prominent and widely used statistical tools.In the above scenario, you may divide the products into two parts – A and B. Here A will remain unchanged while you make significant changes in B’s packaging. Now, on the basis of the response from customer groups who used A and B respectively, you try to decide which is performing better.  It is a hypothetical testing methodology for making decisions that estimate population parameters based on sample statistics. The population refers to all the customers buying your product, while the sample refers to the number of customers that participated in the test.  **Project**In this project, we will analyze the result of an A/B test where the first gate in Cookie Cats was moved from level 30 to level 40. In particular, we will analyze the impact on player retention and game rounds.  **AB Testing Process**- Understanding business problem & data- Detect and resolve problems in the data (Missing Value, Outliers, Unexpected Value)- Look summary stats and plots- Apply hypothesis testing and check assumptions- Check Normality & Homogeneity- Apply tests - Evaluate the results- Make inferences- Recommend business decision to your customer/director/ceo etc.  **Data Preprocessing**<jupyter_code>import pandas as pd

data_train = pd.read_csv('/content/control_data.csv')
experiment_data = pd.read_csv('/content/experiment_data.csv')<jupyter_output><empty_output><jupyter_text>Dataset Information<jupyter_code>data_train.head()
experiment_data.head()
data_train.info()
experiment_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 37 entries, 0 to 36
Data columns (total 5 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   Date         37 non-null     object 
 1   Pageviews    37 non-null     int64  
 2   Clicks       37 non-null     int64  
 3   Enrollments  23 non-null     float64
 4   Payments     23 non-null     float64
dtypes: float64(2), int64(2), object(1)
memory usage: 1.6+ KB<jupyter_text>Checking the number of null values present in each column<jupyter_code>data_train.isna().sum()
experiment_data.isna().sum()
data_train[data_train['Enrollments'].isna()]
data_total = pd.concat([data_train, experiment_data])
data_total.sample(5)
import numpy as np
np.random.seed(7)
import sklearn.utils

data_total['row_id'] = data_total.index
# Create a Day of Week feature
data_total['DOW'] = data_total['Date'].str.slice(start=0, stop=3)
# Remove missing data
data_total.dropna(inplace=True)
data_total['Experiment'] = np.random.randint(2, size=len(data_total))
# Remove missing data
data_total.dropna(inplace=True)
del data_total['Date'], data_total['Payments']
data_total = sklearn.utils.shuffle(data_total)<jupyter_output><empty_output><jupyter_text>Now Check the New data<jupyter_code>data_total.head()<jupyter_output><empty_output><jupyter_text>Reordering Columns<jupyter_code>data_total = data_total[['row_id', 'Experiment', 'Pageviews', 'Clicks', 'DOW', 'Enrollments']]<jupyter_output><empty_output><jupyter_text>Splitting the datasets into training sets and test sets<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(data_total.loc[:, data_total.columns != 'Enrollments'],\
                                                    data_total['Enrollments'], test_size=0.3)<jupyter_output><empty_output><jupyter_text>Encoding Cateogrical Values<jupyter_code>from sklearn.preprocessing import LabelEncoder

lb = LabelEncoder()
X_train['DOW'] = lb.fit_transform(X_train['DOW'])
X_test['DOW'] = lb.transform(X_test['DOW'])
X_train.head()
X_test.head()
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error

def calculate_metrics(y_test, y_preds):
    rmse = np.sqrt(mean_squared_error(y_test, y_preds))
    r_sq = r2_score(y_test, y_preds)
    mae = mean_absolute_error(y_test, y_preds)

    print('RMSE Score: {}'.format(rmse))
    print('R2_Squared: {}'.format(r_sq))
    print('MAE Score: {}'.format(mae))
import matplotlib.pyplot as plt
plt.style.use('ggplot')
%matplotlib inline

def plot_preds(y_test, y_preds, model_name):
    N = len(y_test)
    plt.figure(figsize=(10,5))
    original = plt.scatter(np.arange(1, N+1), y_test, c='blue')
    prediction = plt.scatter(np.arange(1, N+1), y_preds, c='red')
    plt.xticks(np.arange(1, N+1))
    plt.xlabel('# Oberservation')
    plt.ylabel('Enrollments')
    title = 'True labels vs. Predicted Labels ({})'.format(model_name)
    plt.title(title)
    plt.legend((original, prediction), ('Original', 'Prediction'))
    plt.show()
import statsmodels.api as sm

X_train_refined = X_train.drop(columns=['row_id'], axis=1)
linear_regression = sm.OLS(y_train, X_train_refined)
linear_regression = linear_regression.fit()
X_test_refined = X_test.drop(columns=['row_id'], axis=1)
y_preds = linear_regression.predict(X_test_refined)
calculate_metrics(y_test, y_preds)
plot_preds(y_test, y_preds, 'Linear Regression')
print(linear_regression.summary())
pd.DataFrame(linear_regression.pvalues)\
    .reset_index()\
    .rename(columns={'index':'Terms', 0:'p_value'})\
    .sort_values('p_value')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10029452798015812, 'fraction_numerical': 0.012091148659122617, 'mean_word_length': 5.06390977443609, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '86813', 'license': 'MIT', 'n_tokens_mistral': 2035, 'n_tokens_neox': 1958, 'n_words': 645, 'path': 'Machine_Learning/Statistics_for_Machine_Learning/ML_1_8_AB Testing_(D).ipynb', 'repo_name': 'SathiyaEthi/winter-of-contributing', 'size': '6451'}