<jupyter_start><jupyter_text>Bayesian A-B testing of online conversion data (...or, any random process where counting is involved)Suppose you are selling something online.  You are testing a new webpage design to see if it results in more sales ("conversions" of web page visitors to customers).  You divert a portion of your customers to the new website, at random (say, 10% of them). Easiet way to install pymc: `conda install pymc` from your Anaconda directory (Windows)Book: [Bayesian Methods for Hackers by Davidson-Pilon](https://www.amazon.com/Bayesian-Methods-Hackers-Probabilistic-Addison-Wesley/dp/0133902838/ref=sr_1_1?ie=UTF8&qid=1470965242&sr=8-1&keywords=bayesian+methods+hackers)There is also a [free Jupyter notebook version of the book on GitHub!](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers)<jupyter_code>import pymc
import pandas as pd
import numpy as np
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Generate simulated data for purposes of demonstrating the A-B testing technique.<jupyter_code>conv_rate_a = 0.06
conv_rate_b = 0.07
data_a = pymc.rbernoulli(conv_rate_a, 1800)
data_b = pymc.rbernoulli(conv_rate_b, 100)
print data_a[:30]
print len(data_a)
pd.Series(data_a).sum()
pd.Series(data_b).sum()<jupyter_output><empty_output><jupyter_text>Set up the test apparatus.<jupyter_code>p_a = pymc.Uniform("p_a", 0, 1)
p_b = pymc.Uniform("p_b", 0, 1)

@pymc.deterministic
def delta(first = p_a, second = p_b):
    answer = second - first
    return answer

observed_a = pymc.Bernoulli("obs_a", p_a, value=data_a, observed=True)
observed_b = pymc.Bernoulli("obs_b", p_b, value=data_b, observed=True)<jupyter_output><empty_output><jupyter_text>Run the simulation to determine the most likely parameter values.<jupyter_code>model = pymc.MCMC([p_a, p_b, delta, observed_a, observed_b])
model.sample(100000,10000,100)
pymc.Matplot.plot(model)<jupyter_output>Plotting p_a
Plotting p_b
Plotting delta
{'fraction_non_alphanumeric': 0.10510358767054068, 'fraction_numerical': 0.03031834259727135, 'mean_word_length': 6.644787644787645, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '363236', 'license': 'MIT', 'n_tokens_mistral': 775, 'n_tokens_neox': 708, 'n_words': 179, 'path': 'bayesian_ab.ipynb', 'repo_name': 'peterfig/intro2ml', 'size': '1979'}