<commit_before># GAE/Bingo

GAE/Bingo is a drop-in split testing framework for App Engine, built for [Khan Academy](http://www.khanacademy.org) and *heavily* inspired by [<NAME>](http://www.kalzumeus.com)'s [A/Bingo](http://www.bingocardcreator.com/abingo). If you're on App Engine, GAE/Bingo can get your A/B tests up and running in minutes.

You can read [more about the initial inspiration and design of
GAE/Bingo](http://bjk5.com/coming-soon).

GAE/Bingo is [MIT licensed](http://en.wikipedia.org/wiki/MIT_License).

* <a href="#features">Features</a>  
* <a href="#screens">Experiment Dashboard</a>
* <a href="#usage">Usage and Code Samples</a>  
* <a href="#principles">Design Principles</a>  
* <a href="#start">Getting Started</a>  
* <a href="#non-features">Non-features so far</a>  
* <a href="#bonus">Bonus</a>  
* <a href="#faq">FAQ</a>  

## <a name="features">Features</a>

Free features [inherited quite directly from A/Bingo's design](http://www.bingocardcreator.com/abingo):

* Test display or behavioral differences in one line of code.
* Measure any event as a conversion in one line of code.
* Eliminate guesswork: automatically test for statistical significance.
* Blazingly fast, with minimal impact on page load times or server load.
* Written by programmers, for programmers. Marketing is an engineering discipline!

Plus extra goodies:

* Drop into App Engine with minimal configuration
* Framework agnostic -- works with webapp, Django, Flask, whatever.
* Persistent storage of test results -- if you're running experiments
  that take a long time like, say, [testing your software's effects on a student's education](http://www.khanacademy.org), that's no problem.
* Performance optimized for App Engine

## <a name="screens">Experiment Dashboard</a>

<img src="http://i.imgur.com/x4Hew.png"/><br/>

Your dashboard lets you control all experiments along with statistical analysis of the results.</em>

## <a name="usage">Usage and Code Samples</a>

### Starting an experiment
This line of code will automatically set up an A/B test named "new button
design" (if necessary) and return True or False by default. Use this anywhere
you can run Python code, it's highly optimized.

<pre>from gae_bingo.gae_bingo import ab_test

...

if ab_test("new button design"):
    return "new_button_class"
else:
    return "old_button_class"
</pre>

You can also specify an identifier for the conversion metric you're expecting
to analyze.

<pre>
if ab_test("crazy new type of animal", conversion_name="animals escaped"):
    return Gorillas()
else:
    return Monkeys()
</pre>

If you don't specify a conversion_name when starting a test, GAE/Bingo will
automatically listen for conversions with the same name as the experiment.

### Scoring a conversion
This line of code will score a conversion in the "new button design" experiment
for the current user.

<pre>from gae_bingo.gae_bingo import bingo

...
bingo("new button design")
</pre>

...or, in the case of the above "crazy new type of animal" experiment,
<pre>bingo("animals escaped")
</pre>

### Specifying alternatives
Even though the above two lines are all you need to start running some pretty
useful A/B tests, you've got more power than that. Choose from any of the
following lines of code to return various alternatives for your tests.
Remember: each individual user will get consistent results from these
functions.

<pre>from gae_bingo.gae_bingo import ab_test

...

# SIMPLE
#
# Returns True or False
#
use_new_button_design = ab_test("new button design")

...

# LIST OF ALTERNATIVES
#
# Returns "old" or "shiny"
#
button_class = ab_test("new button class", ["old", "shiny"])

...

# LIST OF >2 ALTERNATIVES (multivariate testing)
#
# Returns 10, 15, or 20
#
answers_required = ab_test("answers required", [10, 15, 20])

...

# WEIGHTED ALTERNATIVES
# (use a dictionary with alternatives as keys and weights as values)
#
# Returns "crazy" to 1/5 of your users and "normal" to 4/5 of your users
#
crazy_experiment = ab_test("crazy experiment", {"crazy": 1, "normal": 4})

</pre>

### Testing multiple results for a single experiment
You may want to statistically examine different dimensions of an experiment's
effects. You can do this by passing an array to the conversion_name parameter.

<pre>
breed_new_animal = ab_test("breed new animal", conversion_name=["animals escaped", "talking animals"])
</pre>

This syntactic sugar will automatically create multiple experiments for you.
Your conversions will be tracked independently with their own statistical
analysis, so you can independently call bingo() when appropriate:

<pre>
bingo("animals escaped")
</pre>

...and...

<pre>
bingo("talking animals")
</pre>

## <a name="principles">Design Principles</a>

Just go read through [<NAME>'s slides on A/B testing design principles](http://www.bingocardcreator.com/abingo/design). This implementation only tweaks those to achieve:

* Persistence to datastore for very-long-lasting records and very-long-running
  experiments without sacrificing performance.
* Quick to drop-in for any App Engine (Python) developer, with
  strong-but-customizable ties to existing App Engine user identities.

## <a name="start">Getting Started</a>

1. Download this repository's source and copy the `gae_bingo/` folder into your App Engine project's root directory.

2. Add the following handler definitions (found in `yaml/app.yaml`) to your app's `app.yaml`:
<pre>
handlers:
&ndash; url: /gae_bingo/static
&nbsp;&nbsp;static_dir: gae_bingo/static<br/>
&ndash; url: /gae_bingo/tests/.*
&nbsp;&nbsp;script: gae_bingo/tests/main.py<br/>
&ndash; url: /gae_bingo/.*
&nbsp;&nbsp;script: gae_bingo/main.py
</pre>
...and the following job definitions (found in `yaml/cron.yaml`) to your app's `cron.yaml`:
<pre>
cron:
&ndash; description: persist gae bingo experiments to datastore
&nbsp;&nbsp;url: /gae_bingo/persist
&nbsp;&nbsp;schedule: every 5 minutes
</pre>

3. If you want, modify the contents of config.py to match your application's usage. There
   are two functions to modify: can_control_experiments() and
   current_logged_in_identity()
<pre>
# Customize can_see_experiments however you want to specify
# whether or not the currently-logged-in user has access
# to the experiment dashboard.
#
def can_control_experiments():
    # This default implementation will be fine for most
    return users.is_current_user_admin()
</pre><br/>
<pre>
# Customize current_logged_in_identity to make your a/b sessions
# stickier and more persistent per user.
#
# This should return one of the following:<commit_msg>documentation work<commit_after># GAE/Bingo

GAE/Bingo is a drop-in split testing framework for App Engine, built for [Khan Academy](http://www.khanacademy.org) and *heavily* inspired by [<NAME>](http://www.kalzumeus.com)'s [A/Bingo](http://www.bingocardcreator.com/abingo). If you're on App Engine, GAE/Bingo can get your A/B tests up and running in minutes.

You can read [more about the initial inspiration and design of
GAE/Bingo](http://bjk5.com/coming-soon).

GAE/Bingo is [MIT licensed](http://en.wikipedia.org/wiki/MIT_License).

* <a href="#features">Features</a>  
* <a href="#screens">Experiment Dashboard</a>
* <a href="#usage">Usage and Code Samples</a>  
* <a href="#principles">Design Principles</a>  
* <a href="#start">Getting Started</a>  
* <a href="#non-features">Non-features so far</a>  
* <a href="#bonus">Bonus</a>  
* <a href="#faq">FAQ</a>  

## <a name="features">Features</a>

Free features [inherited quite directly from A/Bingo's design](http://www.bingocardcreator.com/abingo):

* Test display or behavioral differences in one line of code.
* Measure any event as a conversion in one line of code.
* Eliminate guesswork: automatically test for statistical significance.
* Blazingly fast, with minimal impact on page load times or server load.
* Written by programmers, for programmers. Marketing is an engineering discipline!

Plus extra goodies:

* Drop into App Engine with minimal configuration
* Framework agnostic -- works with webapp, Django, Flask, whatever.
* Persistent storage of test results -- if you're running experiments
  that take a long time like, say, [testing your software's effects on a student's education](http://www.khanacademy.org), that's no problem.
* Performance optimized for App Engine

## <a name="screens">Experiment Dashboard</a>

<img src="http://i.imgur.com/x4Hew.png"/><br/>

Your dashboard lets you control all experiments along with statistical analysis of the results.</em>

## <a name="usage">Usage and Code Samples</a>

### Starting an experiment
This line of code will automatically set up an A/B test named "new button
design" (if necessary) and return True or False by default. Use this anywhere
you can run Python code, it's highly optimized.

<pre>from gae_bingo.gae_bingo import ab_test

...

if ab_test("new button design"):
    return "new_button_class"
else:
    return "old_button_class"
</pre>

You can also specify an identifier for the conversion metric you're expecting
to analyze.

<pre>
if ab_test("crazy new type of animal", conversion_name="animals escaped"):
    return Gorillas()
else:
    return Monkeys()
</pre>

If you don't specify a conversion_name when starting a test, GAE/Bingo will
automatically listen for conversions with the same name as the experiment.

### Scoring a conversion
This line of code will score a conversion in the "new button design" experiment
for the current user.

<pre>from gae_bingo.gae_bingo import bingo

...
bingo("new button design")
</pre>

...or, in the case of the above "crazy new type of animal" experiment,
<pre>bingo("animals escaped")
</pre>

### Specifying alternatives
Even though the above two lines are all you need to start running some pretty
useful A/B tests, you've got more power than that. Choose from any of the
following lines of code to return various alternatives for your tests.
Remember: each individual user will get consistent results from these
functions.

<pre>from gae_bingo.gae_bingo import ab_test

...

# SIMPLE
#
# Returns True or False
#
use_new_button_design = ab_test("new button design")

...

# LIST OF ALTERNATIVES
#
# Returns "old" or "shiny"
#
button_class = ab_test("new button class", ["old", "shiny"])

...

# LIST OF >2 ALTERNATIVES (multivariate testing)
#
# Returns 10, 15, or 20
#
answers_required = ab_test("answers required", [10, 15, 20])

...

# WEIGHTED ALTERNATIVES
# (use a dictionary with alternatives as keys and weights as values)
#
# Returns "crazy" to 1/5 of your users and "normal" to 4/5 of your users
#
crazy_experiment = ab_test("crazy experiment", {"crazy": 1, "normal": 4})

</pre>

### Testing multiple results for a single experiment
You may want to statistically examine different dimensions of an experiment's
effects. You can do this by passing an array to the conversion_name parameter.

<pre>
breed_new_animal = ab_test("breed new animal", conversion_name=["animals escaped", "talking animals"])
</pre>

This syntactic sugar will automatically create multiple experiments for you.
Your conversions will be tracked independently with their own statistical
analysis, so you can independently call bingo() when appropriate:

<pre>
bingo("animals escaped")
</pre>

...and...

<pre>
bingo("talking animals")
</pre>

## <a name="principles">Design Principles</a>

Just go read through [<NAME>'s slides on A/B testing design principles](http://www.bingocardcreator.com/abingo/design). This implementation only tweaks those to achieve:

* Persistence to datastore for very-long-lasting records and very-long-running
  experiments without sacrificing performance.
* Quick to drop-in for any App Engine (Python) developer, with
  strong-but-customizable ties to existing App Engine user identities.

## <a name="start">Getting Started</a>

1. Download this repository's source and copy the `gae_bingo/` folder into your App Engine project's root directory.

2. Add the following handler definitions (found in `yaml/app.yaml`) to your app's `app.yaml`:
<pre>
handlers:
&ndash; url: /gae_bingo/static
&nbsp;&nbsp;static_dir: gae_bingo/static<br/>
&ndash; url: /gae_bingo/tests/.*
&nbsp;&nbsp;script: gae_bingo/tests/main.py<br/>
&ndash; url: /gae_bingo/.*
&nbsp;&nbsp;script: gae_bingo/main.py
</pre>
...and the following job definitions (found in `yaml/cron.yaml`) to your app's `cron.yaml`:
<pre>
cron:
&ndash; description: persist gae bingo experiments to datastore
&nbsp;&nbsp;url: /gae_bingo/persist
&nbsp;&nbsp;schedule: every 5 minutes
</pre>

3. If you want, modify the contents of config.py to match your application's usage. There
   are two functions to modify: can_control_experiments() and
   current_logged_in_identity()
<pre>
# Customize can_see_experiments however you want to specify<br/>
# whether or not the currently-logged-in user has access<br/>
# to the experiment dashboard.<br/>
#<br/>
def can_control_experiments():
    # This default implementation will be fine for most
    return users.is_current_user_admin()
</pre><br/>
<pre>
# Customize current_logged_in_identity to make your a/b sessions
# stickier and more persistent per user.
#
# This should return one of the following:
{'fraction_non_alphanumeric': 0.10889209507175944, 'fraction_numerical': 0.0037967955045941225, 'mean_word_length': 4.756118881118881, 'pattern_counts': {'":': 4, '<': 121, '<?xml version=': 0, '>': 123, 'https://': 0, 'lorem ipsum': 0, 'www.': 12, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '153527', 'n_tokens_mistral': 4104, 'n_tokens_neox': 3930, 'n_words': 1559}