<commit_before>---
layout: post
title: Measuring the impact of advertising
date:   2017-08-11
tag: data-science
categories:
 - data-science
keywords: "controlled trials, a/b test, measure, advertising, data science"
description: In this post we discuss the method we developed to measure the impact of advertising in re targeting campaigns
author: pablow
---$
<!--excerpt.start-->

At [Jampp] [jampp], we boost mobile sales through advertising. That’s why measuring the impact of advertising is key to our business. At first glance, this may sound like an easy task, but it turns out it is more than a little complicated. A quick Google scholar search for randomized control trials might show how much research is going on in this area.
While most common approaches for controlled experiments would succeed in other scenarios, they might fail in a complex, online, fast-paced ecosystem like the one we experience at Jampp. 
Bias selection, noisy data, or not having enough statistical power, could make the experiment unsuccessful.
After extensive research on the subject in both academia and the industry, we developed a method that facilitates measuring this impact in complex settings.
In this post, we dive deeper into some of the possible problems for these kind of experiments, and we show the key findings that will be implemented in our design. Finally, we describe the details of the method and the next steps towards improving its design.

<!--excerpt.end-->

## Introduction

Every advertiser wants to know whether they are having an impact on their users or not. More precisely, the question they're trying to answer is perfectly captured in a blog post by [Google] [google-blog]: "Did showing my ads affect customers' behavior, relative to not showing my ads?". With that in mind, we established a method to address this question and its underlying implications for the way retargeting is measured.

The best approach to this issue is to run a randomized controlled trial ([RCT]) [rct] or, more specifically, an [A/B test] [ABtest] [Deng, et al. 2013 ][deng-cuped], which is basically a controlled experiment where two probabilistically equivalent groups (A and B, or treatment and control) are compared. They are identical except for one variation or factor that might affect a user's behavior. In our case, the factor would be to *show an ad* of the specific advertiser we want to analyze. As a result, any difference between the two groups would be because of the factor applied, the only thing that changed. Therefore, we can establish a causal relation between *showing an ad* and the difference in the behavior.  

In order to measure the behavior we need to establish a metric. It will be used to compare the differences between the groups. In general, this metric is based on a specific demand of the advertiser, depending on the kind of behavior they want to measure. For example, it can be a click on an impression, a search, a purchase, etc. 

The metric can vary depending on the type of campaign:

1. In engagement campaigns (i.e. we target users who already downloaded the app to get them to use it more frequently), if we define a *key event* in an app, such as: a purchase, booking a taxi ride, booking a trip, etc.; the metric can be: volume of key events per user, or the total revenue generated by user purchases.
2. In user acquisition campaigns (i.e. non-users becoming users of the app), the key event is *an install*. In this case, we will have a unique event per user.

It is important to notice that events that go deeper into the funnel have less rate 
of occurrence and more volatility among users. (the graph below shows a possible funnel of events) The work of [Gordon et al. 2016] [fcb-kellog] showed that commonly-used approaches for this kind of experiments are unreliable for lower funnel conversion outcomes (e.g., purchases) but somewhat more reliable for upper funnel ones (e.g., search of a product in the app). 

![ Funnel of Events ]({{site.url}}/assets/images/RCT/funnel_events.png){: .center-image } 


In this post we focus on our method for *engagement campaigns* and, for the sake of simplicity, the sum of *key events* per user in the period of analysis will characterize the engagement of a particular user. The mean of that sum for all users will be the key metric of the group under analysis. 
 <commit_msg>corrected wrong header<commit_after>---
layout: post
title: Measuring the impact of advertising
date:   2017-08-11
tag: data-science
categories:
 - data-science
keywords: "controlled trials, a/b test, measure, advertising, data science"
description: In this post we discuss the method we developed to measure the impact of advertising in re targeting campaigns
author: pablow
---



<!--excerpt.start-->

At [Jampp] [jampp], we boost mobile sales through advertising. That’s why measuring the impact of advertising is key to our business. At first glance, this may sound like an easy task, but it turns out it is more than a little complicated. A quick Google scholar search for randomized control trials might show how much research is going on in this area.
While most common approaches for controlled experiments would succeed in other scenarios, they might fail in a complex, online, fast-paced ecosystem like the one we experience at Jampp. 
Bias selection, noisy data, or not having enough statistical power, could make the experiment unsuccessful.
After extensive research on the subject in both academia and the industry, we developed a method that facilitates measuring this impact in complex settings.
In this post, we dive deeper into some of the possible problems for these kind of experiments, and we show the key findings that will be implemented in our design. Finally, we describe the details of the method and the next steps towards improving its design.

<!--excerpt.end-->

## Introduction

Every advertiser wants to know whether they are having an impact on their users or not. More precisely, the question they're trying to answer is perfectly captured in a blog post by [Google] [google-blog]: "Did showing my ads affect customers' behavior, relative to not showing my ads?". With that in mind, we established a method to address this question and its underlying implications for the way retargeting is measured.

The best approach to this issue is to run a randomized controlled trial ([RCT]) [rct] or, more specifically, an [A/B test] [ABtest] [Deng, et al. 2013 ][deng-cuped], which is basically a controlled experiment where two probabilistically equivalent groups (A and B, or treatment and control) are compared. They are identical except for one variation or factor that might affect a user's behavior. In our case, the factor would be to *show an ad* of the specific advertiser we want to analyze. As a result, any difference between the two groups would be because of the factor applied, the only thing that changed. Therefore, we can establish a causal relation between *showing an ad* and the difference in the behavior.  

In order to measure the behavior we need to establish a metric. It will be used to compare the differences between the groups. In general, this metric is based on a specific demand of the advertiser, depending on the kind of behavior they want to measure. For example, it can be a click on an impression, a search, a purchase, etc. 

The metric can vary depending on the type of campaign:

1. In engagement campaigns (i.e. we target users who already downloaded the app to get them to use it more frequently), if we define a *key event* in an app, such as: a purchase, booking a taxi ride, booking a trip, etc.; the metric can be: volume of key events per user, or the total revenue generated by user purchases.
2. In user acquisition campaigns (i.e. non-users becoming users of the app), the key event is *an install*. In this case, we will have a unique event per user.

It is important to notice that events that go deeper into the funnel have less rate 
of occurrence and more volatility among users. (the graph below shows a possible funnel of events) The work of [Gordon et al. 2016] [fcb-kellog] showed that commonly-used approaches for this kind of experiments are unreliable for lower funnel conversion outcomes (e.g., purchases) but somewhat more reliable for upper funnel ones (e.g., search of a product in the app). 

![ Funnel of Events ]({{site.url}}/assets/images/RCT/funnel_events.png){: .center-image } 


In this post we focus on our method for *engagement campaigns* and, for the sake of simplicity, the sum of *key events* per user in the period of analysis will characterize the engagement of a particular user. The mean of that sum for all users will be the key metric of the group under analysis. 
 
{'fraction_non_alphanumeric': 0.050417285926603404, 'fraction_numerical': 0.004115696810334972, 'mean_word_length': 4.360294117647059, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2365604', 'n_tokens_mistral': 2110, 'n_tokens_neox': 2002, 'n_words': 1414}