The problem of measuring the true incremental effectiveness of a digital advertising campaign is of increasing importance to marketers. With a large and increasing percentage of digital advertising delivered via Demand-Side-Platforms (DSPs) executing campaigns via Real-Time-Bidding (RTB) auctions and programmatic approaches, a measurement solution that satisfies both advertiser concerns and the constraints of a DSP is of particular interest. MediaMath (a DSP) has developed the first practical, statistically sound randomization-based methodology for causal ad effectiveness (or Ad Lift) measurement by a DSP (or similar digital advertising execution system that may not have full control over the advertising transaction mechanisms). We describe our solution and establish its soundness within the causal framework of counterfactuals and potential outcomes, and present a Gibbs-sampling procedure for estimating confidence intervals around the estimated Ad Lift. We also address practical complications (unique to the digital advertising setting) that stem from the fact that digital advertising is targeted and measured via identifiers (e.g., cookies, mobile advertising IDs) that may not be stable over time. One such complication is the repeated occurrence of identifiers, leading to interference among observations. Another is due to the possibility of multiple identifiers being associated with the same consumer, leading to "contamination" with some of their identifiers being assigned to the Treatment group and others to the Control group. Complications such as these have severely impaired previous efforts to derive accurate measurements of lift in practice. In contrast to a few other papers on the subject, this paper has an expository aim as well, and provides a rigorous, self-contained, and readily-implementable treatment of all relevant concepts.
In this paper, we develop an experimental analysis to estimate the causal effect of online marketing campaigns as a whole, and not just the media ad design. We analyze the causal effects on user conversion probability. We run experiments based on A B testing to perform this evaluation. We also estimate the causal effect of the media ad design given this randomization approach. We discuss the framework of a marketing campaign in the context of targeted display advertising, and incorporate the main elements of this framework in the evaluation. We consider budget constraints, the auction process, and the targeting engine in the analysis and the experimental set up. For the effects of this evaluation, we assume the targeting engine to be a black box that incorporates the impression delivery policy, the budget constraints, and the bidding process. Our method to disaggregate the campaign causal analysis is inspired on randomized experiments with imperfect compliance and the intention-to-treat (ITT) analysis. In this framework, individuals assigned randomly to the study group might refuse to take the treatment. For estimation, we present a Bayesian approach and provide credible intervals for the causal estimates. We analyze the effects of 2 independent campaigns for different products from the Advertising.com ad network for 20M+ users each campaign. We describe a computer program to assist a clinician with assessing the efficacy of treatments in experimental studies for which treatment assignment is random but subject compliance is imperfect. The major difficulty in such studies is that treatment efficacy is not "identifiable", that is, it cannot be estimated from the data, even when the number of subjects is infinite, unless additional knowledge is provided. Our system combines Bayesian learning with Gibbs sampling using two inputs: (1) the investigator's prior probabilities of the relative sizes of subpopulations and (2) the observed data from the experiment. The system outputs a histogram depicting the posterior distribution of the average treatment effect, that is, the probability that the average outcome (e.g., survival) would attain a given level, had the treatment been taken uniformly by the entire population. This paper describes the theoretical basis for the proposed approach and presents experimental results on both simulated and real data, showing agreement with the theoretical asymptotic bounds.