After observational studies in the lab, running panels, and mining logs, we can begin to consider running large-scale experiments. Online randomized controlled trials split users into two groups and provide them with different versions of the system, logging their responses, to identify how user responses change. Such trials are run by many types of product such as search engines, recommenders, social networks, and online marketplaces. Authors from a broad range of such products [1]} identified a number of important steps, including establishing an overall evaluation criterion (OEC). The OEC should align with long-term business goals, it should evaluate all parts of the product, encourage a diverse set of improvements, and be sufficiently sensitive and affordable.
