Over the last four years we have operated a public smartphone platform testbed called PHONELAB. PHONELAB consists of up to several-hundred participants who run an experimental platform image on their primary smartphone. The experimental platform consists of both instrumentation and experimental changes to platform components, including core Android services and Linux. This paper describes the design of the testbed, the process of conducting PHONELAB experiments, and some of the research the testbed has supported. We also offer many lessons learned along the way, almost all of which have been learned the hard way--through trial and a lot of error. We expect our experiences will help those contemplating operating large user-facing testbeds, anyone conducting experiments on smartphones, and many mobile systems researchers.
LiveLabs @cite_10 @cite_26 @cite_18 is a human behavioral experiment testbed utilizing smartphones. They do not hand out smartphones nor control the platform, but instead deploy experiment software on participants' own devices. Because of this, LiveLabs is able to scale up to several thousands of participants spanning three venues, including university campus, a resort island and a large convention center. LiveLabs has different aims than . Its goal is to enable more pervasive computing experiments, rather than work on smartphone systems.
@cite_18 @cite_26 @cite_10
2473243880 1994718979 2089790844
In this paper, we present LiveLabs, a first-of-its-kind testbed that is deployed across a university campus, convention centre, and resort island and collects real-time attributes such as location, group context etc., from hundreds of opt-in participants. These venues, data, and participants are then made available for running rich human-centric behavioural experiments that could test new mobile sensing infrastructure, applications, analytics, or more social-science type hypotheses that influence and then observe actual user behaviour. We share case studies of how researchers from around the world have and are using LiveLabs, and our experiences and lessons learned from building, maintaining, and expanding Live-Labs over the last three years. We present LiveLabs, a mobile experimentation testbed that is currently deployed across our university campus with further deployments at a large shopping mall, a commercial airport, and a resort island soon to follow. The key goal of LiveLabs is to allow in-situ real-time experimentation of mobile applications and services that require context-specific triggers with real participants on their actual smart phones. We describe how LiveLabs works, and then explain the novel R&D required to realise it. We end with a description of the current LiveLabs status (> 700 active participants to date) as well as present some key lessons learned. We believe that, for successful adoption of novel mobile technologies and applications, it is important to be able to test them under real usage patterns, and with real users. To implement this vision, we present our initial effort in building LiveLabs, a large-scale mobile testbed for in-situ experimentation. LiveLabs is unique in two aspects. First, LiveLabs operates on a scale much larger than most research testbeds it is being deployed in four different public spaces in Singapore (a university campus, a shopping mall, an airport and a leisure resort), and is expected to have a pool of over 30,000 opt-in participants. Second, LiveLabs not only instruments smartphones and the infrastructure to gather deep individual and collective context, but also provides a unique experimentation platform that automates many aspects of behavioral experimentation, such as subject selection and context-triggered delivery of interventions. We briefly describe some of the research challenges associated with building such a large-scale deep-context collection testbed, as well as the current status of LiveLabs. We then share our perspectives on the challenges of setting up and operating such testbeds, with the expectation that our experiences will prove useful to other researchers looking to build similar testbeds elsewhere.