TECHNICAL FIELD

Embodiments of the invention relate to the field of software application development and deployment, and more particularly, to distributed software feature experimentation.

BACKGROUND

Software applications, such as web page based applications, mobile device applications, desktop computer system applications, etc., are created by software developers. The software applications are then distributed to end users and run on end user computer systems (e.g., run on a user's personal computer, mobile computing device, served to a user from a web server, etc.). From usage of a software application, a developer may determine that changes to the application are necessary and/or desired for various reasons. For example, a software application may contain a bug causing the application to act in undesirable or unforeseen ways, and may even prevent the software application from running. As another example, a developer may want to change a user interface element based on experiences and/or feedback of real world users of the application. As yet another example, a software application update intended to allow a real world user to improve service bookings may actually result in fewer service bookings. As yet another example, a new version of a web page may be intended to decrease page load time, when in fact it increases page load time. Thus, the effectiveness of the application in terms of operational performance and user experience may influence updates to the application.

Software applications, however, are becoming increasingly complex in terms of the number and characteristic of user interface elements, user interface layout, functional elements, options that may be deployed in an application, different deployments to various user groups, as well as other application implementation elements that may impact user experience with an application. Thus, measuring whether an update, potentially containing several new features (e.g., layout changes, feature changes, additional/new elements, etc.) and/or different version deployed to different groups (e.g., a mobile user group, a geographic user group, a specific customer base, etc.), is effective in terms of operational performance and/or application performance metrics is also increasingly difficult. Furthermore, when multiple metrics are measured simultaneously during deployment of one or more new and/or updated application features, the potential for error in the analysis of the impact on the metrics attributable to the new and/or updated application features increases to an unacceptable level. Thus, testing of the deployment of new and/or updated application features is inefficient and often not sufficiently accurate.

