BACKGROUND

Many networked systems interact with large numbers of devices that have different contexts, capabilities, and users. Because of the large variety of endpoint devices that a system may interact with, not all of the same software, web pages, and content are compatible with or even useful for different devices. For most day-to-day interactions, such as streaming a video or loading a web page, the impact is small in magnitude and short in duration, for the server as well as the client device and its user. In these common interactions, there is an interaction between devices without a significant commitment of future resources by the client or server. The amount of computing resources expended is small and only of a brief duration, so that if the interaction is ended, e.g., a user stops a video or the user does not click on search results, the exchange has resulted in a very limited consumption of computing resources.

However, when remotely configuring devices or enrolling them in ongoing monitoring programs, applying an incorrect configuration or carrying out excessive monitoring can have a high impact. Enrolling a device in a monitoring program can often involve changes to the configuration of a device, causing the device to perform ongoing or repeated monitoring actions over days, weeks, or months. These actions can significantly affect the performance and usability of a device, by affecting battery life, network bandwidth usage, data storage, processor utilization, and so on. It is important that monitoring programs and their configuration changes are distributed where the monitoring will be appropriate and effective, to avoid wasting limited resources of endpoint devices such as phones, smartwatches, tablet computers, and so on. In addition, in many cases server systems and other systems may rely on the data collected from devices enrolled in monitoring programs, and may be ineffective at completing their monitoring objectives if client devices do not carry out monitoring as required. Similarly, when client devices provide incomplete, inaccurate, or monitoring data this can result in significant costs to servers in terms of power consumption, network bandwidth usage, data storage, and other resources for data collection and processing that does not achieve the monitoring objective.

SUMMARY

In some implementations, a computer system provides a platform for publishing and distributing configuration data to involve different remote devices in various monitoring programs. The system enables different parties, e.g., organizations, administrators, third-parties, etc., to create or register monitoring programs to be distributed and managed by the system. The system uses various techniques, including machine learning models and various types of profiles, to selectively determine which monitoring programs, if any, are appropriate for different remote devices. The processing involves analysis of factors such as characteristics of the remote devices and their users, the requirements and objectives of the monitoring programs, and predictions regarding the outcomes of enrolling users and their devices in specific monitoring programs (e.g., likelihoods of completion of the program, retention in the program, achieving desired levels of data quality or data collection completeness, etc.).

The system uses various information about monitoring programs, devices, users, and historical outcomes to identify and provide a customized set of monitoring programs for each candidate device or user. The set of monitoring programs can also be adaptively changed and adjusted as circumstances change, for example, as the context of a user or device changes, more information about the user or device is received, or as the enrollment of different monitoring programs rises or falls. The analysis by the system can identify customized sets of candidate monitoring programs for different remote devices, which can greatly improve the efficiency with which the system distributes and conducts monitoring with remote devices. The system identifies and distributes the programs that are most compatible with and beneficial to a device and its user and/or those which are predicted to be most likely to be carried out according to the requirements of the monitoring program. This selective distribution helps ensure that resources are focused on the limited set of monitoring programs that will provide useful results and the requirements of the programs will be met (e.g., the monitoring can be carried out in the appropriate manner and for the full duration needed). The system filters out or avoids providing monitoring programs to a device when the system determines that monitoring program would represent a waste of computing resources if initiated, such as where the program has a low likelihood of being completed successfully for the device or user, or where the data that would be collected would be unnecessary (e.g., redundant with existing monitoring of other similarly-situated devices) or would be of low quality.

As a result, unlike many systems that seek to maximize the number of devices involved in monitoring regardless of the predicted effectiveness, the system can tailor the monitoring program options for each device or user, limiting the monitoring programs that are distributed to and initiated by specific devices to the programs for which monitoring is most likely to be completed effectively and for which the results will be valuable and useful. This conserves resources for both client devices and servers by avoiding the distribution and running of monitoring programs that are redundant, ineffective, or unlikely to be carried out correctly.

Systems that manage and distribute monitoring programs over the Internet commonly encounter the problem of poor responsiveness and poor reliability among the devices that receive and begin monitoring programs. Frequently, from the set of devices that begin a monitoring program, the percentage of devices that reliably perform monitoring and report the needed data back to the server is much lower than needed. This often represents significant wasted resources, since partial data or low-precision data often fails to meet the needs of a monitoring scheme—the data often needs to be discarded and the resources consumed (previously and on an ongoing basis) is wasted. In many monitoring programs, especially where users may opt-in or opt-out, there are many devices that begin a monitoring program and do not continue for the full duration needed. This represents a significant inefficiency, where devices may be consuming power, network bandwidth, processor utilization, and storage space on an ongoing basis even when the data collected is incomplete and will not achieve the objective of the monitoring.

The present system conserves resources and improves the rate at which devices carry out monitoring programs by intelligently selecting which monitoring programs are appropriate for different devices, users, and contexts. This includes using information about characteristics of devices, users, and monitoring programs, as well as historical information about compliance with monitoring programs, to predict how well different user and devices will comply with the needs of different monitoring programs, the system can train and use machine learning models to determine these predictions. With the predicted compliance and retention information, the system can distribute and initiate monitoring programs where they are most likely to be effectively performed and avoid enrolling devices that will not likely comply with the requirements of the monitoring program. For example, one monitoring program may require hourly updates with sensor measurements over a period of three months. The system can generate models of how different devices and users in different contexts have interacted and provided data in this monitoring program and other monitoring programs. With the models, the system can weight or prioritize the recommendation of and distribution of the monitoring program for devices, users, and contexts having characteristics similar to those that have resulted in effective compliance with the sensor data collection, hourly reporting, and 3-month duration of the program. Conversely, for devices, users, and contexts where monitoring has not been successful for one or more of the needed program elements, the system can demote or avoid providing the monitoring program to avoid incurring unnecessary, inefficient resource usage to initiate and continue monitoring that is unlikely to be completed properly.

Systems that manage and distribute monitoring programs over the Internet also commonly encounter the problem of failing to distribute monitoring programs widely enough to meet the objectives of the monitoring programs and in a sufficiently targeted manner to monitor the right types of devices and users. Often, a monitoring scheme has an objective that requires a minimum number of devices in varying contexts in order for the data to be reliable and representative for a population. Many monitoring programs also need devices and users with specific contexts or characteristics in order to be effective, which limits the available pool of candidates and makes the automated recruitment of monitored subject more challenging. For example, some monitoring programs may require measurements using a specific combination of sensors or a specific level of accuracy that limits which devices can be used. Similarly, some monitoring programs may be tailored to monitor specific events, conditions, or situations and so devices and users having specific contexts and backgrounds are needed. Many distribution systems, such as typical application stores and content distribution networks, are unable to assess the suitability of different devices and users for different monitoring programs or ineffective at it, leading them to recommend and distribute programs to devices where the programs are incompatible, redundant, ineffective to achieve monitoring objectives, or are unlikely to be used successfully.

The systems discussed herein are able to more effectively recruit the numbers and types of devices and users needed for various monitoring programs, and to target the recruitment to devices and users most likely to perform the needed types of monitoring for the duration of the monitoring program. The system can use information about the needs of the different monitoring programs, including the numbers of devices and users to monitor for each monitoring program, as well as the attributes, contexts, history, and other factors that are needed to make a candidate a good fit for a monitoring program. For example, the system can select and rank monitoring programs for a device or user taking into account the total number of monitored subjects the monitoring program needs, the current level of enrollment relative to the needed total, the factors (e.g., monitored subject attributes) for each monitoring program that make subjects more or less valuable to achieving the objectives of each monitoring program, and so on. The analysis by the system can identify, from among a large pool of candidates, which candidates provide the most valuable data (e.g., due to the subject's context, attributes, history, etc.) and the most likelihood, allowing the system to better fill each monitoring program with the number of subject needed and with the types of subject needed. In some implementations, this results in customized recommendations of monitoring programs to different devices or users, each being selectively provided a set of monitoring programs dynamically and adaptively determined to optimize the criteria for value of the collected data to the researcher or program administrator and value of the monitoring to the subject, while accounting for the likelihood of compliance and retention in the program and limits for the burden placed on devices and users (e.g., actively accounting for the resource consumption a program places on devices and the time commitments required of users to participate in a monitoring program).

Unlike many distribution platforms, the present system can balance the needs of different monitoring programs from among a pool of limited candidates. While many different organizations may desire to conduct monitoring through the system, there is often a limited pool of devices and users that can participate, and each device or user may be limited to participating in a small number of the monitoring programs available to avoid overburdening individual devices or users. Monitoring programs are not compatible with every device or user, and some programs are more strict in their requirements than others. Users appropriately have control their own devices and should select which monitoring programs to participate in, but may not understand which of many (e.g., often hundreds or thousands) monitoring programs are best suited for their devices, habits, and needs, or which monitoring programs their participation in would yield the greatest value. The system discussed herein, however, can account for these factors in the manner in which it selects, ranks, and prioritizes monitoring programs for different remote devices and their users, to allocate the limited opportunities available for monitoring programs to be carried out by candidates among the many monitoring programs to be carried out. This helps tailor the monitoring program options presented for each device and user to those that are appropriate for and valuable to the device and user, while also enabling the system to recruit the needed numbers and types of devices and users that monitoring programs need to meet their overall objectives for aggregating data from many devices and users.

The system can facilitate remote configuration of the sensors, user interfaces, network communication settings, and other aspects of remote devices so that monitoring programs are started and continued by the remote devices. By selectively involving remote devices in monitoring programs predicted by models to have the highest likelihood of success, the system can achieve monitoring with much higher efficiency, effectively reducing the consumption of resources of monitoring that will be only partial, of too short of a duration, or with unacceptable data quality.

In some implementations, the monitoring programs can be used for scientific research or health research. The systems and techniques herein can use machine learning models to select and recommend monitoring program opportunities that are likely to satisfy the needs and preferences of a potential participant in a program as well as the research requirements associated with participation in the program. As described throughout, programs can represent research studies that users can choose to participate in and initiate monitoring using their electronic devices, with the software modules and device configuration needed being distributed through an Internet-based software platform. Researchers use the software platform to design research studies and allow users of the software platform to participate in the research studies. The software platform may provide program opportunities when, for example, a user is interested in participating in one or more programs that are seeking enrollment of additional participants.

For example, the system can identify a research study for which (i) a user's participation is likely to be beneficial to the user and (ii) the user's participation is also likely to benefit one or more research objectives of a researcher that conducts the study. In this way, the predictions generated by the system can be applied to align the requirements of participants and researchers, thereby increasing the likelihood that user participation will be mutually beneficial to the user and the researcher.

The system can provide customized predictions that improve the alignment of user interests, needs or preferences, and research outcomes. For example, the system can evaluate the attributes of different users and their level of compliance with different monitoring program requirements (e.g., types of data collected, frequency of data collection, duration of participation, consistency, activities needed, etc.). With the analysis results or a machine learning model trained based on the data, the system can predict how the probability and level of compliance of other users with specific requirements of a monitoring program or for total set of requirements of the monitoring program. Using these models, and by evaluating a user's historical study participation data in relation to research requirements related to participant compliance, the system can predict how likely a user is to successfully complete the various requirements of different monitoring programs. The system can then use these predictions to rank and prioritize studies to recommend and distribute to each device and user, which can avoid many of the inefficiencies and ineffective monitoring that result from monitoring programs that are initiated and not performed effectively.

For example, the system may identify a first monitoring program for a research study that aligns moderately with user's interests, where the user has a background making collected data useful to achieving the research study's objective, and where the predicted probability of user compliance in the research study is high. A second monitoring program may better align with the user's interests but have a lower predicted probability of compliance with that monitoring program's requirements. As a result, even though second monitoring program is better aligned to the user's preferences, the system prioritizes the first monitoring program that is less aligned to a user's preferences because the system's analysis indicates that the user is not likely to be compliant in the better-aligned program. Thus, the system guides the user to participate in a research study that still aligns with his/her interests but has a higher probability of compliance, which increases the probability of a successful study outcome (e.g., retention for the duration needed and compliance with study requirements).

The architecture of the system provides various technological improvements in predictively evaluating user data (e.g., user attributes, user preferences, user interaction data) and research requirement data (e.g., study design parameters) for a program in selectively generating recommendations for the program. Using different types of data, the system generates recommendations in a manner that improves the likelihood of a beneficial outcome for a user that is provided a recommendation to participate in a program and a researcher that manages the program. To accomplish this, the system processes user data and research requirement data using learning models to identify program opportunities (e.g., identifying a program that a specific user is likely to benefit from by participating in the program and for which the specific user's participation is likely to satisfy one or more research requirements of the program). The system can apply the learning models on an ongoing basis so that recommendations accurately represent any changes in program availability, user preferences, research requirements, among other factors.

In one general aspect, a method performed by one or more computers, the method comprising: accessing, by the one or more computers, a candidate profile for a candidate to participate in interactive programs involving collection of data from participants using remote devices and reporting of the collected data over a communication network, the candidate profile describing attributes of the candidate; identifying, by the one or more computers, program profiles for multiple programs in which the candidate is eligible to enroll as a participant, wherein the program profiles (i) describe the respective programs and (ii) indicate types of data to be collected in the respective programs; determining, by the one or more computers, one or more scores for each of the programs with respect to the candidate, wherein the one or more scores for each program are based on: (i) a relevance of the program to the candidate determined based on the attributes of the candidate as indicated by the candidate profile; and (ii) a predicted level of compliance of the candidate in providing the types of data to be collected in the program, wherein the predicted level of compliance is based on the attributes of the participant; selecting, by the one or more computers, one or more of the programs based on the scores for the programs; and providing, by the one or more computers, selection results over the communication network to a client device associated with the candidate, the selection results being provided for presentation by the client device to indicate the selected one or more programs on an interface of the client device.

In some implementations, the method includes: selecting, by the one or more computers, a configuration data package corresponding to a particular monitoring program indicated by the selection results; and delivering, by the one or more computers, the configuration data package to the client device associated with the candidate, the configuration data package including one or more settings to adjust a configuration of the client device to initiate monitoring by the client device according to the particular monitoring program.

In some implementations, the configuration data package comprises configuration data configured to adjust operation of one or more sensors of the client device to perform a series of measurements to collect, at the client device, one or more types of data specified by the monitoring program.

In some implementations, the configuration data package is configured to adjust communication of the client device to receive measurement data from a second device over a wired or wireless interface and to report the measurement data to the one or more computers over the communication network.

In some implementations, the one or more computers are part of a distribution platform configured to (i) selectively provide monitoring programs to remote devices of users registered with the distribution platform, the monitoring programs including monitoring designed by different third-party organizations, and (ii) receive, process, and store data for the respective monitoring programs from the remote devices and provide aggregated data for each of the monitoring programs to the respective third-party organizations corresponding to the monitoring programs.

In some implementations, the candidate profile indicates interests of the candidate; the program profiles describe the topics or subjects of the programs; and the scores are based on a level of similarity of the interests of the candidate indicated by the profile to the respective topics or subject of the programs as indicated in the program profiles.

In some implementations, the method includes determining a preferred level of interaction of the candidate; and determining that the participant actions for a particular program exceed the preferred level of interaction of the candidate. Determining the scores comprises applying a penalty to a score for the particular program based on determining that the participant actions for a particular program exceed the preferred level of interaction of the candidate.

In some implementations, the method includes determining, for each of the programs and based on the attributes of the candidate indicated in the candidate profile for the candidate, a predicted likelihood of compliance or predicted level of compliance of the candidate in performing the participant actions for the programs. The scores are based on the predicted likelihoods of compliance or predicted levels of compliance.

In some implementations, the predicted likelihood of compliance or predicted level of compliance is generated by providing information indicating the attributes of the candidate to a machine learning model trained based on the attributes of other individuals and actions of the other individuals while enrolled in one or more programs as participants.

In some implementations, the programs include clinical treatment programs.

In some implementations, the programs correspond to different health research studies.

In some implementations, the programs are research studies that each have cohort inclusion criteria or cohort exclusion criteria. The method comprises determining whether the candidate is eligible to participate each of the research studies based on whether the user satisfies cohort inclusion criteria for the respective research studies or whether the user satisfies cohort exclusion criteria for the respective studies. The multiple studies are identified by filtering a set of programs to identify a subset of the programs for which the candidate is determined to be eligible.

In some implementations, determining the scores comprises: assessing a composition of sets of participants enrolled in the respective programs; determining that the candidate has one or more attributes that are underrepresented in the set of participants for a particular program; and weighting one or more scores for the candidate for the particular program based on determining that the candidate has one or more attributes that are underrepresented in the set of participants for a particular program.

In some implementations, program profiles indicate participant actions that participants are requested to perform as part of participation in the respective programs.

In some implementations, the one or more scores for each program are determined based on a level of compatibility between participant actions of the program and a level of engagement preferred to be predicted for the candidate.

The details of one or more implementations are set forth in the accompanying drawings and the description below. Other potential features and advantages will become apparent from the description, the drawings, and the claims.

