CROSS REFERENCE TO RELATED APPLICATIONS

The present application is related to commonly-assigned U.S. patent application Ser. No. 15/677,925, entitled “METHOD AND SYSTEM FOR DETECTING GAPS IN DATA BUCKETS FOR A/B EXPERIMENTATION,” and U.S. patent application Ser. No. 15/677,917, entitled “METHOD AND SYSTEM FOR DETECTING DATA BUCKET INCONSISTENCIES FOR A/B EXPERIMENTATION,” each of which being filed on the same day, and the disclosures of each being incorporated herein by reference in their entirety.

BACKGROUND

1. Technical Field

The present teaching generally relates to online experimentation. More specifically, the present teachings relate to providing pre-approved A/A data buckets. Further, the present teachings relate to detecting and monitoring a gap in data buckets for online experimentation. Still further, the present teachings relate to detecting and monitoring inconsistencies in data buckets for online experimentation.

2. Technical Background

In the age of the Internet, online experimentation, and in particular controlled online experimentation, is a commonly used and effective tool for product development. One such type of controlled online experiment is A/B testing. A/B testing in a more classical sense corresponds to having two (or more) groups, where one group—the control group—is given a controlled experience, while another group (or groups) are given a test experience. For example, in drug tests, a control group may receive a placebo while a test group may receive a test drug. In online experimentation, one set of users may receive one user experience at their user device, while another set of users may receive a different user experience. This allows a service provider to gauge an effectiveness of the user experience based on various user metrics computed for each of the various users of each set of users. For example, a service provider may test an effectiveness of a new website format (experimental user experience) against an original website format (control user experience). Users accessing the website via their user devices may randomly be provided with one of the new website format or the original website format, and user interaction metrics (e.g., clicks, scrolls, advertisement impressions, click-through-rate, etc.) may be computed for each website format for each user to determine how “effective” the new website format is as compared to the original website experience.

There are some requirements to these types of online experiments in order for the results to be accurate. For example, the two or more groups (e.g., control group and experimental group(s)) should include users that are randomly selected without any predisposition. Additionally, a user placed into one group should remain in that group throughout the duration of the experiment. Further still, the size (e.g., number of users) in each group should be substantially equal. Failure to adhere to these conditions may result in compromised results that do not accurately reflect the outcome that is obtained.

One way to prevent potential errors in the experimentation process is to perform A/A validation. A/A validation, for example, serves to try and validate the control and experimentation groups, or in the case of online experimentation, the control data bucket and the experimentation data bucket(s). This includes determining whether or not there are any pre-existing differences in the control and/or experimentation buckets, as well as determining whether or not there are any systematic errors in the experimentation system that would lead to unexpected results.

Typically, A/A validation takes time. For example, in order for data to be obtained and analyzed, four to five days' worth of data may be needed. Additionally, it may be expected for some data buckets of online experiments to fail the A/A validation process. Therefore, it is common for experimenters to open more data buckets for the A/A validation process then may actually be needed. This may cause the experimenter to have to decide which data bucket, if any, to use for the experiment. These issues, amongst various others, may cause a delay in the start of an online experiment, which in the ever evolving and fast paced online world, is undesirable.

When online experiments are performed, as mentioned above, users may be placed into one of a control group or an experimentation group. In certain scenarios, a single experiment may include two or more experimentation groups, depending on the conditions and aspects sought to be tested by the experimenter. Each of the control group and the experimentation group (or groups) is designed such that they have a same size. For example, a control data bucket may be designed such that it is to include 10,000 randomly selected users, while an experimental data bucket may also be designed such that it is to include 10,000 randomly selected, but different, users. If, during or after experimentation it is determined that one or more of these data buckets included substantially less than the designed number of users, the results of the experiment may be compromised. For example, if the data buckets were designed to include 10,000 users, but after the experiment it is determined that the data bucket online includes 7,000 users, this may lead to inaccurate results. Inaccurate results may corrupt the data, and undermine the findings of that experiment.

Further still, when placing users in one of the control data buckets or one of the one or more experimental data buckets, it is believed that each user will only be placed in one of these data buckets. Intuitively, if a user is to be placed in both the control data bucket and the experimental data bucket, this may lead to compromised results. Similarly, if a user is placed in two different experimental data buckets, inconsistent results across these two data buckets will arise.

Each of these issues, as described above, are further exacerbated by the platform of online experimentation as the number of users is extremely large, and the time scale of randomly selecting and placing a user into a data bucket, and rolling out various user experiences is very small.

Therefore, providing techniques to reduce the amount of time needed to validate and accuracy of data buckets for use create online experiments is needed. Further, providing techniques to identify a gap between an expected data bucket size and an actual data bucket size is needed. Still further, providing techniques to identify inconsistencies between data buckets is needed.

SUMMARY

The teachings disclosed herein generally relate to methods, systems, and programming for providing data buckets for online experiments. The teachings disclosed herein further generally relate to methods, systems, and programming for detecting data bucket discrepancies associated with online experiments. The teachings disclosed herein still further generally relate to methods, systems, and programming for identifying data bucket overlap with online experiments.

In one example, a method for providing data buckets for online experiments may be implemented on at least one machine including at least one processor, memory, and communications circuitry is described. User activity data representing user activity for a first plurality of user identifiers may be obtained. A first set of values representing a first user engagement parameter for each user identifier of the first plurality may be generated based on the user activity data, and a second set of values representing a second user engagement parameter for each user identifier of the first plurality may also be generated based on the user activity data. A first ranking for the first set may be determined, and a second ranking for the second set may also be determined. A first exclusion range including a first number of values to be removed from the first set and the second set may be determined. A homogenous value set may be generated by removing the first number of values from the first set and the second set, where each value from the homogenous value set corresponds to a user identifier that is available to be placed in a data bucket for an online experiment.

In another example, a system for providing data buckets for online experimentation is described. The system includes a metric computation system configured to obtain user activity data representing user activity for a first plurality of user identifiers, generate a first set of values representing a first user engagement parameter for each user identifier of the first plurality based on the user activity, and generate a second set of values representing a second user engagement parameter for each user identifier of the first plurality based on the user activity. The system also includes a hash value ranker configured to determine a first ranking for the first set, and determine a second ranking for the second set. The system further includes a hash value exclusion system configured to determine a first exclusion range including a first number of values to be removed from the first set and the second set. The system still further includes a hash value selector system configured to generate a homogenous value set by removing the first number of values from the first set and the second set, wherein each value from the homogenous value set corresponds to a user identifier that is available to be placed in a data bucket for an online experiment.

Other concepts relate to software for implementing the present teaching on providing data buckets for online experiments. A software product, in accord with this concept, includes at least one machine-readable non-transitory medium and information carried by the medium. The information carried by the medium may be executable program code data, parameters in association with the executable program code, and/or information related to a user, a request, content, or information related to a social group, etc.

In one example, a non-transitory computer readable medium having instructions recorded thereon for providing data buckets for online experiments is described. The instructions, when read by a computer, causes the computer to obtain user activity data representing user activity for a first plurality of user identifiers, generate a first set of values representing a first user engagement parameter for each user identifier of the first plurality based on the user activity, and generate a second set of values representing a second user engagement parameter for each user identifier of the first plurality based on the user activity. The instructions further, when read by the computer, causes the computer to determine a first ranking for the first set, and determine a second ranking for the second set. The instructions further, when read by the computer, causes the computer to determine a first exclusion range including a first number of values to be removed from the first set and the second set, and to generate a homogenous value set by removing the first number of values from the first set and the second set, where each value from the homogenous value set corresponds to a user identifier that is available to be placed in a data bucket for an online experiment.

In yet another example, a method for detecting data bucket discrepancies associated with online experiments implemented on at least one machine including at least one processor, memory, and communications circuitry is described. A monitoring layer may be generated within an online experimentation platform. The online experimentation platform may include at least a first layer, where a first online experiment is associated with the first layer, the monitoring layer includes a monitoring layer data bucket, and the first layer includes at least a first data bucket. First data representing user activity associated with a first plurality of identifiers may be obtained, the user activity being associated with the first layer. Second data including representing at least one user engagement parameter may be generated based on the first data. A first discrepancy between the first data and the second data may be determined, where the first discrepancy indicates a first amount of identifiers that include a first metadata tag associated with the first layer and lack a second metadata tag associated with the monitoring layer.

In still yet another example, a system for detecting data bucket discrepancies associated with online experiments is described. The system includes an experimentation system configured to generate a monitoring layer within an online experimentation platform. The experimentation platform includes at least a first layer, and where a first online experiment is associated with the first layer, the monitoring layer includes a monitoring layer data bucket, and the first layer comprises at least a first data bucket. The system also includes a discrepancy detection system configured to obtain first data representing user activity associated with a first plurality of identifiers, the user activity being associated with the first layer. The discrepancy detection system is further configured to determine a first discrepancy between the first data and the second data, where the first discrepancy indicates a first amount of identifiers that include a first metadata tag associated with the first layer and lack a second metadata tag associated with the monitoring layer.

Other concepts relate to software for implementing the present teaching on detecting data bucket discrepancies associated with online experiments. A software product, in accord with this concept, includes at least one machine-readable non-transitory medium and information carried by the medium. The information carried by the medium may be executable program code data, parameters in association with the executable program code, and/or information related to a user, a request, content, or information related to a social group, etc.

In one example, a non-transitory computer readable medium having instructions recorded thereon for detecting data bucket discrepancies associated with online experiments. The instructions, when read by a computer, cause the computer to generate a monitoring layer within an online experimentation platform. The experimentation platform may include at least a first layer, and where a first online experiment is associated with the first layer, the monitoring layer includes a monitoring layer data bucket, and the first layer includes at least a first data bucket. The instructions, when read by the computer, also causes the computer to obtain first data representing user activity associated with a first plurality of identifiers, the user activity being associated with the first layer. The instructions, when read by the computer, further cause the computer to generate second data representing at least one user engagement parameter based on the first data. The instructions, when read by the computer, still further cause the computer to determine a first discrepancy between the first data and the second data, where the first discrepancy indicates a first amount of identifiers that include a first metadata tag associated with the first layer and lack a second metadata tag associated with the monitoring layer.

In still yet another example, a method for identifying data bucket overlap with online experiments implemented on at least one machine including at least one processor, memory, and communications circuitry is described. First data representing a first set of identifiers associated with a first data bucket of a first online experiment may be obtained. Second data representing a second set of identifiers associated with a second data bucket of the first online experiment may be obtained. Based on the first data and the second data, a first number of identifiers that are associated with the first data bucket and the second data bucket may be determined. In response to determining that the first number exceeds a threshold, a data flag may be generated that indicates that results associated with the first online experiment are inconsistent.

In still further yet another example, a system for identifying data bucket overlap with online experiments is described. The system includes a user identifier extraction system configured to obtain first data representing a first set of identifiers associated with a first data bucket of a first online experiment, and obtain second data representing a second set of identifiers associated with a second data bucket of the first online experiment. The system further includes a user identification comparison system configured to determine, based on the first data and the second data, a first number of identifiers that are associated with the first data bucket and the second data bucket. The system still further includes a data bucket abnormality system configured to generate, in response to determining that the first number exceeds a threshold, a data flag indicating that the results associated with the first online experiment are inconsistent.

Other concepts relate to software for implementing the present teaching on identifying data bucket overlap with online experiments. A software product, in accord with this concept, includes at least one machine-readable non-transitory medium and information carried by the medium. The information carried by the medium may be executable program code data, parameters in association with the executable program code, and/or information related to a user, a request, content, or information related to a social group, etc.

In one example, a non-transitory computer readable medium having information recorded thereon for identifying data bucket overlap with online experiments. The information, when read by a computer, causes the computer to obtain first data representing a first set of identifiers associated with a first data bucket of a first online experiment, and obtain second data representing a second set of identifiers associated with a second data bucket of the first online experiment. The information, when read by the computer, further causes the computer determine, based on the first data and the second data, a first number of identifiers that are associated with the first data bucket and the second data bucket. The information, when read by the computer, still further causes the computer to generate, in response to determining that the first number exceeds a threshold, a data flag indicating that results associated with the first online experiment are inconsistent.

Additional novel features will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings or may be learned by production or operation of the examples. The novel features of the present teachings may be realized and attained by practice or use of various aspects of the methodologies, instrumentalities and combinations set forth in the detailed examples discussed below.

