<commit_before># Universal Search Metrics

A summary of the metrics the Universal Search add-on and recommendation server will record.

**Definitions:**

- **URL bar** — the unified address + search bar at the top of the browser.
- **Popup** — the popup that appears when you begin typing into the URL bar.
- **Search** — the full lifecycle, beginning when the user types a key into the url bar, and ending when the user either chooses an item from or otherwise dismisses the popup.
- **Recommendation** - the recommendation UI inserted into the popup by this add-on.

## Data analysis

The collected data will primarily be used to answer the following questions. Images are used for visualization and are not composed of actual data.


### Key performance indicators


#### Recommendation usage rates

_When a result is chosen, how often is it a recommendation? How often do users choose the different types of recommendations?_

This will allow us to understand the overall effectiveness of our recommendations along with how the different types perform so that we can decide what to show and when to show it.

![](images/kpi_1.png)


#### User retention

_Do users continue to use recommendations? As we improve the recommendations, are they more likely to stick around?_

This will allow us to understand overall retention. A user is considered retained if they use at least one recommendation each week.

![](images/kpi_2.png)


### Additional analysis


#### Result types

_What are the most common uses of the Awesome Bar?_

This will give us a better understanding of how people are using the Awesome Bar now so that we can see how recommendations affect that. It will also allow us to devise new tests in the future.

![](images/addtl_1.png)


#### Rich data

_Does richer data entice users to click more often?_

We want to validate that rich data (icons and images) makes it easier for users to choose a recommendation. By tracking click through rates based on the amount of rich data, we can get an understanding of which data types are most impactful.

![](images/addtl_2.png)


## Data collection

Metrics gathered from the Universal Search add-on will be reported to the Test Pilot add-on, which will augment that information and relay it to Firefox's telemetry system via `submitExternalPing`. This is not currently implemented on the Test Pilot side; those efforts are encompassed by [issue #234](https://github.com/mozilla/testpilot/issues/234).

### On the client

#### The popup lifecycle

- The popup usually opens when the user types in the urlbar. It also opens in some less common circumstances, like clicking the history drop marker (the small down caret to the right of the urlbar).
- The popup closes when the user navigates or dismisses the popup without navigating.
  - The popup can be dismissed without navigating by hitting the 'Escape', left arrow, or right arrow keys.
  - The user can navigate by clicking a result in the popup, or by using the arrow or Tab keys to highlight a result in the popup, then hitting Enter.
  - The user can also navigate by typing in the urlbar and hitting Enter, without ever interacting with the popup.

#### Measuring the popup lifecycle

Each time the popup is closed, we record some facts about Firefox's state:

- `didNavigate`: was the popup closed because the user navigated, or because the popup was dismissed without navigating (for example, using the Escape key)?
- `interactionType`: if the user did navigate, was the selected item clicked or keyed?
- `recommendationShown`: was a recommendation visible when the popup was closed?
- `recommendationType`: if a recommendation was shown, was it a wikipedia or a top-level domain recommendation?
  - These types will grow over time, as we add new kinds of recommendations.
- `recommendationSelected`: if the user navigated, and if the recommendation was visible, did the user navigate to the recommendation URL?
- `selectedIndex`: if the user navigated to an item in the popup, which item was selected?
  - For instance, if the third non-recommendation result was clicked, `selectedIndex` would have a value of 2.
  - If the recommendation was selected, the `selectedIndex` value will be -1.

Here's a rough schema:

```js
{
  didNavigate: true or false
  interactionType: 'click' or 'key'
  recommendationShown: true or false
  recommendationSelected: true or false
	recommendationType: 'tld' or 'wikipedia'
  selectedIndex: an integer between -1 and 30 inclusive
}
```

Here's an example of the `payload` portion of a Test Pilot telemetry ping:

```js
{
	"test": "<EMAIL>",  // The em:id field from the add-on
	"agent": "User Agent String",
	"payload": {
		"didNavigate": true,
		"interactionType": "click",
		"recommendationShown": true,
		"recommendationType": "tld",
		"recommendationSelected": true,
		"selectedIndex": -1
	}
}
```

A Redshift schema for the payload:

```js
local schema = {
--   column name                   field type   length  attributes   field name
    {"timestamp",                  "TIMESTAMP", nil,    "SORTKEY",   "Timestamp"},
    {"uuid",                       "VARCHAR",   36,      nil,         get_uuid},

    {"test",                       "VARCHAR",   255,     nil,         "test"},
    {"agent",                      "VARCHAR",   45,      nil,         "agent"},
    {"didNavigate",                "BOOLEAN",   nil,     nil,         "payload[didNavigate]"},
    {"interactionType",            "VARCHAR",   255,     nil,         "payload[interactionType]"},
    {"recommendationShown",        "BOOLEAN",   nil,     nil,         "payload[recommendationShown]"},
    {"recommendationType",         "VARCHAR",   255,     nil,         "payload[recommendationType]"},
    {"recommendationSelected",     "BOOLEAN",   nil,     nil,         "payload[recommendationSelected]"},
    {"selectedIndex",              "INTEGER",   nil,     nil,         "payload[selectedIndex]"}
}
```


#### Data Analysis

We will use this data to calculate click-through rates (CTR) for the popup as shown below.

- CTR of the popup overall: number of events with `didNavigate = true` / total event count
- CTR when recommendations are shown: (number of events with `didNavigate = true` and `recommendationShown = true`) / (total events with `recommendationShown = true`)
- CTR when recommendations are not shown: (number of events with `didNavigate = true` and `recommendationShown = false`) / (total events with `recommendationShown = false`)
- CTR when wikipedia recommendations are shown: (number of events with `didNavigate = true` and `recommendationType = wikipedia`) / (total events with `recommendationType = wikipedia`)
- CTR when TLD recommendations are shown: (number of events with `didNavigate = true` and `recommendationType = tld`) / (total events with `recommendationType = tld`)
- When users navigate via the popup, how often do they use the mouse: (number of events with `didNavigate = true` and `interactionType = click`) / (total events with `didNavigate = true`)
- When users navigate via the popup, how often do they use the keyboard: (number of events with `didNavigate = true` and `interactionType = key`) / (total events with `didNavigate = true`)
- CTR for the nth result down the list: (number of events with `didNavigate = true` and `selectedIndex == n`) / (total events with `didNavigate = true`)
  - We'll use this formula to generate a histogram for the various values of `selectedIndex`.

By using CTR as a proxy for helpfulness, we can use these CTRs to measure whether recommendations are helpful overall, and which types are most or least helpful.

### On the server
<commit_msg>agent field is broken out in redshift<commit_after># Universal Search Metrics

A summary of the metrics the Universal Search add-on and recommendation server will record.

**Definitions:**

- **URL bar** — the unified address + search bar at the top of the browser.
- **Popup** — the popup that appears when you begin typing into the URL bar.
- **Search** — the full lifecycle, beginning when the user types a key into the url bar, and ending when the user either chooses an item from or otherwise dismisses the popup.
- **Recommendation** - the recommendation UI inserted into the popup by this add-on.

## Data analysis

The collected data will primarily be used to answer the following questions. Images are used for visualization and are not composed of actual data.


### Key performance indicators


#### Recommendation usage rates

_When a result is chosen, how often is it a recommendation? How often do users choose the different types of recommendations?_

This will allow us to understand the overall effectiveness of our recommendations along with how the different types perform so that we can decide what to show and when to show it.

![](images/kpi_1.png)


#### User retention

_Do users continue to use recommendations? As we improve the recommendations, are they more likely to stick around?_

This will allow us to understand overall retention. A user is considered retained if they use at least one recommendation each week.

![](images/kpi_2.png)


### Additional analysis


#### Result types

_What are the most common uses of the Awesome Bar?_

This will give us a better understanding of how people are using the Awesome Bar now so that we can see how recommendations affect that. It will also allow us to devise new tests in the future.

![](images/addtl_1.png)


#### Rich data

_Does richer data entice users to click more often?_

We want to validate that rich data (icons and images) makes it easier for users to choose a recommendation. By tracking click through rates based on the amount of rich data, we can get an understanding of which data types are most impactful.

![](images/addtl_2.png)


## Data collection

Metrics gathered from the Universal Search add-on will be reported to the Test Pilot add-on, which will augment that information and relay it to Firefox's telemetry system via `submitExternalPing`. This is not currently implemented on the Test Pilot side; those efforts are encompassed by [issue #234](https://github.com/mozilla/testpilot/issues/234).

### On the client

#### The popup lifecycle

- The popup usually opens when the user types in the urlbar. It also opens in some less common circumstances, like clicking the history drop marker (the small down caret to the right of the urlbar).
- The popup closes when the user navigates or dismisses the popup without navigating.
  - The popup can be dismissed without navigating by hitting the 'Escape', left arrow, or right arrow keys.
  - The user can navigate by clicking a result in the popup, or by using the arrow or Tab keys to highlight a result in the popup, then hitting Enter.
  - The user can also navigate by typing in the urlbar and hitting Enter, without ever interacting with the popup.

#### Measuring the popup lifecycle

Each time the popup is closed, we record some facts about Firefox's state:

- `didNavigate`: was the popup closed because the user navigated, or because the popup was dismissed without navigating (for example, using the Escape key)?
- `interactionType`: if the user did navigate, was the selected item clicked or keyed?
- `recommendationShown`: was a recommendation visible when the popup was closed?
- `recommendationType`: if a recommendation was shown, was it a wikipedia or a top-level domain recommendation?
  - These types will grow over time, as we add new kinds of recommendations.
- `recommendationSelected`: if the user navigated, and if the recommendation was visible, did the user navigate to the recommendation URL?
- `selectedIndex`: if the user navigated to an item in the popup, which item was selected?
  - For instance, if the third non-recommendation result was clicked, `selectedIndex` would have a value of 2.
  - If the recommendation was selected, the `selectedIndex` value will be -1.

Here's a rough schema:

```js
{
  didNavigate: true or false
  interactionType: 'click' or 'key'
  recommendationShown: true or false
  recommendationSelected: true or false
	recommendationType: 'tld' or 'wikipedia'
  selectedIndex: an integer between -1 and 30 inclusive
}
```

Here's an example of the `payload` portion of a Test Pilot telemetry ping:

```js
{
	"test": "<EMAIL>",  // The em:id field from the add-on
	"agent": "User Agent String",
	"payload": {
		"didNavigate": true,
		"interactionType": "click",
		"recommendationShown": true,
		"recommendationType": "tld",
		"recommendationSelected": true,
		"selectedIndex": -1
	}
}
```

A Redshift schema for the payload:

```js
local schema = {
--   column name                   field type   length  attributes   field name
    {"timestamp",                  "TIMESTAMP", nil,    "SORTKEY",   "Timestamp"},
    {"uuid",                       "VARCHAR",   36,      nil,         get_uuid},

    {"test",                       "VARCHAR",   255,     nil,         "test"},

    -- Parsed automatically from the `agent` field
    {"user_agent_browser",         "VARCHAR",   255,     nil,         "Fields[user_agent_browser]"},
    {"user_agent_os",              "VARCHAR",   255,     nil,         "Fields[user_agent_os]"},
    {"user_agent_version",         "VARCHAR",   255,     nil,         "Fields[user_agent_version]"},

    {"didNavigate",                "BOOLEAN",   nil,     nil,         "payload[didNavigate]"},
    {"interactionType",            "VARCHAR",   255,     nil,         "payload[interactionType]"},
    {"recommendationShown",        "BOOLEAN",   nil,     nil,         "payload[recommendationShown]"},
    {"recommendationType",         "VARCHAR",   255,     nil,         "payload[recommendationType]"},
    {"recommendationSelected",     "BOOLEAN",   nil,     nil,         "payload[recommendationSelected]"},
    {"selectedIndex",              "INTEGER",   nil,     nil,         "payload[selectedIndex]"}
}
```


#### Data Analysis

We will use this data to calculate click-through rates (CTR) for the popup as shown below.

- CTR of the popup overall: number of events with `didNavigate = true` / total event count
- CTR when recommendations are shown: (number of events with `didNavigate = true` and `recommendationShown = true`) / (total events with `recommendationShown = true`)
- CTR when recommendations are not shown: (number of events with `didNavigate = true` and `recommendationShown = false`) / (total events with `recommendationShown = false`)
- CTR when wikipedia recommendations are shown: (number of events with `didNavigate = true` and `recommendationType = wikipedia`) / (total events with `recommendationType = wikipedia`)
- CTR when TLD recommendations are shown: (number of events with `didNavigate = true` and `recommendationType = tld`) / (total events with `recommendationType = tld`)
- When users navigate via the popup, how often do they use the mouse: (number of events with `didNavigate = true` and `interactionType = click`) / (total events with `didNavigate = true`)
- When users navigate via the popup, how often do they use the keyboard: (number of events with `didNavigate = true` and `interactionType = key`) / (total events with `didNavigate = true`)
- CTR for the nth result down the list: (number of events with `didNavigate = true` and `selectedIndex == n`) / (total events with `didNavigate = true`)
  - We'll use this formula to generate a histogram for the various values of `selectedIndex`.

By using CTR as a proxy for helpfulness, we can use these CTRs to measure whether recommendations are helpful overall, and which types are most or least helpful.

### On the server

{'fraction_non_alphanumeric': 0.08070828254314492, 'fraction_numerical': 0.004170141784820684, 'mean_word_length': 3.685302073940487, 'pattern_counts': {'":': 18, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2077532', 'n_tokens_mistral': 4245, 'n_tokens_neox': 3963, 'n_words': 2052}