<jupyter_start><jupyter_text>Metrics with elandUse the [`eland` library](https://eland.readthedocs.io) to generate search metrics based on data in an Elasticsearch index. All indices and transforms should have already been created and run before this notebook can be used. We operate on the two main indices: events in `ecs-search-metrics` and the post-transform query-level metrics in `ecs-search-metrics_transform_queryid`.<jupyter_code>%matplotlib inline

import eland as el
import numpy as np
ES_URL = 'http://localhost:9200/'<jupyter_output><empty_output><jupyter_text>Data Loading and Preparation  Event IndexThe `ecs-search-metrics` index contains the raw behavioural events: query, page and click.<jupyter_code>df = el.read_es(ES_URL, 'ecs-search-metrics')
df.dtypes
print(df.info_es())
df.head()<jupyter_output><empty_output><jupyter_text>What is the distribution of ranks of results clicked on?<jupyter_code>df['SearchMetrics.click.result.rank'].describe()
df['SearchMetrics.click.result.rank'].hist()<jupyter_output><empty_output><jupyter_text>How many users are in the dataset?<jupyter_code>df['source.user.id'].nunique()<jupyter_output><empty_output><jupyter_text>How many of each event type?<jupyter_code>df['event.action'].value_counts()<jupyter_output><empty_output><jupyter_text>Split dataset into two dataframes based on action type.<jupyter_code>df_queries = df[df['event.action'] == 'SearchMetrics.query']
df_pages = df[df['event.action'] == 'SearchMetrics.page']
df_clicks = df[df['event.action'] == 'SearchMetrics.click']<jupyter_output><empty_output><jupyter_text>What is the distribution of search result sizes in query events?<jupyter_code>df_queries[['SearchMetrics.results.size']].hist(figsize=[10,5], bins=10)<jupyter_output><empty_output><jupyter_text>Metrics IndexThe `ecs-search-metrics_transform_queryid` index is used to store per-query metrics.<jupyter_code>df_tf_query = el.read_es(ES_URL, 'ecs-search-metrics_transform_queryid')
df_tf_query.head()<jupyter_output><empty_output><jupyter_text>What are the distributions of the numeric fields?<jupyter_code>df_tf_query.select_dtypes(include=[np.number])\
  .drop(['query_event.SearchMetrics.query.page'], axis=1)\
  .hist(figsize=[17,11], bins=10)<jupyter_output><empty_output><jupyter_text>MetricsWe're going to start with some basic query metrics [1] (no session metrics):* Zero Result Rate: The fraction of queries for which no results were returned.* Abandonment Rate^: The fraction of queries for which no results were clicked on.* Clicks per Query^: The mean number of results that are clicked for each query.* Max Reciprocal Rank^†: The mean value of *1/r*, where *r* is the rank of the highest ranked result clicked on.* Mean Reciprocal Rank^†: The mean value of *Σ 1/rᵢ*, summing over the ranks *rᵢ* of all clicks for each query.* Time to First Click^†: The mean time from query being issued until first click on any result.* Time to Last Click^†: The mean time from query being issued until last click on any result.^ When computing the metrics marked with ^, we exclude queries with no results to avoid conflating these measures with zero result rate.  † When computing the metrics marked with †, we exclude queries with no clicks to avoid conflating these measures with abandonment rate.[1] <NAME>, <NAME>, <NAME>. [*How Does Clickthrough Data Reflect Retrieval Quality?*](https://dl.acm.org/citation.cfm?id=1458092). CIKM '08, 2008. Given the above definitions, build datasets to base metrics off of.<jupyter_code># queries that have no results
df_tf_query_without_results = df_tf_query[df_tf_query['query_event.SearchMetrics.results.size'] == 0]

# queries that have results
df_tf_query_with_results = df_tf_query[df_tf_query['query_event.SearchMetrics.results.size'] > 0]

# queries that have results but no clicks
df_tf_query_without_clicks = df_tf_query_with_results[df_tf_query_with_results['metrics.clicks.count'] == 0]

# queries that have results and clicks
df_tf_query_with_clicks = df_tf_query_with_results[df_tf_query_with_results['metrics.clicks.count'] > 0]<jupyter_output><empty_output><jupyter_text>Provide basic counts for all datasets.<jupyter_code>num_queries = df_tf_query.shape[0]
num_queries_without_results = df_tf_query_without_results.shape[0]
num_queries_with_results = df_tf_query_with_results.shape[0]
num_queries_without_clicks = df_tf_query_without_clicks.shape[0]
num_queries_with_clicks = df_tf_query_with_clicks.shape[0]<jupyter_output><empty_output><jupyter_text>Zero Result RateLower is better.  Best possible value: 0.0%<jupyter_code>zero_result_rate = num_queries_without_results / num_queries * 100

print(f"Zero result rate: {round(zero_result_rate, 2)}%")<jupyter_output>Zero result rate: 9.38%<jupyter_text>Abandonment RateLower is better.  Best possible value: 0.0%<jupyter_code>abandonment_rate = num_queries_without_clicks / num_queries * 100

print(f"Abandonment rate: {round(abandonment_rate, 2)}%")<jupyter_output>Abandonment rate: 12.5%<jupyter_text>Clicks per QueryLower is better.  Best possible value: 1.0<jupyter_code>mean_clicks_per_query = df_tf_query_with_results['metrics.clicks.count'].mean()

print(f"Clicks per Query: {round(mean_clicks_per_query, 2)}")<jupyter_output>Clicks per Query: 2.9<jupyter_text>Click Through Rate at Position 3 (CTR@3)Higher is better.  Best possible value: 1.0<jupyter_code>num_queries_with_clicks_at_3 = df_tf_query_with_clicks[df_tf_query_with_clicks['metrics.clicks.exist_at_3'] == True].shape[0]
ctr_at_3 = num_queries_with_clicks_at_3 / num_queries_with_clicks

print(f"CTR@3: {round(ctr_at_3, 2)}")<jupyter_output>CTR@3: 0.56<jupyter_text>Max Reciprocal RankHigher is better.  Best possible value: 1.0<jupyter_code>max_reciprocal_rank = df_tf_query_with_clicks['metrics.clicks.max_reciprocal_rank'].mean()

print(f"Max Reciprocal Rank: {round(max_reciprocal_rank, 2)}")<jupyter_output>Max Reciprocal Rank: 0.46<jupyter_text>Mean Reciprocal RankHigher is better.  Best possible value: 1.0<jupyter_code>mean_mean_reciprocal_rank = df_tf_query_with_clicks['metrics.clicks.mean_reciprocal_rank'].mean()

print(f"Mean Per-Query Mean Reciprocal Rank: {round(mean_mean_reciprocal_rank, 2)}")<jupyter_output>Mean Per-Query Mean Reciprocal Rank: 0.28<jupyter_text>Time to First ClickLower is better.  Best possible value: 0.0<jupyter_code>time_to_first_click = df_tf_query_with_clicks['metrics.clicks.time_to_first_click'].mean()

print(f"Time to First Click: {round(time_to_first_click / 1000, 2)} seconds")<jupyter_output>Time to First Click: 21.2 seconds<jupyter_text>Time to Last ClickHigher is better (infers many interesting results).<jupyter_code>time_to_last_click = df_tf_query_with_clicks['metrics.clicks.time_to_last_click'].mean()

print(f"Time to Last Click: {round(time_to_last_click / 1000, 2)} seconds")<jupyter_output>Time to Last Click: 50.52 seconds
{'fraction_non_alphanumeric': 0.10371019573473561, 'fraction_numerical': 0.016359918200409, 'mean_word_length': 7.892207792207792, 'pattern_counts': {'":': 0, '<': 72, '<?xml version=': 0, '>': 74, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '570592', 'license': 'Apache-2.0', 'n_tokens_mistral': 2443, 'n_tokens_neox': 2327, 'n_words': 552, 'path': 'Machine Learning/Online Search Relevance Metrics/notebooks/Metrics with eland.ipynb', 'repo_name': 'zhuohuwu0603/elastic-examples', 'size': '6859'}