<jupyter_start><jupyter_text>Aggregating Axiom Preferences for Learning to RankThe notebook below exemplifies how `ir_axioms` can be used to generate features for arbitrary learning-to-rank approaches like LambdMART in PyTerrier by aggregating pairwise preferences from different axioms.We use run files and qrels from the passage retrieval task of the TREC Deep Learning track in 2019 and 2020 as example (using BM25 as a baseline).In this notebook, we first generate learning-to-rank features by aggregating preferences from 2019 qrels and topics. Then we train a LambdaMART re-ranker with the generated axiomatic features, re-rank using the trained LambdaMART re-ranker, and evaluate nDCG@10, reciprocal rank, and average precision for the baseline and the re-ranked pipeline using PyTerrier's standard `Experiment` functionality.  PreparationInstall the `ir_axioms` framework and [PyTerrier](https://github.com/terrier-org/pyterrier). In Google Colab, we do this automatically.<jupyter_code>from sys import modules

if 'google.colab' in modules:
    !pip install -q ir_axioms[examples] python-terrier<jupyter_output><empty_output><jupyter_text>We initialize PyTerrier and import all required libraries and load the data from [ir_datasets](https://ir-datasets.com/).<jupyter_code>from pyterrier import started, init

if not started():
    init(tqdm="auto")<jupyter_output>PyTerrier 0.8.0 has loaded Terrier 5.6 (built by craigmacdonald on 2021-09-17 13:27)

No etc/terrier.properties, using terrier.default.properties for bootstrap configuration.<jupyter_text>Datasets and IndexUsing PyTerrier's `get_dataset()`, we load the MS MARCO passage ranking dataset.<jupyter_code>from pyterrier.datasets import get_dataset, Dataset

# Load dataset.
dataset_name = "msmarco-passage"
dataset: Dataset = get_dataset(f"irds:{dataset_name}")
dataset_train: Dataset = get_dataset(f"irds:{dataset_name}/trec-dl-2019/judged")
dataset_test: Dataset = get_dataset(f"irds:{dataset_name}/trec-dl-2020/judged")<jupyter_output><empty_output><jupyter_text>Now define paths where we will store temporary files, datasets, and the search index.<jupyter_code>from pathlib import Path

cache_dir = Path("cache/")
index_dir = cache_dir / "indices" / dataset_name.split("/")[0]<jupyter_output><empty_output><jupyter_text>If the index is not ready yet, now is a good time to create it and index the MS MARCO passages.(Lean back and relax as this may take a while...)<jupyter_code>from pyterrier.index import IterDictIndexer

if not index_dir.exists():
    indexer = IterDictIndexer(str(index_dir.absolute()))
    indexer.index(
        dataset.get_corpus_iter(),
        fields=["text"]
    )<jupyter_output><empty_output><jupyter_text>Baseline RunWe use PyTerrier's `BatchRetrieve` to create a baseline search pipeline for retrieving with BM25 from the index we just created.<jupyter_code>from pyterrier.batchretrieve import BatchRetrieve

bm25 = BatchRetrieve(str(index_dir.absolute()), wmodel="BM25")<jupyter_output><empty_output><jupyter_text>Import AxiomsHere we're listing which axioms we want to use in our experiments.Because some axioms require API calls or are computationally expensive, we cache all axioms using `ir_axiom`'s tilde operator (`~`).<jupyter_code>from ir_axioms.axiom import (
    ArgUC, QTArg, QTPArg, aSL, PROX1, PROX2, PROX3, PROX4, PROX5, TFC1, TFC3, RS_TF, RS_TF_IDF, RS_BM25, RS_PL2, RS_QL,
    AND, LEN_AND, M_AND, LEN_M_AND, DIV, LEN_DIV, M_TDC, LEN_M_TDC, STMC1, STMC1_f, STMC2, STMC2_f, LNC1, TF_LNC, LB1,
    REG, ANTI_REG, REG_f, ANTI_REG_f, ASPECT_REG, ASPECT_REG_f, ORIG
)

axioms = [
    ~ArgUC(), ~QTArg(), ~QTPArg(), ~aSL(),
    ~LNC1(), ~TF_LNC(), ~LB1(),
    ~PROX1(), ~PROX2(), ~PROX3(), ~PROX4(), ~PROX5(),
    ~REG(), ~REG_f(), ~ANTI_REG(), ~ANTI_REG_f(), ~ASPECT_REG(), ~ASPECT_REG_f(),
    ~AND(), ~LEN_AND(), ~M_AND(), ~LEN_M_AND(), ~DIV(), ~LEN_DIV(),
    ~RS_TF(), ~RS_TF_IDF(), ~RS_BM25(), ~RS_PL2(), ~RS_QL(),
    ~TFC1(), ~TFC3(), ~M_TDC(), ~LEN_M_TDC(),
    ~STMC1(), ~STMC1_f(), ~STMC2(), ~STMC2_f(),
    ORIG()
]<jupyter_output><empty_output><jupyter_text>To reduce the preference matrices to lists of three features, we define three aggregation functions.In this exemplary aggregations, we just count how often an axiom wants to change the original ordering in either direction.<jupyter_code>aggregations = [
    lambda prefs: sum(p >= 0 for p in prefs) / len(prefs),
    lambda prefs: sum(p == 0 for p in prefs) / len(prefs),
    lambda prefs: sum(p <= 0 for p in prefs) / len(prefs),
]<jupyter_output><empty_output><jupyter_text>Aggregating Axiomatic FeaturesWith the axioms and aggregation functions, we create axiomatic preference features for the top-20 results from the baseline ranking.<jupyter_code>from ir_axioms.backend.pyterrier.transformers import AggregatedAxiomaticPreferences

features = bm25 % 20 >> AggregatedAxiomaticPreferences(
    axioms=axioms,
    index=index_dir,
    aggregations=aggregations,
    dataset=dataset_name,
    cache_dir=cache_dir,
    verbose=True,
)<jupyter_output><empty_output><jupyter_text>This example shows how the features look like for the first topic of the training dataset.<jupyter_code>features.transform(dataset_train.get_topics()[:1])["features"]<jupyter_output><empty_output><jupyter_text>Learning to RankAfter aggregating the preference features, we can use the features with any learning-to-rank approach.In this case, we initialize a LambdaMART ranker for optimizing nDCG@10 and apply it to our PyTerrier pipeline.<jupyter_code>from lightgbm import LGBMRanker
from pyterrier.ltr import apply_learned_model

lambda_mart = LGBMRanker(
    num_iterations=1000,
    metric="ndcg",
    eval_at=[10],
    importance_type="gain",
)
ltr = features >> apply_learned_model(lambda_mart, form="ltr")<jupyter_output><empty_output><jupyter_text>We train the LamdaMART ranker with the training dataset (using the last 5 topics as the validation dataset).<jupyter_code>ltr.fit(
    dataset_train.get_topics()[:-5],
    dataset_train.get_qrels(),
    dataset_train.get_topics()[-5:],
    dataset_train.get_qrels()
)<jupyter_output><empty_output><jupyter_text>Experimental EvaluationBecause our axiomatic re-rankers are PyTerrier modules, we can now use PyTerrier's `Experiment` interface to evaluate various metrics and to compare our new approach to the BM25 baseline ranking.Refer to the PyTerrier [documentation](https://pyterrier.readthedocs.io/en/latest/experiments.html) to learn more about running experiments.(We concatenate results from the Baseline ranking for the ranking positions after the top-20 using the `^` operator.)<jupyter_code>from pyterrier.pipelines import Experiment
from ir_measures import nDCG, MAP, RR

experiment = Experiment(
    [bm25, ltr ^ bm25],
    dataset_test.get_topics(),
    dataset_test.get_qrels(),
    [nDCG @ 10, RR, MAP],
    ["BM25", "Axiomatic LTR"],
    verbose=True,
)
experiment.sort_values(by="nDCG@10", ascending=False, inplace=True)
experiment<jupyter_output><empty_output><jupyter_text>Extra: Feature ImportancesInspecting the feature importances from LambdaMART can help to identify axioms or aggregations that are not used for re-ranking.If an axiom's feature importance is zero for most of your applications, you may consider omitting it from the ranking pipeline.<jupyter_code>from numpy import ndarray

feature_importance: ndarray = lambda_mart.feature_importances_.reshape(-1, len(aggregations))
feature_importance
feature_importance.sum(0)
feature_importance.sum(1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09992027637523253, 'fraction_numerical': 0.015678979537602975, 'mean_word_length': 6.060975609756097, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 61, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '266273', 'license': 'MIT', 'n_tokens_mistral': 2590, 'n_tokens_neox': 2434, 'n_words': 718, 'path': 'examples/pyterrier_ltr_features.ipynb', 'repo_name': 'webis-de/ir_axioms', 'size': '7526'}