<jupyter_start><jupyter_text>Homework 2: Learning to Rank  **Submission instructions**:- Only the code `TODO: Implement this!` denotes that these sections are graded. Please do not delete the comment YOUR CODE HERE.- Vectoriize your code instead of using for-loop when implementing Neural Nets.- The notebook you submit has to have the student ids, separated by underscores (E.g., `12341234_12341234_12341234.ipynb`). - This will be parsed by a regexp, so please double check your filename.- Only one member of each group has to submit the file to canvas.- Make sure to check that your notebook runs before submission. A quick way to do this is to restart the kernel and run all the cells.  - Please do not delete/add new cells. Removing cells can lead to grade deduction. Also do not change the number of parameters in the pre-defined functions.- Note, that you are not allowed to use Google Colab.**Learning Goals**:- Part 1: Offline LTR  - Learn how to implement pointwise, pairwise and listwise algorithms for learning to rank - Part 2: Online LTR  - Implement learning to rank algorithms from historical clicks and online evaluation of ranking algorithms.- Learn their weaknesses & strengths and when each method is suitable. **Files to submit along with the completed notebook**:- `pointwise_regression.json`- `pointwise_classification.json`- `pairwise.json`- `listwise.json`- `biased_model.json'- `unbiased_model.json'---**Recommended Reading**:- Part 1:  - <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, and <NAME>. Learning to rank using gradient descent. InProceedings of the 22nd international conference on Machine learning, pages 89–96, 2005.  - <NAME>, <NAME>, and <NAME>. Learning to rank with nonsmooth cost functions. In Advances inneural information processing systems, pages 193–200, 2007  - (Sections 1, 2 and 4) Christopher JC Burges. From ranknet to lambdarank to lambdamart: An overview. Learning, 11(23-581):81, 2010  Additional Resources: - This assignment requires knowledge of [PyTorch](https://pytorch.org/). If you are unfamiliar with PyTorch, you can go over [these series of tutorials](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html)In the previous assignment, you experimented with retrieval with different ranking functions and in addition, different document representations. This assignment deals directly with learning to rank (LTR). In offline LTR (Part 1), You will learn how to implement methods from the three approaches associated with learning to rank: pointwise, pairwise and listwise. In Part 2, you will learn about online LTR. Instead of using manually judged datasets, in online LTR, we learn from user interactions. You will learn how to simulate clicks using click models, how to learn unbiasedly from historical clicks and how to evaluate different rankers in an online environment using multileaving methods. **Note:**  - The dataset used in this assignment is +100Mb in size. You may need around 2Gb of RAM for running the whole notebook.  Table of Contents  [Back to top](toptop)Table of contents: - [Chapter 1: Offline LTR](o_LTR) (345 points)     - [Section 1: Dataset and Utility](dataU)      - [Section 2: Pointwtise LTR](pointwiseLTR) (55 points)     - [Section 3: Pairwise LTR](pairwiseLTR) (60 points)     - [Section 4: Pairwise Speed-up RankNet](SpairwiseLTR) (70 points)     - [Section 5: Listwise LTR](listwiseLTR) (80 points)     - [Section 6: Evaluation](evaluation1) (70 points) - [Chapter 2: Online LTR](onLTR) (180 points)     - [Section 1: Clicks Simulation](clicks) (15 points)     - [Section 2: Counterfactual](cLTR) (90 points)     - [Section 3: Online Evaluation](on_eval) (75 points)<jupyter_code>import os
import json
import itertools
from argparse import Namespace
from collections import OrderedDict
from functools import partial


import torch
import numpy as np
from torch import nn
from torch.optim import Adam
import torch.nn.functional as F
from tqdm.notebook import tqdm, trange
from torch.utils.data import Dataset, DataLoader


import matplotlib.pyplot as plt
from matplotlib.pyplot import cm

import pandas as pd

import dataset
import evaluate<jupyter_output><empty_output><jupyter_text>Chapter 1: Offline LTR  A typical setup of learning to rank involves a feature vector constructed using a query-document pair, and a set of relevance judgements. You are given a set of triples (`query`, `document`, `relevance grade`); where relevance grade is an *ordinal* variable  with  5  grades,  for example: {`perfect`,`excellent`,`good`,`fair`,`bad`),  typically  labeled  by human annotators.  In this assignment, you are already given the feature vector for a given document and query pair. To access these vectors, see the following code cells (note: the dataset will be automatically downloaded & the first time the next cell runs, it will take a while!)  Section 1: Data and Utility [Back to TOC](top)First let's get familiar with the dataset and some utility methods for our implementations. Section 1.1 Dataset stats| Split Name | \ queries | \ docs | \ features || :- | :--: | :--: | :--: || train | 2735 | 85227 | 501 || validation | 403 | 12794 | 501 || test | 949 | 29881 | 501 |<jupyter_code>dataset.download_dataset()
data = dataset.get_dataset()
# there is only 1 fold for this dataset 
data = data.get_data_folds()[0]
# read in the data
data.read_data()
print(f"Number of features: {data.num_features}")
# print some statistics
for split in ["train", "validation", "test"]:
    print(f"Split: {split}")
    split = getattr(data, split)
    print(f"\tNumber of queries {split.num_queries()}")
    print(f"\tNumber of docs {split.num_docs()}")<jupyter_output>Number of features: 501
Split: train
	Number of queries 2735
	Number of docs 85227
Split: validation
	Number of queries 403
	Number of docs 12794
Split: test
	Number of queries 949
	Number of docs 29881<jupyter_text>Section 1.2 Utility classes/methodsThe following cells contain code that will be useful for the assigment.<jupyter_code># these is a useful class to create torch DataLoaders, and can be used during training
class LTRData(Dataset):
    def __init__(self, data, split):
        split = {
            "train": data.train,
            "validation": data.validation,
            "test": data.test
        }.get(split)
        assert split is not None, "Invalid split!"
        features, labels = split.feature_matrix, split.label_vector
        self.features = torch.FloatTensor(features)
        self.labels = torch.FloatTensor(labels)
    
    def __len__(self):
        return self.features.size(0)

    def __getitem__(self, i):
        return self.features[i], self.labels[i]
## example 
train_dl = DataLoader(LTRData(data, "train"), batch_size=32, shuffle=True)
# this is how you would use it to quickly iterate over the train/val/test sets 
# - (of course, without the break statement!)
for (x, y) in train_dl:
    print(x.size(), y.size())
    break<jupyter_output>torch.Size([32, 501]) torch.Size([32])<jupyter_text>`evaluate_model` evaluates a model, on a given split.<jupyter_code># this function evaluates a model, on a given split
def evaluate_model(pred_fn, split, batch_size=256, print_results=False, q_level=False):
    dl = DataLoader(LTRData(data, split), batch_size=batch_size)
    all_scores = []
    all_labels = []
    for (x, y) in tqdm(dl, desc=f'Eval ({split})', leave=False):
        all_labels.append(y.squeeze().numpy())
        
        with torch.no_grad():
            output = pred_fn(x)
            all_scores.append(output.squeeze().numpy())
            
    split = {
            "train": data.train,
            "validation": data.validation,
            "test": data.test
    }.get(split)   
    results = evaluate.evaluate2(np.asarray(all_scores), np.asarray(all_labels), print_results=print_results, q_level=q_level)

    return results

## example 
# function that scores a given feature vector e.g a network
net = nn.Linear(501, 1)
# the evaluate method accepts a function. more specifically, a callable (such as pytorch modules) 
def notwork(x):
    return net(x)
# evaluate the function
_ = evaluate_model(notwork, "validation", print_results=True)<jupyter_output><empty_output><jupyter_text>The next cell is used to generate reproducible results:<jupyter_code># use to get reproducible results
def seed(random_seed):
    import random
    torch.manual_seed(random_seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    np.random.seed(random_seed)
    random.seed(random_seed)<jupyter_output><empty_output><jupyter_text>Section 2: Pointwise LTR (55 points) [Back to TOC](top)Let $x \in \mathbb{R}^d$ be an input feature vector, containing features for a query-document pair. Let $f: \mathbb{R}^d \rightarrow \mathbb{R} $ be a function that maps this feature vector to a number $f(x)$ - either a relevance score (regression) or label (classification). The data $\{x \}$ are treated as feature vectors and the relevance judgements are treated as the target which we want to predict. In this section, you will implement a simple Pointwise model using either a regression or classification loss, and use the train set to train this model to predict (or classify) the relevance score.   Section 2.1: Neural Model (5 points)In the following cell, you will implement a simple pointwise LTR model: - Use a neural network to learn a Pointwise model using both a regression and a classification loss, using the relevance grades as the label. Use the following parameters:   - Layers: $501 (input) \rightarrow 256 \rightarrow o$ where $o$ is either 5 for classification or 1 for regression, where each layer is a linear layer (`nn.Linear`) with a ReLu activation function (`nn.ReLU`) in between the layers. Use the default weight initialization scheme. (Hint: use `nn.Sequential` for a one-line forward function!)  - Note: Do not use a `nn.Softmax` function here - it will be taken care of later!  - This network will also be used by other methods i.e Pairwise<jupyter_code># TODO: Implement this! (5 points)
class NeuralModule(nn.Module):
    def __init__(self, output_dim):
        """
        Initializes the Pointwise neural network. 
        Input: output_dim: The dimension of the output layer. In this assignment, 
                it is either 1 (regression) or 5 (classification)
        """
        self.output_dim = output_dim
        
        # YOUR CODE HERE
        super(NeuralModule, self).__init__()
        
        self.model = nn.Sequential(
          nn.Linear(501, 256),
          nn.ReLU(),
          nn.Linear(256, self.output_dim),
          nn.ReLU()
        )
    
    def forward(self, x):
        """
        Takes in an input feature vector (of size 501) and produces the (regression/classification) output 
        Input: x: a [N, 501] tensor
        Output: a [N, output_dim] tensor
        """
        # YOUR CODE HERE
        out = self.model(x)
        return out
point_nn_clf = NeuralModule(5)
point_nn_reg = NeuralModule(1)
# YOUR CODE HERE
raise NotImplementedError()
# test the forward function
n = 10
inp = torch.rand(n, data.num_features)
out = point_nn_clf(inp)
### BEGIN HIDDEN TEST
n = 20
inp = torch.rand(n, data.num_features)
out = point_nn_clf(inp)
assert out.size(0) == n
assert out.size(1) == 5
### END HIDDEN TEST
# test the forward function
n = 10
inp = torch.rand(n, data.num_features)
out = point_nn_reg(inp)
### BEGIN HIDDEN TEST
n = 20
inp = torch.rand(n, data.num_features)
out = point_nn_reg(inp)
assert out.size(0) == n
assert out.size(1) == 1
### END HIDDEN TEST<jupyter_output><empty_output><jupyter_text>Section 2.2: Loss Functions (5 points)Pointwise LTR algorithms use pointwise loss functions.Usually, the popular loss functions for pointwise LTR are: - Cross entropy loss for classification (3 points) - Regression loss (2 points) **Implementation (3 points):**Implement cross entropy loss and and then cross entropy prediction.<jupyter_code># TODO: Implement this! (2 points)
def clf_loss(output, target):
    """
    Cross entropy loss - returns a single number. 
    output: (float) tensor, shape - [N, 5] 
    target: (float/long) tensor, shape - [N]. 
    
    Hint: This function should also handle cases when target is either long/float types 
    """
    assert output.size(0) == target.size(0)
    assert output.size(1) == 5
    # YOUR CODE HERE
    criterion = nn.CrossEntropyLoss()
    return criterion(output, target)
## Test clf_loss
g = torch.manual_seed(42)
tests = [torch.rand(5, 5, generator=g) for _ in range(5)]
target = torch.LongTensor([1, 2, 3, 4, 0])

results = [1.5910069942474365, 
           1.600582480430603, 
           1.6723783016204834, 
           1.8065979480743408, 
           1.6385562419891357]

l1 = [clf_loss(output, target).item() for output in tests]
print(f'your results: {l1}')
print(f'expected results: {results}')

3# TODO: Implement this! (1 points)
def clf_pred(inp, net):
    """
    The output of the classifier network produces a [Nx5] output corresponding to 
    the relevance labels (each row does *not* add to 1!)
    This function should predict the most probable relevance from the relevance labels
    
    inp: The input [N, num_features]
    net: the neural network, takes in [N, num_features] and outputs [N, 5]
    
    return: a [N, 1] (long) tensor, the relevance labels  
    """
    # YOUR CODE HERE
    outputs = net(inp)
    _, predicted = torch.max(outputs, 1)
    return predicted 
g = torch.manual_seed(42)
def clf_(inp):
    return torch.rand(inp.size(0), 5, generator=g)

inp = torch.rand(10, 5, generator=g)
r = np.array([3, 3, 2, 1, 2, 4, 4, 1, 3, 1])
p = clf_pred(inp, clf_).numpy()
print(f'your results:{p}')
print(f'expected:{r}')

# NOTE:
# to construct a predictor for a particular network, see this example
# (this will be required in the next cell)
clf_pred_fn = partial(clf_pred, net=point_nn_clf)
# the 'net' argument doesn't need to be provided anymore!
clf_pred_fn(torch.rand(5, data.num_features)).numpy()<jupyter_output><empty_output><jupyter_text>**Implementation (2 points):**Implement regression loss.<jupyter_code># TODO: Implement this! (2 points)
def reg_loss(output, target):
    """
    Regression loss - returns a single number. 
    Make sure to use the TODO loss!
    output: (float) tensor, shape - [N, 1] 
    target: (float) tensor, shape - [N]. 
    """
    assert target.dim() == 1
    assert output.size(0) == target.size(0)
    assert output.size(1) == 1
    
    # YOUR CODE HERE 
    criterion =nn.MSELoss ()
    return criterion (output.squeeze(), target)
## Test reg_loss
g = torch.manual_seed(42)
output = [torch.randint(low=0, high=5, size=(5, 1), generator=g).float() for _ in range(5)]
target = torch.randint(low=0, high=5, size=(5,), generator=g).float()

l = [reg_loss(o, target).item() for o in output]
r = [4.800000190734863, 
     3.0, 
     7.599999904632568, 
     5.400000095367432, 
     0.6000000238418579]
print(f'your results:{l}')
print(f'expected:{r}')<jupyter_output>your results:[4.800000190734863, 3.0, 7.599999904632568, 5.400000095367432, 0.6000000238418579]
expected:[4.800000190734863, 3.0, 7.599999904632568, 5.400000095367432, 0.6000000238418579]<jupyter_text>**Implementation (45 points):**Now implement a wrapper for training a pointwise LTR, that takes the model and the loss function as input and trains the model.**Rubric:** - Network is trained for specified epochs, and iterates over the entire dataset and (train) data is shuffled : 5 points - Evaluation on the validation set: 5 points - Training (e.g optimizer, zero_grad, backward): 10 points - Appropriate loss function & prediction function: 5 points - Both classification / regression models handled appropriately: 5 points - Performance as expected: 15 points<jupyter_code># TODO: Implement this! (45 points)
def train_pointwise(net, loss, params):
    """
    This function should train a Pointwise network, 
    trained based on the loss (either "clf" / "reg"). 
    
    The network is trained using the Adam optimizer
        
    
    Note: Do not change the function definition! 
    
    
    Hints:
    1. Use the LTRData class defined above
    2. You will have to construct a partial function if loss="clf" 
       before using it in evaluate_model() (see cells after the defn of clf_pred)
    
    net: the neural network to be trained
    
    loss: one of "clf" or "reg"
    
    params: params is an object which contains config used in training 
        (eg. params.epochs - the number of epochs to train). 
        For a full list of these params, see the next cell. 
    
    Returns: a dictionary containing: "metrics_val" (a list of dictionaries) and 
             "metrics_train" (a list of dictionaries). 
             
             "metrics_val" should contain metrics (the metrics in params.metrics) computed
             after each epoch on the validation set (metrics_train is similar). 
             You can use this to debug your models.
    
    """
    
    assert loss in {"clf", "reg"}
    
    val_metrics_epoch = []
    train_metrics_epoch = []
    # YOUR CODE HERE

    optimizer = torch.optim.Adam(net.parameters(), lr = params.lr)
    train_dl = DataLoader(LTRData(data, "train"), batch_size=params.batch_size, shuffle=True)


    for epoch in range(params.epochs): 
        running_loss = 0.0
        for (inputs, labels) in train_dl:  # inputs: 256*501  labels:256*1
            optimizer.zero_grad()
            outputs = net(inputs)       
            if loss == "clf":
                labels = labels.long() #convert float tensor to long tensor
                myloss = clf_loss(outputs, labels)
            elif loss == "reg":
                myloss = reg_loss (outputs, labels)
            myloss.backward()
            optimizer.step()
        train_metrics_epoch.append (evaluate_model(notwork, "train", print_results=False))
        val_metrics_epoch.append (evaluate_model(notwork, "validation", print_results=False))
            
    return {
        "metrics_val": val_metrics_epoch,
        "metrics_train": train_metrics_epoch
    }
# Change this to test your code!
pointwise_test_params = Namespace(epochs=1, 
                    lr=1e-3,
                    batch_size=256,
                    metrics={"ndcg"})
# uncomment to test your code
## train a regression model
met_reg = train_pointwise(point_nn_reg, "reg", pointwise_test_params)
## train a classification model
met_clf = train_pointwise (point_nn_clf, "clf", pointwise_test_params)<jupyter_output>/home/manolis/miniconda3/envs/ir1/lib/python3.9/site-packages/torch/autograd/__init__.py:130: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370175239/work/c10/cuda/CUDAFunctions.cpp:100.)
  Variable._execution_engine.run_backward(<jupyter_text>The next cell is used to generate reproducible results which should be submitted with the assignment:<jupyter_code>def create_results(net, train_fn, prediction_fn, results_file, *train_params):
    
    print("Training Model")
    metrics = train_fn(net, *train_params)
    net.eval()
    test_metrics, test_qq = evaluate_model(prediction_fn, "test", print_results=True, q_level=True)
    
    
    test_q = {}
    for m in {"ndcg", "precision@05", "recall@05"}:
        test_q[m] = test_qq[m]
    
    with open(results_file, "w") as writer:
        json.dump({
            "metrics": metrics,
            "test_metrics": test_metrics,
            "test_query_level_metrics": test_q,
        }, writer, indent=2)<jupyter_output><empty_output><jupyter_text>Now use the above functions to generate your `json` files for submission:<jupyter_code>seed(42)
params_regr = Namespace(epochs=11, 
                    lr=1e-3,
                    batch_size=256,
                    metrics={"ndcg", "precision@05", "recall@05"})

pointwise_regression_model = NeuralModule(1)
create_results(pointwise_regression_model, 
               train_pointwise, 
               pointwise_regression_model,
               "./pointwise_regression.json", 
               "reg", params_regr)
# persist models
torch.save(pointwise_regression_model.state_dict(), "./pointwise_regr_wt")
seed(42)
params_clf = Namespace(epochs=13, 
                    lr=1e-3,
                    batch_size=256,
                    metrics={"ndcg", "precision@05", "recall@05"})

pointwise_classification_model = NeuralModule(5)
create_results(pointwise_classification_model,
               train_pointwise,
               partial(clf_pred, net=pointwise_classification_model),
               "./pointwise_classification.json", "clf", params_clf)

torch.save(pointwise_classification_model.state_dict(), "./pointwise_clf_wt")<jupyter_output>Training Model<jupyter_text>YOUR ANSWER HERE<jupyter_code>assert os.path.exists("./pointwise_regression.json")
assert os.path.exists("./pointwise_classification.json")<jupyter_output><empty_output><jupyter_text>Section 3: Pairwise LTR (60 points) [Back to TOC](top)In this section,  you will learn and implement RankNet, a  pairwise learning to rank algorithm.For a given query, consider two documents $D_i$ and $D_j$ with two different ground truth relevance  labels,  with  feature  vectors $x_i$ and $x_j$ respectively.   The  RankNet  model,  just  like  the pointwise model, uses $f$ to predict scores i.e $s_i=f(x_i)$ and $s_j=f(x_j)$, but uses a different loss during  training. $D_i \triangleright D_j$ denotes  the  event  that $D_i$ should  be  ranked  higher  than $D_j$.   The  two outputs $s_i$ and $s_j$ are mapped to a learned probability that $D_i \triangleright D_j$: $$        P_{ij} = \frac{1}{1 + e^{-\sigma(s_i - s_j)}} $$  where $\sigma$ is a parameter that determines the shape of the sigmoid. The loss of the RankNet model is the cross entropy cost function:$$        C = - \bar{P}_{ij} \log P_{ij} - (1-\bar{P}_{ij}) \log (1 - P_{ij}) $$As the name suggests, in the pairwise approach to LTR, we optimize a loss $l$ over pairs of documents. Let $S_{ij} \in \{0, \pm1 \}$ be equal to $1$ if the relevance of document $i$ is greater than document $j$; $-1$ if document $j$ is more relevant than document $i$; and 0 if they have the same relevance. This gives us $\bar{P}_{ij} = \frac{1}{2} (1 + S_{ij})$ so that $\bar{P}_{ij} = 1$ if $D_i \triangleright D_j$; $\bar{P}_{ij} = 0$ if $D_j \triangleright D_i$; and finally $\bar{P}_{ij} = \frac{1}{2}$ if the relevance is identical. This gives us:$$        C = \frac{1}{2}(1- S_{ij})\sigma(s_i - s_j) + \log(1+ e^{-\sigma(s_i - s_j)}) $$Now, consider a single query for which $n$ documents have been returned. Let the output scores of the ranker be $s_j$ ; $j=\{1, \dots, n \}$, the model parameters be $w_k \in \mathbb{R}^W$, and let the set of pairs of document indices used for training be $\mathcal{P}$. Then, the total cost is $C_T = \sum_{i,j \in \mathcal{P}} C(s_i; s_j)$. - Implement RankNet. You should construct training samples by creating all possible pairs of documents for a given query and optimizing the loss above. Use the following parameters:  - Layers: $501 (input) \rightarrow 256 \rightarrow 1$, where each layer is a linear layer (`nn.Linear`) with a ReLu activation function (`nn.ReLu`) in between the layers. Use the default weight initialization scheme. (Hint: use `nn.Sequential` for a one-line forward function!) For the pairwise loss, we need to have a structured **dataloader** which detects the documents associated with a specific query:<jupyter_code>class QueryGroupedLTRData(Dataset):
    def __init__(self, data, split):
        self.split = {
            "train": data.train,
            "validation": data.validation,
            "test": data.test
        }.get(split)
        assert self.split is not None, "Invalid split!"
    
    def __len__(self):
        return self.split.num_queries()

    def __getitem__(self, q_i):
        feature = torch.FloatTensor(self.split.query_feat(q_i))
        labels = torch.FloatTensor(self.split.query_labels(q_i))
        return q_i, feature, labels

# the return types are different from what pytorch expects, 
# so we will define a custom collate function which takes in
# a batch and returns tensors (qids, features, labels) 
def qg_collate_fn(batch):
    
    qids = []
    features = []
    labels = []
    
    for (q, f, l) in batch:
        qids.append(q)
        features.append(f)
        labels.append(l)
    
    return qids, features, labels
    
    
## example - NOTE the collate_fn argument!
train_dl = DataLoader(QueryGroupedLTRData(data, "train"), batch_size=1, shuffle=True, collate_fn=qg_collate_fn)
# this is how you would use it to quickly iterate over the train/val/test sets 
for (qids, x, y) in train_dl:
    # different from the previous data loader, qids, x and y aren't tensors, but lists!
    for q_i, features_i, labels_i in zip(qids, x, y):
        print(f"Query {q_i} has {len(features_i)} query-document pairs")
        print(f"Shape of features for Query {q_i}: {features_i.size()}")
        break
    break<jupyter_output>Query 1048 has 16 query-document pairs
Shape of features for Query 1048: torch.Size([16, 501])<jupyter_text>**Implementation (25 points):**First, implement the pairwaise loss, described above.**Rubric:** - Each ordering  combination is considered: 10 points - Proper application of the formula: 10 points - Mean loss: 5 points<jupyter_code># # TODO: Implement this! (25 points)
def pairwise_loss (scores, labels):
    """
    Compute and return the pairwise loss *for a single query*. 
    To compute this, compute the loss for each ordering in a query, 
    and then return the mean. Use sigma=1.
    
    For a query, consider all possible ways of comparing 2 document-query pairs.
    
    Hint: See the next cell for an example which should make it clear how the inputs look like
    
    scores: tensor of size [N, 1] (the output of a neural network), where N = length of <query, document> pairs
    labels: tensor of size [N], contains the relevance labels 
    
    """
    if labels.size(0) < 2:
        return None
    Si_Sj = torch.FloatTensor ([i[0]-i[1] for i in itertools.combinations (scores,2)])
    label_comb =[i for i in itertools.combinations (labels,2)]
    
    Sij = []
    for l in label_comb:
        if l[0]>l[1]:
            Sij.append (1)
        elif l[0]<l[1]:
            Sij.append (-1)
        elif l[0]==l[1]:
            Sij.append (0)
    Sij= torch.FloatTensor(Sij)
    sigma =1
    loss = 0.5*(1-Sij)*sigma*(Si_Sj)+ torch.log (1+ torch.exp(-sigma*(Si_Sj)))
    loss = torch.mean (loss)
    return loss


# Let's say we have 2 queries, the first one with 5 <document, query> pairs 
#    and the second one with 2 <document, query> pairs. The two variables can
#    look something like this (note the shape, not the values):

scores_1 = torch.FloatTensor([0.2, 2.3, 4.5, 0.2, 1.0])
labels_1 = torch.FloatTensor([1, 2, 3, 0, 4])


scores_2 = torch.FloatTensor([3.2, 1.7])
labels_2 = torch.FloatTensor([3, 1])

print(pairwise_loss(scores_1, labels_1))
print(pairwise_loss(scores_2, labels_2))
# YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>**Implementation (35 points):**Now implement the wrapper for the pairwise LTR.**Rubric:** - Network is trained for specified epochs, and iterates over the entire dataset - and (train) data is shuffled : 10 points - Loss calculation: 10 points - Evaluation on the validation set: 5 points - Training (e.g optimizer, zero_grad, backward): 10 points<jupyter_code># TODO: Implement this! (35 points)
def train_pairwise(net, params):
    """
    This function should train the given network using the pairwise loss
    
    Returns: a dictionary containing: "metrics_val" (a list of dictionaries) and 
             "metrics_train" (a list of dictionaries). 
             
             "metrics_val" should contain metrics (the metrics in params.metrics) computed
             after each epoch on the validation set (metrics_train is similar). 
             You can use this to debug your models
    
    Note: Do not change the function definition! 
    Note: You can assume params.batch_size will always be equal to 1
    
    Hint: Consider the case when the loss function returns 'None'
    
    net: the neural network to be trained
    
    params: params is an object which contains config used in training 
        (eg. params.epochs - the number of epochs to train). 
        For a full list of these params, see the next cell. 
    """

    val_metrics_epoch = []
    train_metrics_epoch = []
    # YOUR CODE HERE
    optimizer = torch.optim.Adam(net.parameters(), lr = params.lr)
    train_dl = DataLoader(QueryGroupedLTRData(data, "train"), 
                          batch_size=params.batch_size, shuffle=True,  collate_fn=qg_collate_fn)
    
    for epoch in range(params.epochs): 
        for (qids, x, y) in train_dl:
            for q_i, features_i, labels_i in zip(qids, x, y):                                          # features_i=x=N*501,  labels_i=y=N  N:total num of query-document pairs
                
                net.zero_grad()
                scores = net(features_i)
                loss = pairwise_loss(scores, labels_i)
                if loss is None:
                    continue
                loss.requires_grad_(True)
                loss.backward()
                optimizer.step()
        train_metrics_epoch.append (evaluate_model(notwork, "train", print_results=False))
        val_metrics_epoch.append (evaluate_model(notwork, "validation", print_results=False))

    return {
        "metrics_val": val_metrics_epoch,
        "metrics_train": train_metrics_epoch
    }
pairwise_params_test = Namespace(epochs=1, lr=1e-3, batch_size=1, metrics={"ndcg"})
## uncomment to test your code
pairwise_net = NeuralModule(1)
train_pairwise(pairwise_net, pairwise_params_test)
pairwise_test, pairwise_q = evaluate_model(pairwise_net,
                                         "test", print_results=True, q_level=True)
pairwise_params_test = Namespace(epochs=1, lr=1e-3, batch_size=1, metrics={"ndcg"})
## uncomment to test your code
pairwise_net = NeuralModule(1)
train_pairwise(pairwise_net, pairwise_params_test)
pairwise_test, pairwise_q = evaluate_model(pairwise_net,
                                         "test", print_results=True, q_level=True)
pairwise_params_test = Namespace(epochs=1, lr=1e-3, batch_size=1, metrics={"ndcg"})
## uncomment to test your code
# pairwise_net = NeuralModule(1)
# train_pairwise(pairwise_net, pairwise_params_test)
# pairwise_test, pairwise_q = evaluate_model(pairwise_net,
#                                          "test", print_results=True, q_level=True)
# YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>Section 4: Pairwise: Speed-up RankNet (70 points) [Back to TOC](top)To speed up training of the previous model, we can consider a sped up version of the model, where instead of `.backward` on the loss, we use `torch.backward(lambda_i)`. The derivative of the total cost $C_T$ with respect to the model parameters $w_k$ is:$$        \frac{\partial C_T}{\partial w_k} = \sum_{(i,j) \in \mathcal{P}} \frac{\partial C(s_i, s_j)}{\partial s_i} \frac{\partial s_i}{\partial w_k} + \frac{\partial C(s_i, s_j)}{\partial s_j} \frac{\partial s_j}{\partial w_k} $$We can rewrite this sum by considering the set of indices $j$ , for which $\{i,j\}$ is a valid pair, denoted by $\mathcal{P}_i$, and the set of document indices $\mathcal{D}$:$$\frac{\partial C_T}{\partial w_k} = \sum_{i \in \mathcal{D}}\frac{\partial s_i}{\partial w_k} \sum_{j \in \mathcal{P}_i} \frac{\partial C(s_i, s_j)}{\partial s_i} $$This sped of version of the algorithm first computes scores $s_i$ for all the documents. Then for each $j= 1, \dots, n$, compute:$$\lambda_{ij} = \frac{\partial C(s_i, s_j)}{\partial s_i} = \sigma \bigg( \frac{1}{2}(1 - S_{ij}) -  \frac{1}{1 + e^{\sigma(s_i -s_j))}} \bigg) \\\lambda_i = \sum_{j \in \mathcal{P}_i} \frac{\partial C(s_i, s_j)}{\partial s_i} = \sum_{j \in \mathcal{P}_i} \lambda_{ij}$$That gives us:$$\frac{\partial C_T}{\partial w_k} = \sum_{i \in \mathcal{D}}\frac{\partial s_i}{\partial w_k} \lambda_i$$This can be directly optimized in pytorch using: `torch.autograd.backward(scores, lambda_i)`   **Implementation (20 points):**Implement the sped-up version of pairwise loss, described above.**Rubric:** - Each ordering  combination is considered: 10 points - Proper application of the formula: 10 points<jupyter_code># TODO: Implement this! (20 points)
def compute_lambda_i(scores, labels):
    """
    Compute \lambda_i (defined in the previous cell). (assume sigma=1.)
    
    scores: tensor of size [N, 1] (the output of a neural network), where N = length of <query, document> pairs
    labels: tensor of size [N], contains the relevance labels 
    
    return: \lambda_i, a tensor of shape: [N, 1]
    """
    
    # YOUR CODE HERE
    
    if labels.size(0) < 2:
        return None

    scores_transp = scores.view(len(scores), 1)
    si_sj = scores-scores_transp

    Sij = torch.empty(len (labels), len (labels))
    for i in range (0, len (labels)):
        for j in range (0, len(labels)):
                if labels[i]>labels[j]:
                    Sij[i][j]=1
                elif labels[i]<labels[j]:
                    Sij[i][j]=-1
                elif labels[i]==labels[j]:
                    Sij[i][j]=0
    Sij= torch.FloatTensor(Sij)

    sigma =1 
    lambda_ij= sigma * (0.5*(1- Sij)- 1/(1+torch.exp( -sigma*(si_sj))) )
    lambda_i = torch.sum(lambda_ij, dim=1)
    lambda_i = lambda_i.view(len(lambda_i),1) 

    return lambda_i
# YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>**Implementation (50 points):**Using the sped-up loss function `compute_lambda_i`, implement the spedup wrapper for pairwise training.**Rubric:** - Network is trained for specified epochs, and iterates over the entire dataset and (train) data is shuffled : 10 points - Loss calculation: 10 points - Evaluation on the validation set: 5 points - Training (e.g optimizer, zero_grad, backward): 10 points - Performance as expected: 15 points<jupyter_code># TODO: Implement this! (50 points)
def train_pairwise_spedup(net, params):
    """
    This function should train the given network using the sped up pairwise loss
    
    
    Note: Do not change the function definition! 
    Note: You can assume params.batch_size will always be equal to 1
    
    
    net: the neural network to be trained
    
    params: params is an object which contains config used in training 
        (eg. params.epochs - the number of epochs to train). 
        For a full list of these params, see the next cell. 
    
    Returns: a dictionary containing: "metrics_val" (a list of dictionaries) and 
             "metrics_train" (a list of dictionaries). 
             
             "metrics_val" should contain metrics (the metrics in params.metrics) computed
             after each epoch on the validation set (metrics_train is similar). 
             You can use this to debug your models
    """
    val_metrics_epoch = []
    train_metrics_epoch = []
    # YOUR CODE HERE
    optimizer = torch.optim.Adam(net.parameters(), lr = params.lr)
    train_dl = DataLoader(QueryGroupedLTRData(data, "train"), 
                          batch_size=params.batch_size, shuffle=True,  collate_fn=qg_collate_fn)
    
    for epoch in range(params.epochs): 
        for (qids, x, y) in train_dl:
            for q_i, features_i, labels_i in zip(qids, x, y):     # features_i=x=N*501,  labels_i=y=N  N:total num of query-document pairs
                net.zero_grad()
                scores = net(features_i)
                lambda_i = compute_lambda_i (scores, labels_i)  
                torch.autograd.backward(scores, lambda_i)  
                optimizer.step()
        train_metrics_epoch.append (evaluate_model(notwork, "train", print_results=False))
        val_metrics_epoch.append (evaluate_model(notwork, "validation", print_results=False))    
    
    return {
        "metrics_val": val_metrics_epoch,
        "metrics_train": train_metrics_epoch
    }
# uncomment to test your code
pairwise_spedup_params_test = Namespace(epochs=1, lr=1e-3, batch_size=1, metrics={"ndcg@10","ndcg"})
pairwise_net_spedup = NeuralModule(1)
train_pairwise_spedup(pairwise_net_spedup, pairwise_spedup_params_test)
# uncomment to test your code
# pairwise_spedup_params_test = Namespace(epochs=1, lr=1e-3, batch_size=1, metrics={"ndcg@10","ndcg"})
# pairwise_net_spedup = NeuralModule(1)
# train_pairwise_spedup(pairwise_net_spedup, pairwise_spedup_params_test)<jupyter_output><empty_output><jupyter_text>The next cell creates the results file you will have to submit. *Note that the next cell trains and evaluates only the sped-up version - this is intentional!*<jupyter_code>seed(42)
params = Namespace(epochs=8, 
                    lr=1e-3,
                    batch_size=1,
                    metrics={"ndcg", "precision@05", "recall@05"})
pairwise_model = NeuralModule(1)

create_results(pairwise_model, 
               train_pairwise_spedup, 
               pairwise_model,
               "./pairwise.json",
               params)
# persist model
torch.save(pairwise_model.state_dict(), "./pairwise_wt")<jupyter_output>Training Model<jupyter_text>YOUR ANSWER HERE<jupyter_code>assert os.path.exists("./pairwise.json")<jupyter_output><empty_output><jupyter_text>Section 5: Listwise LTR (80 points) [Back to TOC](top)In this section, you will implement LambdaRank, a listwise approach to LTR. Consider the computation of $\lambda$ for sped-up RankNet (that you've already implemented). $\lambda$ here amounts to the 'force' on a document given its neighbours in the ranked list. The design of $\lambda$ in LambdaRank is similar to RankNet, but is scaled by DCG gain from swapping the two documents in question. Let's suppose that the corresponding ranks of doucment $D_i$ and $D_j$ are $r_i$ and $r_j$ respectively. Given a ranking measure $IRM$, such as $NDCG$ or $ERR$, the lambda function in LambdaRank is defined as:$$        \frac{\partial C}{\partial s_i} = \sum_{j \in D} \lambda_{ij} \cdot |\bigtriangleup IRM (i,j)| $$Where $|\bigtriangleup IRM(i,j)|$ is the absolute difference in $IRM$ after swapping the rank positions $r_i$ and $r_j$ while leaving everything else unchanged ($| \cdot |$ denotes the absolute value). Note that we do not backpropogate $|\bigtriangleup IRM|$, it is treated as a constant that scales the gradients. In this assignment we will use $|\bigtriangleup NDCG|$ **Implementation (30 points):**Implement the listwise loss.**Rubric:** - Each ordering  combination is considered: 10 points - Computing $|\bigtriangleup NDCG|$: 10 points  - Proper application of the formula: 10 points<jupyter_code># TODO: Implement this! (30 points)

def DCG (labels, denom):                                 # label:[N] # denominator:[N] 
    return torch.sum ((2**labels-1)/denom)               # return a scalar

def lambda_ij (scores, labels):
    
    scores_transp = scores.view(len(scores), 1)
    si_sj = scores-scores_transp

    Sij = torch.empty(len (labels), len (labels))
    for i in range (0, len (labels)):
        for j in range (0, len(labels)):
                if labels[i]>labels[j]:
                    Sij[i][j]=1
                elif labels[i]<labels[j]:
                    Sij[i][j]=-1
                elif labels[i]==labels[j]:
                    Sij[i][j]=0
    Sij= torch.FloatTensor(Sij)

    sigma =1 
    lambda_ij= sigma * (0.5*(1- Sij)- 1/(1+torch.exp( -sigma*(si_sj))) )
    return lambda_ij


def listwise_loss(scores, labels):
    
    """
    Compute the LambdaRank loss. (assume sigma=1.)
    
    scores: tensor of size [N, 1] (the output of a neural network), where N = length of <query, document> pairs
    labels: tensor of size [N], contains the relevance labels 
    
    returns: a tensor of size [N, 1]
    """
    
    # YOUR CODE HERE
    denom = torch.log2 (torch.range(1, len(labels))+1)

    odcg = DCG (labels, denom)       # oDCG: original labels, unsorted labels
    ideal_labels= torch.sort(labels, descending=True).values
    idcg= DCG (ideal_labels, denom)


    swaped_DCG = torch.zeros(len(labels), len(labels))
    for i in range (0, len(labels)):
        for j in range (0, len(labels)):  
            li=labels[i]
            lj=labels[j]
            swaped_labels = labels.clone().detach()
            swaped_labels[j]=li
            swaped_labels[i]=lj
            swaped_DCG[i][j] = DCG (swaped_labels, denom)
    delta_DCG = odcg - swaped_DCG
    delta_nDCG = torch.abs (delta_DCG/idcg)
    
    lambda_ij_matrix =lambda_ij (scores, labels)
    new_grad = delta_nDCG * lambda_ij_matrix
    lambda_i = torch.sum(new_grad, dim=1)
    lambda_i = lambda_i.view(len(lambda_i),1)
    return lambda_i
scores_1 = torch.FloatTensor([0.2, 2.3, 4.5, 0.2, 1.0])
labels_1 = torch.FloatTensor([1, 2, 3, 0, 4])


scores_2 = torch.FloatTensor([3.2, 1.7])
labels_2 = torch.FloatTensor([3, 1])


print(listwise_loss(scores_1, labels_1))



print(listwise_loss(scores_2, labels_2))
# YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>**Implementation (50 points):**And use the loss function above to train a listwise LTR.**Rubric:** - Network is trained for specified epochs, and iterates over the entire dataset and (train) data is shuffled : 10 points - Loss calculation: 10 points - Evaluation on the validation set: 5 points - Training (e.g optimizer, zero_grad, backward): 10 points - Performance as expected: 15 points<jupyter_code># TODO: Implement this! (50 points)
def train_listwise(net, params):
    """
    This function should train the given network using the listwise (LambdaRank) loss
    
    Note: Do not change the function definition! 
    Note: You can assume params.batch_size will always be equal to 1
    
    
    net: the neural network to be trained
    
    params: params is an object which contains config used in training 
        (eg. params.epochs - the number of epochs to train). 
        For a full list of these params, see the next cell. 
        
    Returns: a dictionary containing: "metrics_val" (a list of dictionaries) and 
             "metrics_train" (a list of dictionaries). 
             
             "metrics_val" should contain metrics (the metrics in params.metrics) computed
             after each epoch on the validation set (metrics_train is similar). 
             You can use this to debug your models
    """
    
    val_metrics_epoch = []
    train_metrics_epoch = []
    # YOUR CODE HERE
    optimizer = torch.optim.Adam(net.parameters(), lr = params.lr)
    train_dl = DataLoader(QueryGroupedLTRData(data, "train"), 
                      batch_size=params.batch_size, shuffle=True,  collate_fn=qg_collate_fn)
    
    for epoch in range(params.epochs): 
        for (qids, x, y) in train_dl:
            for q_i, features_i, labels_i in zip(qids, x, y):                                          # features_i=x=N*501,  labels_i=y=N  N:total num of query-document pairs
                net.zero_grad()
                scores = net(features_i)
                lambda_i = listwise_loss(scores, labels_i)
                torch.autograd.backward(scores, lambda_i)  
                optimizer.step()
        train_metrics_epoch.append (evaluate_model(notwork, "train", print_results=False))
        val_metrics_epoch.append (evaluate_model(notwork, "validation", print_results=False))    
    
    return {
        "metrics_val": val_metrics_epoch,
        "metrics_train": train_metrics_epoch
    }
# uncomment to test your code
listwise_params_test = Namespace(epochs=1, lr=1e-3, batch_size=1, metrics={"ndcg"})
listwise_net = NeuralModule(1)
train_listwise(listwise_net, listwise_params_test)
evaluate_model(listwise_net, "test", print_results=True)
# uncomment to test your code
# listwise_params_test = Namespace(epochs=1, lr=1e-3, batch_size=1, metrics={"ndcg"})
# listwise_net = NeuralModule(1)
# train_listwise(listwise_net, listwise_params_test)
# evaluate_model(listwise_net, "test", print_results=True)<jupyter_output><empty_output><jupyter_text>The following cell saves your results in `listwise.json` file for submission.<jupyter_code>seed(42)
params = Namespace(epochs=8, 
                    lr=1e-4,
                    batch_size=1,
                    metrics={"ndcg", "precision@05", "recall@05"})
listwise_model = NeuralModule(1)

create_results(listwise_model, 
               train_listwise, 
               listwise_model,
               "./listwise.json",
               params)

# persist model
torch.save(listwise_model.state_dict(), "./listwise_wt")<jupyter_output>Training Model<jupyter_text>YOUR ANSWER HERE<jupyter_code>assert os.path.exists("./listwise.json")<jupyter_output><empty_output><jupyter_text>Section 6: Comparing Pointwise, Pairwise and Listwise (70 points) [Back to TOC](top)In the next few cells, we will compare the methods you've implemented. Helper functions are provided for you, which you can use to make some conclusions. You can modify the code as needed (you are encouraged to do so!)<jupyter_code>def autolabel(ax, rects, labels):
    """Attach a text label above each bar in *rects*, displaying its height."""
    for l, rect in zip(labels, rects):
        height = rect.get_height()
        ax.annotate('{}'.format(l),
                    xy=(rect.get_x() + rect.get_width() / 2, height),
                    xytext=(0, 3),  # 3 points vertical offset
                    textcoords="offset points",
                    ha='center', va='bottom',
                   fontsize=7)

def compare_methods(labels, metrics, metrics_to_plot={"ndcg", "precision@05", "recall@05"}):
    """
    Constructs bar plots to compare different methods. 
    
    labels: list/tuple of length N
    metrics: list/tuple of length N, containing dictionary containing the test set results 
    metrics_to_plot: set of metrics to plot - each metric creates a separate plot 
    """
    assert len(metrics) == len(labels)
    
    x = np.arange(len(metrics_to_plot)) 
    fig, axes = plt.subplots(nrows=1, ncols=len(metrics_to_plot), sharey=False)
    fig.set_figheight(7)
    fig.set_figwidth(15)
    
    colors = cm.get_cmap("Set1").colors

    for metric, ax, c in zip(metrics_to_plot, axes, colors):
        m = [_[metric][0] for _ in metrics]  
        std = [_[metric][1] for _ in metrics]
        x = np.arange(len(labels))
        rects = ax.bar(x, m, label=metric, color=c)
        
        l = ["{0:.4f}({1:.4f})".format(_[metric][0], _[metric][1]) for _ in metrics]
        autolabel(ax, rects, l)
        
        ax.set_xticks(x)
        ax.set_xticklabels(labels, rotation=45)
        ax.set_yticks(np.linspace(0, 1, num=11))
        ax.set_ylim(ymin=min(m) - 0.05, ymax= max(m) + 0.05)
        ax.set_title(metric)
    
    

def plot_distribution(labels, q_metrics, metric="ndcg"):
    """
    Plots the distribution of NDCG scores
    
    labels: list/tuple of length N
    q_metrics: list/tuple of dictionaries with length N, containing the query level results
    metric: the metric to plot
    
    """
    
    n = len(labels)
    # nC2
    n_plots = int((n*(n-1))/2)
    
    fig, axes = plt.subplots(nrows=n_plots, ncols=1)
    fig.set_figheight(8 * n_plots)
    fig.set_figwidth(10)
    
    colors = cm.get_cmap("Set1").colors
    
    for idx, (i, j) in enumerate(itertools.combinations(range(n), 2)):
        ax = axes[idx]
        
        im = q_metrics[i][metric]
        jm = q_metrics[j][metric]
        
        ax.hist(im, bins=50, label=labels[i], color=colors[i], alpha=0.55)
        ax.hist(jm, bins=50, label=labels[j], color=colors[j], alpha=0.55)
        
        ax.set_title(f"{labels[i]} vs {labels[j]}")
        ax.legend()
        ax.set_ylabel("Count")
        ax.set_xlabel("NDCG (binned)")
        
# Load metrics and models

pointwise_regr_model = NeuralModule(1)
pointwise_regr_model.load_state_dict(torch.load("./pointwise_regr_wt"))

pointwise_clf_model = NeuralModule(5)
pointwise_clf_model.load_state_dict(torch.load("./pointwise_clf_wt"))
pointwise_clf_pred_fn = partial(clf_pred, net=pointwise_clf_model)

pairwise_model = NeuralModule(1)
pairwise_model.load_state_dict(torch.load("./pairwise_wt"))

listwise_model = NeuralModule(1)
listwise_model.load_state_dict(torch.load("./listwise_wt"))


methods = [
    {"results_file": "./pointwise_regression.json", "label": "Pointwise (R)"},
    {"results_file": "./pointwise_classification.json", "label": "Pointwise (C)"},
    {"results_file": "./pairwise.json", "label": "Pairwise"},
    {"results_file": "./listwise.json", "label": "Listwise"}
]

labels = []
results = []
q_results = []
for m in methods:
    labels.append(m["label"])
    
    with open(m["results_file"]) as reader:
        r = json.load(reader)
    
    results.append(r["test_metrics"])
    q_results.append(r["test_query_level_metrics"])


compare_methods(labels, results)
plot_distribution(labels, q_results)<jupyter_output><empty_output><jupyter_text>In the next cell, report the performance metrics for the methods (25 points):|     Model    | NDCG (s.d)| Precision@5 (s.d) | Recall@5 (s.d) ||:------------:|:----------:|:-----------:|:--------:|| Pointwise(R) |            |             |          || Pointwise(C) |            |             |          ||   Pairwise   |            |             |          ||   Listwise   |            |             |          |**Rubric:** Each reported  carries 2 points.<jupyter_code>|     Model    | NDCG (s.d)      |  Precision@5 (s.d)  | Recall@5 (s.d) |
|:------------:|:----------:     |:-----------:        |:--------:      |
| Pointwise(R) |  0.8723(0.05308)|   0.6974(0.27626)   |0.1538 (0.08467)|          
| Pointwise(C) |  0.8354(0.05743)|   0.6085 (0.27598)  |0.1328 (0.07951)|          
|   Pairwise   |  0.7666(0.05517)|   0.2991 (0.26490)  |0.0627 (0.05952)|
|   Listwise   |  0.6808(0.04983)|   0.1265 (0.15819)  |0.0271 (0.03642)|<jupyter_output><empty_output><jupyter_text>Generally we can say that our neural network converges when we reach the optimum update of our weights that lead to the minimum loss value. We  train our neural network and update the Weights tensor up until the point that the loss value in the training set starts to increase instead of decreasing. In our case the number of updating rounds, epochs are 8. The above approach may lead to overfitting, thus we may want to train our artificial neural network up to the point when the accuracy of validation loss increases and the loss does not increase that much. PointWise LTR performs better and generally gives better results across all metrics in comparison to Pairwise LTR. On the pairwise ltr method we mainly focus on optimising being close to the label of the document (relevant or not)  and not on its ranking. On the contrary, pairwise LTR model produces a relevant document score and computes the value of loss function, based on pairs of documents with difference in relevance. The main idea behind listwise ltr is that we optimise the ranking of the metrics directly, for example ndcg per say. Listwise LTR is the method that performs by far worse than all the others across all metrics if we take the mean value of them across all queries also. Though we can see that its s.d value is the least one(especially for precision@5 it is almost half of all the other models s.d) so we can conclude that it is the model with the least bias in it. Pointwise R model is the best, then comes pointwise C, pairwise and listwise at Mean value of Precision@5 for Pointwise R is approximately 70% for classification around 50%, for pairwise 30% and for listwise 10%.For ncdg Pointwise R gives 87,5% for C 80%, for pairwise 76% and for listwise around 67%.Recall@5 mean values are 15%, 10%, 6% and 2% respectively for Pointwise R,C pairwise and listwise LTR models. Pointwise Regression vs Pointwise Classification Plotting their NDCG together we can infer that pointwise R gives a better score . Write a conclusion in the next cell, considering (45 points):- rates of convergence- time complexity- performance wrt the 3 metrics- performance across queries- ... any other observations YOUR ANSWER HERE  Chapter 2: Online LTR (180 points) [Back to TOC](top)In this part we want to use user interactions for training learning to rank algorithms.This part consists of the following sections: - Clicks Simulation (15 points) - Counterfactual Learning to Rank (80 points) - Online Evaluation (50 points)  Section 1: Clicks Simulation (15 points) [Back to TOC](top)In online LTR, we work with user interactions such as clicks.One way to test online LTR algorithms is to conduce semi-synthetic experiments.In semi-synthetic experiments, the feature vectors of a LTR dataset (similar to what you used in Part 1) are used.But instead of using the relevance labels for training the LTR algorithm, **simulated clicks** are used.In this section we want to simulate clicks based on the labels and use the generated clicks in later sections to train our Counterfactual LTR (CLTR) algorithm.First we need to have a *production ranker* that determines the order of documents for each query.The click probability of each document depends on its rank in the results list as well as its relevance.<jupyter_code>ranks = dataset.load_production_ranker()
print(f'ranks has {len(ranks.keys())} keys: {ranks.keys()}')
for key in ranks:
    print(f'{key} shape:{ranks[key].shape}')
    
data.train.ranks = ranks['train']
data.validation.ranks = ranks['valid']
data.test.ranks = ranks['test']<jupyter_output><empty_output><jupyter_text>**Implementation (5 points):**Now use the `ranks` to generate clicks.We assume documents with label $[3,4]$ are relevant and labels $[0,1,2]$ are non-relevant.We also assume the examination probability of a document at rank $r$ is $\frac{1}{r}$.Finally, we assume a $0.05$ noise.This means that the click probability of a document with label $[3,4]$ at rank $r$ is $\frac{1}{r}$, while the click probability of a document with label $[0,1,2]$ at rank $r$ is $\frac{0.05}{r}$.<jupyter_code># TODO: Implement this! (5 points)
def generate_clicks(ranking, query_labels):
    """
    Generates random clicks based on the given list of ranking and relevance labels for the documents of one query.
    Input:
        ranking: contains the rank of documents, eg. ranking[5]=0 means that document 5 is shown at rank 0.
        query_labels: contains the labels of documents, eg. query_labels[5]=3 means that document 5 has a label equal to 3.
    Output:
        A np.array of clicked positions, eq. clicked=[1,4] meanse that documents at ranks 1 and 4 have been clicked. Note that document at rank 1 may differ from document 1 (see description of ranking input).
        
    Hint: Use the np.random.binomial function for generating click from probability.
    """
    theta = 1. / np.arange(1, ranking.shape[0] + 1)
    noise_prob = 0.05 * theta
    
    zipped_ranking = list(zip(ranking,query_labels))
    zipped_ranking = sorted(zipped_ranking, key=lambda tup: tup[0])
    click_prob = []
    
    #print(zipped_ranking)
    
    for i,q in enumerate(zipped_ranking):
        if q[1] in [3,4]: 
            click_prob.append((theta[i],q[0]))
        else:
            click_prob.append((noise_prob[i],q[0]))
            
    clicked = []
            
    for i in click_prob:
        s = np.random.binomial(1,i[0],1)
        if s[0] !=0 :
            clicked.append(i[1])
        
           
    clicked = np.array(clicked)    
   
    #raise NotImplementedError()
    return clicked.astype(np.int32)
 
ranking = np.array([2, 1, 3, 0, 4, 7, 5, 6])
labels = np.array([0, 3, 4, 1, 1, 2, 4, 3])

np.random.seed(4)
print(f'clicks: {generate_clicks(ranking, labels)}')
print('expected clicks for seed 4: [0 1 6]')<jupyter_output><empty_output><jupyter_text>**Implementation (10 points):**Use `generate_clicks` to simulate clicks for different query sessions.For this, randomly select a query id, generate random clicks on documents of that query using your `generate_clicks` implementation and only keep the clicks on `topk` positions (i.e. do not consider clicks on ranks after `topk` rank).Repeat this process until you have `click_count` number of clicks.<jupyter_code># TODO: Implement this! (10 points)
def simulate_clicks(data, split_name, click_count, topk=20):
    """
    Simulate click_count clicks on topk results on the specified split of the data.
    Output:
        A list L=[L_0, L_1, ..., L_n] of lists. Each L_i is itself a list of clicks over the documents of query number i, eg. L_0=[0,2,0,1,0,5,2] means that in the sessions of the first query, the documents at positions 0,1,2 and 5 were clicked 3,1,2 and 1 times, respectively.
    """
    data_split = getattr(data, split_name)
#     argsorted, doclist_ranges = dataset.clean_and_sort(ranks[split_name], data_split, topk)

#     click_data_split = Namespace(feature_matrix = data_split.feature_matrix[argsorted,:],
#                                  doclist_ranges = doclist_ranges,
#                                  clicks = None)
#     labels = data_split.label_vector[argsorted]

    #num_queries = click_data_split.doclist_ranges.shape[0] - 1
    
    num_queries = data_split.doclist_ranges.shape[0] - 1
    clicks = [[] for _ in range(num_queries)]
    np.random.seed(42)
       
    # YOUR CODE HERE

    
    for i,c in enumerate(clicks):
        rng = data_split.doclist_ranges[i:i+2]
        c.extend([gl for gl in generate_clicks(ranks[split_name][rng[0]:rng[1]], data_split.query_labels(i)).tolist() if gl<=topk])
             
    return clicks    
   
    #raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>Use `simulate_clicks` function to simulate clicks for train and validation sets.**Note:**You can dump your generated clicks using pickle or numby and load them in your future runs to avoid long waiting times.<jupyter_code>train_clicks = simulate_clicks(data,'train', click_count = 10000)

assert isinstance(train_clicks, list)
session_clicks = np.array(list(map(len, train_clicks)))
print(f'{session_clicks.sum()} clicks simulated over {session_clicks.shape[0]} queries.')
print(f'On average, each session has {session_clicks.mean()} clicks.')
print(f'min number of clicks per session:{session_clicks.min()}')
print(f'max number of clicks per session:{session_clicks.max()}')<jupyter_output><empty_output><jupyter_text>For our new click data, we need a new dataloader.`QueryGroupedOnlineLTRData` provides what you will need for training an LTR model from the clicks:<jupyter_code>class QueryGroupedOnlineLTRData(Dataset):
    def __init__(self, data, split, clicks, topk):
        self.split = {
            "train": data.train,
            "validation": data.validation,
            "test": data.test
        }.get(split)
        assert self.split is not None, "Invalid split!"
        self.clicks = clicks
        self.topk = topk
    
    def __len__(self):
        return self.split.num_queries()

    def __getitem__(self, q_i):
        s_i = self.split.doclist_ranges[q_i]
        e_i = self.split.doclist_ranges[q_i + 1]
        feature_ = self.split.feature_matrix[s_i:e_i,:]
        ranking = self.split.ranks[s_i:e_i]
        inverse_ranking = np.argsort(ranking)
#       We re-order the rows of feature matrix so that in coincides with the ranking of documents.
#       We also cut the documents after rank topk.
        feature = torch.FloatTensor(feature_[inverse_ranking[:self.topk],:])
        clicks = self.clicks[q_i]
        return q_i, feature, clicks
    
def qgo_collate_fn(batch):
    
    qids = []
    features = []
    clicks = []
    
    for (q, f, l) in batch:
        qids.append(q)
        features.append(f)
        clicks.append(l)
    return qids, features, clicks<jupyter_output><empty_output><jupyter_text>Section 2: Counterfactual LTR (90 points) [Back to TOC](top)Now we want to use the simulated clicks and train a counterfactual LTR algorithm.We will use a simple loss function:$$L=\frac{1}{|Q|} \sum_{q \in Q} \sum_{d \in D_q} r(d, q) \cdot \lambda(\bar{rank}(d, D_q))$$where $Q$ is the set of queries, $D_q$ is the list of documents shown to the user for query $q$, $r(d, q)$ is the relevance of document $d$ to query $q$ and $rank(d, D_q)$ is the rank of $d$ in $D_q$.In this assignment, we use sigmoid-like bound for $\lambda(\bar{rank}(d, D_q))$:$$\lambda(\bar{rank}(d, D_q)) = \sum_{d' \in D_q} log\left(1+exp\left(-2\cdot(f(d)-f(d')\right)\right)$$  Section 2.1. Biased LTR (20 points)Remember that we do not have the explicit relevance in the online settings.Instead, we have to learn from the clicks.Clicks, implicitly indicate relevance.A naive approach for learning from clicks would be to replace relevance with clicks and train a model with the loss function:$$L_{biased}=\frac{1}{|Q|} \sum_{q \in Q} \sum_{d \in D_q} c(d, q) \cdot \lambda(\bar{rank}(d, D_q))$$where $c(d, q)$ indicates the number of times $d$ was clicked for query $q$.<jupyter_code># TODO: Implement this! (20 points)
def online_loss_biased(scores, clicks):
    
    from collections import Counter
    click_counts = Counter(clicks)
    
    
    # YOUR CODE HERE
    c_lmd = 0
    
    scores_list = scores.tolist()
    for i,x1  in enumerate(scores_list):
        lmd = 0
        for i2,x2 in enumerate(scores_list):
             lmd += (np.log(1 + np.exp(-2*(x1[0] - x2[0]))))
        c_lmd += click_counts[i] * lmd
        
    return c_lmd
    #raise NotImplementedError()
scores = torch.FloatTensor([1.2, -2, 1.1, 2.4, 0, 1.2])[:,None]
clicks = [0, 0, 1, 3, 1, 4, 0, 5, 0, 1, 2]
print(f'biased loss: {online_loss_biased(scores, clicks)}')
print(f'expected: -6.205410003662109')<jupyter_output><empty_output><jupyter_text>Section 2.2. Unbiased LTR (10 points)Now modify `online_loss_biased` to debias the clicks, using IPS method:$$L_{unbiased}=\frac{1}{|Q|} \sum_{q \in Q} \sum_{d \in D_q} \frac{c(d, q)}{P\left(o(d,q)=1\right)} \cdot \lambda(\bar{rank}(d, D_q))$$where $P\left(o(d,q)=1\right)$ indicates the probability that $d$ was observed by the user for query $q$.<jupyter_code># TODO: Implement this! (10 points)
def online_loss_unbiased(scores, clicks):
    """
    Compute and return the unbiased online loss *for a single query*. To compute this, use L_{unbiased} formula above.
    
    scores: tensor of size [N, 1] (the output of a neural network), where N = length of <query, document> pairs
    clicks: list of clicked ranks. 
    
    Note 1: the scores are aligned with the click positions, i.e. scores[0] corresponds to the 0 rand, etc.
    Note 2: the weights are provided.
    
    """
    from collections import Counter
    click_counts = Counter(clicks)
    weights = torch.Tensor((1./np.arange(1,scores.shape[0]+1))[:,None])
    weights_list = weights.tolist()
    
    # YOUR CODE HERE
    
    c_lmd = 0
    
    scores_list = scores.tolist()
    for i,x1  in enumerate(scores_list):
        lmd = 0
        for i2,x2 in enumerate(scores_list):
             lmd += (np.log(1 + np.exp(-2*(x1[0] - x2[0]))))
        c_lmd += click_counts[i] * lmd / weights_list[i][0]
                
    return c_lmd
scores = torch.FloatTensor([1.2, -2, 1.1, 2.4, 0, 1.2])[:,None]
clicks = [0, 0, 1, 3, 1, 4, 0, 5, 0, 1, 2]
print(f'biased loss: {online_loss_unbiased(scores, clicks)}')
print(f'expected: -17.11062240600586')<jupyter_output><empty_output><jupyter_text>**Implementation (60 points):**In the next cell, write a wrapper that uses your loss functions and simulated clicks to train an online LTR method (no evaluation on validation set is required):**Rubric:** - Network is trained for specified epochs, and iterates over the entire dataset and (train) data is shuffled : 10 points - Loss calculation: 10 points - Training (e.g optimizer, zero_grad, backward): 10 points - Performance as expected for biased loss: 15 points - Performance as expected for unbiased loss: 15 points<jupyter_code># TODO: Implement this! (60 points)
def train_online(net, train_clicks, loss_fn, params):
    """
    Use QueryGroupedOnlineLTRData to load the data train split and clicks.
    Use the appropriate loss_fn (biased/unbiased).
    No need to use validation set.
    """
    # YOUR CODE HERE
    from collections import Counter
  
    train_dl = DataLoader(QueryGroupedOnlineLTRData(data, "train", train_clicks, params.topk), batch_size=1, shuffle=True, collate_fn=qgo_collate_fn) 
    optimizer = torch.optim.Adam(net.parameters(), lr = params.lr)
    
    count_new_click = []

   
    for epoch in range(params.epochs):
        for (_, features,train_clicks) in train_dl:
            count_new_click.clear()
            count_lst = Counter(train_clicks[0])
            for i in range(0,features[0].shape[0]):
                if i in count_lst.keys():
                    count_new_click.append(count_lst[i])
                else:
                    count_new_click.append(0)
            
            optimizer.zero_grad()
            outputs = net(features[0])
            myloss = loss_fn(outputs,count_new_click)
            #print ("myloss", type(myloss) , myloss)
            myloss = torch.tensor(myloss)
            myloss.requires_grad=True
            #print("myloss", type(myloss) , myloss)
            myloss.backward()
            optimizer.step()

params = Namespace(epochs=3, lr=1e-3, topk=20)
train_clicks = simulate_clicks(data,'train', click_count = 50000)
biased_net = NeuralModule(1)
train_online(biased_net, train_clicks, online_loss_biased, params)
unbiased_net = NeuralModule(1)
train_online(unbiased_net, train_clicks, online_loss_unbiased, params)
evaluate_model(biased_net, 'test', print_results=True)
evaluate_model(unbiased_net, 'test', print_results=True)

seed(42)
train_clicks = simulate_clicks(data,'train', click_count = 50000)
params = Namespace(epochs=3, 
                    lr=1e-3,
                    batch_size=1,
                    topk=20,
                    metrics={"ndcg", "precision@05", "recall@05"})
biased_model = NeuralModule(1)

create_results(biased_model, 
               train_online, 
               biased_model,
               "./biased_model.json",
               train_clicks,
               online_loss_biased,
               params)
# persist model
torch.save(biased_model.state_dict(), "./biased_wt")
seed(42)

params = Namespace(epochs=3, 
                    lr=1e-3,
                    batch_size=1,
                    topk=20,
                    metrics={"ndcg", "precision@05", "recall@05"})
unbiased_model = NeuralModule(1)

create_results(unbiased_model, 
               train_online, 
               unbiased_model,
               "./unbiased_model.json",
               train_clicks,
               online_loss_unbiased,
               params)
# persist model
torch.save(biased_model.state_dict(), "./unbiased_wt")
# YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>Section 2.3: Comparison (20 points)Now we can compare two loss functions: biased and unbiased.We want to see how they can improve by increasing the number of training clicks.Train both biased and unbaised models on $[2000, 10000, 50000]$ number of clicks and compare the results.Plot the results in a **single** figure with x-axis showing the number of trainin clicks and y-axis showing the nDCG@10.Discuss your observations.**Rubric:**- Two curves are plotted in the figure: 10 points- Clear titles, x label, y labels and legends (if applicable): 5 points- Explain what you observe: 5 points<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

biased= [1,2,3]
unbiased = [4,5,6]

def plot (biased, unbiased):
    clicks = [2000,10000,50000]
    plt.plot (clicks, biased, 'r-', label='biased')
    plt.plot (clicks, unbiased, 'b-', label='unbiased')
    plt.xlabel("clicks")
    plt.ylabel("nDCG@10")
    plt.title ("biased and unbiased ranking model")
    plt.legend()
    plt.savefig("biased_unbiased_comparison", dpi=300)
    plt.show()
plot (biased, unbiased)<jupyter_output><empty_output><jupyter_text>Section 3: Online Evaluation (75 points) [Back to TOC](top)Sometimes, in online search engines, we want to compare two or more different ranking functions based on the user interactions.This comparison is done via online evaluation.In this section we implement one of the online evaluation methods: probabilistic multileaving.We compare three rankers: - Production ranker: the ranks that are provided to you for doing the click simulation. - Biased method from previous section. - Unbiased method from previous section Before proceeding, we need some auxiliary functions: - `invert_ranking` gets the ranking and gives the inverted rankings. This means that for an input with `rank[d]=r`, the output would be `i_rank[r]=d`. - `get_predictions` gives a dictionary of predictions, i.e. `predictions[qid]` is the list of scores produced by the given model.<jupyter_code>def get_predictions(net, test_clicks, topk):
    test_dl = DataLoader(QueryGroupedOnlineLTRData(data, 'test', test_clicks, topk), 
                          batch_size=1, 
                          shuffle=False,
                          collate_fn=qg_collate_fn)
    predictions = {}
    for qids, x, _ in test_dl:
        predictions[qids[0]] = net(x[0]).detach().numpy()[:,0]
    return predictions


def invert_rankings(rankings):
    '''
    Invert indices in a matrix of rankings, ranking per row.
    '''
    inverted = np.zeros(rankings.shape)
    
    inverted[np.arange(rankings.shape[0])[:,None],rankings] = np.arange(rankings.shape[1])[None,:]
    return inverted<jupyter_output><empty_output><jupyter_text>The next cell helps you build the ranking matrix for each query in the test set.The first row is the production ranker, the second row is the biased ranker and the third row is the unbiased ranker.<jupyter_code># we only need the ranking of the production ranker for the test set. No clicks have to be simulated here.
test_clicks = simulate_clicks(data, 'test', click_count = 0)
biased_predictions = get_predictions(biased_net, test_clicks, 20)
unbiased_prediction = get_predictions(unbiased_net, test_clicks, 20)

def get_ranking_matrix(qid, topk):
    ranking_matrix = np.empty([3, min(data.test.query_size(qid), topk)], dtype=np.int32)
    ranking = data.test.ranks[data.test.doclist_ranges[qid]:data.test.doclist_ranges[qid+1]]
    ranking_matrix[0,:] = np.arange(ranking_matrix.shape[1])
    ranking_matrix[1,:] = np.argsort(-biased_predictions[qid])
    ranking_matrix[2,:] = np.argsort(-unbiased_prediction[qid])
    return ranking_matrix
print(get_ranking_matrix(0, 20))<jupyter_output><empty_output><jupyter_text>Section 3.1 Multileaving (10 points)Given the rankings of multiple rankers, we want to decide how to fill the results list and show it to the users.Implement the following function.<jupyter_code># TODO: Implement this! (10 points)
def make_multileaving(inverted_rankings, topk):
    '''
    ARGS: (all np.array of docids)
    - inverted_rankings: matrix (rankers x documents) where [x,y] corresponds to the rank of doc y in ranker x
    RETURNS
    - ranking of indices corresponding to inverted_rankings
    '''
     n_rankers = inverted_rankings.shape[0]
    n = inverted_rankings.shape[1]
    k = min(n, topk)

    # YOUR CODE HERE
    tau = 3
    
    #calculation of initial P(A), P(B)
    P_all = []
    sum_all = []
    result = []
    
    for rank in range(n_rankers):
        sum_per_rank = 0
        for i in range(k):
            sum_per_rank += 1/(inverted_rankings[rank][i]**tau)
        sum_all.append(sum_per_rank)
        
    for rank in range(n_rankers):
        all_doc = []
        for i in range(k):
            d_i = (1/(inverted_rankings[rank][i]**tau))/sum_all[rank]
            all_doc.append(d_i)
        P_all.append(all_doc)
    
    #placing the docs
    choices = np.arange(n_rankers)
    for doc_ranked in range(k): 
        choice = np.random.choice(choices)
        best_doc = max(P_all[choice])
        index = P_all[choice].index(best_doc)
        result.append(index)
        
        new_array
        for i in inverted_rankings:
            np.delete(i, [index]) #deleting part does not work properly, since I cannot just delete elements in a loop here
            
        #recalculating P_all
        P_all = []
        sum_all = []
        
        for rank in range(n_rankers):
            sum_per_rank = 0
            for i in range(k):
                sum_per_rank += 1/(inverted_rankings[rank][i]**tau)
            sum_all.append(sum_per_rank)
            
        for rank in range(n_rankers):
            all_doc = []
            for i in range(k):
                d_i = (1/(inverted_rankings[rank][i]**tau))/sum_all[rank]
                all_doc.append(d_i)
            P_all.append(all_doc)
        
    return result

print(make_multileaving(invert_rankings(get_ranking_matrix(0, 20)), 20))
# YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>Section 3.2: Probability of rankers (15 points)We have shown the multileaved list to the user and they clicked on some documents.In this function we want to calculate the probability that the clicked documents belong to a specific ranker.Note that the results of rankers are not unique and each document in the results list may belong to different rankers.So we need to assign a *probability* to each click belonging to each ranker.<jupyter_code># TODO: Implement this! (15 points)
def probability_of_list(result_list, inverted_rankings, clicked_docs):
    '''
    ARGS: (all np.array of docids)
    - result_list: the multileaved list
    - inverted_rankings: matrix (rankers x documents) where [x,y] corresponds to the rank of doc y in ranker x
    - clicked_docs: boolean array of result_list length indicating clicks
    RETURNS
    -sigmas: matrix (rankers x clicked_docs) with probabilty ranker added clicked doc
    '''
    n_docs = inverted_rankings.shape[1]
    n_rankers = inverted_rankings.shape[0]

    click_doc_ind = result_list[clicked_docs]
    
    # YOUR CODE HERE
    raise NotImplementedError()
ranking_matrix = get_ranking_matrix(0, 20)
result_list = make_multileaving(invert_rankings(ranking_matrix), 20)
clicks = [0,4,9,10]
probabilities = probability_of_list(result_list, invert_rankings(ranking_matrix), clicks)
print(f'ranking matrix:\n {ranking_matrix}')
print(f'results list (shown to user):\n {result_list}')
print(f'clicked documents: {clicks}')
print(f'probabilities:\n {probabilities}')
# YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>Section 3.3: Preference matrix (10 points)Given the probabilities of each ranker being clicked, we want to calculate a preference matrix that for each pair of rankers tells us which one is preferred by the clicks.<jupyter_code># TODO: Implement this! (10 points)
def preferences_of_list(probs):
    '''
    ARGS:
    -probs: clicked docs x rankers matrix with probabilities ranker added clicked doc  (use probability_of_list)
    -n_samples: number of samples to base preference matrix on
    RETURNS:
    - preference matrix: matrix (rankers x rankers) in this matrix [x,y] > 0 means x won over y and [x,y] < 0 means x lost from y
      the value is analogous to the (average) degree of preference
    '''
    n_samples = 10
    n_clicks = probs.shape[0]
    n_rankers = probs.shape[1]
    
    # YOUR CODE HERE
    raise NotImplementedError()

def infer_preferences(inverted_rankings, result_list, clicked_docs):
    n_rankers = inverted_rankings.shape[0]
    if np.any(clicked_docs):
        return preferences_of_list(probability_of_list(result_list,
                                        inverted_rankings,
                                        clicked_docs))
    else:
        return np.zeros((n_rankers, n_rankers))

print(preferences_of_list(probabilities))
# YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output><jupyter_text>Now we can simulate the multileaving to see how our target rankers are evaluated by it.<jupyter_code>def simulate_multileaving(n_impressions, topk):
    n_samples = 10
    impressions = 0
    pref_update = 0
    total_pref = np.zeros((3, 3))
    for step_i in range(n_impressions):
        qid = np.random.randint(0, data.test.doclist_ranges.shape[0] - 1)

        start_i = data.test.doclist_ranges[qid]
        end_i = data.test.doclist_ranges[qid + 1]
        n_query_docs = end_i - start_i
        query_labels = data.test.label_vector[start_i:end_i]

        inverted_rankings = invert_rankings(get_ranking_matrix(qid, topk))
        multileaving = make_multileaving(inverted_rankings, topk)

        cur_clicks = generate_clicks(multileaving, query_labels)

        if np.any(cur_clicks):
            pref = infer_preferences(inverted_rankings, multileaving, cur_clicks)
            total_pref += pref
            pref_update += 1
    return total_pref / pref_update

simulate_multileaving(10000, 20)<jupyter_output><empty_output><jupyter_text>Section 3.4: Analysis (40 points)Analyze the behavior of your multileaving implementation by testing it with two different `topk` values: $[5,20]$ and with different number of evaluation clicks: $[2000, 10000, 50000]$.Put the preference of the unbiased method over the biased method (i.e. `pref[2,1]` in the matrix output of `simulate_multileaving` function) for these experiments in a table.**Rubric:**- Six experiments: 12 points- Analysis of the observations: 28 points<jupyter_code># YOUR CODE HERE
raise NotImplementedError()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1055532713162358, 'fraction_numerical': 0.022215655034177932, 'mean_word_length': 3.3448333612460224, 'pattern_counts': {'":': 33, '<': 180, '<?xml version=': 0, '>': 177, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 24, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '41', 'dir': 'jupyter-structured-clean-dedup', 'id': '216102', 'license': 'MIT', 'n_tokens_mistral': 25646, 'n_tokens_neox': 23887, 'n_words': 7832, 'path': 'lab_2/lab_2.ipynb', 'repo_name': 'Kacper-Sawicz/Information_Retrieval_labs', 'size': '77892'}