philipphager
commited on
Commit
•
79b73b4
1
Parent(s):
59d07ef
Update README.md
Browse files
README.md
CHANGED
@@ -13,9 +13,18 @@ metrics:
|
|
13 |
---
|
14 |
|
15 |
# Naive Listwise MonoBERT trained on Baidu-ULTR
|
16 |
-
A flax-based MonoBERT cross encoder trained on the Baidu-ULTR dataset with a **listwise softmax cross-entropy loss on clicks**. The loss is called "naive" as we use user clicks as a signal of relevance without any additional position bias correction. For more info, read our paper here.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
17 |
|
18 |
## Usage
|
|
|
|
|
19 |
```Python
|
20 |
import jax.numpy as jnp
|
21 |
|
@@ -58,9 +67,5 @@ outputs = model(batch)
|
|
58 |
print(outputs)
|
59 |
```
|
60 |
|
61 |
-
|
62 |
|
63 |
-
| Model | log-likelihood | DCG@1 | DCG@3 | DCG@5 | DCG@10 | nDCG@10 | MRR@10 |
|
64 |
-
|---------------------|----------------|--------|--------|--------|--------|---------|--------|
|
65 |
-
| Naive Pointwise | 0.2272 | 1.6836 | 3.5616 | 4.8822 | 7.4244 | 0.3640 | 0.6096 |
|
66 |
-
| **Naive Listwise** | - | 1.9738 | 4.1609 | 5.6861 | 8.5432 | 0.4091 | 0.6436 |
|
|
|
13 |
---
|
14 |
|
15 |
# Naive Listwise MonoBERT trained on Baidu-ULTR
|
16 |
+
A flax-based MonoBERT cross encoder trained on the [Baidu-ULTR](https://arxiv.org/abs/2207.03051) dataset with a **listwise softmax cross-entropy loss on clicks**. The loss is called "naive" as we use user clicks as a signal of relevance without any additional position bias correction. For more info, [read our paper here](https://arxiv.org/abs/2404.02543).
|
17 |
+
|
18 |
+
## Test Results on Baidu-ULTR Expert Annotations
|
19 |
+
|
20 |
+
| Model | log-likelihood | DCG@1 | DCG@3 | DCG@5 | DCG@10 | nDCG@10 | MRR@10 |
|
21 |
+
|---------------------|----------------|--------|--------|--------|--------|---------|--------|
|
22 |
+
| Naive Pointwise | 0.2272 | 1.6836 | 3.5616 | 4.8822 | 7.4244 | 0.3640 | 0.6096 |
|
23 |
+
| **Naive Listwise** | - | 1.9738 | 4.1609 | 5.6861 | 8.5432 | 0.4091 | 0.6436 |
|
24 |
|
25 |
## Usage
|
26 |
+
Here is an example with a mock input batch for how to download and call the model:
|
27 |
+
|
28 |
```Python
|
29 |
import jax.numpy as jnp
|
30 |
|
|
|
67 |
print(outputs)
|
68 |
```
|
69 |
|
70 |
+
For more details on how to use the model with real data from Baidu-ULTR, take a look at the [evaluation script of our model repository](https://github.com/philipphager/baidu-bert-model/blob/main/eval.py).
|
71 |
|
|
|
|
|
|
|
|