simonhughes22
commited on
Commit
•
afedd94
1
Parent(s):
c27faa6
Update README.md
Browse files
README.md
CHANGED
@@ -19,6 +19,7 @@ widget:
|
|
19 |
example_title: "Positive"
|
20 |
- text: "A boy is jumping on skateboard in the middle of a red bridge. [SEP] The boy skates down the sidewalk on a blue bridge"
|
21 |
example_title: "Negative"
|
|
|
22 |
---
|
23 |
# Cross-Encoder for Hallucination Detection
|
24 |
This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.
|
@@ -48,6 +49,7 @@ etc. See examples below for expected probability scores.
|
|
48 |
|
49 |
## Usage with Sentencer Transformers (Recommended)
|
50 |
|
|
|
51 |
The model can be used like this, on pairs of documents, passed as a list of list of strings (```List[List[str]]]```):
|
52 |
|
53 |
```python
|
@@ -73,6 +75,36 @@ array([0.61051559, 0.00047493709, 0.99639291, 0.00021221573, 0.99599433, 0.00141
|
|
73 |
Note that the model is designed to work with entire documents, so long as they fit into the 512 token context window (across both documents).
|
74 |
Also note that the order of the documents is important, the first document is the source document, and the second document is validated against the first for factual consistency, e.g. as a summary of the first or a claim drawn from the source.
|
75 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
76 |
## Usage with Transformers AutoModel
|
77 |
You can use the model also directly with Transformers library (without the SentenceTransformers library):
|
78 |
|
|
|
19 |
example_title: "Positive"
|
20 |
- text: "A boy is jumping on skateboard in the middle of a red bridge. [SEP] The boy skates down the sidewalk on a blue bridge"
|
21 |
example_title: "Negative"
|
22 |
+
|
23 |
---
|
24 |
# Cross-Encoder for Hallucination Detection
|
25 |
This model was trained using [SentenceTransformers](https://sbert.net) [Cross-Encoder](https://www.sbert.net/examples/applications/cross-encoder/README.html) class.
|
|
|
49 |
|
50 |
## Usage with Sentencer Transformers (Recommended)
|
51 |
|
52 |
+
### Inference
|
53 |
The model can be used like this, on pairs of documents, passed as a list of list of strings (```List[List[str]]]```):
|
54 |
|
55 |
```python
|
|
|
75 |
Note that the model is designed to work with entire documents, so long as they fit into the 512 token context window (across both documents).
|
76 |
Also note that the order of the documents is important, the first document is the source document, and the second document is validated against the first for factual consistency, e.g. as a summary of the first or a claim drawn from the source.
|
77 |
|
78 |
+
### Training
|
79 |
+
|
80 |
+
```python
|
81 |
+
from sentence_transformers.cross_encoder.evaluation import CEBinaryClassificationEvaluator
|
82 |
+
from sentence_transformers import SentenceTransformer, InputExample, losses
|
83 |
+
|
84 |
+
num_epochs = 5
|
85 |
+
model_save_path = "./model_dump"
|
86 |
+
|
87 |
+
# Load some training examples as such, using a pandas dataframe with source and summary columns:
|
88 |
+
train_examples, test_examples = [], []
|
89 |
+
for i, row in df_train.iterrows():
|
90 |
+
train_examples.append(InputExample(texts=[row['source'], row['summary']], label=int(row['label'])))
|
91 |
+
|
92 |
+
for i, row in df_test.iterrows():
|
93 |
+
test_examples.append(InputExample(texts=[row['source'], row['summary']], label=int(row['label'])))
|
94 |
+
test_evaluator = CEBinaryClassificationEvaluator.from_input_examples(test_examples, name='test_eval')
|
95 |
+
|
96 |
+
# Then train the model as such as per the Cross Encoder API:
|
97 |
+
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=train_batch_size)
|
98 |
+
warmup_steps = math.ceil(len(train_dataloader) * num_epochs * 0.1) #10% of train data for warm-up
|
99 |
+
model.fit(train_dataloader=train_dataloader,
|
100 |
+
evaluator=test_evaluator,
|
101 |
+
epochs=num_epochs,
|
102 |
+
evaluation_steps=10_000,
|
103 |
+
warmup_steps=warmup_steps,
|
104 |
+
output_path=model_save_path,
|
105 |
+
show_progress_bar=True)
|
106 |
+
```
|
107 |
+
|
108 |
## Usage with Transformers AutoModel
|
109 |
You can use the model also directly with Transformers library (without the SentenceTransformers library):
|
110 |
|