Update README.md
Browse files
README.md
CHANGED
@@ -129,27 +129,31 @@ python -m src.prediction.evaluate_veracity
|
|
129 |
|
130 |
## Format for submission files
|
131 |
|
132 |
-
To facilitate human evaluation, the submission file should include the text of the evidence documents used, retrieved through the `url` field. If external knowledge is utilized, please provide the scraped text. If our provided knowledge store is used, this can be achieved by running the following code block (see [veracity_with_scraped_text.py](https://huggingface.co/chenxwh/AVeriTeC/blob/main/src/prediction/veracity_with_scraped_text.py) for adding the text to the previous prediction file. An example output for the dev set is [here](https://huggingface.co/chenxwh/AVeriTeC/blob/main/data_store/dev_veracity_prediction_for_submission.json).
|
133 |
```bash
|
134 |
python -m src.prediction.veracity_with_scraped_text --knowledge_store_dir <directory_of_the_knowledge_store>
|
135 |
```
|
136 |
|
137 |
-
|
138 |
```json
|
139 |
-
|
140 |
-
|
141 |
-
|
142 |
-
|
143 |
-
|
144 |
-
|
145 |
-
|
146 |
-
|
147 |
-
|
148 |
-
|
149 |
-
|
150 |
-
|
151 |
-
|
152 |
-
|
|
|
|
|
|
|
|
|
153 |
```
|
154 |
|
155 |
## Citation
|
|
|
129 |
|
130 |
## Format for submission files
|
131 |
|
132 |
+
To facilitate human evaluation, the submission file should include the text of the evidence documents used, retrieved through the `url` field. If external knowledge is utilized, please provide the scraped text. If our provided knowledge store is used, this can be achieved by running the following code block (see [veracity_with_scraped_text.py](https://huggingface.co/chenxwh/AVeriTeC/blob/main/src/prediction/veracity_with_scraped_text.py) for adding the text to the previous prediction file). An example output for the dev set is [here](https://huggingface.co/chenxwh/AVeriTeC/blob/main/data_store/dev_veracity_prediction_for_submission.json).
|
133 |
```bash
|
134 |
python -m src.prediction.veracity_with_scraped_text --knowledge_store_dir <directory_of_the_knowledge_store>
|
135 |
```
|
136 |
|
137 |
+
The final submission file should include the following information:
|
138 |
```json
|
139 |
+
[
|
140 |
+
{
|
141 |
+
"claim_id": "The ID of the sample.",
|
142 |
+
"claim": "The claim text itself.",
|
143 |
+
"pred_label": "The predicted label of the claim.",
|
144 |
+
"evidence": [
|
145 |
+
{
|
146 |
+
"question": "The text of the generated question.",
|
147 |
+
"answer": "The text of the answer to the generated question.",
|
148 |
+
"url": "The source URL for the answer.",
|
149 |
+
"scraped_text": "The text scraped from the URL."
|
150 |
+
}
|
151 |
+
]
|
152 |
+
},
|
153 |
+
{
|
154 |
+
...
|
155 |
+
},
|
156 |
+
]
|
157 |
```
|
158 |
|
159 |
## Citation
|