Update README.md

README.md

@@ -8,14 +8,32 @@ This model was trained using [SentenceTransformers](https://sbert.net) [Cross-En
 The model was trained on the NLI data and a variety of datasets evaluating summarization accuracy for factual consistency, including [FEVER](https://huggingface.co/datasets/fever), [Vitamin C](https://huggingface.co/datasets/tals/vitaminc) and [PAWS](https://huggingface.co/datasets/paws).
 
 ## Performance
-TODO
+
+TRUE Dataset (Minus Vitamin C, FEVER and PAWS) - 0.872 AUC Score
+SummaC Benchmark (Test) - 0.764 Balanced Accuracy
+SummaC Benchmark (Test) - 0.831 AUC Score
+[AnyScale Ranking Test](https://www.anyscale.com/blog/llama-2-is-about-as-factually-accurate-as-gpt-4-for-summaries-and-is-30x-cheaper) - 86.6 % Accuracy
+
 ## Usage
 
-Pre-trained models can be used like this:
+The model can be used like this:
+
 ```python
 from sentence_transformers import CrossEncoder
 model = CrossEncoder('vectara/hallucination_evaluation_model')
-scores = model.predict([('A man is eating pizza', 'A man eats something'), ('A black race car starts up in front of a crowd of people.', 'A man is driving down a lonely road.')])
+model.predict([
+    ["A man walks into a bar and buys a drink", "A bloke swigs alcohol at a pub"],
+    ["A person on a horse jumps over a broken down airplane.", "A person is at a diner, ordering an omelette."],
+    ["A person on a horse jumps over a broken down airplane.", "A person is outdoors, on a horse."],
+    ["A boy is jumping on skateboard in the middle of a red bridge.", "The boy skates down the sidewalk on a blue bridge"],
+    ["A man with blond-hair, and a brown shirt drinking out of a public water fountain.", "A blond drinking water in public."],
+    ["A man with blond-hair, and a brown shirt drinking out of a public water fountain.", "A blond man wearing a brown shirt is reading a book."],    
+])
+```
+
+This returns a numpy array:
+```
+array([6.1051625e-01, 4.7493601e-04, 9.9639291e-01, 2.1221593e-04, 9.9599433e-01, 1.4126947e-03], dtype=float32)
 ```
 
 ## Usage with Transformers AutoModel
@@ -27,9 +45,25 @@ import torch
 model = AutoModelForSequenceClassification.from_pretrained('vectara/hallucination_evaluation_model')
 tokenizer = AutoTokenizer.from_pretrained('vectara/hallucination_evaluation_model')
 
-features = tokenizer(['A man is eating pizza', 'A black race car starts up in front of a crowd of people.'], ['A man eats something', 'A man is driving down a lonely road.'],  padding=True, truncation=True, return_tensors="pt")
+pairs = [
+    ["A man walks into a bar and buys a drink", "A bloke swigs alcohol at a pub"],
+    ["A person on a horse jumps over a broken down airplane.", "A person is at a diner, ordering an omelette."],
+    ["A person on a horse jumps over a broken down airplane.", "A person is outdoors, on a horse."],
+    ["A boy is jumping on skateboard in the middle of a red bridge.", "The boy skates down the sidewalk on a blue bridge"],
+    ["A man with blond-hair, and a brown shirt drinking out of a public water fountain.", "A blond drinking water in public."],
+    ["A man with blond-hair, and a brown shirt drinking out of a public water fountain.", "A blond man wearing a brown shirt is reading a book."],    
+]
+
+inputs = tokenizer.batch_encode_plus(pairs, return_tensors='pt', padding=True)
 
 model.eval()
 with torch.no_grad():
-    scores = model(**features)
+    outputs = model(**inputs)
+    logits = outputs.logits.cpu().detach().numpy()
+    scores = 1 / (1 + np.exp(-logits)).flatten()
+```
+
+This returns a numpy array:
+```
+array([6.1051559e-01, 4.7493709e-04, 9.9639291e-01, 2.1221573e-04, 9.9599433e-01, 1.4127002e-03], dtype=float32)
 ```