siddartha-abacus
commited on
Commit
•
8233a1a
1
Parent(s):
0770313
Update README.md
Browse files
README.md
CHANGED
@@ -25,6 +25,24 @@ There are our Needle-in-a-Haystack heatmap results. We are conducting further ev
|
|
25 |
|
26 |
![image/png](https://cdn-uploads.huggingface.co/production/uploads/64c14f6b02e1f8f67c73bd05/Z4uUhcjgf1P7EPGQyRLkW.png)
|
27 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
28 |
## Training Methodology
|
29 |
|
30 |
The methodology for training uses [PoSE](https://arxiv.org/abs/2309.10400) and dynamic-NTK interpolation.
|
|
|
25 |
|
26 |
![image/png](https://cdn-uploads.huggingface.co/production/uploads/64c14f6b02e1f8f67c73bd05/Z4uUhcjgf1P7EPGQyRLkW.png)
|
27 |
|
28 |
+
### MT-Bench Evaluation
|
29 |
+
|
30 |
+
We also measured performance on MT-Bench to verify that the context extension did not significantly impact performance on instruct tasks:
|
31 |
+
|
32 |
+
```
|
33 |
+
####### 1st turn:
|
34 |
+
Meta-Llama-3-70B-Instruct 9.21
|
35 |
+
Llama-3-Giraffe-70B-Instruct 9.19
|
36 |
+
|
37 |
+
####### 2nd turn:
|
38 |
+
Meta-Llama-3-70B-Instruct 2 8.80
|
39 |
+
Llama-3-Giraffe-70B-Instruct 2 8.54
|
40 |
+
|
41 |
+
####### average:
|
42 |
+
Meta-Llama-3-70B-Instruct 9.00
|
43 |
+
Llama-3-Giraffe-70B-Instruct 8.87
|
44 |
+
```
|
45 |
+
|
46 |
## Training Methodology
|
47 |
|
48 |
The methodology for training uses [PoSE](https://arxiv.org/abs/2309.10400) and dynamic-NTK interpolation.
|