Commit
•
152cbb9
1
Parent(s):
01dc13a
Update README.md
Browse files
README.md
CHANGED
|
@@ -96,6 +96,15 @@ After training on two RTX A6000s for \~120,050 steps over a period of 91 hours,
|
|
| 96 |
|
| 97 |
Naturally, as the optimiser state had been lost, the model's learning rate descended slower than it had been previously. Nevertheless, after completing an epoch of training, the model was able to achieve a validation loss of 2.04.
|
| 98 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
## Limitations 🚧
|
| 100 |
Although the model has not been tested for bias, one would expect it to exhibit much of the same, if not all, the biases of [GPT2-XL](https://huggingface.co/gpt2-xl).
|
| 101 |
|
|
|
|
| 96 |
|
| 97 |
Naturally, as the optimiser state had been lost, the model's learning rate descended slower than it had been previously. Nevertheless, after completing an epoch of training, the model was able to achieve a validation loss of 2.04.
|
| 98 |
|
| 99 |
+
## Benchmarks 📊
|
| 100 |
+
Tested against version 2.0.0 of the [Open Australian Legal QA](https://huggingface.co/datasets/umarbutler/open-australian-legal-qa) dataset, the model achieved a perplexity of 8.01, outperforming all known language models for Australian law.
|
| 101 |
+
| Model | Parameters | Perplexity |
|
| 102 |
+
|--|--|--|
|
| 103 |
+
| **Open Australian Legal LLM** | **1.5B** | **8.01** |
|
| 104 |
+
| [Open Australian Legal Phi 1.5](https://huggingface.co/umarbutler/open-australian-legal-phi-1_5) | 1.3B | 8.69 |
|
| 105 |
+
| [Open Australian Legal GPT2](https://huggingface.co/umarbutler/open-australian-legal-gpt2) | 124M | 16.37 |
|
| 106 |
+
| [Open Australian Legal DistilGPT2](https://huggingface.co/umarbutler/open-australian-legal-distilgpt2) | 88.2M | 23.9 |
|
| 107 |
+
|
| 108 |
## Limitations 🚧
|
| 109 |
Although the model has not been tested for bias, one would expect it to exhibit much of the same, if not all, the biases of [GPT2-XL](https://huggingface.co/gpt2-xl).
|
| 110 |
|