Expanded benchmarks.

README.md

@@ -23,6 +23,7 @@ widget:
 - text: >-
     A <mask> of trade is valid to the extent to which it is not against public
     policy, whether it is in severable terms or not.
+- text: Norfolk Island is an Australian <mask>.
 - text: >-
     In Mabo v <mask> (No 2) (1992) 175 CLR 1, the Court found that the doctrine
     of terra nullius was not applicable to Australia at the time of British
@@ -52,7 +53,8 @@ pipeline_tag: fill-mask
 
 # EmuBert
 <img src="https://huggingface.co/umarbutler/emubert/resolve/main/logo.png" width="100" align="left" />
-EmuBert is the largest open-source masked language model for Australian law.
+
+EmuBert is the **largest** and **most accurate** open-source masked language model for Australian law.
 
 Trained on 180,000 laws, regulations and decisions across six Australian jurisdictions, totalling 1.4 billion tokens, taken from the [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus), EmuBert is well suited for a diverse range of downstream natural language processing tasks applied to the Australian legal domain, including **text classification**, **named entity recognition** and **question answering**. It can also be used as-is for **semantic similarity**, **vector search** and general **sentence embedding**.
 
@@ -147,7 +149,7 @@ Whereas the final block of the training set would have been dropped if it did no
 
 The final training set comprised 2,885,839 blocks totalling 1,477,549,568 tokens, the validation set comprised 155,563 blocks totalling 79,648,256 tokens, and the test set comprised 155,696 blocks totalling 79,716,352 tokens.
 
-Instead of training EmuBert from scratch, [Roberta](https://huggingface.co/roberta-base)'s weights were all reused, except for its token embeddings which were either replaced with the average token embedding or, if a token was shared between Roberta and EmuBert's vocabularies, moved to its new position in EmuBert's vocabulary.
+Instead of training EmuBert from scratch, [Roberta](https://huggingface.co/roberta-base)'s weights were all reused, except for its token embeddings which were either replaced with the average token embedding or, if a token was shared between Roberta and EmuBert's vocabularies, moved to its new position in EmuBert's vocabulary, as described by Umar Butler in his blog post, [*How to reuse model weights when training with a new tokeniser*](https://umarbutler.com/how-to-reuse-model-weights-when-training-with-a-new-tokeniser/).
 
 In order to reduce training time, [Better Transformer](https://huggingface.co/docs/optimum/en/bettertransformer/overview) was used to enable fast path execution and scaled dot-product attention, alongside automatic mixed 16-bit precision and [bitsandbytes](https://huggingface.co/docs/bitsandbytes/main/en/reference/optim/adamw#bitsandbytes.optim.AdamW8bit)' 8-bit implementation of AdamW, all of which have been shown to have little to no detrimental effect on performance.
 
@@ -177,12 +179,15 @@ The code used to create EmuBert may be found [here](https://github.com/umarbutle
 ## Benchmarks 📊
 EmuBert achieves a [(pseudo-)perplexity](https://doi.org/10.18653/v1/2020.acl-main.240) of 2.05 against [version 2.0.0](https://huggingface.co/datasets/umarbutler/open-australian-legal-qa/tree/b53a24f8edf5eb33d033a53b5b53d0a4a220d4ae) of the [Open Australian Legal QA](https://huggingface.co/datasets/umarbutler/open-australian-legal-qa) dataset, outperforming all known state-of-the-art masked language models, as shown below:
 
-| Model          | Perplexity |
-| -------------- | ---------- |
-| **EmuBert**    | **2.05**   |
-| Roberta        | 2.38       |
-| Bert (cased)   | 2.18       |
-| Bert (uncased) | 2.41       |
+| Model                   | Perplexity |
+| ----------------------- | ---------- |
+| **EmuBert**             | **2.05**   |
+| Bert (cased)            | 2.18       |
+| Legal-Bert              | 2.33       |
+| Roberta                 | 2.38       |
+| Bert (uncased)          | 2.41       |
+| Legalbert (casehold)    | 3.08       |
+| Legalbert (pile-of-law) | 4.41       |
 
 ## Limitations 🚧
 Although informal testing has not revealed any racial, sexual, gender or other social biases, given that Roberta's weights were reused, it is possible that there may be some biases that have been transferred over to EmuBert. It is also possible that there are social biases present in the Corpus that may have been introduced via training. More rigorous testing is necessary to determine the true extent of any biases present in EmuBert.