task_categories:text-generation task_categories:fill-mask task_ids:language-modeling task_ids:masked-language-modeling annotations_creators:no-annotation language_creators:found size_categories:10K<n<100K source_datasets:Federal Register of Legislation source_datasets:Federal Court of Australia source_datasets:NSW Legislation source_datasets:Queensland Legislation source_datasets:Western Australian Legislation source_datasets:South Australian Legislation source_datasets:Tasmanian Legislation language:en license:cc-by-4.0 legal doi:10.57967/hf/1111 region:us
---
language:
- en
license: cc-by-4.0
license_details: https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus/blob/main/LICENCE.md
tags:
- legal
annotations_creators:
- no-annotation
language_creators:
- found
language_details: en-AU, en-GB
pretty_name: Open Australian Legal Corpus
size_categories:
- 10K<n<100K
source_datasets:
- Federal Register of Legislation
- Federal Court of Australia
- NSW Legislation
- Queensland Legislation
- Western Australian Legislation
- South Australian Legislation
- Tasmanian Legislation
task_categories:
- text-generation
- fill-mask
task_ids:
- language-modeling
- masked-language-modeling
dataset_info:
  features:
  - name: text
    dtype: string
  - name: type
    dtype: string
  - name: jurisdiction
    dtype: string
  - name: source
    dtype: string
  - name: citation
    dtype: string
  - name: url
    dtype: string
  config_name: train
  splits:
  - name: train
    num_bytes: 5210239612
    num_examples: 98424
  download_size: 5261513500
  dataset_size: 5210239612
---
<!-- To update the above `dataset_info` section, please run the following command: `datasets-cli test open_australian_legal_corpus.py --save_info --all_configs`. -->

# **Open Australian Legal Corpus ‍⚖️**
<a href="https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus" alt="Release"><img src="https://img.shields.io/badge/release-v3.1.0-green"></a>

The Open Australian Legal Corpus is the first and only multijurisdictional open corpus of Australian legislative and judicial documents.

Comprised of 98,424 texts totalling over 40 million lines and 500 million tokens, the Corpus includes almost every in force statute and regulation in the Commonwealth, New South Wales, Queensland, Western Australia, South Australia, Tasmania and Norfolk Island, in addition to thousands of bills and tens of thousands of court and tribunal decisions.

As the largest free and open dataset of its kind to-date, the Corpus is intended to progress the burgeoning field of legal AI research in Australia by allowing researchers to pretrain and finetune machine learning models for downstream natural language processing tasks applied to the Australian legal domain such as document classification, summarisation, information retrieval and question answering.

To ensure its accessibility to as wide an audience as possible, the Corpus and all of its documents are distributed under permissive licences that allow for both commercial and non-commercial usage (see the [Licence 📄](LICENCE.md)).

## Statistics 📊
The Corpus is comprised of 98,424 documents, totalling 45,149,530 lines and 553,023,599 tokens.

A breakdown of the number of documents by type and source is provided below:
| Source                          |   Primary Legislation |   Secondary Legislation |   Bill |   Decision |   **Total** |
|:--------------------------------|----------------------:|------------------------:|-------:|-----------:|--------:|
| Federal Register of Legislation |                  3,407 |                   25,535 |   5,669 |          0 |**34,611**|
| Federal Court of Australia      |                     0 |                       0 |      0 |      55,667 |**55,667**|
| NSW Legislation                 |                   876 |                     739 |      0 |          0 |**1,615**|
| Queensland Legislation          |                   565 |                     431 |    385 |          0 |**1,381**|
| Western Australian Legislation  |                   811 |                     756 |      0 |          0 |**1,567**|
| South Australian Legislation    |                   561 |                     494 |      0 |          0 |**1,055**|
| Tasmanian Legislation           |                   859 |                    1,669 |      0 |          0 |**2,528**|
| **Total**                           |**7,079**|**29,624**|**6,054**|**55,667**|**98,424**|

## Structure 🗂️
The Corpus is stored in [corpus.jsonl](corpus.jsonl), a json lines file where each line represents a document consisting of five keys:

| Key | Description |
| --- | --- |
| text | The UTF-8 encoded text of the document. |
| type | The type of the document. Possible values are `primary_legislation`, `secondary_legislation`, `bill` and `decision`. |
| jurisdiction | The jurisdiction of the document. Possible values are `commonwealth`, `new_south_wales`, `queensland`, `western_australia`, `south_australia`, `tasmania` and `norfolk_island`. |
| source | The source of the document. Possible values are `federal_register_of_legislation`, `federal_court_of_australia`, `nsw_legislation`, `queensland_legislation`, `western_australian_legislation`, `south_australian_legislation` and `tasmanian_legislation`. |
| citation | The title of the document with, in the case of legislation and bills, an abbreviated form of the document's jurisdiction enclosed in parentheses appended. |
| url | A hyperlink to the document. |

## Collection 📥
Documents were sourced from the [Federal Register of Legislation](https://www.legislation.gov.au/), [Federal Court of Australia](https://www.fedcourt.gov.au/digital-law-library/judgments/search), [NSW Legislation](https://legislation.nsw.gov.au/), [Queensland Legislation](https://www.legislation.qld.gov.au/), [Western Australian Legislation](https://www.legislation.wa.gov.au/), [South Australian Legislation](https://www.legislation.sa.gov.au/) and [Tasmanian Legislation](https://www.legislation.tas.gov.au/) databases. Unfortunately, due to copyright restrictions as well as refusals to grant permission to scrape their websites, the [High Court of Australia](https://eresources.hcourt.gov.au/), [Victorian Legislation](https://www.legislation.vic.gov.au/copyright), [ACT Legislation](https://www.legislation.act.gov.au) and [NT Legislation](https://legislation.nt.gov.au/) databases could not be incorporated into the Corpus.

The text of these documents were extracted using [inscriptis](https://github.com/weblyzard/inscriptis) or, in the case of the [South Australian Legislation](https://www.legislation.sa.gov.au/) database which was provided as an archive of rtf files, [striprtf](https://github.com/joshy/striprtf). No post-processing was applied.

The below table shows the types of documents taken from each source and the date upon which they were collected (for the [South Australian Legislation](https://www.legislation.sa.gov.au/) database, the date provided is when the database was archived):
| Source | Date | Documents |
| --- | --- | --- |
| Federal Register of Legislation | 9 August 2023 | <ul><li>The most recent versions of all in force acts and the Constitution (primary legislation);</li> <li>The most recent versions of all in force legislative instruments, notifiable instruments, administrative arrangements orders and prerogative instruments (secondary legislation); and</li> <li>The as made versions of all bills.</li></ul> |
| Federal Court of Australia | 9 August 2023 | <ul><li>All decisions of the Federal Court of Australia, Industrial Relations Court of Australia, Australian Competition Tribunal, Copyright Tribunal, Defence Force Discipline Appeal Tribunal, Federal Police Disciplinary Tribunal, Trade Practices Tribunal and Supreme Court of Norfolk Island.</li></ul> |
| NSW Legislation | 9 August 2023 | <ul><li>The most recent versions of all in force public and private acts (primary legislation); and</li> <li>The most recent versions of all in force statutory instruments and environmental planning instruments (secondary legislation).</li></ul> |
| Queensland Legislation | 9 August 2023 | <ul><li>The most recent versions of all in force acts (primary legislation);</li> <li>The most recent versions of all in force statutory instruments (secondary legislation); and</li> <li>The as introduced versions of all bills.</li></ul> |
| Western Australian Legislation | 14 September 2023 | <ul><li>The most recent versions of all in force acts (primary legislation); and</li> <li>The most recent versions of all in force subsidiary legislation (secondary legislation).</li></ul> |
| South Australian Legislation | 3 July 2023 | <ul><li>The most recent versions of all in force acts (primary legislation); and</li> <li>The most recent versions of all in force proclamations, policies and regulations (secondary legislation).</li></ul> |
| Tasmanian Legislation | 9 August 2023 | <ul><li>The most recent versions of all in force acts (primary legislation); and</li> <li>The most recent versions of all in force statutory rules (secondary legislation).</li></ul> |

The code used to collect these documents and create the Corpus can be found [here](https://github.com/umarbutler/open-australian-legal-corpus-creator).

## Citation 🔖
If you rely on the Corpus, please cite:
```bibtex
@misc{butler-2023-open-australian-legal-corpus,
    author = {Butler, Umar},
    year = {2023},
    title = {Open Australian Legal Corpus},
    publisher = {Hugging Face},
    version = {3.1.0},
    doi = {10.57967/hf/1111},
    url = {https://huggingface.co/datasets/umarbutler/open-australian-legal-corpus}
}
```

## Acknowledgements 🙏
In the spirit of reconciliation, the author acknowledges the Traditional Custodians of Country throughout Australia and their connections to land, sea and community. He pays his respect to their Elders past and present and extends that respect to all Aboriginal and Torres Strait Islander peoples today.

The author thanks the [Federal Register of Legislation](https://www.legislation.gov.au/), [Federal Court of Australia](https://www.fedcourt.gov.au/digital-law-library/judgments/search), [NSW Legislation](https://legislation.nsw.gov.au/), [Queensland Legislation](https://www.legislation.qld.gov.au/), [Western Australian Legislation](https://www.legislation.wa.gov.au/) and [Tasmanian Legislation](https://www.legislation.tas.gov.au/) for all granting him permission to scrape their websites, as well as [South Australian Legislation](https://www.legislation.sa.gov.au/) for providing him with a copy of their legislative database.

The author also thanks the makers of [Visual Studio Code](https://github.com/microsoft/vscode), [Python](https://github.com/python/cpython), [Jupyter Notebook](https://github.com/jupyter/notebook), [urllib3](https://github.com/urllib3/urllib3), [certifi](https://github.com/certifi/python-certifi), [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/), [lxml](https://github.com/lxml/lxml), [inscriptis](https://github.com/weblyzard/inscriptis), [striprtf](https://github.com/joshy/striprtf) and [pytz](https://github.com/stub42/pytz), as well as the creators of the [pile-of-law](https://huggingface.co/datasets/pile-of-law/pile-of-law) corpus which served as a large source of inspiration for this project.

Finally, the author is eternally grateful for the endless support of his wife and her willingness to put up with many a late night spent writing code and quashing bugs.

## Changelog 🔄
All notable changes to the Corpus will be documented here. The format of this changelog is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

### 3.1.0 - 2023-09-14
### Changed
- Rescraped the [Western Australian Legislation](https://www.legislation.wa.gov.au/) database on 14 September 2023.

### Fixed
- Ensured that documents from the [Western Australian Legislation](https://www.legislation.wa.gov.au/) database are decoded correctly.

### 3.0.0 - 2023-08-09
#### Added
- Added the `jurisdiction` field.

#### Changed
- Recollected the Corpus on 9 August 2023.
- Set the citation of NSW legislation to be everything before the first occurance of '(NSW)' in title, with '(NSW)' then being appended to the citation.

#### Fixed
- Removed extra whitespace characters from citations.
- '(Cth)' is not longer added to the end of citations of Norfolk Island legislation.
- '(NSW)' is no longer added to the end of citations that already have '(NSW)' in them.
- Created citations for a number of decisions that were missing them.

### 2.0.0 - 2023-07-01
#### Added
- Added the `citation` field.
- Added this changelog.

#### Changed
- Recollected the Corpus on 1 July 2023.

### 1.0.0 - 2023-06-25
#### Added
- Added 97,750 texts collected from the [Federal Register of Legislation](https://www.legislation.gov.au/), [Federal Court of Australia](https://www.fedcourt.gov.au/digital-law-library/judgments/search), [NSW Legislation](https://legislation.nsw.gov.au/), [Queensland Legislation](https://www.legislation.qld.gov.au/), [Western Australian Legislation](https://www.legislation.wa.gov.au/), [South Australian Legislation](https://www.legislation.sa.gov.au/) and [Tasmanian Legislation](https://www.legislation.tas.gov.au/) databases.
- Created the `text`, `type`, `source` and `url` fields.

## Licence 📄
The Corpus itself is licensed under a [Creative Commons Attribution 4.0 International Licence](https://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, on the condition that you give appropriate credit to the original author and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

Documents sourced from the [Federal Register of Legislation](https://www.legislation.gov.au/Content/Disclaimer#copyright), [NSW Legislation](https://legislation.nsw.gov.au/copyright), [Queensland Legislation](https://www.legislation.qld.gov.au/copyright), [Western Australian Legislation](https://www.legislation.wa.gov.au/legislation/statutes.nsf/copyright.html), [South Australian Legislation](https://www.legislation.sa.gov.au/copyright) and [Tasmanian Legislation](https://www.legislation.tas.gov.au/copyrightanddisclaimer) databases were also all licensed under a [Creative Commons Attribution 4.0 International Licence](https://creativecommons.org/licenses/by/4.0/) at the time of their inclusion in the Corpus. They require specific wordings to be used for attribution, which are provided [here](LICENCE.md). 

With regard to documents from the [Federal Court of Australia](https://www.fedcourt.gov.au/copyright), at the time of scraping, their licence permitted users to download, display, print and reproduce material in an unaltered form for personal, non-commercial use or use within their organisation. It also permitted judgements and decisions or excerpts thereof to be reproduced or published in an unaltered form (including for commercial use), provided that they are acknowledged to be judgements or decisions of the Court or Tribunal, any commentary, head notes or additional information added is clearly attributed to the publisher or organisation and not the Court or Tribunals, and the source from which the judgement was copied is acknowledged.

The complete version of this licence is available [here](LICENCE.md).