<commit_before># dga-sync README


## Sync data.gov.au datsets easily

The Australian government's data.gov.au website references a growing
abundance of public and open data government data resources - more
than 3700 datasets at the time of writing. While in some cases, data.gov.au
provides an API to access a dataset, it doesn't always. For this reason and
others, there are often advantages in downloading the data for local use or
to re-package it. dga-sync utility eases the task of synchronising 
that data to a local file system.

dga-sync uses the JSON metadata stored on data.gov.au for
each dataset to ensure that data files are only downloaded if they are 
newer than what has previously been downloaded. A local copy of the metadata 
is also stored.


## Getting started

```
npm install dga-sync
```

### Simple usage

For each data.gov.au dataset, there is a JSON metadata file (accessed from<commit_msg>minor changes to README and project metadata (2)<commit_after># dga-sync README


## Sync data.gov.au datsets easily

The Australian government's data.gov.au website references a growing
abundance of public and open data government data resources - more
than 3700 datasets at the time of writing. While in some cases, data.gov.au
provides an API to access a dataset, it doesn't always. For this reason and
others, there are often advantages in downloading the data for local use or
to re-package it. The dga-sync utility eases the task of synchronising 
that data to a local file system.

dga-sync uses the JSON metadata stored on data.gov.au for
each dataset to ensure that data files are only downloaded if they are 
newer than what has previously been downloaded. A local copy of the metadata 
is also stored.


## Getting started

```
npm install dga-sync
```

### Simple usage

For each data.gov.au dataset, there is a JSON metadata file (accessed from
{'fraction_non_alphanumeric': 0.050133333333333335, 'fraction_numerical': 0.0048, 'mean_word_length': 4.13972602739726, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3151292', 'n_tokens_mistral': 517, 'n_tokens_neox': 488, 'n_words': 285}