<commit_before>## Testing

To test bash scripts, we use [bats](https://github.com/sstephenson/bats), then generate tests. This is to speed up both running the tests (as they're not generated every time), and writing them (simple YAML configuration).

### To run the tests:
- Install [bats](https://github.com/sstephenson/bats) first.
- Install the submodules if needed:
    - [bats-assert](https://github.com/ztombol/bats-assert)
    - [bats-support](https://github.com/ztombol/bats-support)

```shell
$ ./test/run-tests.sh 
$ ./test/run-argparse-tests.sh
```

### Generating tests:

1. Write a spec in `test/spec.yml`.
2. Run `test/run-tests.sh`.

- `test/run-tests.sh` calls a Python script called `bats-test-gen.py` and relies on stdout and redirection to write to a file in a new, temporary directory called `".tmp-dir"` in the repo root. It copies necessary files then runs the generated tests within.
<commit_msg>Update bats links in readme<commit_after>## Testing

To test bash scripts, we use [bats](https://github.com/bats-core/bats-core), then generate tests. This is to speed up both running the tests (as they're not generated every time), and writing them (simple YAML configuration).

### To run the tests:
- Install [bats](https://github.com/bats-core/bats-core) first.
- Install the submodules if needed:
    - [bats-assert](https://github.com/ztombol/bats-assert)
    - [bats-support](https://github.com/ztombol/bats-support)

```shell
$ ./test/run-tests.sh 
$ ./test/run-argparse-tests.sh
```

### Generating tests:

1. Write a spec in `test/spec.yml`.
2. Run `test/run-tests.sh`.

- `test/run-tests.sh` calls a Python script called `bats-test-gen.py` and relies on stdout and redirection to write to a file in a new, temporary directory called `".tmp-dir"` in the repo root. It copies necessary files then runs the generated tests within.

{'fraction_non_alphanumeric': 0.13670613562970937, 'fraction_numerical': 0.002152852529601722, 'mean_word_length': 3.85378590078329, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2431991', 'n_tokens_mistral': 609, 'n_tokens_neox': 591, 'n_words': 204}