<commit_before># Machine-readable version of Bibliographical Survey of Prose Fiction Published in the British Isles

Scripts and copies of original data required to generate machine-readable versions
of the following sources:

* "British Fiction 1800-1829," ed. Garside et al. http://www.british-fiction.cf.ac.uk
* "The English Novel, 1830-1836," ed. Garside et al. http://www.cardiff.ac.uk/encap/journals/corvey/1830s/index.html
* Updates 1-6. http://www.romtext.cf.ac.uk/reports/index.html

The following files contain machine-readable versions of these sources:

- `data/british-fiction-1800-1829.json`

## Recreating the dataset

1. Retrieve the index and detail pages from  `http://www.british-fiction.cf.ac.uk` using `scripts/download-british-fiction-1800-1836.py`.
2. Process the downloaded pages with `scripts/process-british-fiction-1800-1829.py`.

## TODO

- Updates 1830-1836
- Add 1770-99<commit_msg>Add additional details to README<commit_after># Machine-readable version of *Bibliographical Survey of Prose Fiction Published in the British Isles*

Scripts and copies of original data required to generate machine-readable versions
of the following sources:

* "British Fiction 1800-1829," ed. Garside et al. http://www.british-fiction.cf.ac.uk
* "The English Novel, 1830-1836," ed. Garside et al. http://www.cardiff.ac.uk/encap/journals/corvey/1830s/index.html
* Updates 1-6. http://www.romtext.cf.ac.uk/reports/index.html

The following files contain machine-readable versions of these sources:

- `data/british-fiction-1800-1829.json`

## Recreating the dataset

1. Retrieve the index and detail pages from  `http://www.british-fiction.cf.ac.uk` using `scripts/download-british-fiction-1800-1836.py`.
2. Process the downloaded pages with `scripts/process-british-fiction-1800-1829.py`.

## Mirror of www.british-fiction.cf.ac.uk

The partial mirror of `www.british-fiction.cf.ac.uk` created by
`downloaded-british-fiction-1800-1836.py` was retrieved January 30th, 2015.

## Correcting errors

**Do not edit the `data/british-fiction-1800-1829.json` directly.** If there is
an error that is the result of parsing the mirror of
`http://www.british-fiction.cf.ac.uk` the errors need to be corrected in the
parsing scripts.

## TODO

- Add 1830-1836
- Add "updates" if not already included (a small number)
- Add 1770-99 (not yet trivially machine readable)
{'fraction_non_alphanumeric': 0.11101730688054032, 'fraction_numerical': 0.061629379485014776, 'mean_word_length': 4.266666666666667, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 11, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2663598', 'n_tokens_mistral': 882, 'n_tokens_neox': 743, 'n_words': 236}