<commit_before>This is a directory for local data.

It contains our content info as flat JSON files, keyed by ID.
(Note: uppercase letters get an underscore prefix, to work around
case-insensitive filesystems.)

Example data follows.

**TEMP:** We are temporarily checking these files into the git repo,
in order to test lookups of existing URLs (that are known in the local db).
We can/should remove these files (and update our tests) once we have
new content acceptance hooked up.

`100.json`:

```json
{
    "id": "100",
    "url": "http://e.i.uol.com.br/outros/0907/090731cielao1.jpg",
    "ready": true,
    "failed": false,
    "progress": 1,
    "mime": "image/jpeg",
    "size": 33998,
    "dzi": {
        "tileSize": 254,
        "tileOverlap": 1,
        "tileFormat": "jpg",
        "width": 208,
        "height": 208
    }
}
```

`100_U.json`:

```json
{
    "id": "100U",
    "url": "http://www.archdrawing.ireland.anglican.org/archive/files/a0630962f3164841684eea66b6552445.JPG",
    "ready": true,
    "failed": false,
    "progress": 1,
    "mime": "image/jpeg",
    "size": 202199,
    "dzi": {
        "tileSize": 254,
        "tileOverlap": 1,
        "tileFormat": "jpg",
        "width": 1824,
        "height": 1368
    }
}
```

`4rcn.json`:

```json
{
    "id": "4rcn",
    "url": "http://media.stenaline.com/media_SE/lalandia-map-zoomit/lalandia-map.jpg",
    "ready": true,
    "failed": false,
    "progress": 1,
    "mime": "image/jpeg",
    "size": 9115770,
    "dzi": {
        "tileSize": 254,
        "tileOverlap": 1,
        "tileFormat": "jpg",
        "width": 5058,
        "height": 3750
    }
}
```

`hdfm.json`:

```json
{
    "id": "hdfm",
    "url": "http://wdh.blob.core.windows.net/deepzoom-sources/hampi/HampiCB/02-Lakshmi,%20The%20Temple%20Elephant/09-Lakshmi%20and%20little%20girl.jpg",
    "ready": true,
    "failed": false,
    "progress": 1,
    "mime": "image/jpeg",
    "size": 15533083,
    "dzi": {
        "tileSize": 254,
        "tileOverlap": 1,
        "tileFormat": "jpg",
        "width": 4032,
        "height": 6048
    }
}
```<commit_msg>Data: remove duplication from IDs readme.<commit_after>This is a directory for local data.

It contains our content info as flat JSON files, keyed by ID.
(Note: uppercase letters get an underscore prefix, to work around
case-insensitive filesystems.)

**TEMP:** We are temporarily checking some sample data into the git repo,
in order to test lookups of existing URLs (that are known in the local db).
We can/should remove these files (and update our tests) once we have
new content acceptance hooked up.
{'fraction_non_alphanumeric': 0.15414110429447853, 'fraction_numerical': 0.054831288343558285, 'mean_word_length': 2.8882265275707897, 'pattern_counts': {'":': 52, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6931910', 'n_tokens_mistral': 1038, 'n_tokens_neox': 907, 'n_words': 265}