region:us
---
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  - split: validation
    path: data/validation-*
  - split: test
    path: data/test-*
dataset_info:
  features:
  - name: section
    dtype: string
  - name: filename
    dtype: string
  - name: text
    dtype: string
  splits:
  - name: train
    num_bytes: 36995955.01408451
    num_examples: 127
  - name: validation
    num_bytes: 2039147.1267605633
    num_examples: 7
  - name: test
    num_bytes: 2330453.8591549294
    num_examples: 8
  download_size: 23775111
  dataset_size: 41365556.00000001
---
# Dataset Card for "bees-internal"

Full-length OCRs of Bee material. Documents were split into multiple chunks if over 1 MB of text to not destroy the CPU when tokenizing.




Tokens:

```json
{
"metadata": {
    "model": "gpt-3.5-turbo",
    "clean_text": true,
    "extension": "mmd",
    "recursive": true,
    "global_token_count": 9105492
  }
}
```


Files:

```yml
splits:
- name: train
  num_bytes: 36027579.5882353
  num_examples: 122
- name: validation
  num_bytes: 2067156.205882353
  num_examples: 7
- name: test
  num_bytes: 2067156.205882353
  num_examples: 7
```

[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)