<commit_before># Binary Time Series Format

## Record format, all 32 bit integers:

- `N`: number of data points
- `L`: length of the name in **bytes**
- `T`: start time int64 in unix epoch seconds (including negative times for before 1970)
- `D`: int32 epoch seconds between data points
- `X#`: `F-3` extra integers padding out to the size of the record header specified in the file header.
- `S`: `L` bytes of a string representing the name of the record, in UTF-8
- `P`: `N` int32 representing the data points (`MIN_INT` represents missing data point)

## File format:

Designated extension: `.btsf`

- A header containing:
  - `V` a version number (currently 1)
  - `F` the number of **bytes** in the file header (currently 4*4)
  - `H` the number of **bytes** in a record header (currently `3*4+1*8`)
  - `R` the number of records in the file
- `R` concatenated records

## Pro tips for using the format

When reading, store an integer corresponding to `H` and add that to the byte offset of a record to get the index of the name.
Skip over an additional `L` bytes to get to the data.
Skip over `N*4+L+H` bytes to get to the next record in the file.

When reading the header, read in the numbers and then start at index `F` in the file for reading records.
<commit_msg>New format<commit_after># Binary Time Series Format

All numbers are little-endian.

## Record format, all 32 bit integers:

- `N`: uint32 number of data points
- `L`: uint32 length of the name in **bytes**
- `X#`: `H-8` extra bytes padding out to the size of the record header specified in the file header.
- `S`: `L` bytes of a string representing the name of the record, in UTF-8
- `P`: `N` data points
  - `T`: int32 time in unix epoch
  - `D`: float32 data point

## File format:

Designated extension: `.btsf`

- A header containing:
  - `V` uint32 version number (currently 1)
  - `F` uint32 the number of **bytes** in the file header (currently `4*4`)
  - `H` uint32 the number of **bytes** in a record header (currently `2*4`)
  - `R` uint32 the number of records in the file
- `R` concatenated records

## Pro tips for using the format

When reading, store an integer corresponding to `H` and add that to the byte offset of a record to get the index of the name.
Skip over an additional `L` bytes to get to the data.
Skip over `N*8+L+H` bytes to get to the next record in the file.

When reading the header, read in the numbers and then start at index `F` in the file for reading records.

{'fraction_non_alphanumeric': 0.09543065103113627, 'fraction_numerical': 0.01940962393853619, 'mean_word_length': 3.7123809523809523, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2576944', 'n_tokens_mistral': 785, 'n_tokens_neox': 732, 'n_words': 411}