<commit_before># **File Formats**

All the file formats are outlined below including the header and record 
contents.

## **Log file**

Log files contain all the log records.

#### *Log File Header*

Log files are prefixed with an 8-byte header. The header contains the file signature `LOG`, an 8-bit version followed by an unsigned 32-bit integer for boolean flags. The file header also contains the TTL for all the records in the file. There are pros and cons in having the TTL in the file header. For instance, the individual records do not have separate TTLs. While this removes some flexibility in the data model, performance remains extremely high as each record does not need to be evaluated for expiration.

TTL is a duration specified in nanoseconds. 

```
3-byte signature
1-byte version
4-byte flags
8-byte expiration / ttl

0        1        2        3        4        5        6        7        8
+--------+--------+--------+--------+--------+--------+--------+--------+
|      File Signature      |   Ver  |               Flags               |
+--------+--------+--------+--------+--------+--------+--------+--------+
|                   Expiration Time / Time To Live                      |
+--------+--------+--------+--------+--------+--------+--------+--------+

```

- `IDX` file signature
- an unsigned 8-bit integer to represent the file version
- an unsigned 32-bit integer for boolean flags
- a signed 64-bit integer for time to live
  - **Units:** duration in nanoseconds

#### *Log Records*

Each log record has a 16-byte header followed by the record data. The header
consists of:

```
4-byte size
4-byte flags
8-byte time
data

0        8        16
+--------+--------+--------+--------+--------+
| s + f  |  time  |           data           |
+--------+--------+--------+--------+--------+<commit_msg>Update file signature<commit_after># **File Formats**

All the file formats are outlined below including the header and record 
contents.

## **Log file**

Log files contain all the log records.

#### *Log File Header*

Log files are prefixed with an 8-byte header. The header contains the file signature `LOG`, an 8-bit version followed by an unsigned 32-bit integer for boolean flags. The file header also contains the TTL for all the records in the file. There are pros and cons in having the TTL in the file header. For instance, the individual records do not have separate TTLs. While this removes some flexibility in the data model, performance remains extremely high as each record does not need to be evaluated for expiration.

TTL is a duration specified in nanoseconds. 

```
3-byte signature
1-byte version
4-byte flags
8-byte expiration / ttl

0        1        2        3        4        5        6        7        8
+--------+--------+--------+--------+--------+--------+--------+--------+
|      File Signature      |   Ver  |               Flags               |
+--------+--------+--------+--------+--------+--------+--------+--------+
|                   Expiration Time / Time To Live                      |
+--------+--------+--------+--------+--------+--------+--------+--------+

```

- `LOG` file signature
- an unsigned 8-bit integer to represent the file version
- an unsigned 32-bit integer for boolean flags
- a signed 64-bit integer for time to live
  - **Units:** duration in nanoseconds

#### *Log Records*

Each log record has a 16-byte header followed by the record data. The header
consists of:

```
4-byte size
4-byte flags
8-byte time
data

0        8        16
+--------+--------+--------+--------+--------+
| s + f  |  time  |           data           |
+--------+--------+--------+--------+--------+
{'fraction_non_alphanumeric': 0.21888646288209607, 'fraction_numerical': 0.016921397379912665, 'mean_word_length': 1.4881194840461642, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3244560', 'n_tokens_mistral': 978, 'n_tokens_neox': 929, 'n_words': 428}