<commit_before># LZMA2 Format

LZMA2 is a container of chunks. Each chunk is lead by a control byte.

Following the C implementation in the LZMA SDK the control byte can be
described as such:

Chunk header         | Description
:------------------- | :--------------------------------------------------
`00000000`           | End of LZMA2 stream
`00000001 U U`       | Uncompressed chunk, reset dictionary
`00000010 U U`       | Uncompressed chunk, no reset of dictionary
`100uuuuu U U P P`   | LZMA, no reset
`101uuuuu U U P P`   | LZMA, reset state
`110uuuuu U U P P S` | LZMA, reset state, new properties
`111uuuuu U U P P S` | LZMA, reset state, new properties, reset dictionary

The symbols used are described by following table.

A dictionary reset requires always new properties. If this is an
uncompressed chunk the properties need to be provided in the next
compressed chunk. New properties require a reset of the state.

Uncompressed data is written into the dictionary.

Symbol | Description
:----- | :-----------------
u      | unpacked size bit
U      | unpacked size byte
P      | packed size byte
S      | properties byte

The unpacked size and packed size are written in big-endian byte order.
The actual size is one more. So a chunk with 1 byte unpacked data will
store size 0 in the unpacked byte.

The properties byte provides the parameters pb, lc, lp using following
formula:

    S = (pb * 5 + lp) * 9 + lc

For LZMA2 following limitation has been introduced:

    lc + lp <= 4.

The parameters are defined as follows:

Name  | Range  | Description
:---- | :----- | :------------------------------
lc    | [0,8]  | number of literal context bits
lp    | [0,4]  | number of literal pos bits
pb    | [0,4]  | the number of pos bits

<commit_msg>doc: added information about dictionary size to LZMA2.md<commit_after># LZMA2 Format

LZMA2 is a container of chunks. Each chunk is lead by a control byte.

Additional the LZMA2 format requires a byte that encodes the dictionary
size using following encoding.

Bits | Mask | Description
----:|-----:|:------------------------------------------------
 0-5 | 0x3F | Dictionary Size
 6-7 | 0xC0 | Reserved for future use; Must be zero

The dictionary size is encoded with a one-bit mantissa and five-bit
exponent. The smalles dictionary size is 4 KiB and the bigges is 4 GiB
- 1 B.

|Raw Value | Mantissa | Exponent | Dictionary size|
|---------:|---------:|---------:|---------------:|
|        0 |        2 |       11 |          4 KiB |
|        1 |        3 |       11 |          6 KiB |
|        2 |        2 |       12 |          8 KiB |
|        3 |        3 |       12 |         12 KiB |
|      ... |      ... |      ... |            ... |
|       36 |        2 |       29 |       1024 MiB |
|       37 |        3 |       29 |       1536 MiB |
|       38 |        2 |       30 |       2048 MiB |
|       39 |        3 |       30 |       3072 MiB |
|       40 |        2 |       31 |  4096 MiB - 1B |

For test purposes we add the dictionary size byte as first byte of a
LZMA2 stream.

## Chunks

Following the C implementation in the LZMA SDK the control byte can be
described as such:

Chunk header         | Description
:------------------- | :--------------------------------------------------
`00000000`           | End of LZMA2 stream
`00000001 U U`       | Uncompressed chunk, reset dictionary
`00000010 U U`       | Uncompressed chunk, no reset of dictionary
`100uuuuu U U P P`   | LZMA, no reset
`101uuuuu U U P P`   | LZMA, reset state
`110uuuuu U U P P S` | LZMA, reset state, new properties
`111uuuuu U U P P S` | LZMA, reset state, new properties, reset dictionary

The symbols used are described by following table.

A dictionary reset requires always new properties. If this is an
uncompressed chunk the properties need to be provided in the next
compressed chunk. New properties require a reset of the state.

Uncompressed data is written into the dictionary.

Symbol | Description
:----- | :-----------------
u      | unpacked size bit
U      | unpacked size byte
P      | packed size byte
S      | properties byte

The unpacked size and packed size are written in big-endian byte order.
The actual size is one more. So a chunk with 1 byte unpacked data will
store size 0 in the unpacked byte.

The properties byte provides the parameters pb, lc, lp using following
formula:

    S = (pb * 5 + lp) * 9 + lc

For LZMA2 following limitation has been introduced:

    lc + lp <= 4.

The parameters are defined as follows:

Name  | Range  | Description
:---- | :----- | :------------------------------
lc    | [0,8]  | number of literal context bits
lp    | [0,4]  | number of literal pos bits
pb    | [0,4]  | the number of pos bits


{'fraction_non_alphanumeric': 0.13846480067854114, 'fraction_numerical': 0.03880407124681934, 'mean_word_length': 1.869221411192214, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2070847', 'n_tokens_mistral': 1434, 'n_tokens_neox': 1309, 'n_words': 640}