<commit_before>utf [![travis](https://travis-ci.org/LB--/utf.svg?branch=0)](https://travis-ci.org/LB--/utf)
===

My personal C++14 UTF-8 library in the public domain.
It supports arbitrary header length (and therefore supports code points that require more than four code units in UTF-8).
Interaction is via iterators - as long as `operator*()` returns values with native endianness, everything works.

### UTF-8
This library is fully compatible with UTF-8, but supports an extension that allows storing code points of any arbitrary value (even huge values of over 16 bytes).
The same basic rules of UTF-8 apply:
* Code units that start with `0b0` are always 7-bit ASCII
* Code units that start with `0b11` are always the start of a multi-code-unit sequence
* Code units that start with `0b10` should always be a continuation of a multi-code-unit sequence

However, the official UTF-8 standard limits code points to at most U+10FFFF, which uses four 8-bit code units.
Even the original proposal was limited to six 8-bit code units.
In order to understand what happens when we want to store really large values in a code point, we need to talk about header overflow.

### Arbitrary Code Point Storage
The header of a multi-code-unit sequence starts with `0b11`, meaning two code units, and appends `0b1` for each additional code unit that makes up the code point.
The header ends with `0b0` and the remaining bits make up the actual code point.
With 8-bits, you can have up to `0b11111110` (7 code units) before the header can no longer fit in a single code unit.
To store larger values, the header needs to overflow into the next code unit, but we also want to keep the property that code units starting with `0b10` are a continuation, since this allows backtracking in a UTF-8 sequence.
Thus, the first header size that spills over into multiple 8-bit code units looks like `0b11111111, 0b100xxxxx` and means there will be 9 code units.
Note that it is not possible for there to be 8 code units - Either the header doesn't spill over and we have 7 code units, or the header does spill over and we have 9.
This is intentional, for it retains the ability to visually inspect the binary and know how many code points there will be by simply counting the `1`s until the terminating `0` (ignoring the `0` in the second most significant bit).

Using this technique we retain full compatibility with UTF-8 while also supporting excessively large code points.
Although the Unicode standard doesn't currently have enough code points to warrant such support, this library will be ready for the day it does (assuming the problem is still relevant by then and is solved the same way then as it is here).
Just be aware that currently if you actually store any code points that require more than four or six 8-bit code units, the resulting encoded code points will likely not be recognized or supported by 99% of software that exists today.

### UTF-16
**This library is NOT compatible with traditional UTF-16 encodings!**
If you use a 16-bit code unit type, it is treated as if UTF-8 simply had more bits, which unfortunately is not how UTF-16 works.
UTF-16 uses a more complicated encoding which does not share the same properties as UTF-8.
[The only time you should use traditional UTF-16 is when interacting with the Windows API.](http://utf8everywhere.org/#windows)
If you have UTF-16 strings that you want to use with this library, first [convert them to UTF-8](https://gist.github.com/LB--/735a911302ee9891a431514f6978e0a6) and then pass them to this library. 


### Other uses
UTF-style encoding isn't just useful for storage of Unicode code points - you can store anything you want with this library.
UTF-style encoding is useful when the values can normally fit in a single code unit but may occasionally need to occupy multiple code units.
Text storage is just one such example of a good use case.
Additionally, there is relatively low overhead - on average just 3 bits per additional code unit.

### UTF-32
Sure!
After 31 bits, you'll start to use more than one code unit per code point.

### Endianness
If you are working with single-byte code units (e.g. UTF-8), you don't need to worry about endianness.
Otherwise, always ensure that the code units you pass in have _native_ endianness.
This library makes no attempt to detect or otherwise acknowledge endianness, and leaves it up to the user to convert to or from native endianness as necessary.

### UTF-64
Sure!
After 63 bits, you'll start to use more than one code unit per code point.
To really take advantage of this, though, you will want to use a custom unsigned numeric type, most likely from the C++ bignum library of your choice.
Just make sure they type you use overloads the proper C++ operators.

### UTF-128, UTF-256, UTF-512, UTF-1024, etc
Sure, I guess?
Does your compiler even have any primitive integer types that large?
Code point types have to be primitive integer types so that they can be passed to [`std::make_unsigned`](http://en.cppreference.com/w/cpp/types/make_unsigned), though I guess you could throw conformance out the window and specialize `std::make_unsigned` on your own types.
If your compiler has primitive integer types that large or you want to break conformance, go for it.
It _should_ work as long as `sizeof(your_type)` is correct.
I think.

### UTF-not-a-power-of-2
If you have a weird system where `CHAR_BIT` is not `8`, this library should still work.
You only need at least 3 bits per code unit to use UTF-style encoding, though at that point the headers would take up over half of the code points.
If you have a weirder system where some primitive integer types are not multiples `CHAR_BIT`, then sorry, you won't be able to use this library, or a C++ compiler for that matter.

## Usage<commit_msg>Fix a couple typos<commit_after>utf [![travis](https://travis-ci.org/LB--/utf.svg?branch=0)](https://travis-ci.org/LB--/utf)
===

My personal C++14 UTF-8 library in the public domain.
It supports arbitrary header length (and therefore supports code points that require more than four code units in UTF-8).
Interaction is via iterators - as long as `operator*()` returns values with native endianness, everything works.

### UTF-8
This library is fully compatible with UTF-8, but supports an extension that allows storing code points of any arbitrary value (even huge values of over 16 bytes).
The same basic rules of UTF-8 apply:
* Code units that start with `0b0` are always 7-bit ASCII
* Code units that start with `0b11` are always the start of a multi-code-unit sequence
* Code units that start with `0b10` should always be a continuation of a multi-code-unit sequence

However, the official UTF-8 standard limits code points to at most U+10FFFF, which uses four 8-bit code units.
Even the original proposal was limited to six 8-bit code units.
In order to understand what happens when we want to store really large values in a code point, we need to talk about header overflow.

### Arbitrary Code Point Storage
The header of a multi-code-unit sequence starts with `0b11`, meaning two code units, and appends `0b1` for each additional code unit that makes up the code point.
The header ends with `0b0` and the remaining bits make up the actual code point.
With 8-bits, you can have up to `0b11111110` (7 code units) before the header can no longer fit in a single code unit.
To store larger values, the header needs to overflow into the next code unit, but we also want to keep the property that code units starting with `0b10` are a continuation, since this allows backtracking in a UTF-8 sequence.
Thus, the first header size that spills over into multiple 8-bit code units looks like `0b11111111, 0b100xxxxx` and means there will be 9 code units.
Note that it is not possible for there to be 8 code units - Either the header doesn't spill over and we have 7 code units, or the header does spill over and we have 9.
This is intentional, for it retains the ability to visually inspect the binary and know how many code points there will be by simply counting the `1`s until the terminating `0` (ignoring the `0` in the second most significant bit).

Using this technique we retain full compatibility with UTF-8 while also supporting excessively large code points.
Although the Unicode standard doesn't currently have enough code points to warrant such support, this library will be ready for the day it does (assuming the problem is still relevant by then and is solved the same way then as it is here).
Just be aware that currently if you actually store any code points that require more than four or six 8-bit code units, the resulting encoded code points will likely not be recognized or supported by 99% of software that exists today.

### UTF-16
**This library is NOT compatible with traditional UTF-16 encodings!**
If you use a 16-bit code unit type, it is treated as if UTF-8 simply had more bits, which unfortunately is not how UTF-16 works.
UTF-16 uses a more complicated encoding which does not share the same properties as UTF-8.
[The only time you should use traditional UTF-16 is when interacting with the Windows API.](http://utf8everywhere.org/#windows)
If you have UTF-16 strings that you want to use with this library, first [convert them to UTF-8](https://gist.github.com/LB--/735a911302ee9891a431514f6978e0a6) and then pass them to this library. 


### Other uses
UTF-style encoding isn't just useful for storage of Unicode code points - you can store anything you want with this library.
UTF-style encoding is useful when the values can normally fit in a single code unit but may occasionally need to occupy multiple code units.
Text storage is just one such example of a good use case.
Additionally, there is relatively low overhead - on average just 3 bits per additional code unit.

### UTF-32
Sure!
After 31 bits, you'll start to use more than one code unit per code point.

### Endianness
If you are working with single-byte code units (e.g. UTF-8), you don't need to worry about endianness.
Otherwise, always ensure that the code units you pass in have _native_ endianness.
This library makes no attempt to detect or otherwise acknowledge endianness, and leaves it up to the user to convert to or from native endianness as necessary.

### UTF-64
Sure!
After 63 bits, you'll start to use more than one code unit per code point.
To really take advantage of this, though, you will want to use a custom unsigned numeric type, most likely from the C++ bignum library of your choice.
Just make sure the type you use overloads the proper C++ operators.

### UTF-128, UTF-256, UTF-512, UTF-1024, etc
Sure, I guess?
Does your compiler even have any primitive integer types that large?
Code point types have to be primitive integer types so that they can be passed to [`std::make_unsigned`](http://en.cppreference.com/w/cpp/types/make_unsigned), though I guess you could defenestrate conformance and specialize `std::make_unsigned` on your own types.
If your compiler has primitive integer types that large or you want to break conformance, go for it.
It _should_ work as long as `sizeof(your_type)` is correct.
I think.

### UTF-not-a-power-of-2
If you have a weird system where `CHAR_BIT` is not `8`, this library should still work.
You only need at least 3 bits per code unit to use UTF-style encoding, though at that point the headers would take up over half of the code points.
If you have a weirder system where some primitive integer types are not multiples of `CHAR_BIT`, then sorry, you won't be able to use this library, or a C++ compiler for that matter.

## Usage
{'fraction_non_alphanumeric': 0.055541181543768865, 'fraction_numerical': 0.02414833980163864, 'mean_word_length': 4.285323609845032, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 4, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2783741', 'n_tokens_mistral': 3190, 'n_tokens_neox': 2919, 'n_words': 1872}