<commit_before>BaseNcoding
===========

Library for encoding of binary data into strings using base32, base64, base128 and other algorithms.
<commit_msg>Init Readme<commit_after>BaseNcoding
===========

There are well-known algorithms for binary data to string encoding exist, such as algorithms with radix of power of 2 (base32, base64) and algorithms with not of power of 2 ([base85](http://en.wikipedia.org/wiki/Ascii85), [base91](http://sourceforge.net/projects/base91/)).

This library implement algorithm for general case, that is custom alphabet can be used (alphabet with custom length). Here is [**BaseNcoding DEMO**](http://kvanttt.github.io/BaseNcoding/).

Idea of developed algorithm based on base85 encoding, except of block size is not constant, but calculated depending on alphabet length.

###Steps of algorithm
 * Calculation of block size in bits and chars.
 * Conversation of input string to byte array (using UTF8 Encoding).
 * Splitting byte array on n-bit groups.
 * Conversation of every group to radix n.
 * Tail bits processing.

For optimal block size calculation following considerations has been used:

![System for щptimal block size calculation](http://habrastorage.org/getpro/habr/post_images/6af/516/5a4/6af5165a4b4516d3deaa8736375238ff.png)

In this system:

* **a** - Length of alphabet **A**.
* **k** - Count of encoding chars.
* **b** - One digit radix base (2 in most cases).
* **n** - Bits count in radix **b** for representation of **k** chars of alphabet **A**.
* **r** - Compression ratio (greater is better).
* **⌊x⌋** — the largest integer not greater than x (floor).
* **⌈x⌉** — the smallest integer not less than x (ceiling).

Diagram of optimal block size and alphabet length dependence has been calculated with help of system above:
![](http://habrastorage.org/getpro/habr/post_images/910/d57/8b8/910d578b87c79d7ca121584e277de221.png)

One can see that known base64, base85, base91 encodings has been developed in good points (minimal block size with good compression ratio).

For bits block to chars block BigNumbers are using.
Algorithm has parallel version.

###License

BaseN algorithm is licensed under the Apache 2.0 License.

More detail explanation available [on Russian](http://habrahabr.ru/post/204778/).

{'fraction_non_alphanumeric': 0.10565870910698497, 'fraction_numerical': 0.042440318302387266, 'mean_word_length': 4.758269720101781, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3151090', 'n_tokens_mistral': 727, 'n_tokens_neox': 645, 'n_words': 277}