<commit_before># Shortening
A Base64-like encoder for creating short URLs


## Padding significant
Most encoders use number bases to convert characters.
This can waste space by refusing to use the zero-value in the
most significant position.

For example, in base<sub>10</sub> the numbers **1**, **01**, **00001**,
are all equivalent despite being padded with 0's.

This is the reason encoders like Radix will skip values
such as **A** and **AAA** in favor of possibly longer strings.

By using these values we can recover 64<sup>(n-1)</sup> extra URLs
of length *n*.

#### URL Counts
| Length | Radix<sub>64</sub> |   Shortening  | difference |
|--------|--------------------|---------------|------------|
|    1   |                 63 |            64 |          1 |
|    2   |              4,032 |         4,096 |         64 |
|    3   |            258,048 |       262,144 |       4096 |
|    4   |         16,515,072 |    16,777,216 |    262,144 |
|    5   |      1,056,964,608 | 1,073,741,824 | 16,777,216 |


## Performance
| Benchmark | ns/op |
|-----------|-------|
|  Encode   |  68.3 |
|  Decode   |  40.8 |

* go1.5 darwin/amd64
* Intel Core i7 920 - 2.66 GHz<commit_msg>add benchmarks for big encoders<commit_after># Shortening
A Base64-like encoder for creating short URLs


## Padding significant
Most encoders use number bases to convert characters.
This can waste space by refusing to use the zero-value in the
most significant position.

For example, in base<sub>10</sub> the numbers **1**, **01**, **00001**,
are all equivalent despite being padded with 0's.

This is the reason encoders like Radix will skip values
such as **A** and **AAA** in favor of possibly longer strings.

By using these values we can recover 64<sup>(n-1)</sup> extra URLs
of length *n*.

#### URL Counts
| Length | Radix<sub>64</sub> |   Shortening  | difference |
|--------|--------------------|---------------|------------|
|    1   |                 63 |            64 |          1 |
|    2   |              4,032 |         4,096 |         64 |
|    3   |            258,048 |       262,144 |       4096 |
|    4   |         16,515,072 |    16,777,216 |    262,144 |
|    5   |      1,056,964,608 | 1,073,741,824 | 16,777,216 |


## Performance
|  Benchmark   |  ns/op |
|--------------|--------|
|  Encode      |   69.3 |
|  Decode      |   40.8 |
|  BigEncode   |  139.0 |
|  BigDecode   | 1050.0 |

* go1.5.1 darwin/amd64
* Intel Core i7 920 - 2.66 GHz
{'fraction_non_alphanumeric': 0.17673469387755103, 'fraction_numerical': 0.10285714285714286, 'mean_word_length': 1.8140068886337544, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '435281', 'n_tokens_mistral': 938, 'n_tokens_neox': 766, 'n_words': 259}