<commit_before>Performance
===========

to be written


Data size comparison
====================

The directory [*tests/jsonSample*](https://github.com/arangodb/velocypack/tree/master/tests/jsonSample)
contains several example JSON files which can be converted to VPack values. 
The following section shows how much can be gained by converting these JSON 
data into VPack values.

The resulting VPack sizes are compared against some other popular schema-free 
serialization formats such as [MessagePack / MsgPack](http://msgpack.org/)
and [BSON](http://bsonspec.org/).

The following table contains the size comparisons for the beforementioned 
files and formats. All values in the table are byte sizes. Smaller values are
better. 

The columns in the table have the following meanings:

* *original JSON*: size of the original JSON data (potentially including whitespace)
* *net JSON*: size of the remaining JSON data after stripping whitespace
* *BSON*: size after converting JSON input to BSON
* *MsgPack*: size after converting JSON input to MsgPack
* *VPack*: size after converting JSON input to VPack
* *VPack-c*: size after converting JSON input to VPack, using *compact* option in VPack

```
test case         original JSON    net JSON     MsgPack        BSON       VPack     VPack-c
------------------------------------------------------------------------------------------- 
api-docs.json         1,205,964   1,054,928   1,003,337   1,012,938   1,004,012     994,160
commits.json             25,216      25,216      23,091      30,907      22,935      20,789 
countries.json        1,134,029   1,134,029   1,161,461   1,367,765   1,061,291     956,786 
directory-tree.json     297,695     297,695     306,534     368,518     276,862     244,716 
doubles.json          1,187,062   1,187,062     899,981   1,488,895   1,299,984     899,982 
doubles-small.json      158,706     108,705      89,995     138,895     130,001      89,998  
file-list.json          151,317     151,316     139,979     195,268     150,177     133,536
object.json             157,781     157,781     118,519     178,895     158,633     118,630
pass1.json                1,441         961         844       1,181         916         804
pass2.json                   52          52          32         169          51          51
pass3.json                  148         115         124         123         110         108
random1.json              9,672       7,577       7,007       8,209       7,310       6,836
random2.json              8,239       6,443       5,960       6,992       6,222       5,815
random3.json             72,953      57,112      52,780      61,903      55,066      51,515
sample.json             687,491     168,089     150,697     155,462     162,113     153,187
small.json                   82          58          35          79          35          30
```

Data size comparison, with Object key compression
=================================================

Object keys can optionally be compressed in the VPack format.
Object key compression works by storing a (small) integer value instead of the original 
(potentially much longer) object attribute name. Especially when attribute names are long,
this compression can save lots of storage space. Attribute name translation is also fast
as attribute names and their integer replacement values are maintained in a hash table
which has O(1) lookup performance.

The compression may or may not be suitable for all use cases. VelocyPack data is schema-free,
so arbitrary types of objects can be stored with it. However, in many use cases many objects
share a common set of attribute names. In the most extreme use case, all objects even have the
same attribute names. If this is known in advance, the object key compression makes sense.<commit_msg>New columns in Performance.md.<commit_after>Performance
===========

to be written


Data size comparison
====================

The directory [*tests/jsonSample*](https://github.com/arangodb/velocypack/tree/master/tests/jsonSample)
contains several example JSON files which can be converted to VPack values. 
The following section shows how much can be gained by converting these JSON 
data into VPack values.

The resulting VPack sizes are compared against some other popular schema-free 
serialization formats such as [MessagePack / MsgPack](http://msgpack.org/)
and [BSON](http://bsonspec.org/).

The following table contains the size comparisons for the beforementioned 
files and formats. All values in the table are byte sizes. Smaller values are
better. 

The columns in the table have the following meanings:

* *original JSON*: size of the original JSON data (potentially including whitespace)
* *net JSON*: size of the remaining JSON data after stripping whitespace
* *BSON*: size after converting JSON input to BSON
* *MsgPack*: size after converting JSON input to MsgPack
* *VPack*: size after converting JSON input to VPack
* *VPack-c*: size after converting JSON input to VPack, using *compact* option in VPack
* *VPc-gzip*: size after running `gzip -9` on the VPack-c output
* *VPc-snappy*: size after using snappy-compression on the VPack-c output

```
test case         original JSON    net JSON     MsgPack        BSON       VPack     VPack-c  VPc-gzip VPc-snappy
----------------------------------------------------------------------------------------------------------------
api-docs.json         1,205,964   1,054,928   1,003,337   1,012,938   1,004,012     994,160   101,954    188,967
commits.json             25,216      25,216      23,091      30,907      22,935      20,789     3,863      5,617 
countries.json        1,134,029   1,134,029   1,161,461   1,367,765   1,061,291     956,786   230,890    350,236 
directory-tree.json     297,695     297,695     306,534     368,518     276,862     244,716    67,015     94,354 
doubles.json          1,187,062   1,187,062     899,981   1,488,895   1,299,984     899,982   418,291    837,753 
doubles-small.json      158,706     108,705      89,995     138,895     130,001      89,998    52,321     77,465  
file-list.json          151,317     151,316     139,979     195,268     150,177     133,536    42,463     62,585
object.json             157,781     157,781     118,519     178,895     158,633     118,630    30,396     61,269
pass1.json                1,441         961         844       1,181         916         804       687        822
pass2.json                   52          52          32         169          51          51        88         69
pass3.json                  148         115         124         123         110         108       132        126
random1.json              9,672       7,577       7,007       8,209       7,310       6,836     2,931      3,762
random2.json              8,239       6,443       5,960       6,992       6,222       5,815     2,520      3,249
random3.json             72,953      57,112      52,780      61,903      55,066      51,515    17,781     24,513
sample.json             687,491     168,089     150,697     155,462     162,113     153,187    22,891     25,300
small.json                   82          58          35          79          35          30        67         48
```

Data size comparison, with Object key compression
=================================================

Object keys can optionally be compressed in the VPack format.
Object key compression works by storing a (small) integer value instead of the original 
(potentially much longer) object attribute name. Especially when attribute names are long,
this compression can save lots of storage space. Attribute name translation is also fast
as attribute names and their integer replacement values are maintained in a hash table
which has O(1) lookup performance.

The compression may or may not be suitable for all use cases. VelocyPack data is schema-free,
so arbitrary types of objects can be stored with it. However, in many use cases many objects
share a common set of attribute names. In the most extreme use case, all objects even have the
same attribute names. If this is known in advance, the object key compression makes sense.
{'fraction_non_alphanumeric': 0.10246656031414897, 'fraction_numerical': 0.13461774450852865, 'mean_word_length': 2.0207561156412157, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '600852', 'n_tokens_mistral': 2846, 'n_tokens_neox': 2167, 'n_words': 885}