<commit_before>VelocyPack (VPack) - a fast and compact format for serialization and storage
============================================================================

TravisCI: [![Build Status](https://secure.travis-ci.org/arangodb/velocypack.png?branch=master)](http://travis-ci.org/arangodb/velocypack)   AppVeyor: [![Build status](https://ci.appveyor.com/api/projects/status/pkbl4t7vey88bqud?svg=true)](https://ci.appveyor.com/project/jsteemann/velocypack)

Motivation
----------

These days, JSON (JavaScript Object Notation, see [ECMA-404]
(http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf))
is used in many cases where data has to be exchanged.
Lots of protocols between different services use it, databases store
JSON (document stores naturally, but others increasingly as well). It
is popular, because it is simple, human-readable, and yet surprisingly
versatile, despite its limitations.

At the same time there are a plethora of alternatives ranging from XML
over Universal Binary JSON, MongoDB's BSON, MessagePack, BJSON (binary
JSON), Apache Thrift till Google's protocol buffers and ArangoDB's
shaped JSON.

When looking into this, we were surprised to find that none of these
formats manages to combine compactness, platform independence, fast
access to subobjects and rapid conversion from and to JSON.

We have invented VPack because we need a binary format that

  - is compact
  - covers all of JSON plus dates, integers, binary data and arbitrary
    precision numbers
  - can be used in a database kernel to access subdocuments for
    example for indexes, so it must be possible to access subdocuments
    (array and object members) efficiently
  - can be transferred to JSON and from JSON rapidly
  - avoids too many memory allocations
  - gives flexibility to assemble objects, such that subobjects reside
    in the database in an unchanged way
  - allows to use an external table for frequently used attribute names
  - quickly allows to read off the type and length of a given object

This data format must be backed by good C++ classes to allow

  - easy and fast parsing from JSON
  - easy and convenient buildup without too many memory allocations
  - fast access of subobjects (arrays and objects)
  - flexible memory management
  - fast dumping to JSON

The VPack format is an attempt to achieve all this.

This repository contains a C++ library for building, manipulating and
serializing VPack data. It is the *reference implementation for the VPack
format*.
<commit_msg>Explain rationale behind VelocyPack in more detail.<commit_after>VelocyPack (VPack) - a fast and compact format for serialization and storage
============================================================================

TravisCI: [![Build Status](https://secure.travis-ci.org/arangodb/velocypack.png?branch=master)](http://travis-ci.org/arangodb/velocypack)   AppVeyor: [![Build status](https://ci.appveyor.com/api/projects/status/pkbl4t7vey88bqud?svg=true)](https://ci.appveyor.com/project/jsteemann/velocypack)

Motivation
----------

These days, JSON (JavaScript Object Notation, see [ECMA-404]
(http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf))
is used in many cases where data has to be exchanged.
Lots of protocols between different services use it, databases store
JSON (document stores naturally, but others increasingly as well). It
is popular, because it is simple, human-readable, and yet surprisingly
versatile, despite its limitations.

At the same time there are a plethora of alternatives ranging from XML
over Universal Binary JSON, MongoDB's BSON, MessagePack, BJSON (binary
JSON), Apache Thrift till Google's protocol buffers and ArangoDB's
shaped JSON.

When looking into this, we were surprised to find that none of these
formats manages to combine compactness, platform independence, fast
access to subobjects and rapid conversion from and to JSON.

We have invented VPack because we need a binary format that

  - is self-contained, schemaless and platform independent
  - is compact
  - covers all of JSON plus dates, integers, binary data and arbitrary
    precision numbers
  - can be used in a database kernel to access subdocuments for
    example for indexes, so it must be possible to access subdocuments
    (array and object members) efficiently
  - can be transferred to JSON and from JSON rapidly
  - avoids too many memory allocations
  - gives flexibility to assemble objects, such that subobjects reside
    in the database in an unchanged way
  - allows to use an external table for frequently used attribute names
  - quickly allows to read off the type and length of a given object
    from its beginning

All this gives us the possibility to use *the same byte sequence of
data* for **transport**, **storage** and (read-only) **work**.

The other popular formats we looked at have all some deficiency with
respect to the above list. To name but a few:

  - JSON itself lacks some data types (dates and binary data) and does
    not provide quick subvalue access without parsing
  - XML is not compact and is not good with binary data, it also lacks
    quick subvalue access
  - BSON is relatively compact, gets quite a lot right with respect to
    data types, but is seriously lacking w.r.t. subvalue access
  - Apache Thrift and Google's Protocol Buffers are not schemaless and 
    self-contained and the transport format is a serialization that is
    not good for rapid subvalue access
  - MessagePack is probably the closest to our shopping list, it is
    quite compact, has decent data types but again no quick subvalue
    access
  - Our own shaped JSON (used in ArangoDB as internal storage format)
    has very quick subvalue access, but the shape data is kept outside
    the actual data, so the data markers are not self-contained.
    Furthermore, we have run into scalability issues on multi-core
    because of the shared data structures for the shapes.

Any new data format must be backed by good C++ classes to allow

  - easy and fast parsing from JSON
  - easy and convenient buildup without too many memory allocations
  - fast access of subobjects (arrays and objects)
  - flexible memory management
  - fast dumping to JSON

The VelocyPack format is an attempt to achieve all this and our first
experiments and usage attempts are very encouraging..

This repository contains a C++ library for building, manipulating and
serializing VPack data. It is the *reference implementation for the VPack
format*.

{'fraction_non_alphanumeric': 0.07801961986511342, 'fraction_numerical': 0.0030656039239730227, 'mean_word_length': 4.245176848874598, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '545400', 'n_tokens_mistral': 1696, 'n_tokens_neox': 1603, 'n_words': 857}