<commit_before># simdjson : Parsing gigabytes of JSON per second

## A C++  library to see how fast we can parse JSON with complete validation.

JSON documents are everywhere on the Internet. Servers spend a lot of time parsing these documents. We want to accelerate the parsing of JSON per se using commonly available SIMD instructions as much as possible while doing full validation (including character encoding).


## Some performance results

We can use a quarter or fewer instructions than a state-of-the-art parser like RapidJSON, and half as many as sajson. To our knowledge, simdjson is the first fully-validating JSON parser to run at gigabytes per second on commodity processors.

<img src="doc/gbps.png" width="90%">


On a skylake processor, the parsing speeds (in GB/s) of various processors on the twitter.json file are as follows.

| parser   |  GB/s |
|---|---|
| simdjson | 2.2	|
| RapidJSON encoding-validation | 0.51|
| RapidJSON encoding-validation, insitu | 0.71|
| sajson (insitu, dynamic) | 0.70|
| sajson (insitu, static) | 0.97|
| dropbox | 0.14|
| fastjson | 0.26|
| gason | 0.85|
| ultrajson | 0.42|
| jsmn | 0.28|
|cJSON | 0.34|

## Requirements

- We support platforms like Linux or macOS, as well as Windows through Visual Studio 2017 or better.
- A processor with AVX2 (i.e., Intel processors starting with the Haswell microarchitecture released 2013, and processors from AMD starting with the Rizen)
- A recent C++ compiler (e.g., GNU GCC or LLVM CLANG or Visual Studio 2017), we assume C++17. GNU GCC 7 or better or LLVM's clang 6 or better.
- Some benchmark scripts assume bash and other common utilities, but they are  optional.

## License

This code is made available under the Apache License 2.0. 

Under Windows, we build some tools using the  windows/dirent_portable.h  file (which is outside our library code): it under the liberal (business-friendly) MIT license.

## Code example

```C
#include "simdjson/jsonparser.h"

/...
<commit_msg>Update README.md (#45)<commit_after># simdjson : Parsing gigabytes of JSON per second

## A C++  library to see how fast we can parse JSON with complete validation.

JSON documents are everywhere on the Internet. Servers spend a lot of time parsing these documents. We want to accelerate the parsing of JSON per se using commonly available SIMD instructions as much as possible while doing full validation (including character encoding).


## Some performance results

We can use a quarter or fewer instructions than a state-of-the-art parser like RapidJSON, and half as many as sajson. To our knowledge, simdjson is the first fully-validating JSON parser to run at gigabytes per second on commodity processors.

<img src="doc/gbps.png" width="90%">


On a skylake processor, the parsing speeds (in GB/s) of various processors on the twitter.json file are as follows.

| parser   |  GB/s |
|---|---|
| simdjson | 2.2	|
| RapidJSON encoding-validation | 0.51|
| RapidJSON encoding-validation, insitu | 0.71|
| sajson (insitu, dynamic) | 0.70|
| sajson (insitu, static) | 0.97|
| dropbox | 0.14|
| fastjson | 0.26|
| gason | 0.85|
| ultrajson | 0.42|
| jsmn | 0.28|
|cJSON | 0.34|

## Requirements

- We support platforms like Linux or macOS, as well as Windows through Visual Studio 2017 or better.
- A processor with AVX2 (i.e., Intel processors starting with the Haswell microarchitecture released 2013, and processors from AMD starting with the Ryzen)
- A recent C++ compiler (e.g., GNU GCC or LLVM CLANG or Visual Studio 2017), we assume C++17. GNU GCC 7 or better or LLVM's clang 6 or better.
- Some benchmark scripts assume bash and other common utilities, but they are  optional.

## License

This code is made available under the Apache License 2.0. 

Under Windows, we build some tools using the  windows/dirent_portable.h  file (which is outside our library code): it under the liberal (business-friendly) MIT license.

## Code example

```C
#include "simdjson/jsonparser.h"

/...

{'fraction_non_alphanumeric': 0.08410979602115336, 'fraction_numerical': 0.02719717955175019, 'mean_word_length': 3.9402985074626864, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '436313', 'n_tokens_mistral': 1264, 'n_tokens_neox': 1134, 'n_words': 560}