<commit_before># FlatCC FlatBuffers interface for C

The project includes:

- an executable `flatcc` FlatBuffers schema compiler for C and a
  corresponding library `libflatcc.a`. The compiler generates C header
  files or a binary flatbuffers schema.
- a typeless runtime library `libflatccbuilder.a` for building
  flatbuffers from C. Generated builder headers depend on this library.
  It may also be useful for other language interfaces. The library
  maintains a stack state to make it easy to build buffers from a parser
  or similar.
- an optional portability layer for non-C11 compliant compilers. This is
  not mature and needs target specific testing but should be a good
  starting point for using the generated code with most C compilers
  around.

The `flatcc` compiler is implemented as a standalone tool instead of
extending Googles `flatc` compiler in order to have a pure portable C
library implementation of the schema compiler that is designed to fail
graciously on abusive input in long running processes. It is also
believed a C version may help provide schema parsing to other language
interfaces that find interfacing with C easier than C++. The FlatBuffers
team at Googles FPL lab has been very helpful in providing feedback and
answering many questions to help ensure the best possible compatibility.
Notice the name `flatcc` (FlatBuffers C Compiler) vs Googles `flatc`.

The `flatcc` compiler has some extra features such as allowing for
referencing structs defined later in the schema - which makes sense
since it is already possible with other types. The binary schema format
also does not preserve the order of structs. The option is controlled by
a flag in `config.h` The generated source supports both bottom-up and
top-down construction mixed freely.

`flatcc` does not support parsing JSON objects.


## Time / Space / Usability Tradeoff

The priority has been to design an easy to use C builder interface that
is reasonably fast, suitable for both servers and embedded devices, but
with usability over absolute performance - still the small buffer output
rate is measured in millons per second and read access 10-100 millon
buffers per second from a rough estimate.

For 100MB buffers with 1000 monsters, dynamically extended monster
names, monster vector, and inventory vector, the bandwidth reaches about
2.2GB/s and 45ms/buffer on 2.2GHz Haswell Core i7 CPU. This inlcudes
reading back and validating all data. Reading only a few key fields
increases bandwidth to 2.7GB/s and 37ms/op. For 10MB buffers bandwidth
may be higher but eventually smaller buffers will be hit by call
overhead and thus we down to 300MB/s at about 150ns/op encoding small
buffers. These numbers are just a rough guideline - they obviously
depend on hardware, compiler, and data encoded. Measurements are
excluding an ininitial warmup step.

See also benchmark below.

The client C code can avoid almost any kind of allocation to build<commit_msg>Add link to FlatBuffers homepage<commit_after># FlatCC FlatBuffers interface for C

The project includes:

- an executable `flatcc` FlatBuffers schema compiler for C and a
  corresponding library `libflatcc.a`. The compiler generates C header
  files or a binary flatbuffers schema.
- a typeless runtime library `libflatccbuilder.a` for building
  flatbuffers from C. Generated builder headers depend on this library.
  It may also be useful for other language interfaces. The library
  maintains a stack state to make it easy to build buffers from a parser
  or similar.
- an optional portability layer for non-C11 compliant compilers. This is
  not mature and needs target specific testing but should be a good
  starting point for using the generated code with most C compilers
  around.

The `flatcc` compiler is implemented as a standalone tool instead of
extending Googles `flatc` compiler in order to have a pure portable C
library implementation of the schema compiler that is designed to fail
graciously on abusive input in long running processes. It is also
believed a C version may help provide schema parsing to other language
interfaces that find interfacing with C easier than C++. The FlatBuffers
team at Googles FPL lab has been very helpful in providing feedback and
answering many questions to help ensure the best possible compatibility.
Notice the name `flatcc` (FlatBuffers C Compiler) vs Googles `flatc`.

The `flatcc` compiler has some extra features such as allowing for
referencing structs defined later in the schema - which makes sense
since it is already possible with other types. The binary schema format
also does not preserve the order of structs. The option is controlled by
a flag in `config.h` The generated source supports both bottom-up and
top-down construction mixed freely.

`flatcc` does not support parsing JSON objects.

See also the Google FPL
[[FlatBuffers](http://google.github.io/flatbuffers/) homepage.

## Time / Space / Usability Tradeoff

The priority has been to design an easy to use C builder interface that
is reasonably fast, suitable for both servers and embedded devices, but
with usability over absolute performance - still the small buffer output
rate is measured in millons per second and read access 10-100 millon
buffers per second from a rough estimate.

For 100MB buffers with 1000 monsters, dynamically extended monster
names, monster vector, and inventory vector, the bandwidth reaches about
2.2GB/s and 45ms/buffer on 2.2GHz Haswell Core i7 CPU. This inlcudes
reading back and validating all data. Reading only a few key fields
increases bandwidth to 2.7GB/s and 37ms/op. For 10MB buffers bandwidth
may be higher but eventually smaller buffers will be hit by call
overhead and thus we down to 300MB/s at about 150ns/op encoding small
buffers. These numbers are just a rough guideline - they obviously
depend on hardware, compiler, and data encoded. Measurements are
excluding an ininitial warmup step.

See also benchmark below.

The client C code can avoid almost any kind of allocation to build
{'fraction_non_alphanumeric': 0.031052806376619065, 'fraction_numerical': 0.010959814015277316, 'mean_word_length': 4.480436760691537, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4381251', 'n_tokens_mistral': 1550, 'n_tokens_neox': 1435, 'n_words': 920}