<commit_before>OS-X & Ubuntu: [![Build Status](https://travis-ci.org/dvidelabs/flatcc.svg?branch=master)](https://travis-ci.org/dvidelabs/flatcc)
Windows: [![Windows Build Status](https://ci.appveyor.com/api/projects/status/github/dvidelabs/flatcc?branch=master&svg=true)](https://ci.appveyor.com/project/dvidelabs/flatcc)

# FlatCC FlatBuffers in C for C

`flatcc` has no external dependencies except for build and compiler
tools, and the C runtime library. With concurrent Ninja builds, a small
client project can build flatcc with libraries, generate schema code,
link the project and execute a test case in less than 2 seconds (4 incl.
flatcc clone), rebuild in less than 0.2 seconds and produce binaries
between 15K and 60K, read small buffers in 30ns, build FlatBuffers in
about 600ns, and with a larger executable handle optional json parsing
or printing in less than 2 us for a 10 field mixed type message.


<!-- vim-markdown-toc GFM -->
* [Project Details](#project-details)
* [Reporting Bugs](#reporting-bugs)
* [Status](#status)
* [Time / Space / Usability Tradeoff](#time--space--usability-tradeoff)
* [Generated Files](#generated-files)
* [Using flatcc](#using-flatcc)
* [Quickstart](#quickstart)
* [Quickstart - reading a buffer](#quickstart---reading-a-buffer)
* [Quickstart - compiling for read-only](#quickstart---compiling-for-read-only)
* [Quickstart - building a buffer](#quickstart---building-a-buffer)
* [Verifying a Buffer](#verifying-a-buffer)
* [File and Type Identifiers](#file-and-type-identifiers)
    * [File Identifiers](#file-identifiers)
    * [Type Identifiers](#type-identifiers)
* [JSON Parsing and Printing](#json-parsing-and-printing)
    * [Performance Notes](#performance-notes)
* [Global Scope and Included Schema](#global-scope-and-included-schema)
* [Required Fields and Duplicate Fields](#required-fields-and-duplicate-fields)
* [Fast Buffers](#fast-buffers)
* [Types](#types)
* [Endianness](#endianness)
* [Offset Sizes and Direction](#offset-sizes-and-direction)
* [Pitfalls in Error Handling](#pitfalls-in-error-handling)
* [Searching and Sorting](#searching-and-sorting)
* [Null Values](#null-values)
* [Portability Layer](#portability-layer)
* [Building](#building)
    * [Unix Build (OS-X, Linux, related)](#unix-build-os-x-linux-related)
    * [Windows Build (MSVC)](#windows-build-msvc)
    * [Cross-compilation](#cross-compilation)
* [Distribution](#distribution)
    * [Unix Files](#unix-files)
    * [Windows Files](#windows-files)
* [Running Tests on Unix](#running-tests-on-unix)
* [Running Tests on Windows](#running-tests-on-windows)
* [Configuration](#configuration)
* [Using the Compiler and Builder library](#using-the-compiler-and-builder-library)
* [Benchmarks](#benchmarks)

<!-- vim-markdown-toc -->

## Project Details

NOTE: see
[CHANGELOG](https://github.com/dvidelabs/flatcc/blob/master/CHANGELOG.md).
There are occassionally minor breaking changes as API inconsistencies
are discovered. Unless clearly stated, breaking changes will not affect
the compiled runtime library, only the header files. In case of trouble,
make sure the `flatcc` tool is same version as the `include/flatcc`
path.

The project includes:

- an executable `flatcc` FlatBuffers schema compiler for C and a
  corresponding library `libflatcc.a`. The compiler generates C header
  files or a binary flatbuffers schema.
- a typeless runtime library `libflatccrt.a` for building and verifying
  flatbuffers from C. Generated builder headers depend on this library.
  It may also be useful for other language interfaces. The library
  maintains a stack state to make it easy to build buffers from a parser
  or similar.
- a small `flatcc/portable` header only library for non-C11 compliant
  compilers, and small helpers for all compilers including endian
  handling and numeric printing and parsing.

See also:

- [Reporting Bugs](https://github.com/dvidelabs/flatcc#reporting-bugs)

- [Google FPL FlatBuffers](http://google.github.io/flatbuffers/)

- [Online Forum](https://groups.google.com/forum/#!forum/flatbuffers)

- [Build Instructions](https://github.com/dvidelabs/flatcc#building)

- [Quickstart](https://github.com/dvidelabs/flatcc#quickstart)

- [Builder Interface Reference]

- [Benchmarks]

The `flatcc` compiler is implemented as a standalone tool instead of
extending Googles `flatc` compiler in order to have a pure portable C
library implementation of the schema compiler that is designed to fail
graciously on abusive input in long running processes. It is also
believed a C version may help provide schema parsing to other language
interfaces that find interfacing with C easier than C++. The FlatBuffers
team at Googles FPL lab has been very helpful in providing feedback and
answering many questions to help ensure the best possible compatibility.
Notice the name `flatcc` (FlatBuffers C Compiler) vs Googles `flatc`.

The `flatcc` compiler has some extra features such as allowing for
referencing structs defined later in the schema - which makes sense
since it is already possible with other types. The binary schema format
also does not preserve the order of structs. The option is controlled by
a flag in `config.h` The generated source supports both bottom-up and
top-down construction mixed freely.

The JSON format is compatible with Googles `flatc` tool. The `flatc`
tool converts JSON from the command line using a schema and a buffer as
input. `flatcc` generates schema specific code to read and write JSON
at runtime. While the `flatcc` approach is likely much faster and also
easier to deploy, the `flatc` approach is likely more convenient when
manually working with JSON such as editing game scenes. Both tools have
their place. 

**NOTE: Big-endian platforms are only supported as of release 0.4.0.**

## Reporting Bugs

If possible, please provide a short reproducible schema and
source file using [issue4](https://github.com/dvidelabs/flatcc/issues/4)
as an example. The first comment in this issue details how to quickly
set up a new temporary project using the `scripts/setup.sh` script.


## Status

0.4.2 is featurewise on par with 0.4.1 but improves compatibility with
C++ in portable headers and fixes `aligned_alloc` for older GCC
versions. 0.4.2. also fixes a memory corruption bug when building with a
nesting level of 8 or above (number of open buffers, tables, and
vectors).


Main features supported as of 0.4.2

- generated FlatBuffers reader and builder headers for C
- generated FlatBuffers verifier headers for C
- generated FlatBuffers JSON parser and printer for C
- ability to concatenate all output into one file, or to stdout
- robust dependency file generation for build systems
- binary schema (.bfbs) generation
- pre-generated reflection headers for handling .bfbs files
- cli schema compiler and library for compiling schema
- runtime library for builder, verifier and JSON support
- thorough test cases
- monster sample project
- fast build times
- support for big endian platforms (as of 0.4.0)
- support for big endian encoded flatbuffers on both le and be platforms. Enabled on `be` branch.
- size prefixed buffers - see also [Builder Interface Reference]

Supported platforms:

- Ubuntu gcc 4.4-4.8 and clang 3.5-3.8
- OS-X current clang / gcc
- Windows MSVC 2010, 2013, 2015, 2015 Win64 
- IBM XLC on AIX big endian Power PC has been tested for release 0.4.0
  but is not part of regular release tests.

The monster sample does not work with MSVC 2010 because it intentionally
uses C99 style code to better follow the C++ version.

There is no reason why other or older compilers cannot be supported, but
it may require some work in the build configuration and possibly
updates to the portable library. The above is simply what has been
tested and configured.

The portability layer has some features that are generally important for
things like endian handling, and others to provide compatibility for
optional and missing C11 features. Together this should support most C
compilers around, but relies on community feedback for maturity.

The necessary size of the runtime include files can be reduced
significantly by using -std=c11 and avoiding JSON (which needs a lot of
numeric parsing support), and by removing `include/flatcc/reflection`
which is present to support handling of binary schema files and can be
generated from `reflection/reflection.fbs`, and removing
`include/flatcc/support` which is only used for tests and samples. The
exact set of required files may change from release to release, and it
doesn't really matter with respect to the compiled code size.

There are no plans to make frequent updates once the project becomes
stable, but input from the community will always be welcome and included
in releases where relevant, especially with respect to testing on
different target platforms.


## Time / Space / Usability Tradeoff

The priority has been to design an easy to use C builder interface that
is reasonably fast, suitable for both servers and embedded devices, but
with usability over absolute performance - still the small buffer output
rate is measured in millons per second and read access 10-100 millon
buffers per second from a rough estimate. Reading FlatBuffers is more
than an order of magnitude faster than building them.

For 100MB buffers with 1000 monsters, dynamically extended monster
names, monster vector, and inventory vector, the bandwidth reaches about
2.2GB/s and 45ms/buffer on 2.2GHz Haswell Core i7 CPU. This includes
reading back and validating all data. Reading only a few key fields
increases bandwidth to 2.7GB/s and 37ms/op. For 10MB buffers bandwidth
may be higher but eventually smaller buffers will be hit by call
overhead and thus we get down to 300MB/s at about 150ns/op encoding
small buffers. These numbers are just a rough guideline - they obviously
depend on hardware, compiler, and data encoded. Measurements are
excluding an ininitial warmup step. 

The generated JSON parsers are roughly 4 times slower than building a
FlatBuffer directly in C or C++, or about 2200ns vs 600ns for a 700 byte
JSON message. JSON parsing is thus roughly two orders of magnitude faster
than reading the equivalent Protocol Buffer, as reported on the [Google
FlatBuffers
Benchmarks](http://google.github.io/flatbuffers/flatbuffers_benchmarks.html)
page. LZ4 compression would estimated double the overall processing time
of JSON parsing. JSON printing is faster than parsing but not very
significantly so. JSON compresses to roughly half the size of compressed
FlatBuffers on large buffers, but compresses worse on small buffers (not
to mention when not compressing at all).

It should be noted that FlatBuffer read performance exclude verification
which JSON parsers and Protocol Buffers inherently include by their
nature. Verification has not been benchmarked, but would presumably add
less than 50% read overhead unless only a fraction of a large buffer is to
be read.

See also [benchmark](https://github.com/dvidelabs/flatcc#benchmark)
below.

The client C code can avoid almost any kind of allocation to build
buffers as a builder stack provides an extensible arena before
committing objects - for example appending strings or vectors piecemeal.
The stack is mostly bypassed when a complete object can be constructed
directly such as a vector from integer array on little endian platforms.

The reader interface should be pretty fast as is with less room for
improvement performance wise. It is also much simpler than the builder.

Usability has also been prioritized over smallest possible generated
source code and compile time. It shouldn't affect the compiled size
by much.

The compiled binary output should be reasonably small for everything but
the most restrictive microcontrollers. A 33K monster source test file
(in addition to the generated headers and the builder library) results
in a less than 50K optimized binary executable file including overhead
for printf statements and other support logic, or a 30K object file
excluding the builder library.

Read-only binaries are smaller but not necessarily much smaller than
builders considering they do less work: The compatibility test reads a
pre-generated binary `monsterdata_test.golden` monster file and verifies
that all content is as expected. This results in a 13K optimized binary
executable or a 6K object file. The source for this check is 5K
excluding header files. Readers do not need to link with a library. 

JSON parsers bloat the compiled C binary compared to pure Flatbuffer
usage because they inline the parser decision tree. A JSON parser for
monster.fbs may add 100K +/- optimization settings to the executable
binary.


## Generated Files

The generated code for building flatbuffers,
and for parsing and printing flatbuffers, all need access to
`include/flatcc`. The reader does no rely on any library but all other
generated files rely on the `libflatccrt.a` runtime library. Note that
`libflatcc.a` is only required if the flatcc compiler itself is required
as a library.

The reader and builder rely on generated common reader and builder
header files. These common file makes it possible to change the global
namespace and redefine basic types (`uoffset_t` etc.). In the future
this _might_ move into library code and use macros for these
abstractions and eventually have a set of predefined files for types
beyond the standard 32-bit unsigned offset (`uoffset_t`). The runtime
library is specific to one set of type definitions.

Refer to `monster_test.c` and the generated files for detailed guidance
on use. The monster schema used in this project is a slight adaptation
to the original to test some additional edge cases.

For building flatbuffers a separate builder header file is generated per
schema. It requires a `flatbuffers_common_builder.h` file also generated
by the compiler and a small runtime library `libflatccrt.a`. It is
because of this requirement that the reader and builder generated code
is kept separate. Typical uses can be seen in the `monster_test.c` file.
The builder allows of repeated pushing of content to a vector or a
string while a containing table is being updated which simplifies
parsing of external formats. It is also possible to build nested buffers
in-line - at first this may sound excessive but it is useful when
wrapping a union of buffers in a network interface and it ensures proper
alignment of all buffer levels.

For verifying flatbuffers, a `myschema_verifier.h` is generated. It
depends on the runtime library and the reader header.

Json parsers and printers generate one file per schema file and included
schema will have their own parsers and printers which including parsers
and printers will depend upon, rather similar to how builders work.

Low level note: the builder generates all vtables at the end of the
buffer instead of ad-hoc in front of each table but otherwise does the
same deduplication of vtables. This makes it possible to cluster vtables
in hot cache or to make sure all vtables are available when partially
transmitting a buffer. This behavior can be disabled by a runtime flag.

Because some use cases may include very constrained embedded devices,
the builder library can be customized with an allocator object and a
buffer emitter object. The separate emitter ensures a buffer can be
constructed without requiring a full buffer to be present in memory at
once, if so desired.

The typeless builder library is documented in `flatcc_builder.h` and
`flatcc_emitter.h` while the generated typed builder api for C is
documented in [Builder Interface Reference].


## Using flatcc

Refer to `flatcc -h` for details.

The compiler can either generate a single header file or headers for all
included schema and a common file and with or without support for both
reading (default) and writing (-w) flatbuffers. The simplest option is
to use (-a) for all and include the `myschema_builder.h` file.

The (-a) or (-v) also generates a verifier file.

Make sure `flatcc` under the `include` folder is visible in the C
compilers include path when compiling flatbuffer builders.

The `flatcc` (-I) include path will assume all files with same base name
(case insentive) are identical and only include the first. All generated
files use the input basename and land in working directory or the path
set by (-o).

Note that the binary schema output can be with or without namespace
prefixes and the default differs from `flatc` which strips namespaces.
The binary schema can also have a non-standard size field prefixed so
multiple schema can be outfileenated in a single file if so desired (see
also the bfbs2json example).

Files can be generated to stdout using (--stdout). C headers will be
ordered and outfileenated, but otherwise identical to the separate file
output. Each include statement is guarded so this will not lead to
missing include files.

The generated code, especially with all combined with --stdout, may
appear large, but only the parts actually used will take space up the
the final executable or object file. Modern compilers inline and include
only necessary parts of the statically linked builder library.

JSON printer and parser can be generated using the --json flag or
--json-printer or json-parser if only one of them is required. There are
some certain runtime library compile time flags that can optimize out
printing symbolic enums, but these can also be disabled at runtime.


## Quickstart

After [building](https://github.com/dvidelabs/flatcc#building) the `flatcc tool`,
binaries are located in the `bin` and `lib` directories under the
`flatcc` source tree.

You can either jump directly to the [monster
example](https://github.com/dvidelabs/flatcc/tree/master/samples/monster)
that follows
[Googles FlatBuffers Tutorial](https://google.github.io/flatbuffers/flatbuffers_guide_tutorial.html), or you can read along the quickstart guide below. If you follow
the monster tutorial, you may want to clone and build flatcc and copy
the source to a separate project directory as follows:

    git clone https://github.com/dvidelabs/flatcc.git
    flatcc/scripts/setup.sh -a mymonster
    cd mymonster
    scripts/build.sh
    build/mymonster

`scripts/setup.sh` will as a minimum link the library and tool into a
custom directory, here `mymonster`. With (-a) it also adds a simple
build script, copies the example, and updates `.gitignore` - see
`scripts/setup.sh -h`. Setup can also build flatcc, but you still have
to ensure the build environment is configured for your system.

To write your own schema files please follow the main FlatBuffers
project documentation on [writing schema
files](https://google.github.io/flatbuffers/flatbuffers_guide_writing_schema.html).

The [Builder Interface Reference] may be useful after studying the
monster sample and quickstart below.

When looking for advanced examples such as sorting vectors and finding
elements by a key, you should find these in the
[`test/monster_test`](https://github.com/dvidelabs/flatcc/tree/master/test/monster_test) project.

The following quickstart guide is a broad simplification of the
`test/monster_test` project - note that the schema is slightly different
from the tutorial. Focus is on the C specific framework rather
than general FlatBuffers concepts.

You can still use the setup tool to create an empty project and
follow along, but there are no assumptions about that in the text below.

## Quickstart - reading a buffer

Here we provide a quick example of read-only access to Monster flatbuffer -
it is an adapted extract of the `monster_test.c` file.

First we compile the schema read-only with common (-c) support header and we
add the recursion because `monster_test.fbs` includes other files.

    flatcc -cr test/monster_test/monster_test.fbs

For simplicity we assume you build an example project in the project
root folder, but in praxis you would want to change some paths, for
example:

    mkdir -p build/example
    flatcc -cr -o build/example test/monster_test/monster_test.fbs
    cd build/example

We get:

    flatbuffers_common_reader.h
    include_test1_reader.h
    include_test2_reader.h
    monster_test_reader.h

(There is also the simpler `samples/monster/monster.fbs` but then you won't get
included schema files).

Namespaces can be long so we optionally use a macro to manage this.

    #include "monster_test_reader.h"

    #undef ns
    #define ns(x) FLATBUFFERS_WRAP_NAMESPACE(MyGame_Example, x)

    int verify_monster(void *buffer)
    {
        ns(Monster_table_t) monster;
        /* This is a read-only reference to a flatbuffer encoded struct. */
        ns(Vec3_struct_t) vec;
        flatbuffers_string_t name;
        size_t offset;

        if (!(monster = ns(Monster_as_root(buffer)))) {
            printf("Monster not available\n");
            return -1;
        }
        if (ns(Monster_hp(monster)) != 80) {
            printf("Health points are not as expected\n");
            return -1;
        }
        if (!(vec = ns(Monster_pos(monster)))) {
            printf("Position is absent\n");
            return -1;
        }

        /* -3.2f is actually -3.20000005 and not -3.2 due to representation loss. */
        if (ns(Vec3_z(vec)) != -3.2f) {
            printf("Position failing on z coordinate\n");
            return -1;
        }

        /* Verify force_align relative to buffer start. */
        offset = (char *)vec - (char *)buffer;
        if (offset & 15) {
            printf("Force align of Vec3 struct not correct\n");
            return -1;
        }

        /*
         * If we retrieved the buffer using `flatcc_builder_finalize_aligned_buffer` or
         * `flatcc_builder_get_direct_buffer` the struct should also
         * be aligned without subtracting the buffer.
         */
        if (vec & 15) {
            printf("warning: buffer not aligned in memory\n");
        }

        /* ... */
        return 0;
    }
    /* main() {...} */


## Quickstart - compiling for read-only

Assuming our above file is `monster_example.c` the following are a few
ways to compile the project for read-only - compilation with runtime
library is shown later on.

    cc -I include monster_example.c -o monster_example

    cc -std=c11 -I include monster_example.c -o monster_example

    cc -D FLATCC_PORTABLE -I include monster_example.c -o monster_example

The include path or source path is likely different. Some files in
`include/flatcc/portable` are always used, but the `-D FLATCC_PORTABLE`
flag includes additional files to support compilers lacking c11
features.


## Quickstart - building a buffer

Here we provide a very limited example of how to build a buffer - only a few
fields are updated. Pleaser refer to `monster_test.c` and the doc directory
for more information.

First we must generate the files:

    flatcc -a monster_test.fbs

This produces:

    flatbuffers_common_reader.h
    flatbuffers_common_builder.h<commit_msg>Update Quickstart subsections<commit_after>OS-X & Ubuntu: [![Build Status](https://travis-ci.org/dvidelabs/flatcc.svg?branch=master)](https://travis-ci.org/dvidelabs/flatcc)
Windows: [![Windows Build Status](https://ci.appveyor.com/api/projects/status/github/dvidelabs/flatcc?branch=master&svg=true)](https://ci.appveyor.com/project/dvidelabs/flatcc)

# FlatCC FlatBuffers in C for C

`flatcc` has no external dependencies except for build and compiler
tools, and the C runtime library. With concurrent Ninja builds, a small
client project can build flatcc with libraries, generate schema code,
link the project and execute a test case in less than 2 seconds (4 incl.
flatcc clone), rebuild in less than 0.2 seconds and produce binaries
between 15K and 60K, read small buffers in 30ns, build FlatBuffers in
about 600ns, and with a larger executable handle optional json parsing
or printing in less than 2 us for a 10 field mixed type message.


<!-- vim-markdown-toc GFM -->
* [Project Details](#project-details)
* [Reporting Bugs](#reporting-bugs)
* [Status](#status)
* [Time / Space / Usability Tradeoff](#time--space--usability-tradeoff)
* [Generated Files](#generated-files)
* [Using flatcc](#using-flatcc)
* [Quickstart](#quickstart)
    * [Reading a Buffer](#reading-a-buffer)
    * [Compiling for Read-Only](#compiling-for-read-only)
    * [Building a Buffer](#building-a-buffer)
* [Verifying a Buffer](#verifying-a-buffer)
* [File and Type Identifiers](#file-and-type-identifiers)
    * [File Identifiers](#file-identifiers)
    * [Type Identifiers](#type-identifiers)
* [JSON Parsing and Printing](#json-parsing-and-printing)
    * [Performance Notes](#performance-notes)
* [Global Scope and Included Schema](#global-scope-and-included-schema)
* [Required Fields and Duplicate Fields](#required-fields-and-duplicate-fields)
* [Fast Buffers](#fast-buffers)
* [Types](#types)
* [Endianness](#endianness)
* [Offset Sizes and Direction](#offset-sizes-and-direction)
* [Pitfalls in Error Handling](#pitfalls-in-error-handling)
* [Searching and Sorting](#searching-and-sorting)
* [Null Values](#null-values)
* [Portability Layer](#portability-layer)
* [Building](#building)
    * [Unix Build (OS-X, Linux, related)](#unix-build-os-x-linux-related)
    * [Windows Build (MSVC)](#windows-build-msvc)
    * [Cross-compilation](#cross-compilation)
* [Distribution](#distribution)
    * [Unix Files](#unix-files)
    * [Windows Files](#windows-files)
* [Running Tests on Unix](#running-tests-on-unix)
* [Running Tests on Windows](#running-tests-on-windows)
* [Configuration](#configuration)
* [Using the Compiler and Builder library](#using-the-compiler-and-builder-library)
* [Benchmarks](#benchmarks)

<!-- vim-markdown-toc -->

## Project Details

NOTE: see
[CHANGELOG](https://github.com/dvidelabs/flatcc/blob/master/CHANGELOG.md).
There are occassionally minor breaking changes as API inconsistencies
are discovered. Unless clearly stated, breaking changes will not affect
the compiled runtime library, only the header files. In case of trouble,
make sure the `flatcc` tool is same version as the `include/flatcc`
path.

The project includes:

- an executable `flatcc` FlatBuffers schema compiler for C and a
  corresponding library `libflatcc.a`. The compiler generates C header
  files or a binary flatbuffers schema.
- a typeless runtime library `libflatccrt.a` for building and verifying
  flatbuffers from C. Generated builder headers depend on this library.
  It may also be useful for other language interfaces. The library
  maintains a stack state to make it easy to build buffers from a parser
  or similar.
- a small `flatcc/portable` header only library for non-C11 compliant
  compilers, and small helpers for all compilers including endian
  handling and numeric printing and parsing.

See also:

- [Reporting Bugs](https://github.com/dvidelabs/flatcc#reporting-bugs)

- [Google FPL FlatBuffers](http://google.github.io/flatbuffers/)

- [Online Forum](https://groups.google.com/forum/#!forum/flatbuffers)

- [Build Instructions](https://github.com/dvidelabs/flatcc#building)

- [Quickstart](https://github.com/dvidelabs/flatcc#quickstart)

- [Builder Interface Reference]

- [Benchmarks]

The `flatcc` compiler is implemented as a standalone tool instead of
extending Googles `flatc` compiler in order to have a pure portable C
library implementation of the schema compiler that is designed to fail
graciously on abusive input in long running processes. It is also
believed a C version may help provide schema parsing to other language
interfaces that find interfacing with C easier than C++. The FlatBuffers
team at Googles FPL lab has been very helpful in providing feedback and
answering many questions to help ensure the best possible compatibility.
Notice the name `flatcc` (FlatBuffers C Compiler) vs Googles `flatc`.

The `flatcc` compiler has some extra features such as allowing for
referencing structs defined later in the schema - which makes sense
since it is already possible with other types. The binary schema format
also does not preserve the order of structs. The option is controlled by
a flag in `config.h` The generated source supports both bottom-up and
top-down construction mixed freely.

The JSON format is compatible with Googles `flatc` tool. The `flatc`
tool converts JSON from the command line using a schema and a buffer as
input. `flatcc` generates schema specific code to read and write JSON
at runtime. While the `flatcc` approach is likely much faster and also
easier to deploy, the `flatc` approach is likely more convenient when
manually working with JSON such as editing game scenes. Both tools have
their place. 

**NOTE: Big-endian platforms are only supported as of release 0.4.0.**

## Reporting Bugs

If possible, please provide a short reproducible schema and
source file using [issue4](https://github.com/dvidelabs/flatcc/issues/4)
as an example. The first comment in this issue details how to quickly
set up a new temporary project using the `scripts/setup.sh` script.


## Status

0.4.2 is featurewise on par with 0.4.1 but improves compatibility with
C++ in portable headers and fixes `aligned_alloc` for older GCC
versions. 0.4.2. also fixes a memory corruption bug when building with a
nesting level of 8 or above (number of open buffers, tables, and
vectors).


Main features supported as of 0.4.2

- generated FlatBuffers reader and builder headers for C
- generated FlatBuffers verifier headers for C
- generated FlatBuffers JSON parser and printer for C
- ability to concatenate all output into one file, or to stdout
- robust dependency file generation for build systems
- binary schema (.bfbs) generation
- pre-generated reflection headers for handling .bfbs files
- cli schema compiler and library for compiling schema
- runtime library for builder, verifier and JSON support
- thorough test cases
- monster sample project
- fast build times
- support for big endian platforms (as of 0.4.0)
- support for big endian encoded flatbuffers on both le and be platforms. Enabled on `be` branch.
- size prefixed buffers - see also [Builder Interface Reference]

Supported platforms:

- Ubuntu gcc 4.4-4.8 and clang 3.5-3.8
- OS-X current clang / gcc
- Windows MSVC 2010, 2013, 2015, 2015 Win64 
- IBM XLC on AIX big endian Power PC has been tested for release 0.4.0
  but is not part of regular release tests.

The monster sample does not work with MSVC 2010 because it intentionally
uses C99 style code to better follow the C++ version.

There is no reason why other or older compilers cannot be supported, but
it may require some work in the build configuration and possibly
updates to the portable library. The above is simply what has been
tested and configured.

The portability layer has some features that are generally important for
things like endian handling, and others to provide compatibility for
optional and missing C11 features. Together this should support most C
compilers around, but relies on community feedback for maturity.

The necessary size of the runtime include files can be reduced
significantly by using -std=c11 and avoiding JSON (which needs a lot of
numeric parsing support), and by removing `include/flatcc/reflection`
which is present to support handling of binary schema files and can be
generated from `reflection/reflection.fbs`, and removing
`include/flatcc/support` which is only used for tests and samples. The
exact set of required files may change from release to release, and it
doesn't really matter with respect to the compiled code size.

There are no plans to make frequent updates once the project becomes
stable, but input from the community will always be welcome and included
in releases where relevant, especially with respect to testing on
different target platforms.


## Time / Space / Usability Tradeoff

The priority has been to design an easy to use C builder interface that
is reasonably fast, suitable for both servers and embedded devices, but
with usability over absolute performance - still the small buffer output
rate is measured in millons per second and read access 10-100 millon
buffers per second from a rough estimate. Reading FlatBuffers is more
than an order of magnitude faster than building them.

For 100MB buffers with 1000 monsters, dynamically extended monster
names, monster vector, and inventory vector, the bandwidth reaches about
2.2GB/s and 45ms/buffer on 2.2GHz Haswell Core i7 CPU. This includes
reading back and validating all data. Reading only a few key fields
increases bandwidth to 2.7GB/s and 37ms/op. For 10MB buffers bandwidth
may be higher but eventually smaller buffers will be hit by call
overhead and thus we get down to 300MB/s at about 150ns/op encoding
small buffers. These numbers are just a rough guideline - they obviously
depend on hardware, compiler, and data encoded. Measurements are
excluding an ininitial warmup step. 

The generated JSON parsers are roughly 4 times slower than building a
FlatBuffer directly in C or C++, or about 2200ns vs 600ns for a 700 byte
JSON message. JSON parsing is thus roughly two orders of magnitude faster
than reading the equivalent Protocol Buffer, as reported on the [Google
FlatBuffers
Benchmarks](http://google.github.io/flatbuffers/flatbuffers_benchmarks.html)
page. LZ4 compression would estimated double the overall processing time
of JSON parsing. JSON printing is faster than parsing but not very
significantly so. JSON compresses to roughly half the size of compressed
FlatBuffers on large buffers, but compresses worse on small buffers (not
to mention when not compressing at all).

It should be noted that FlatBuffer read performance exclude verification
which JSON parsers and Protocol Buffers inherently include by their
nature. Verification has not been benchmarked, but would presumably add
less than 50% read overhead unless only a fraction of a large buffer is to
be read.

See also [benchmark](https://github.com/dvidelabs/flatcc#benchmark)
below.

The client C code can avoid almost any kind of allocation to build
buffers as a builder stack provides an extensible arena before
committing objects - for example appending strings or vectors piecemeal.
The stack is mostly bypassed when a complete object can be constructed
directly such as a vector from integer array on little endian platforms.

The reader interface should be pretty fast as is with less room for
improvement performance wise. It is also much simpler than the builder.

Usability has also been prioritized over smallest possible generated
source code and compile time. It shouldn't affect the compiled size
by much.

The compiled binary output should be reasonably small for everything but
the most restrictive microcontrollers. A 33K monster source test file
(in addition to the generated headers and the builder library) results
in a less than 50K optimized binary executable file including overhead
for printf statements and other support logic, or a 30K object file
excluding the builder library.

Read-only binaries are smaller but not necessarily much smaller than
builders considering they do less work: The compatibility test reads a
pre-generated binary `monsterdata_test.golden` monster file and verifies
that all content is as expected. This results in a 13K optimized binary
executable or a 6K object file. The source for this check is 5K
excluding header files. Readers do not need to link with a library. 

JSON parsers bloat the compiled C binary compared to pure Flatbuffer
usage because they inline the parser decision tree. A JSON parser for
monster.fbs may add 100K +/- optimization settings to the executable
binary.


## Generated Files

The generated code for building flatbuffers,
and for parsing and printing flatbuffers, all need access to
`include/flatcc`. The reader does no rely on any library but all other
generated files rely on the `libflatccrt.a` runtime library. Note that
`libflatcc.a` is only required if the flatcc compiler itself is required
as a library.

The reader and builder rely on generated common reader and builder
header files. These common file makes it possible to change the global
namespace and redefine basic types (`uoffset_t` etc.). In the future
this _might_ move into library code and use macros for these
abstractions and eventually have a set of predefined files for types
beyond the standard 32-bit unsigned offset (`uoffset_t`). The runtime
library is specific to one set of type definitions.

Refer to `monster_test.c` and the generated files for detailed guidance
on use. The monster schema used in this project is a slight adaptation
to the original to test some additional edge cases.

For building flatbuffers a separate builder header file is generated per
schema. It requires a `flatbuffers_common_builder.h` file also generated
by the compiler and a small runtime library `libflatccrt.a`. It is
because of this requirement that the reader and builder generated code
is kept separate. Typical uses can be seen in the `monster_test.c` file.
The builder allows of repeated pushing of content to a vector or a
string while a containing table is being updated which simplifies
parsing of external formats. It is also possible to build nested buffers
in-line - at first this may sound excessive but it is useful when
wrapping a union of buffers in a network interface and it ensures proper
alignment of all buffer levels.

For verifying flatbuffers, a `myschema_verifier.h` is generated. It
depends on the runtime library and the reader header.

Json parsers and printers generate one file per schema file and included
schema will have their own parsers and printers which including parsers
and printers will depend upon, rather similar to how builders work.

Low level note: the builder generates all vtables at the end of the
buffer instead of ad-hoc in front of each table but otherwise does the
same deduplication of vtables. This makes it possible to cluster vtables
in hot cache or to make sure all vtables are available when partially
transmitting a buffer. This behavior can be disabled by a runtime flag.

Because some use cases may include very constrained embedded devices,
the builder library can be customized with an allocator object and a
buffer emitter object. The separate emitter ensures a buffer can be
constructed without requiring a full buffer to be present in memory at
once, if so desired.

The typeless builder library is documented in `flatcc_builder.h` and
`flatcc_emitter.h` while the generated typed builder api for C is
documented in [Builder Interface Reference].


## Using flatcc

Refer to `flatcc -h` for details.

The compiler can either generate a single header file or headers for all
included schema and a common file and with or without support for both
reading (default) and writing (-w) flatbuffers. The simplest option is
to use (-a) for all and include the `myschema_builder.h` file.

The (-a) or (-v) also generates a verifier file.

Make sure `flatcc` under the `include` folder is visible in the C
compilers include path when compiling flatbuffer builders.

The `flatcc` (-I) include path will assume all files with same base name
(case insentive) are identical and only include the first. All generated
files use the input basename and land in working directory or the path
set by (-o).

Note that the binary schema output can be with or without namespace
prefixes and the default differs from `flatc` which strips namespaces.
The binary schema can also have a non-standard size field prefixed so
multiple schema can be outfileenated in a single file if so desired (see
also the bfbs2json example).

Files can be generated to stdout using (--stdout). C headers will be
ordered and outfileenated, but otherwise identical to the separate file
output. Each include statement is guarded so this will not lead to
missing include files.

The generated code, especially with all combined with --stdout, may
appear large, but only the parts actually used will take space up the
the final executable or object file. Modern compilers inline and include
only necessary parts of the statically linked builder library.

JSON printer and parser can be generated using the --json flag or
--json-printer or json-parser if only one of them is required. There are
some certain runtime library compile time flags that can optimize out
printing symbolic enums, but these can also be disabled at runtime.


## Quickstart

After [building](https://github.com/dvidelabs/flatcc#building) the `flatcc tool`,
binaries are located in the `bin` and `lib` directories under the
`flatcc` source tree.

You can either jump directly to the [monster
example](https://github.com/dvidelabs/flatcc/tree/master/samples/monster)
that follows
[Googles FlatBuffers Tutorial](https://google.github.io/flatbuffers/flatbuffers_guide_tutorial.html), or you can read along the quickstart guide below. If you follow
the monster tutorial, you may want to clone and build flatcc and copy
the source to a separate project directory as follows:

    git clone https://github.com/dvidelabs/flatcc.git
    flatcc/scripts/setup.sh -a mymonster
    cd mymonster
    scripts/build.sh
    build/mymonster

`scripts/setup.sh` will as a minimum link the library and tool into a
custom directory, here `mymonster`. With (-a) it also adds a simple
build script, copies the example, and updates `.gitignore` - see
`scripts/setup.sh -h`. Setup can also build flatcc, but you still have
to ensure the build environment is configured for your system.

To write your own schema files please follow the main FlatBuffers
project documentation on [writing schema
files](https://google.github.io/flatbuffers/flatbuffers_guide_writing_schema.html).

The [Builder Interface Reference] may be useful after studying the
monster sample and quickstart below.

When looking for advanced examples such as sorting vectors and finding
elements by a key, you should find these in the
[`test/monster_test`](https://github.com/dvidelabs/flatcc/tree/master/test/monster_test) project.

The following quickstart guide is a broad simplification of the
`test/monster_test` project - note that the schema is slightly different
from the tutorial. Focus is on the C specific framework rather
than general FlatBuffers concepts.

You can still use the setup tool to create an empty project and
follow along, but there are no assumptions about that in the text below.

### Reading a Buffer

Here we provide a quick example of read-only access to Monster flatbuffer -
it is an adapted extract of the `monster_test.c` file.

First we compile the schema read-only with common (-c) support header and we
add the recursion because `monster_test.fbs` includes other files.

    flatcc -cr test/monster_test/monster_test.fbs

For simplicity we assume you build an example project in the project
root folder, but in praxis you would want to change some paths, for
example:

    mkdir -p build/example
    flatcc -cr -o build/example test/monster_test/monster_test.fbs
    cd build/example

We get:

    flatbuffers_common_reader.h
    include_test1_reader.h
    include_test2_reader.h
    monster_test_reader.h

(There is also the simpler `samples/monster/monster.fbs` but then you won't get
included schema files).

Namespaces can be long so we optionally use a macro to manage this.

    #include "monster_test_reader.h"

    #undef ns
    #define ns(x) FLATBUFFERS_WRAP_NAMESPACE(MyGame_Example, x)

    int verify_monster(void *buffer)
    {
        ns(Monster_table_t) monster;
        /* This is a read-only reference to a flatbuffer encoded struct. */
        ns(Vec3_struct_t) vec;
        flatbuffers_string_t name;
        size_t offset;

        if (!(monster = ns(Monster_as_root(buffer)))) {
            printf("Monster not available\n");
            return -1;
        }
        if (ns(Monster_hp(monster)) != 80) {
            printf("Health points are not as expected\n");
            return -1;
        }
        if (!(vec = ns(Monster_pos(monster)))) {
            printf("Position is absent\n");
            return -1;
        }

        /* -3.2f is actually -3.20000005 and not -3.2 due to representation loss. */
        if (ns(Vec3_z(vec)) != -3.2f) {
            printf("Position failing on z coordinate\n");
            return -1;
        }

        /* Verify force_align relative to buffer start. */
        offset = (char *)vec - (char *)buffer;
        if (offset & 15) {
            printf("Force align of Vec3 struct not correct\n");
            return -1;
        }

        /*
         * If we retrieved the buffer using `flatcc_builder_finalize_aligned_buffer` or
         * `flatcc_builder_get_direct_buffer` the struct should also
         * be aligned without subtracting the buffer.
         */
        if (vec & 15) {
            printf("warning: buffer not aligned in memory\n");
        }

        /* ... */
        return 0;
    }
    /* main() {...} */


### Compiling for Read-Only

Assuming our above file is `monster_example.c` the following are a few
ways to compile the project for read-only - compilation with runtime
library is shown later on.

    cc -I include monster_example.c -o monster_example

    cc -std=c11 -I include monster_example.c -o monster_example

    cc -D FLATCC_PORTABLE -I include monster_example.c -o monster_example

The include path or source path is likely different. Some files in
`include/flatcc/portable` are always used, but the `-D FLATCC_PORTABLE`
flag includes additional files to support compilers lacking c11
features.


### Building a Buffer

Here we provide a very limited example of how to build a buffer - only a few
fields are updated. Pleaser refer to `monster_test.c` and the doc directory
for more information.

First we must generate the files:

    flatcc -a monster_test.fbs

This produces:

    flatbuffers_common_reader.h
    flatbuffers_common_builder.h
{'fraction_non_alphanumeric': 0.06066561844863732, 'fraction_numerical': 0.007643256464011181, 'mean_word_length': 4.181375876895225, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 34, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1285307', 'n_tokens_mistral': 12493, 'n_tokens_neox': 11750, 'n_words': 6101}