<commit_before>Bidar: Binary Data Representation.
==================================

Oh, no not another BSON?!?!

Actually, yes.

The fact of the proliferation of serialized data formats, and of
serialized JSON-like formats in particular, is a fiat demonstration
that there isn't actually a solid consensus for a single unified and
universal format. As such, I (Danfuzz) think it's a reasonable and
prudent idea to continue the exploration of the design of these
things.

To be clear, I am not about to claim that the format defined here is
the One True Serialized JavaScript Object Format. What I *will* claim,
though, is that this one is at a significantly different spot in the
coordinate-space of possible serialization formats, and as such, it
may be useful to folks in cases where the other existing solutions
don't seem to be particularly apropos.

Here are the properties being aimed for here:

* reasonably space-efficient

* single-representation (non-equal binary forms implies
  non-isomorphic / non-equal object forms)

* without gratuitous quoting (e.g., look what happens when you put a
  JSON-encoded string inside another JSON-encoded string; avoid
  that!)

* supports a buffer (binary blob) type

* embraces as much of the JavaScript data model as is reasonably
  possible. For example, this format differentiates `null` and
  `undefined` (Why? Because you can tell the difference between
  them in the purely-local case, and you shouldn't necessarily have
  to change your code if you decide to interpose a machine
  boundary. I have no love of `undefined` myself and wish I could
  do away with it, but I'm not about to try to do so wishfully.)

* supports "holes" where non-pure-data elements may be filled in.
  More specifically, supports clients that want to be able to set
  up function/callback-like things for over-the-wire communication.

* supports non-tree graphs of objects (because, once you have "holes,"
  object identity matters). This includes both circular structures and
  arbitrary acyclic graphs.

Example
-------

One of the major reasons for the development of this module was to<commit_msg>Non-goal.<commit_after>Bidar: Binary Data Representation.
==================================

Oh, no not another BSON?!?!

Actually, yes.

The fact of the proliferation of serialized data formats, and of
serialized JSON-like formats in particular, is a fiat demonstration
that there isn't actually a solid consensus for a single unified and
universal format. As such, I (Danfuzz) think it's a reasonable and
prudent idea to continue the exploration of the design of these
things.

To be clear, I am not about to claim that the format defined here is
the One True Serialized JavaScript Object Format. What I *will* claim,
though, is that this one is at a significantly different spot in the
coordinate-space of possible serialization formats, and as such, it
may be useful to folks in cases where the other existing solutions
don't seem to be particularly apropos.

Here are the properties being aimed for here:

* reasonably space-efficient

* single-representation (non-equal binary forms implies
  non-isomorphic / non-equal object forms)

* without gratuitous quoting (e.g., look what happens when you put a
  JSON-encoded string inside another JSON-encoded string; avoid
  that!)

* supports a buffer (binary blob) type

* embraces as much of the JavaScript data model as is reasonably
  possible. For example, this format differentiates `null` and
  `undefined` (Why? Because you can tell the difference between
  them in the purely-local case, and you shouldn't necessarily have
  to change your code if you decide to interpose a machine
  boundary. I have no love of `undefined` myself and wish I could
  do away with it, but I'm not about to try to do so wishfully.)

* supports "holes" where non-pure-data elements may be filled in.
  More specifically, supports clients that want to be able to set
  up function/callback-like things for over-the-wire communication.

* supports non-tree graphs of objects (because, once you have "holes,"
  object identity matters). This includes both circular structures and
  arbitrary acyclic graphs.

Non-Goals:

* partially streamable. If you want to incrementally stream a (notionally)
  single structure, that can be done by defining a layer above this one.

Example
-------

One of the major reasons for the development of this module was to
{'fraction_non_alphanumeric': 0.06505534221820646, 'fraction_numerical': 0.0, 'mean_word_length': 4.277711561382598, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '694199', 'n_tokens_mistral': 1126, 'n_tokens_neox': 1080, 'n_words': 649}