<commit_before>(ns lab)

(comment
  Overview and goal
     schema-based, binary, fast, compact, EDN

  Existing techniques
     schema-based and schemaless, binary and text based

  Technologies
     clojure, clojurescript, java, js, libraries ...

  Frame specification
      parts, reading, writing

  Buffer implementation
     java, js, node, output types

  Schema specification
     parts, structuring, varints

  Code generation
     !!!compile-time!!!
     parsing, validating, code generation

  Data generation

  Testing
     local and cross-platform

  Microbenchmarks
     size and performance

  Publishing
     open source, github repo, clojars repo, license

  Improvements
     performance, extensibility, forward/backward compatibility, memory sharing)

(comment
  Pseudo structure
    Coord
      float x, y

    Fixture
      int id
      Coord[] coords

    Body
      Coord position
      float angle
      string body-type
      Fixture[] fixtures

    Snapshot
      long time<commit_msg>lab notes<commit_after>(ns lab)

(comment
  Overview and goal
     schema-based, binary, fast, compact, EDN

  Existing techniques
     schema-based and schemaless, binary and text based

  Technologies
     clojure, clojurescript, java, js, libraries ...

  Frame specification
      parts, reading, writing

  Buffer implementation
     java, js, node, output types

  Schema specification
     parts, structuring, varints

  Code generation
     !!!compile-time!!!
     parsing, validating, code generation

  Data generation

  Testing
     local and cross-platform

  Microbenchmarks
     size and performance

  Publishing
     open source, github repo, clojars repo, license

  Improvements
     performance, extensibility, forward/backward compatibility, memory sharing)
"
Goals:
 - fast
 - small messages
 - easy to use
 - usable by both server / client (clojure / clojurescript)

Serialization:
 - binary / text
 - with / without schema
 - self-describing
 - platforms?

Overview:
 - define schemas
 - generate code
 - data <-> binary

Techs:
 - java, clojure, js, clojurescript
 - java.nio, js/ByteBuffer
 - atom, git, github

Buffers:
 - clj: java.nio.ByteBuffer -> byte[]
 - cljs: js/ByteBuffer -> ArrayBuffer
 - relative read/write

Binary frame:
 - header: schema id, endianness, diffed?, meta-schema-id
 - payload

Schemas building:
 - primitives: integers, floating point numbers, boolean, char, varint
 - string, keyword, symbol, any, nil, date
 - list, vector, set, map, tuple, record, multi, optional, enum, wrapped

Testing
 - local: numerous unit tests
 - server - browser

Microbenchmarks
 - ~3.5x faster
 - ~2x smaller

Publishing
 - open source
 - github
 - clojars
 - license

Improvements
 - performance
 - versioning
 - extensibility
 - memory sharing

"
(comment
  Pseudo structure
    Coord
      float x, y

    Fixture
      int id
      Coord[] coords

    Body
      Coord position
      float angle
      string body-type
      Fixture[] fixtures

    Snapshot
      long time
{'fraction_non_alphanumeric': 0.06913661925239828, 'fraction_numerical': 0.0009923916639100231, 'mean_word_length': 2.7847309136420524, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1618322', 'n_tokens_mistral': 887, 'n_tokens_neox': 833, 'n_words': 305}