<commit_before># Packet Diary

Getting started late on the diary. I keep one for new projects, but this has
been around for a while, so there is none.

`sizeOf` should be a function, not a property.

## Pre-compilation

Your unwind loop gets fast. Inside the compiled function, everything you need
for a pattern so you get that closure preformance boost from V8. The loop is
something like.

```javascript
while (next = next() && remaining);
```

Or something. Not thinking straight.

This gets rid of all the man conditionals in the parse and serialize loops.
Everything could simply be unwound if not for resume. Thus, closures. But, also,
maybe a switch statement? Fall through? Evil but good.

## Bit Packing

It might be much easier, if you were to put bits in arrays.

## Positional Callbacks

I'm wondering if I want to eliminate positional callbacks, only build a tree.
That simplifies all the data structures.

Or...

Fix it up so that the tokenizer assigns a name &mdash; arg0, arg1, arg2 and if
it has never named anything, then it the callback gets a flattening. You could
even wrap the callback, so flattening costs extra.

What am I trying to save here? What cost? The thought is that array indexes cost
less than object indexes.

Actually, the fear is 

## Pre-Compiled Parsers and Serializers

I've held onto a pointless optimization for far to long. I've believed, based on
nothing but intuition, that parameters are faster than objects. Perhaps at some
point I saw some evidence of this but that's gone.

The logic still supports positional arguments now, but the implementation
expects an object, instead of expecting an array.<commit_msg>Notes on offsets.<commit_after># Packet Diary

Getting started late on the diary. I keep one for new projects, but this has
been around for a while, so there is none.

`sizeOf` should be a function, not a property.

## Pre-compilation

Your unwind loop gets fast. Inside the compiled function, everything you need
for a pattern so you get that closure preformance boost from V8. The loop is
something like.

```javascript
while (next = next() && remaining);
```

Or something. Not thinking straight.

This gets rid of all the man conditionals in the parse and serialize loops.
Everything could simply be unwound if not for resume. Thus, closures. But, also,
maybe a switch statement? Fall through? Evil but good.

## Bit Packing

It might be much easier, if you were to put bits in arrays.

## Positional Callbacks

I'm wondering if I want to eliminate positional callbacks, only build a tree.
That simplifies all the data structures.

Or...

Fix it up so that the tokenizer assigns a name &mdash; arg0, arg1, arg2 and if
it has never named anything, then it the callback gets a flattening. You could
even wrap the callback, so flattening costs extra.

What am I trying to save here? What cost? The thought is that array indexes cost
less than object indexes.

Actually, the fear is 

## Pre-Compiled Parsers and Serializers

I've held onto a pointless optimization for far to long. I've believed, based on
nothing but intuition, that parameters are faster than objects. Perhaps at some
point I saw some evidence of this but that's gone.

The logic still supports positional arguments now, but the implementation
expects an object, instead of expecting an array.

## Offsets

The `Serializer` is just that, it takes an object or parameters and flattens
them to a stream. Currently, I'm facing a problem in that `offsetsOf` returns an
object if the `Serializer` was told to extract from an `Object.` However, the
sturcture of the `offsetsOf` report includes the contianing bytes of of bit
packed fields. There is no name to assign the containing bytes. They are unnamed
in the pattern.

It won't be a problem for arrays of structures, though. The array will most
likely have a name.

Thus, for that one special case, we say that there are no objects coming out of
`offsetsOf`. It is an array for the purpose of visualization. If you want to
find a particular field, you can use `Array.filter` or some such.

Oh, also, for the purpose of display, we might also want to see skipped bytes.
They certainly do not have a name.
{'fraction_non_alphanumeric': 0.04978458592628052, 'fraction_numerical': 0.0019147917663954045, 'mean_word_length': 4.1339066339066335, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '543449', 'n_tokens_mistral': 1121, 'n_tokens_neox': 1062, 'n_words': 688}