<story><title>Add extra stuff to a “standard” encoding? Sure, why not</title><url>https://rachelbythebay.com/w/2023/09/19/badlib/</url></story><parent_chain><item><author>jimmytucson</author><text>&amp;gt; author decided it was a good idea to prepend the message with the message length encoded as a varint.&lt;p&gt;&amp;gt; WHY? Oh, why?!&lt;p&gt;Uh oh. Is this my HN moment?&lt;p&gt;This is exactly how I implemented it at my company. We had to write many protobuf messages to one file in bulk (in parallel). I did a fair amount of research before designing this and didn’t find any standard for separating protobuf messages (in fact, found that there explicitly isn’t a standard in that protobuf doesn’t care). So I thought rather than using  some “special” control character, like a null byte, which would inevitably be not-so-special and collide with somebody else’s (like Schema Registry’s “magic byte”), I’d use something meaningful like the number of bytes the following record is.&lt;p&gt;As for why I chose varint instead of just picking an interger size, well for one I got nerd-sniped by varint encoding and thought it would be cool to try and implement it in Scala. Secondly, I thought if I chose a fixed size integer, no matter what size I pick, my users will always surprise me and exceed it at least once, and when that happens, kaboom! I wanted to future proof this without wasting 64 goddamn bytes in front of each message, and also I got nerd-sniped, OK?!?&lt;p&gt;Someone on my team recently shared one of these files outside the company and so I really hope she’s not talking about me but that’s a crazy coincidence if not!</text></item></parent_chain><comment><author>paulddraper</author><text>Congrats :) you perfectly re-created the actual Protobuf stream format. [1]&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;protobuf.dev&amp;#x2F;reference&amp;#x2F;java&amp;#x2F;api-docs&amp;#x2F;com&amp;#x2F;google&amp;#x2F;protobuf&amp;#x2F;Parser#parseDelimitedFrom-java.io.InputStream-&quot; rel=&quot;nofollow noreferrer&quot;&gt;https:&amp;#x2F;&amp;#x2F;protobuf.dev&amp;#x2F;reference&amp;#x2F;java&amp;#x2F;api-docs&amp;#x2F;com&amp;#x2F;google&amp;#x2F;prot...&lt;/a&gt;</text></comment>
<story><title>Add extra stuff to a “standard” encoding? Sure, why not</title><url>https://rachelbythebay.com/w/2023/09/19/badlib/</url></story><parent_chain><item><author>jimmytucson</author><text>&amp;gt; author decided it was a good idea to prepend the message with the message length encoded as a varint.&lt;p&gt;&amp;gt; WHY? Oh, why?!&lt;p&gt;Uh oh. Is this my HN moment?&lt;p&gt;This is exactly how I implemented it at my company. We had to write many protobuf messages to one file in bulk (in parallel). I did a fair amount of research before designing this and didn’t find any standard for separating protobuf messages (in fact, found that there explicitly isn’t a standard in that protobuf doesn’t care). So I thought rather than using  some “special” control character, like a null byte, which would inevitably be not-so-special and collide with somebody else’s (like Schema Registry’s “magic byte”), I’d use something meaningful like the number of bytes the following record is.&lt;p&gt;As for why I chose varint instead of just picking an interger size, well for one I got nerd-sniped by varint encoding and thought it would be cool to try and implement it in Scala. Secondly, I thought if I chose a fixed size integer, no matter what size I pick, my users will always surprise me and exceed it at least once, and when that happens, kaboom! I wanted to future proof this without wasting 64 goddamn bytes in front of each message, and also I got nerd-sniped, OK?!?&lt;p&gt;Someone on my team recently shared one of these files outside the company and so I really hope she’s not talking about me but that’s a crazy coincidence if not!</text></item></parent_chain><comment><author>NavinF</author><text>&amp;gt; didn’t find any standard for separating protobuf messages&lt;p&gt;The fact that protobufs are not self-delimiting is an endless source of frustration, but I know of 2 standards for doing this:&lt;p&gt;- SerializeDelimited* is part of the protobuf library: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;protocolbuffers&amp;#x2F;protobuf&amp;#x2F;blob&amp;#x2F;main&amp;#x2F;src&amp;#x2F;google&amp;#x2F;protobuf&amp;#x2F;util&amp;#x2F;delimited_message_util.cc&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;protocolbuffers&amp;#x2F;protobuf&amp;#x2F;blob&amp;#x2F;main&amp;#x2F;src&amp;#x2F;go...&lt;/a&gt;&lt;p&gt;- Riegeli is &amp;quot;a file format for storing a sequence of string records, typically serialized protocol buffers. It supports dense compression, fast decoding, seeking, detection and optional skipping of data corruption, filtering of proto message fields for even faster decoding, and parallel encoding&amp;quot;: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;google&amp;#x2F;riegeli&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;google&amp;#x2F;riegeli&lt;/a&gt;</text></comment>