<story><title>How do I delete bytes from the beginning of a file? (2010)</title><url>https://devblogs.microsoft.com/oldnewthing/20101201-00/?p=12153</url></story><parent_chain><item><author>derefr</author><text>I&amp;#x27;ve often thought it would be useful (especially for software that manages container or archive file-formats, or for databases) to have exposed to the user a file-system object that operates &lt;i&gt;like&lt;/i&gt; a file, but which—instead of being exposed by OS as a seekable byte stream that can be appended&amp;#x2F;truncated on one end—is exposed by syscalls as a &lt;i&gt;vector&lt;/i&gt; of arbitrary, non-uniformly-sized extents, where the user is expected to ask the OS to pre-allocate extent buffers (think mmap(2) with anonymous private disk pages), and then, separately, stuff those into this vector object (which would be an operation almost exactly like hard-linking an existing file into a directory, in terms of its time and space complexity.)&lt;p&gt;Of course, for most filesystems (and especially ones with sparse-file support, and &lt;i&gt;especially&lt;/i&gt; ones with copy-on-write support), such an on-disk data structure is exactly what&amp;#x27;s already underlying the byte-stream abstraction. So this would just be a passthrough to allow people to directly manipulate that data structure. (In the process probably breaking certain preconditions the filesystem relies on, though, so it would need to track these &amp;quot;low-level extent vectors&amp;quot; as a separate filesystem object type. Programs would still be able to use regular file-abstraction syscalls on them, though.)&lt;p&gt;Interestingly, despite filesystems themselves not exposing the lower-level extent-vector abstractions, in some other systems that have file-stream-like abstractions, you &lt;i&gt;can&lt;/i&gt; operate on &amp;quot;files&amp;quot; this way.&lt;p&gt;Postgres&amp;#x27;s BLOBs, for example, are seekable byte-streams, which also (at least theoretically) allow you to insert into the middle of them. (I say theoretically because it&amp;#x27;s not an implemented API, but it&amp;#x27;s not exactly hidden from the user, either. BLOBs just get broken out into records in a table representing their extents; you can rewrite the keys of said records in that table to do whatever low-level operations you like.)&lt;p&gt;Or, for another example, S3 and its competitors let you arbitrarily compose objects as “components” of other virtual objects, which then read back as the concatenation of the objects they contain. (Sadly you don’t get any other vector-manipulation ops than this, but if you keep references to the leaf extents around, that’s often enough to rebuild your extent-vectors any time they change.)&lt;p&gt;Also, of course, back on the real filesystem, you can just manage your &amp;quot;vector of parts of a file&amp;quot; as multiple files in a directory, and then abstract over that directory that by using a FUSE server which exposes a view where the files are one contiguous file.</text></item></parent_chain><comment><author>nine_k</author><text>This, and more, has been tried on mainframes. When you could not afford to install a free RDBMS with a click of a mouse (the mouse being not available yet, too), you could really appreciate record-oriented files you could use as a database, and even have indexed access to the records.</text></comment>
<story><title>How do I delete bytes from the beginning of a file? (2010)</title><url>https://devblogs.microsoft.com/oldnewthing/20101201-00/?p=12153</url></story><parent_chain><item><author>derefr</author><text>I&amp;#x27;ve often thought it would be useful (especially for software that manages container or archive file-formats, or for databases) to have exposed to the user a file-system object that operates &lt;i&gt;like&lt;/i&gt; a file, but which—instead of being exposed by OS as a seekable byte stream that can be appended&amp;#x2F;truncated on one end—is exposed by syscalls as a &lt;i&gt;vector&lt;/i&gt; of arbitrary, non-uniformly-sized extents, where the user is expected to ask the OS to pre-allocate extent buffers (think mmap(2) with anonymous private disk pages), and then, separately, stuff those into this vector object (which would be an operation almost exactly like hard-linking an existing file into a directory, in terms of its time and space complexity.)&lt;p&gt;Of course, for most filesystems (and especially ones with sparse-file support, and &lt;i&gt;especially&lt;/i&gt; ones with copy-on-write support), such an on-disk data structure is exactly what&amp;#x27;s already underlying the byte-stream abstraction. So this would just be a passthrough to allow people to directly manipulate that data structure. (In the process probably breaking certain preconditions the filesystem relies on, though, so it would need to track these &amp;quot;low-level extent vectors&amp;quot; as a separate filesystem object type. Programs would still be able to use regular file-abstraction syscalls on them, though.)&lt;p&gt;Interestingly, despite filesystems themselves not exposing the lower-level extent-vector abstractions, in some other systems that have file-stream-like abstractions, you &lt;i&gt;can&lt;/i&gt; operate on &amp;quot;files&amp;quot; this way.&lt;p&gt;Postgres&amp;#x27;s BLOBs, for example, are seekable byte-streams, which also (at least theoretically) allow you to insert into the middle of them. (I say theoretically because it&amp;#x27;s not an implemented API, but it&amp;#x27;s not exactly hidden from the user, either. BLOBs just get broken out into records in a table representing their extents; you can rewrite the keys of said records in that table to do whatever low-level operations you like.)&lt;p&gt;Or, for another example, S3 and its competitors let you arbitrarily compose objects as “components” of other virtual objects, which then read back as the concatenation of the objects they contain. (Sadly you don’t get any other vector-manipulation ops than this, but if you keep references to the leaf extents around, that’s often enough to rebuild your extent-vectors any time they change.)&lt;p&gt;Also, of course, back on the real filesystem, you can just manage your &amp;quot;vector of parts of a file&amp;quot; as multiple files in a directory, and then abstract over that directory that by using a FUSE server which exposes a view where the files are one contiguous file.</text></item></parent_chain><comment><author>JoshTriplett</author><text>Today, though, that filesystem abstraction is block-aligned. So you could delete blocks from the beginning or middle of a file easily, but you couldn&amp;#x27;t delete &lt;i&gt;bytes&lt;/i&gt; from the beginning or middle.</text></comment>