<story><title>Remove first 300M lines from 700 GB file on 1TB disk</title><url>https://unix.stackexchange.com/questions/610494/how-do-i-remove-the-first-300-million-lines-from-a-700-gb-txt-file-on-a-system-w</url></story><parent_chain><item><author>hprotagonist</author><text>the only answer i’d use here is “go buy another 1TB drive and then ...”&lt;p&gt;$100 and a day and a half for shipping is a much better idea than playing russian roulette with work data by adjusting files in-place.</text></item></parent_chain><comment><author>nybble41</author><text>&amp;gt; playing russian roulette with work data by adjusting files in-place&lt;p&gt;That can be avoided if your filesystem supports reflinks &amp;#x2F; copy_file_range (BTRFS, XFS, OCFS2). Just create a reflink of the file, drop all the blocks at the beginning of the reflink &amp;quot;copy&amp;quot; up to the one containing the line you want to start with using fallocate --collapse-range, and overwrite the remaining data up to the first line with some suitable fill character. For the JSON-encoded lines in this case spaces should work. Total disk space used: one data block for the padding plus some metadata for the reflink. No changes to the original file. No copying or relocating 700 GB of data. Quick, easy, and and safe. But you have to start with the right filesystem for it to work.</text></comment>
<story><title>Remove first 300M lines from 700 GB file on 1TB disk</title><url>https://unix.stackexchange.com/questions/610494/how-do-i-remove-the-first-300-million-lines-from-a-700-gb-txt-file-on-a-system-w</url></story><parent_chain><item><author>hprotagonist</author><text>the only answer i’d use here is “go buy another 1TB drive and then ...”&lt;p&gt;$100 and a day and a half for shipping is a much better idea than playing russian roulette with work data by adjusting files in-place.</text></item></parent_chain><comment><author>Demiurge</author><text>Very recently I&amp;#x27;ve been in a similar situation with a dedicated server. There was no way to add storage and even getting another server would have been pretty expensive. Luckily it was a raid, so I just had to double the storage temporarily :)</text></comment>