<story><title>New hard drive rituals (2018)</title><url>https://blog.linuxserver.io/2018/10/29/new-hard-drive-rituals/</url></story><parent_chain><item><author>jolmg</author><text>On the topic of new rituals for storage devices, whenever I buy a microSD card or a USB flash drive and before I setup the filesystem, I&amp;#x27;ve been filling it with random data and reading it back to ensure it does have the storage capacity it&amp;#x27;s supposed to have.&lt;p&gt;I started doing this after reading comments online (probably here) on fraudulent cards and drives circulating around Amazon. Some comments mentioned that there are even some devices that have their firmware set-up such that writes don&amp;#x27;t fail even when you&amp;#x27;ve written more than what the device can actually hold. They supposedly manage this by reusing previously used blocks for new files. I suppose in other words, each block has multiple addresses. The addresses would probably loop around the same blocks until the fake size is reached. That would give it the most realistic appearance that it&amp;#x27;s of the reported size while silently corrupting previously written storage.&lt;p&gt;In any case, so far I&amp;#x27;ve only gotten 4 cards&amp;#x2F;drives since then, and I haven&amp;#x27;t found them to be frauds.&lt;p&gt;I&amp;#x27;ve since wondered how much I&amp;#x27;m subtracting from their product lifespan with my testing. I mean, I imagine that microSD cards and USB flash drives do wear leveling on unused blocks, but without TRIM support on them, after testing them, I&amp;#x27;m probably eliminating the devices&amp;#x27; ability to do wear leveling since I&amp;#x27;m occupying all blocks without the ability to tell the device that they&amp;#x27;re unoccupied now.&lt;p&gt;As far as I understand, the problem with lacking TRIM support on USB flash drives is not that there&amp;#x27;s no protocol for it through USB, since there are USB-SATA adapters that specify support for UASP. Admittedly, though, I&amp;#x27;m assuming that UASP is enough to get TRIM support. I don&amp;#x27;t remember testing that.&lt;p&gt;In any case, I wonder if this new ritual I&amp;#x27;m doing is really worth it, and if it&amp;#x27;s not, when would it be?&lt;p&gt;EDIT: I wonder if it&amp;#x27;s realistic to expect that one day we&amp;#x27;ll get USB flash drives and microSD cards with TRIM support. I don&amp;#x27;t expect the need to check for fraudulent storage devices to go away.</text></item></parent_chain><comment><author>tialaramex</author><text>Filling with random data is advisable anyway if you use encryption for storage, which almost everybody should do (it&amp;#x27;s clearly fine to store stuff that is genuinely public unencrypted, but if you&amp;#x27;re not sure probably not everything is public and so just encrypt everything)&lt;p&gt;The reason to write random data is that otherwise an adversary inspecting the raw device can determine how much you really stored in the encrypted volume, as the chance of a block being full of zeroes _after_ encryption is negligible so such blocks are invariably just untouched.&lt;p&gt;The encryption will ensure that random bits and encrypted data are indistinguishable (in practice, in theory this isn&amp;#x27;t a requirement of things like IND-CPA it just happens that it&amp;#x27;s what you get anyway).</text></comment>
<story><title>New hard drive rituals (2018)</title><url>https://blog.linuxserver.io/2018/10/29/new-hard-drive-rituals/</url></story><parent_chain><item><author>jolmg</author><text>On the topic of new rituals for storage devices, whenever I buy a microSD card or a USB flash drive and before I setup the filesystem, I&amp;#x27;ve been filling it with random data and reading it back to ensure it does have the storage capacity it&amp;#x27;s supposed to have.&lt;p&gt;I started doing this after reading comments online (probably here) on fraudulent cards and drives circulating around Amazon. Some comments mentioned that there are even some devices that have their firmware set-up such that writes don&amp;#x27;t fail even when you&amp;#x27;ve written more than what the device can actually hold. They supposedly manage this by reusing previously used blocks for new files. I suppose in other words, each block has multiple addresses. The addresses would probably loop around the same blocks until the fake size is reached. That would give it the most realistic appearance that it&amp;#x27;s of the reported size while silently corrupting previously written storage.&lt;p&gt;In any case, so far I&amp;#x27;ve only gotten 4 cards&amp;#x2F;drives since then, and I haven&amp;#x27;t found them to be frauds.&lt;p&gt;I&amp;#x27;ve since wondered how much I&amp;#x27;m subtracting from their product lifespan with my testing. I mean, I imagine that microSD cards and USB flash drives do wear leveling on unused blocks, but without TRIM support on them, after testing them, I&amp;#x27;m probably eliminating the devices&amp;#x27; ability to do wear leveling since I&amp;#x27;m occupying all blocks without the ability to tell the device that they&amp;#x27;re unoccupied now.&lt;p&gt;As far as I understand, the problem with lacking TRIM support on USB flash drives is not that there&amp;#x27;s no protocol for it through USB, since there are USB-SATA adapters that specify support for UASP. Admittedly, though, I&amp;#x27;m assuming that UASP is enough to get TRIM support. I don&amp;#x27;t remember testing that.&lt;p&gt;In any case, I wonder if this new ritual I&amp;#x27;m doing is really worth it, and if it&amp;#x27;s not, when would it be?&lt;p&gt;EDIT: I wonder if it&amp;#x27;s realistic to expect that one day we&amp;#x27;ll get USB flash drives and microSD cards with TRIM support. I don&amp;#x27;t expect the need to check for fraudulent storage devices to go away.</text></item></parent_chain><comment><author>mceachen</author><text>There&amp;#x27;s a tool called `f3` that does this automatically: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;fight-flash-fraud.readthedocs.io&amp;#x2F;en&amp;#x2F;latest&amp;#x2F;index.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;fight-flash-fraud.readthedocs.io&amp;#x2F;en&amp;#x2F;latest&amp;#x2F;index.htm...&lt;/a&gt;&lt;p&gt;TL;DR (for Debian variants):&lt;p&gt;&lt;pre&gt;&lt;code&gt;    $ sudo apt install f3
    $ cd &amp;#x2F;MOUNTPOINT-FOR-SD
    $ f3write .
    $ f3read .&lt;/code&gt;&lt;/pre&gt;</text></comment>