<story><title>Japan HP accidentally deleted 77TB data in Kyoto U. supercomputing system</title><url>https://www.iimc.kyoto-u.ac.jp/ja/whatsnew/information/detail/211228056999.html</url></story><parent_chain><item><author>doctor_eval</author><text>So this is something I’ve never understood. If you modify a shell script while it’s running, the shell executes the modified file. This normally but not always causes the script to fail.&lt;p&gt;Now I’ve known about this behaviour for a very long time and it always seemed very broken to me. It’s not how binaries work (at least not when I was doing that kind of thing).&lt;p&gt;So I guess bash or whatever does an mmap of the script it’s running, which is presumably why modifications to the script are visible immediately. But if a new file was installed eg using cp&amp;#x2F;tar&amp;#x2F;unzip, I’m surprised that this didn’t just unlink the old script and create a new one - which would create a new inode and therefore make the operation atomic, right? And this (I assume) is why a recompiled binary doesn’t have the same problem (because the old binary is first unlinked).&lt;p&gt;So, how could this (IMO) bad behaviour be fixed? Presumably mmap is used for efficiency, but isn’t it possible to mark a file as in use so it’s cant be modified? I’ve certainly seen on some old Unices that you can’t overwrite a running binary. Why can’t we do the same with shell scripts?&lt;p&gt;Honestly, while it’s great that HP is accepting responsibility, and we know that this happens, the behaviour seems both arbitrary and unnecessary to me. Is it fixable?</text></item><item><author>rvnx</author><text>I really appreciate the announcement from Hewlett Packard, which is very apologetic: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.iimc.kyoto-u.ac.jp&amp;#x2F;services&amp;#x2F;comp&amp;#x2F;pdf&amp;#x2F;file_loss_insident_20211228.pdf&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.iimc.kyoto-u.ac.jp&amp;#x2F;services&amp;#x2F;comp&amp;#x2F;pdf&amp;#x2F;file_loss_i...&lt;/a&gt;&lt;p&gt;They do not try to blame it on complex systems or other factors.&lt;p&gt;Users lost 1 day and 1&amp;#x2F;2 of recent work (which doesn&amp;#x27;t seem to be that bad).&lt;p&gt;&lt;pre&gt;&lt;code&gt;  About file loss in Luster file system in your supercomputer system, we are 100% responsible.
  We deeply apologize for causing a great deal of inconvenience due to the serious failure of the file loss.

  We would like to report the background of the file disappearance, its root cause and future countermeasures as follows:

  We believe that this file loss is 100% our responsibility.
  We will offer compensation for users who have lost files.

  [...]

  Impact:
  --

  Target file system: &amp;#x2F;LARGE0

  Deleted files: December 14, 2021 17:32 to December 16, 2021 12:43

  Files that were supposed to be deleted: Files that had not been updated since 17:32 on December 3, 2021

  [...]

  Cause:
  --

  The backup script uses the find command to delete log files that are older than 10 days.

  A variable name is passed to the delete process of the find command.

  A new improved version of the script was applied on the system.

  However, during deployment, there was a lack of consideration as the periodical script was not disabled.

  The modified shell script was reloaded from the middle.

  As a result, the find command containing undefined variables was executed and deleted the files.

  [...]

  Further measures:
  --

  In the future, the programs to be applied to the system will be fully verified and applied.

  We will examine the extent of the impact and make improvements so that similar problems do not occur.

  In addition, we will re-educate the engineers in charge of human error and risk prediction &amp;#x2F; prevention to prevent recurrence.

 We will thoroughly implement the measures.&lt;/code&gt;&lt;/pre&gt;</text></item></parent_chain><comment><author>wongarsu</author><text>&amp;gt; isn’t it possible to mark a file as in use so it’s cant be modified?&lt;p&gt;That&amp;#x27;s the route chosen by Windows for binary executables (exe&amp;#x2F;dll) and various other systems. Locking a file against writes, delete&amp;#x2F;rename or even read is just another flag in the windows equivalent of fopen [1]. This makes for software that&amp;#x27;s quite easy to reason about, but hard to update. The reason why you have to restart Windows to install Windows updates or even install some software is largely due to this locking mechanism: you can&amp;#x27;t update files that are open (and rename tricks don&amp;#x27;t work because locks apply to files, not inodes).&lt;p&gt;With about three decades of hindsight I&amp;#x27;m not sure if it&amp;#x27;s a good tradeoff. It makes it easy to prevent the race conditions that are an endless source of security bugs on unix-like systems; but otoh most software doesn&amp;#x27;t use the mechanism because it&amp;#x27;s not in the lowest-common-denominator File APIs of most programming languages; and MS is paying for it with users refusing to install updates because they don&amp;#x27;t want to restart their PC.&lt;p&gt;1: Search for FILE_SHARE_DELETE in &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;docs.microsoft.com&amp;#x2F;en-us&amp;#x2F;windows&amp;#x2F;win32&amp;#x2F;api&amp;#x2F;fileapi&amp;#x2F;nf-fileapi-createfilea&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;docs.microsoft.com&amp;#x2F;en-us&amp;#x2F;windows&amp;#x2F;win32&amp;#x2F;api&amp;#x2F;fileapi&amp;#x2F;n...&lt;/a&gt;</text></comment>
<story><title>Japan HP accidentally deleted 77TB data in Kyoto U. supercomputing system</title><url>https://www.iimc.kyoto-u.ac.jp/ja/whatsnew/information/detail/211228056999.html</url></story><parent_chain><item><author>doctor_eval</author><text>So this is something I’ve never understood. If you modify a shell script while it’s running, the shell executes the modified file. This normally but not always causes the script to fail.&lt;p&gt;Now I’ve known about this behaviour for a very long time and it always seemed very broken to me. It’s not how binaries work (at least not when I was doing that kind of thing).&lt;p&gt;So I guess bash or whatever does an mmap of the script it’s running, which is presumably why modifications to the script are visible immediately. But if a new file was installed eg using cp&amp;#x2F;tar&amp;#x2F;unzip, I’m surprised that this didn’t just unlink the old script and create a new one - which would create a new inode and therefore make the operation atomic, right? And this (I assume) is why a recompiled binary doesn’t have the same problem (because the old binary is first unlinked).&lt;p&gt;So, how could this (IMO) bad behaviour be fixed? Presumably mmap is used for efficiency, but isn’t it possible to mark a file as in use so it’s cant be modified? I’ve certainly seen on some old Unices that you can’t overwrite a running binary. Why can’t we do the same with shell scripts?&lt;p&gt;Honestly, while it’s great that HP is accepting responsibility, and we know that this happens, the behaviour seems both arbitrary and unnecessary to me. Is it fixable?</text></item><item><author>rvnx</author><text>I really appreciate the announcement from Hewlett Packard, which is very apologetic: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.iimc.kyoto-u.ac.jp&amp;#x2F;services&amp;#x2F;comp&amp;#x2F;pdf&amp;#x2F;file_loss_insident_20211228.pdf&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.iimc.kyoto-u.ac.jp&amp;#x2F;services&amp;#x2F;comp&amp;#x2F;pdf&amp;#x2F;file_loss_i...&lt;/a&gt;&lt;p&gt;They do not try to blame it on complex systems or other factors.&lt;p&gt;Users lost 1 day and 1&amp;#x2F;2 of recent work (which doesn&amp;#x27;t seem to be that bad).&lt;p&gt;&lt;pre&gt;&lt;code&gt;  About file loss in Luster file system in your supercomputer system, we are 100% responsible.
  We deeply apologize for causing a great deal of inconvenience due to the serious failure of the file loss.

  We would like to report the background of the file disappearance, its root cause and future countermeasures as follows:

  We believe that this file loss is 100% our responsibility.
  We will offer compensation for users who have lost files.

  [...]

  Impact:
  --

  Target file system: &amp;#x2F;LARGE0

  Deleted files: December 14, 2021 17:32 to December 16, 2021 12:43

  Files that were supposed to be deleted: Files that had not been updated since 17:32 on December 3, 2021

  [...]

  Cause:
  --

  The backup script uses the find command to delete log files that are older than 10 days.

  A variable name is passed to the delete process of the find command.

  A new improved version of the script was applied on the system.

  However, during deployment, there was a lack of consideration as the periodical script was not disabled.

  The modified shell script was reloaded from the middle.

  As a result, the find command containing undefined variables was executed and deleted the files.

  [...]

  Further measures:
  --

  In the future, the programs to be applied to the system will be fully verified and applied.

  We will examine the extent of the impact and make improvements so that similar problems do not occur.

  In addition, we will re-educate the engineers in charge of human error and risk prediction &amp;#x2F; prevention to prevent recurrence.

 We will thoroughly implement the measures.&lt;/code&gt;&lt;/pre&gt;</text></item></parent_chain><comment><author>kragen</author><text>This behavior in shell scripts predates mmap.  In very early versions of Unix it was arguably even useful; there was a goto command which was implemented by seeking on the shell-script file descriptor rather than as a shell builtin, for example.  I don&amp;#x27;t know of any use for it since the transition to the Bourne shell, but my knowledge is far from comprehensive.  (I suppose if your shell script is not small compared to the size of RAM, it might be undesirable to read it all in at the start of execution; shar files are a real-life example even on non-PDP-11 machines.)&lt;p&gt;As I understand it, the reason for ETXTBSY (&amp;quot;on some old Unices...you can&amp;#x27;t overwrite a running binary&amp;quot;) was to prevent segfaults.&lt;p&gt;cp usually just opens the file O_WRONLY|O_TRUNC, which seems like the wrong default; Emacs for example does create a new file and rename it over the old one when you save, usually, allocating a new inode as you say.  By default it makes an exception if there are other hardlinks to the file.&lt;p&gt;Btrfs and xfs have a &amp;quot;reflink&amp;quot; feature that allows you to efficiently make a copy-on-write snapshot of a file, which would be ideal for this sort of thing, since the shell or whatever won&amp;#x27;t see any changes to the original file, even if it&amp;#x27;s overwritten in place.  Unfortunately I don&amp;#x27;t think you can make &lt;i&gt;anonymous&lt;/i&gt; reflinks, so for the shell to reflink a shell script when it starts executing it would need write access to somewhere in the filesystem to put the reflink, and then it would need to know how to find that place, somehow.  And of course that wouldn&amp;#x27;t help if you were running on ext4fs or, I imagine, Lustre, though apparently an implementation was proposed in 02019: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;wiki.lustre.org&amp;#x2F;Lreflink_High_Level_Design&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;wiki.lustre.org&amp;#x2F;Lreflink_High_Level_Design&lt;/a&gt;</text></comment>