<story><title>I Accidentally Deleted 7TB of Videos Before Going to Production</title><url>https://blog.thevinter.com/posts/vimeo</url></story><parent_chain><item><author>tauwauwau</author><text>Once we get used to doing same thing multiple times a day, it doesn&amp;#x27;t matter if the log shows that we&amp;#x27;re about to take a destructive action, we&amp;#x27;ll still do it. 
Only thing that is foolproof is to not take the destructive action because people make mistake, it&amp;#x27;s human nature. I don&amp;#x27;t know how this can be implemented, may be encrypt the files, take a backup in some other location (which may not be allowed).&lt;p&gt;Multiple reviewers here didn&amp;#x27;t catch the mistake&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.bloombergquint.com&amp;#x2F;markets&amp;#x2F;citi-s-900-million-misfire-happened-in-midst-of-software-switch&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.bloombergquint.com&amp;#x2F;markets&amp;#x2F;citi-s-900-million-mi...&lt;/a&gt;</text></item><item><author>dkersten</author><text>I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection.&lt;p&gt;This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don&amp;#x27;t actually do it. Then have multiple people inspect the logs. Once ok&amp;#x27;d, run again, with manual prompts after each log item asking to continue, for the first few files&amp;#x2F;bits of data. Only after that was ok&amp;#x27;d too did it run the remainder.&lt;p&gt;In other things I&amp;#x27;ve worked on, I&amp;#x27;ve taken the terraform-style plan first, then apply the plan approach, with manual inspection of the plan in between.</text></item><item><author>dsego</author><text>&amp;gt; but at the time the code seemed completely correct to me&lt;p&gt;It always does.&lt;p&gt;&amp;gt; Well, it teaches me to do more diverse tests when doing destructive operations.&lt;p&gt;Or add some logging and do a dry run and check the results, literally simple prints statements:&lt;p&gt;&lt;pre&gt;&lt;code&gt;    print(&amp;quot;-----&amp;quot;)
    print(&amp;quot;Downloading videos ids from url: {url}&amp;quot;)
    print(list of ids)
    ...
    ...
    ...
    # delete()  dangerous action commented out until I&amp;#x27;m sure it&amp;#x27;s right
    print(&amp;quot;I&amp;#x27;m about to delete video {id}&amp;quot;)

    print(&amp;quot;Deleted {count} videos&amp;quot;) # maybe even assert
    ...
&lt;/code&gt;&lt;/pre&gt;
Then dump out to a file and spot check it five times before running for real.</text></item></parent_chain><comment><author>HowardStark</author><text>While this is a huge issue, a solution (well, a partial mitigation) I&amp;#x27;ve seen and used is the &amp;quot;Pointing and Calling&amp;quot; technique. The basic idea is that you incorporate more actions beyond reading and typing or pressing a button—generally by having people point at something and say aloud what it is they&amp;#x27;re doing and what they expect to happen.&lt;p&gt;It&amp;#x27;s used rather extensively in safety-critical public transportation in Japan [1] and to a lesser extent in New York (along with many other countries) [2]. This can easily extend to software without overcomplicating by just setting the expectation that engineers, Q&amp;amp;A, etc. do this even when alone.&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.atlasobscura.com&amp;#x2F;articles&amp;#x2F;pointing-and-calling-japan-trains&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.atlasobscura.com&amp;#x2F;articles&amp;#x2F;pointing-and-calling-j...&lt;/a&gt;&lt;p&gt;[2] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Pointing_and_calling&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Pointing_and_calling&lt;/a&gt;</text></comment>
<story><title>I Accidentally Deleted 7TB of Videos Before Going to Production</title><url>https://blog.thevinter.com/posts/vimeo</url></story><parent_chain><item><author>tauwauwau</author><text>Once we get used to doing same thing multiple times a day, it doesn&amp;#x27;t matter if the log shows that we&amp;#x27;re about to take a destructive action, we&amp;#x27;ll still do it. 
Only thing that is foolproof is to not take the destructive action because people make mistake, it&amp;#x27;s human nature. I don&amp;#x27;t know how this can be implemented, may be encrypt the files, take a backup in some other location (which may not be allowed).&lt;p&gt;Multiple reviewers here didn&amp;#x27;t catch the mistake&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.bloombergquint.com&amp;#x2F;markets&amp;#x2F;citi-s-900-million-misfire-happened-in-midst-of-software-switch&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.bloombergquint.com&amp;#x2F;markets&amp;#x2F;citi-s-900-million-mi...&lt;/a&gt;</text></item><item><author>dkersten</author><text>I was involved with archiving of data that was legally required to be retained for PSD2 compliance. So it was pretty important that the data was correctly archived, but it was just as important that it was properly removed from other places due to data protection.&lt;p&gt;This is basically the approach that was taken: log before and after every action exactly what data or files is being acted on and how. Don&amp;#x27;t actually do it. Then have multiple people inspect the logs. Once ok&amp;#x27;d, run again, with manual prompts after each log item asking to continue, for the first few files&amp;#x2F;bits of data. Only after that was ok&amp;#x27;d too did it run the remainder.&lt;p&gt;In other things I&amp;#x27;ve worked on, I&amp;#x27;ve taken the terraform-style plan first, then apply the plan approach, with manual inspection of the plan in between.</text></item><item><author>dsego</author><text>&amp;gt; but at the time the code seemed completely correct to me&lt;p&gt;It always does.&lt;p&gt;&amp;gt; Well, it teaches me to do more diverse tests when doing destructive operations.&lt;p&gt;Or add some logging and do a dry run and check the results, literally simple prints statements:&lt;p&gt;&lt;pre&gt;&lt;code&gt;    print(&amp;quot;-----&amp;quot;)
    print(&amp;quot;Downloading videos ids from url: {url}&amp;quot;)
    print(list of ids)
    ...
    ...
    ...
    # delete()  dangerous action commented out until I&amp;#x27;m sure it&amp;#x27;s right
    print(&amp;quot;I&amp;#x27;m about to delete video {id}&amp;quot;)

    print(&amp;quot;Deleted {count} videos&amp;quot;) # maybe even assert
    ...
&lt;/code&gt;&lt;/pre&gt;
Then dump out to a file and spot check it five times before running for real.</text></item></parent_chain><comment><author>irrational</author><text>Because everyone assumes that everyone else is looking at it more closely than they are. “I’ll just do a cursory look since I’m sure everyone else is doing a in-depth look.” Narrator: nobody did an in-depth search.</text></comment>