<story><title>GitLab Database Incident – Live Report</title><url>https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub</url></story><parent_chain><item><author>hardwaresofton</author><text>The real question is what holds together duct tape?</text></item><item><author>gizmo</author><text>What if I told you all of society is held together by duct tape? If you&amp;#x27;re surprised that startups cut corners you&amp;#x27;re in for a rude awakening. I&amp;#x27;m frequently amazed anything works at all.</text></item><item><author>elementalest</author><text>&amp;gt;1.    LVM snapshots are by default only taken once every 24 hours. YP happened to run one manually about 6 hours prior to the outage&lt;p&gt;&amp;gt;2.    Regular backups seem to also only be taken once per 24 hours, though YP has not yet been able to figure out where they are stored. According to JN these don’t appear to be working, producing files only a few bytes in size.&lt;p&gt;&amp;gt;3.    SH: It looks like pg_dump may be failing because PostgreSQL 9.2 binaries are being run instead of 9.6 binaries. This happens because omnibus only uses Pg 9.6 if data&amp;#x2F;PG_VERSION is set to 9.6, but on workers this file does not exist. As a result it defaults to 9.2, failing silently. No SQL dumps were made as a result. Fog gem may have cleaned out older backups.&lt;p&gt;&amp;gt;4.    Disk snapshots in Azure are enabled for the NFS server, but not for the DB servers.
    The synchronisation process removes webhooks once it has synchronised data to staging. Unless we can pull these from a regular backup from the past 24 hours they will be lost
    The replication procedure is super fragile, prone to error, relies on a handful of random shell scripts, and is badly documented&lt;p&gt;&amp;gt;5.    Our backups to S3 apparently don’t work either: the bucket is empty&lt;p&gt;&amp;gt;So in other words, out of 5 backup&amp;#x2F;replication techniques deployed none are working reliably or set up in the first place.&lt;p&gt;Sounds like it was only a matter of time before something like this happened. How could so many systems be not working and no one notice?</text></item></parent_chain><comment><author>devdas</author><text>Duct tape is like the force. It has a light side and a dark side, and it holds the universe together.</text></comment>
<story><title>GitLab Database Incident – Live Report</title><url>https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub</url></story><parent_chain><item><author>hardwaresofton</author><text>The real question is what holds together duct tape?</text></item><item><author>gizmo</author><text>What if I told you all of society is held together by duct tape? If you&amp;#x27;re surprised that startups cut corners you&amp;#x27;re in for a rude awakening. I&amp;#x27;m frequently amazed anything works at all.</text></item><item><author>elementalest</author><text>&amp;gt;1.    LVM snapshots are by default only taken once every 24 hours. YP happened to run one manually about 6 hours prior to the outage&lt;p&gt;&amp;gt;2.    Regular backups seem to also only be taken once per 24 hours, though YP has not yet been able to figure out where they are stored. According to JN these don’t appear to be working, producing files only a few bytes in size.&lt;p&gt;&amp;gt;3.    SH: It looks like pg_dump may be failing because PostgreSQL 9.2 binaries are being run instead of 9.6 binaries. This happens because omnibus only uses Pg 9.6 if data&amp;#x2F;PG_VERSION is set to 9.6, but on workers this file does not exist. As a result it defaults to 9.2, failing silently. No SQL dumps were made as a result. Fog gem may have cleaned out older backups.&lt;p&gt;&amp;gt;4.    Disk snapshots in Azure are enabled for the NFS server, but not for the DB servers.
    The synchronisation process removes webhooks once it has synchronised data to staging. Unless we can pull these from a regular backup from the past 24 hours they will be lost
    The replication procedure is super fragile, prone to error, relies on a handful of random shell scripts, and is badly documented&lt;p&gt;&amp;gt;5.    Our backups to S3 apparently don’t work either: the bucket is empty&lt;p&gt;&amp;gt;So in other words, out of 5 backup&amp;#x2F;replication techniques deployed none are working reliably or set up in the first place.&lt;p&gt;Sounds like it was only a matter of time before something like this happened. How could so many systems be not working and no one notice?</text></item></parent_chain><comment><author>Vinkekatten</author><text>Good intentions and heartfelt prayer.</text></comment>