<story><title>Anatomy of a Ceph meltdown</title><url>http://chneukirchen.org/blog/archive/2018/01/anatomy-of-a-ceph-meltdown.html</url></story><parent_chain><item><author>saganus</author><text>To be fair, the author says right at the beginning:&lt;p&gt;&amp;quot;Please remember that we are all unpaid volunteers who have our own studies and&amp;#x2F;or day jobs, and no one has had more experience with Ceph than what you get from reading the manual.&amp;quot;&lt;p&gt;So while you might have a valid point, it&amp;#x27;s hard to develop those skills if you are not a proper sysadmin. If you are just an unpaid volunteer, it means that most likely the environment you work in can&amp;#x27;t afford (or won&amp;#x27;t pay) for actual sysadmins whose paid job is to know these things.&lt;p&gt;Maybe the discussion should be, why were there unpaid volunteers doing sysadmin work instead of an actual sysadmin, but I guess that is a normal occurrence in universities all over the world?</text></item><item><author>nimbius</author><text>Most of this post-mortem has nothing to do with Ceph or Gentoo, and everything to do with competent system administration and change management.  No one appears to have stopped to consider the ramifications of unannounced upgrades to the fileserver or OSD&amp;#x27;s.  Once you&amp;#x27;re frantically rebuilding world, its over.  the lessons learned completely ignore communications between sysadmins and management or, god forbid, even customers.  &amp;quot;do everything whenever you like&amp;quot; is not system administration.&lt;p&gt;&amp;gt;If we notice something is going wrong with Ceph, we will not hesitate to shut down the cluster prematurely.&lt;p&gt;once bitten..twice shy?  how does this solve future problems in the HA filesystem for your customer?&lt;p&gt;&amp;gt;We should not update Ceph on all machines at once.&lt;p&gt;HA&amp;#x2F;HPC 101.  rolling upgrades are fine, slow and steady, with lots of testing and a documented rollback procedure if things go screwy.  large enough&amp;#x2F;critical systems often have dev test and prod&lt;p&gt;&amp;gt;We will build glibc with debugging symbols.&lt;p&gt;you will plan your OS upgrades separate from your Ceph upgrades and with properly communicated actions before, during, and after the events.&lt;p&gt;&amp;gt;We will track Ceph releases more closely&lt;p&gt;as opposed to...what? being a sysadmin, Im on more than a dozen release mailing lists.  I track load balancer known issues and patches to make sure i dont roll out, say, an F5 firmware with known failure to properly proxy HTTP2</text></item></parent_chain><comment><author>batbomb</author><text>They also said:&lt;p&gt;&amp;gt; First and foremost, having a multiple-day spanning downtime is completely unacceptable for a central service like this [...]&lt;p&gt;Which goes to your last point a bit. It&amp;#x27;s not a normal occurrence to all universities, but to some. If it was me, I&amp;#x27;d turn this thing off repeatedly until the people used it realized it&amp;#x27;s value and tried to dedicate a bit of manpower to it.</text></comment>
<story><title>Anatomy of a Ceph meltdown</title><url>http://chneukirchen.org/blog/archive/2018/01/anatomy-of-a-ceph-meltdown.html</url></story><parent_chain><item><author>saganus</author><text>To be fair, the author says right at the beginning:&lt;p&gt;&amp;quot;Please remember that we are all unpaid volunteers who have our own studies and&amp;#x2F;or day jobs, and no one has had more experience with Ceph than what you get from reading the manual.&amp;quot;&lt;p&gt;So while you might have a valid point, it&amp;#x27;s hard to develop those skills if you are not a proper sysadmin. If you are just an unpaid volunteer, it means that most likely the environment you work in can&amp;#x27;t afford (or won&amp;#x27;t pay) for actual sysadmins whose paid job is to know these things.&lt;p&gt;Maybe the discussion should be, why were there unpaid volunteers doing sysadmin work instead of an actual sysadmin, but I guess that is a normal occurrence in universities all over the world?</text></item><item><author>nimbius</author><text>Most of this post-mortem has nothing to do with Ceph or Gentoo, and everything to do with competent system administration and change management.  No one appears to have stopped to consider the ramifications of unannounced upgrades to the fileserver or OSD&amp;#x27;s.  Once you&amp;#x27;re frantically rebuilding world, its over.  the lessons learned completely ignore communications between sysadmins and management or, god forbid, even customers.  &amp;quot;do everything whenever you like&amp;quot; is not system administration.&lt;p&gt;&amp;gt;If we notice something is going wrong with Ceph, we will not hesitate to shut down the cluster prematurely.&lt;p&gt;once bitten..twice shy?  how does this solve future problems in the HA filesystem for your customer?&lt;p&gt;&amp;gt;We should not update Ceph on all machines at once.&lt;p&gt;HA&amp;#x2F;HPC 101.  rolling upgrades are fine, slow and steady, with lots of testing and a documented rollback procedure if things go screwy.  large enough&amp;#x2F;critical systems often have dev test and prod&lt;p&gt;&amp;gt;We will build glibc with debugging symbols.&lt;p&gt;you will plan your OS upgrades separate from your Ceph upgrades and with properly communicated actions before, during, and after the events.&lt;p&gt;&amp;gt;We will track Ceph releases more closely&lt;p&gt;as opposed to...what? being a sysadmin, Im on more than a dozen release mailing lists.  I track load balancer known issues and patches to make sure i dont roll out, say, an F5 firmware with known failure to properly proxy HTTP2</text></item></parent_chain><comment><author>kbenson</author><text>Considering sysadmins often have keys to the kingdom, unpaid volunteers for sysadmin work seems &lt;i&gt;really&lt;/i&gt; risky to me. Want volunteers to design a webapp for you? Go ahead, your professional sysadmin will make sure they only get permission to what they need. It doesn&amp;#x27;t work well the other way around.</text></comment>