<story><title>How to build software like an SRE</title><url>https://www.willett.io/posts/precepts/</url></story><parent_chain><item><author>rytis</author><text>&amp;gt; Extremely strict RPC settings. I’m talking zero retries (or MAYBE one) [...]&lt;p&gt;I disagree. If we&amp;#x27;re talking about distributed systems, then one thing is guaranteed - network is not going to be reliable. And if we have 10&amp;#x27;s or 100&amp;#x27;s of services, this policy means that at the smallest blip the whole thing collapses like a house of cards. If the concern is &amp;quot;it&amp;#x27;s hard to troubleshoot&amp;quot;, well then perhaps implement better logging (&amp;quot;connectivity to service X has been unreliable with X% failure rate over the last Xhrs&amp;quot; instead of &amp;quot;connection terminated&amp;quot;, or no logging at all).</text></item></parent_chain><comment><author>jasonhansel</author><text>The answer is to &amp;quot;kick retries up the stack&amp;quot;; when you fail to reach a service, you return a 503 and have &lt;i&gt;your&lt;/i&gt; clients retry, to avoid a case where every service in the stack starts retrying all at once and causes a massive increase in traffic.&lt;p&gt;IMHO you should only add retries if it proves necessary &lt;i&gt;in practice&lt;/i&gt; to reach your SLA, if you&amp;#x27;re building something that isn&amp;#x27;t itself triggered by an RPC, or if you&amp;#x27;re performing an operation that can never be made idempotent.</text></comment>
<story><title>How to build software like an SRE</title><url>https://www.willett.io/posts/precepts/</url></story><parent_chain><item><author>rytis</author><text>&amp;gt; Extremely strict RPC settings. I’m talking zero retries (or MAYBE one) [...]&lt;p&gt;I disagree. If we&amp;#x27;re talking about distributed systems, then one thing is guaranteed - network is not going to be reliable. And if we have 10&amp;#x27;s or 100&amp;#x27;s of services, this policy means that at the smallest blip the whole thing collapses like a house of cards. If the concern is &amp;quot;it&amp;#x27;s hard to troubleshoot&amp;quot;, well then perhaps implement better logging (&amp;quot;connectivity to service X has been unreliable with X% failure rate over the last Xhrs&amp;quot; instead of &amp;quot;connection terminated&amp;quot;, or no logging at all).</text></item></parent_chain><comment><author>saiya-jin</author><text>This particular concern costed some 10 folks 2 mandays per head last week across the globe and due to consequences of the issue got escalated to higher management.&lt;p&gt;Different topic a bit - messaging &amp;amp; routing ecosystem working 10 years without flaw suddenly started exhibiting slowness and randomly would just stop, needing restart of client. We debugged like crazy, java messaging system by me on one end, Tibco ems system on the other. Tibco refused to help due to server being out of support (note for us&amp;#x2F;US team). We had network guys on 13h call too, but they didn&amp;#x27;t have as much experience with WANs.&lt;p&gt;After 2 days, they discovered some internal backbone network system between US and Switzerland just failed out of blue in the worst way possible - dropped some +-20% of the packets, so things kept chugging along somehow, till they didn&amp;#x27;t (some Acks on transaction commits rarely didn&amp;#x27;t happen and then all got blocked without a hint why).&lt;p&gt;2 lessons - don&amp;#x27;t always doubt yourself and your skills when SHTF. And don&amp;#x27;t take things like servers, OS, network for granted. Don&amp;#x27;t expect some clever monitoring already in place will figure out issues for you.</text></comment>