<story><title>Jepsen: RethinkDB 2.2.3 reconfiguration</title><url>https://aphyr.com/posts/330-jepsen-rethinkdb-2-2-3-reconfiguration</url></story><parent_chain></parent_chain><comment><author>coffeemug</author><text>Slava @ RethinkDB here.&lt;p&gt;It&amp;#x27;s been a pleasure working with Kyle on doing an exhaustive analysis of RethinkDB and tracking down bugs revealed in the second test. Our lead engineer wrote a detailed analysis of our findings (linked from the blog post, but I wanted to link it again): &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;rethinkdb&amp;#x2F;rethinkdb&amp;#x2F;issues&amp;#x2F;5289#issuecomment-175394540&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;rethinkdb&amp;#x2F;rethinkdb&amp;#x2F;issues&amp;#x2F;5289#issuecomm...&lt;/a&gt;. For anyone interested in the details, this is a good read!</text></comment>
<story><title>Jepsen: RethinkDB 2.2.3 reconfiguration</title><url>https://aphyr.com/posts/330-jepsen-rethinkdb-2-2-3-reconfiguration</url></story><parent_chain></parent_chain><comment><author>emidln</author><text>As someone employed in building complex systems, I appreciate the takeaways listed during the discussion:&lt;p&gt;&lt;pre&gt;&lt;code&gt;    RethinkDB’s engineers noted three takeaways from tracking
    down this bug. First, fuzz-testing—at both the functional
    and integration-test level, can be a powerful tool for
    verifying systems with complex order dependence. Second,
    runtime invariant assertions were key in identifying the
    underlying cause. A test like Jepsen can tell you that the
    cluster can exhibit split-brain behavior, but can’t tell
    you anything about why. The error messages from those
    leader and log order assertions were key hints in tracking
    down the bug. Rethink plans to introduce additional runtime
    assertions to help identify future problems. Finally, they
    plan to devote more attention to issues which suggest—even
    tangentially—consistency errors.&lt;/code&gt;&lt;/pre&gt;</text></comment>