<story><title>Raft Consensus Protocol</title><url>https://www.consul.io/docs/architecture/consensus</url></story><parent_chain></parent_chain><comment><author>atonse</author><text>I’m going to ask folks here a question.&lt;p&gt;Recently our consul and nomad clusters both blew up within a day of each other.&lt;p&gt;In some crazy twist of luck, Amazon removed the underlying instances that happened to run the leaders to both clusters. This was in our QA environment so they were all t2.nanos.&lt;p&gt;In this situation, shouldn’t we just expect the other two nodes to hold an election and elect a new leader? Isn’t this the most plain vanilla use case there is?&lt;p&gt;In both situations, the clusters got stuck without leaders indefinitely because they got stuck  trying to ping the leader to see if they could hold an election (how could they?? the whole point of the election is because the leader disappeared). And the only way to recover was to do some insane dance of building a JSON file and hard coding IPs.&lt;p&gt;Based on some research, it seems like we need to hard code IPs to avoid this in the future. This seems like a huge smell and goes against everything I’ve ever read about raft and the idea of self healing clusters.&lt;p&gt;What am I missing here? I don’t remember ever having to deal with this with mongodb in 2014.</text></comment>
<story><title>Raft Consensus Protocol</title><url>https://www.consul.io/docs/architecture/consensus</url></story><parent_chain></parent_chain><comment><author>zuj</author><text>For those looking for how raft works&lt;p&gt;&lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;thesecretlivesofdata.com&amp;#x2F;raft&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;thesecretlivesofdata.com&amp;#x2F;raft&amp;#x2F;&lt;/a&gt;</text></comment>