<story><title>NewSQL databases fail to guarantee consistency and I blame Spanner</title><url>http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-failing-to.html</url></story><parent_chain><item><author>Quekid5</author><text>I may be a bit of an old fart, but this is the exact reasoning behind my decision to never go with &amp;quot;distributed X&amp;quot; if there&amp;#x27;s a &amp;quot;single-machine X&amp;quot; where you can just vertically scale.&lt;p&gt;If you can afford 3-5 machines&amp;#x2F;VMs for a cluster you can almost certainly afford a single machine&amp;#x2F;VM with 2-4x the resources&amp;#x2F;CPU and chances are that it&amp;#x27;ll perform just as well (or better) because it doesn&amp;#x27;t have network latency to contend with.&lt;p&gt;Of course if you&amp;#x27;re around N &amp;gt;= 7 or N &amp;gt;= 9, &lt;i&gt;then&lt;/i&gt; you should perhaps start considering &amp;quot;distributed X&amp;quot;.&lt;p&gt;As long as your application is mostly built on very few assumptions about consistency it&amp;#x27;s usually pretty painless to actually go distributed.&lt;p&gt;Of course, there &lt;i&gt;are&lt;/i&gt; legitimate cases where you want to go distributed even with N=3 or N=5, but IME they&amp;#x27;re very few and far between... but if your 3&amp;#x2F;5 machines are co-located then it&amp;#x27;s really likely that the &amp;quot;partitioning&amp;quot; problem is one of the scenarios where they actually &lt;i&gt;all&lt;/i&gt; go down simultaneously or the availability goes to 0 because you can&amp;#x27;t actually reach any of the machines (which are likely to be on the &amp;#x27;same&amp;#x27; network).</text></item><item><author>nemothekid</author><text>&amp;gt;&lt;i&gt;Systems that guarantee consistency only experience a necessary reduction in availability in the event of a network partition. As networks become more redundant, partitions become an increasingly rare event. And even if there is a partition, it is still possible for the majority partition to be available&lt;/i&gt;&lt;p&gt;In my experience, yes network partitions are incredibly rare. However 99% of my distributed ststem partitions have little do with the network. When running databases on a cloud environment network partitions can occur for a variety of reasons that don’t actually include the network link between databases:&lt;p&gt;1. The host database is written in a GC’d language and experiences a pathological GC pause.&lt;p&gt;2. The Virtual machine is migrated and experiences a pathological pause&lt;p&gt;3. Google migrates your machine with local SSDs, fucks up that process and you lose all your data on that machine (you do have backups right?)&lt;p&gt;4. AWS retires your instance and you need to reboot your VM.&lt;p&gt;You may never see these issue if you are running a 3 or 5 cluster database. I began seeing issues like this semi regularly once the cluster grew to 30-40 machines (Cassandra). Now I will agree that none of the issues took down majority, but if your R=3, it really only takes an unlucky partition to fuck up an entire shard</text></item></parent_chain><comment><author>toast0</author><text>I think part of this is that most of the common knowledge about scaling is hard fought from the 90s&amp;#x2F;2000 era. eBay got bigger and bigger Sun boxes to run Oracle, until they couldn&amp;#x27;t get anything bigger -- then they had a problem and had to shard their listings into categories, etc. In the last few Intel cpu generations, computation performance has had small gains, but addressable memory has doubled about every other release, you can now get machines with 2TB of ram, and it&amp;#x27;s not that expensive.&lt;p&gt;You can fit &lt;i&gt;a lot&lt;/i&gt; of data in an in memory database of 2TB.</text></comment>
<story><title>NewSQL databases fail to guarantee consistency and I blame Spanner</title><url>http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-failing-to.html</url></story><parent_chain><item><author>Quekid5</author><text>I may be a bit of an old fart, but this is the exact reasoning behind my decision to never go with &amp;quot;distributed X&amp;quot; if there&amp;#x27;s a &amp;quot;single-machine X&amp;quot; where you can just vertically scale.&lt;p&gt;If you can afford 3-5 machines&amp;#x2F;VMs for a cluster you can almost certainly afford a single machine&amp;#x2F;VM with 2-4x the resources&amp;#x2F;CPU and chances are that it&amp;#x27;ll perform just as well (or better) because it doesn&amp;#x27;t have network latency to contend with.&lt;p&gt;Of course if you&amp;#x27;re around N &amp;gt;= 7 or N &amp;gt;= 9, &lt;i&gt;then&lt;/i&gt; you should perhaps start considering &amp;quot;distributed X&amp;quot;.&lt;p&gt;As long as your application is mostly built on very few assumptions about consistency it&amp;#x27;s usually pretty painless to actually go distributed.&lt;p&gt;Of course, there &lt;i&gt;are&lt;/i&gt; legitimate cases where you want to go distributed even with N=3 or N=5, but IME they&amp;#x27;re very few and far between... but if your 3&amp;#x2F;5 machines are co-located then it&amp;#x27;s really likely that the &amp;quot;partitioning&amp;quot; problem is one of the scenarios where they actually &lt;i&gt;all&lt;/i&gt; go down simultaneously or the availability goes to 0 because you can&amp;#x27;t actually reach any of the machines (which are likely to be on the &amp;#x27;same&amp;#x27; network).</text></item><item><author>nemothekid</author><text>&amp;gt;&lt;i&gt;Systems that guarantee consistency only experience a necessary reduction in availability in the event of a network partition. As networks become more redundant, partitions become an increasingly rare event. And even if there is a partition, it is still possible for the majority partition to be available&lt;/i&gt;&lt;p&gt;In my experience, yes network partitions are incredibly rare. However 99% of my distributed ststem partitions have little do with the network. When running databases on a cloud environment network partitions can occur for a variety of reasons that don’t actually include the network link between databases:&lt;p&gt;1. The host database is written in a GC’d language and experiences a pathological GC pause.&lt;p&gt;2. The Virtual machine is migrated and experiences a pathological pause&lt;p&gt;3. Google migrates your machine with local SSDs, fucks up that process and you lose all your data on that machine (you do have backups right?)&lt;p&gt;4. AWS retires your instance and you need to reboot your VM.&lt;p&gt;You may never see these issue if you are running a 3 or 5 cluster database. I began seeing issues like this semi regularly once the cluster grew to 30-40 machines (Cassandra). Now I will agree that none of the issues took down majority, but if your R=3, it really only takes an unlucky partition to fuck up an entire shard</text></item></parent_chain><comment><author>Serow225</author><text>My experience is that it&amp;#x27;s unfortunately really hard to convince people who don&amp;#x27;t deeply understand distributed systems (but think they know) that just because a system is called &amp;#x27;HA&amp;#x27; or can have an &amp;#x27;HA&amp;#x27; mode turned on, that doing so has downsides. They freak out if you try to propose _not_ running the HA mode, because they don&amp;#x27;t (or aren&amp;#x27;t willing to) understand the potential downsides of dealing with split-brain, bad recovery, etc. They just see it as a &amp;quot;enable checkbox and forget it&amp;quot; kind of thing that should always be turned on. This is a big struggle of mine, any suggestions would be appreciated...</text></comment>