<story><title>Re: MemSQL the “world&apos;s fastest database”? (2012)</title><url>http://www.postgresql.org/message-id/4FE8A2DE.5050806@agliodbs.com</url></story><parent_chain><item><author>ankrgyl</author><text>This post is missing a key trade-off that you get with an in-memory row store (tl;dr ability to build lock-free), but also since 6&amp;#x2F;25&amp;#x2F;2012 several other innovations in MemSQL that have shipped over the past 2+ years.&lt;p&gt;The main advantage of having a main-memory row store is the ability to leverage random-access data in memory. This unlocks you to use random access aggressively and build lock-free indexes (atomic instructions are available only for data that&amp;#x27;s in memory). With these tradeoffs you can build extremely concurrent data structures that simply aren&amp;#x27;t possible with a system that must manage a buffer cache. See &lt;a href=&quot;http://blog.memsql.com/the-story-behind-memsqls-skiplist-indexes/&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;blog.memsql.com&amp;#x2F;the-story-behind-memsqls-skiplist-ind...&lt;/a&gt; for more details.&lt;p&gt;The article also suggests that MemSQL is not durable in the event of a power outage. This is simply not the case - we&amp;#x27;ve had durability since the first ever public release of MemSQL, including both asynchronous and synchronous modes.&lt;p&gt;MemSQL now also has distributed transactions and MPP, supports intra and inter data-center replication, semi-structured (JSON) data, and even has a column store that works as a hybrid between disk and in-memory.&lt;p&gt;And BTW, I actually agree with this mentality in the context of column stores. There&amp;#x27;s no real advantage to having an &amp;quot;in-memory&amp;quot; column store because of the append-only nature of columnar blobs of data. That&amp;#x27;s why ours is a hybrid.</text></item></parent_chain><comment><author>colanderman</author><text>As someone who knows nothing about database implementations but a lot about memory, this makes no sense:&lt;p&gt;&amp;gt; &lt;i&gt;atomic instructions are available only for data that&amp;#x27;s in memory&lt;/i&gt;&lt;p&gt;&lt;i&gt;All&lt;/i&gt; CPU instructions are available only for data that&amp;#x27;s in memory (or registers).  No (sane) database on Earth operates solely by issuing disk I&amp;#x2F;O.&lt;p&gt;That linked blog post doesn&amp;#x27;t support your claim, either.  It makes some points about &amp;quot;indirections&amp;quot; and &amp;quot;buffer caches&amp;quot;, neither of which is relevant to the applicability of lock-free techniques or atomic instructions.&lt;p&gt;(Quick proof of both of the above: &lt;i&gt;all&lt;/i&gt; virtual memory systems abstract both (a) disk access (swap) and (b) pointer indirection (TLBs), without restricting atomic operations.)&lt;p&gt;What &lt;i&gt;is&lt;/i&gt; true, is that atomic operations, and memory barriers for that matter, are not effective across coherency domains.  So, if you have two processes concurrently accessing a memory-mapped block device &lt;i&gt;on different computers&lt;/i&gt;, then yes, you lose atomic operations and lock-free data structures.  But, AFAIK, this is not how most databases operate: most run as one or several processes &lt;i&gt;within the same coherency domain&lt;/i&gt; (e.g. the same computer).  The operating system ensures that multiple processes mapping the same file maintain a coherent view of it (primarily by mapping each disk block to at most one physical memory location).</text></comment>
<story><title>Re: MemSQL the “world&apos;s fastest database”? (2012)</title><url>http://www.postgresql.org/message-id/4FE8A2DE.5050806@agliodbs.com</url></story><parent_chain><item><author>ankrgyl</author><text>This post is missing a key trade-off that you get with an in-memory row store (tl;dr ability to build lock-free), but also since 6&amp;#x2F;25&amp;#x2F;2012 several other innovations in MemSQL that have shipped over the past 2+ years.&lt;p&gt;The main advantage of having a main-memory row store is the ability to leverage random-access data in memory. This unlocks you to use random access aggressively and build lock-free indexes (atomic instructions are available only for data that&amp;#x27;s in memory). With these tradeoffs you can build extremely concurrent data structures that simply aren&amp;#x27;t possible with a system that must manage a buffer cache. See &lt;a href=&quot;http://blog.memsql.com/the-story-behind-memsqls-skiplist-indexes/&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;blog.memsql.com&amp;#x2F;the-story-behind-memsqls-skiplist-ind...&lt;/a&gt; for more details.&lt;p&gt;The article also suggests that MemSQL is not durable in the event of a power outage. This is simply not the case - we&amp;#x27;ve had durability since the first ever public release of MemSQL, including both asynchronous and synchronous modes.&lt;p&gt;MemSQL now also has distributed transactions and MPP, supports intra and inter data-center replication, semi-structured (JSON) data, and even has a column store that works as a hybrid between disk and in-memory.&lt;p&gt;And BTW, I actually agree with this mentality in the context of column stores. There&amp;#x27;s no real advantage to having an &amp;quot;in-memory&amp;quot; column store because of the append-only nature of columnar blobs of data. That&amp;#x27;s why ours is a hybrid.</text></item></parent_chain><comment><author>dschiptsov</author><text>How exactly you have solved the problem of &amp;quot;durabity&amp;quot; given that any disk is orders of magnitude longer to access than RAM? The Mongo way - push buffers to be written to an OS kernel and forgot?)&lt;p&gt;Let&amp;#x27;s put it straight. Words should mean what they originally meant, not how sales or PR guys want them to be meant.&lt;p&gt;For us, a database is piece of software which enforces data consistency and is fail-safe. That&amp;#x27;s why we want it - to reliable store our data.&lt;p&gt;According the laws of physics, the throughput is bounded by the slowest operation (the same way a durability of a chain is bounded by a weakest cell) and in case of a database, this operation is an atomic update in a persistent, failsafe storage with consistency guarantee. That is what we call a transaction. This was the meaning since System R or the first version of Berkeley DB.&lt;p&gt;Transaction &amp;quot;in memory&amp;quot; is a nonsense for us, so are any memory-to-network-stack or memory-to-client-process &amp;quot;benchmarks&amp;quot;.&lt;p&gt;As long as there is no proven (not advertised) consistency guarantee and fail safety guarantee such system should be called a Cache, not a Database.&lt;p&gt;Call your product Cache, and all the confusion would disappear. In other words, &amp;quot;in-memory&amp;quot; 
 is not a database, it contradicts with the meaning of the word, no mater how your sales people are trying to twist the language.&lt;p&gt;There is no &amp;quot;specialized databases&amp;quot;. There are specialized caches. OK.</text></comment>