IMDB showdown - comparing OrigoDB, Redis and SQL Server Hekaton
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Friberg, Robert
Moving data between RAM and disk is expensive both in terms of runtime performance but also developer productivity. A relational model, stored procedures, data access code and O/R mapping all add complexity and require significant time and effort to maintain.A solution based on a pure in-memory technology eliminates all or some of this extra work reducing both cost and time to market. Using a hybrid, such as Sql Server Hekaton, you can benefit from in-memory performance without sacrificing existing investments in existing RDBMS based architecture. In this session, Robert introduces OrigoDB, Redis and Sql Server Hekaton, demonstrates an application based on each of them and discusses the pros and cons of each approach.
 Yeah. Yeah, welcome. Really happy to be here. I'm Robert. I'm from Deverex Labs. It's a small company in Sweden. And I'm a developer. I've been developing since about 20 years, mainly on.NET and a lot of SQL Server and a lot of other things. But at Deverex, we're building a OrigoDB. It's an in-memory database. And we're trying to make a living from it, not really there yet. So we support ourselves doing consulting and training in Sweden, mainly. So today, I'm going to show you a little bit about OrigoDB. I'm going to show you a little bit about Redis and a little bit about SQL Servers in-memory OLTP function. One hour is not a lot of time. So it's going to be pretty brief, except we'll spend a lot of time with Origo, of course. Yeah. OK, so let's get started. So what's this in-memory thing all about? Why is in-memory so special? Why is there so much buzz about in-memory? Well, it's speed, obviously, right? In-memory is really fast. Here's a little table with some numbers. And in the middle, you can see RAM there. We measure the time, access time to RAM, about in nanoseconds, right? You buy a module of RAM. The access time is about 60 nanoseconds. And if we compare that to a rotating disk, then the access time is about 10 milliseconds. That's about right. You probably all have SSDs, but we can use a rotational disk as a reference. So 10 milliseconds. That sounds pretty fast, but if you compare that on a scale, that's 167,000 times slower than reading from memory. So that's kind of a head start right there. So you don't really have to be, you don't have to really do perfect stuff in memory. If you just stick your data in memory, then you can cheat with everything. You don't need good data structures or algorithms because you have a head start at that speed. So yeah, so speed is the primary driver. And just to get some perspective, the last column out there, it says one second for RAM. So one second, that's a pretty short period of time. If I'm reading some value for somewhere and it takes me one second to get the reply. One second is one second. But if we compare that to rotating this, that's 46 hours. I hope I did the math right. I did it on an appkin during Geekbeers yesterday. You might have to check it. All right. But all right, what more? You have the distance. If we have a distance of 240 meters, it's just down the street, buy a beer and go back, 240 meters. That's one trip around the earth, 40,000 kilometers. So at the same time that I can go to the store and buy a beer and go back, I'd have to go around the world, 40,000 kilometers. So memory is really fast. Memory has a different property because it's a random access. You can read at any position in memory. And you pay the same price, about 60 in nanoseconds. And when you access disk, you need to organize your data. So you read it sequentially. Because if you try to do random accesses, a transaction in a relational database, it might do 20 or 30 page modifications. And if we had to read and write all those pages, we could have like 100 reads and writes of a page. And if those were random, then each transaction would take like a second. Then you can do a maximum of one transaction per second. But they're not really like that. We're going to look at the architecture a bit, the current architecture of the relational database systems, SQL Server in particular. And we'll look at that soon. Here's another factor. So OK, so memory is fast, right? That's great. But if I only can fit a tiny bit of my data in RAM, then I still have a problem, right? I still have to keep it on disk. And that's necessary. But if you look at the prices, how they've developed the last 30 years or more, one gigabyte of RAM cost $6.5 million. That's a lot of RAM around the 80s. And if you look at the price today, it's about $8 US for a gigabyte of RAM. And at the same time, the top capacity in servers for RAM is also grown. I remember my first PC, it was a 386S6. You remember those? And I got it, and I thought, well, I better double the amount of RAM, so I got two megabytes of RAM in that machine. And that was awful out at that time. But yeah, this one's got 16 gigs in it, this laptop. And on Azure, the largest VM you can purchase on Azure has 112 gigs of data. And on Amazon, it's twice as big, 244. They are expensive, but you can get the servers with RAM. And you can buy commodity servers from Dell with about two terabytes of RAM. And that's a lot of data, right? How big is your relational database? How big is your OLTP database with your working set of data? Well, if I look back at my experience, the projects I've worked with, health care and that kind of thing in Sweden, a big database is always less than a terabyte. Around 100 gigs or 200 gigs. But then again, there's a lot of extra data in those gigs because there's a lot of index and there's a lot of overhead with the data pages. If I represent that data in memory with an optimal structure for memory, then it will just shrink away. And we don't need all the indexes because we can just scan and seek the data. So we don't need indexes for every type of query. So your data probably fits in RAM. And if it does, it could be a good idea to put it in memory and not just for speed. All right, we'll get to that. That's not really why we built Origo. We didn't build it for speed because most of the time, a relational database, unless you really mess things up, is fast enough. If you know what you're doing, you can make it work well enough for what you need to do. So we kind of built it to save time. OK, there are a lot of in-memory stores out there. A lot of products claiming they're in memory, kind of riding on the buzz and trying to get a market share just by claiming they're in memory. Here are just some of the products. I erased some of them. I had a list about 40 or 50, but just to show you some of them, you've probably seen them. There's one group of in-memory databases called New Sequel. You've probably heard of that, the New Sequel. I'm going to show you the old sequel today with the relational architecture, the B-tree structure. We're going to look at that soon. But some of these new vendors, they're putting all the data in memory, and they redesigned the relational architecture from scratch. And VoltDB is one of them. And another one is MemSequel. And there are probably others as well, if you've heard of New Sequel. And that's pretty smart because they're utilizing the existing knowledge that we have. We all know how to use a relational database. We all know how to type SQL and so on. So they're giving us something that we can get working with quickly, a model we're familiar with. And then we have all these key value cache thingies. And they're not really databases because it's just a key value store. You push in a piece of data and you associate it with a key. They're not a database anymore than a dictionary is, in.NET. Perhaps. You heard of these, Aerospy, KSLcast? They're used mainly for caching, kind of like distributed caches. Does anyone use app fabric? The cache thing in app fabric? Maybe? All right. Well, I think they're going out of business, app fabric. They're getting rid of it, right? I heard that the other day. So you'll probably be choosing something like this, Hazelcast, Aerospy, or Redis in Azure. We're going to be talking about Redis as well. Well, those down there to the left are key value stores. And then on the right, we see these hybrid solutions, the existing, the dinosaurs, the ones that are taking all your money. And they want to keep taking your money. So they have to offer some kind of in-memory thing before they go and die. And hopefully Oracle will soon. Yeah, and we'll be looking specifically at SQL servers in-memory OLTP. It's well a bit. We'll be looking at it. I call it a hybrid because you have all the same behavior, all the regular stuff is there. They just added new features that can take advantage of the in-memory features. SAP, when I see that in the project, I run. But they have some cool in-memory stuff. Have you heard of it? Saupana, the Germans. OK. This is pretty impressive stuff they built there. And then there's Origo. I don't really know where it is, where it fits on this map. It's not new SQL. It's not hybrid. It's not key value store unless you want to. But we'll have a look at that later. OK. So let's start by having a look at SQL server. I'm sure you're all familiar with it. Use it. Or any other relational database. Most of the things I'm talking about here apply to most relational databases with traditional architecture. Well, SQL is old. It's from the 70s. So what else from the 70s do you use? Languages or libraries and stuff? Nothing, right? OK. And the design for the B3, the underlying architecture, the data structure behind relational databases, was conceived during the 70s when we had tiny amounts of RAM and pretty small amounts of disk as well. But our data had to be on disk. And before that, we read from punch cards or tapes like streaming media. And then when disks came, we had to utilize them in some way. Just make things go reasonably fast. Just reading and writing data. It was slow at that time. So the relational database was kind of a solution to those constraints that were really extreme. But none of that is really true today, like I showed you a few slides ago. We have a lot of memory, and it's cheap. So it's kind of old. It's impressive, but it's obsolete. All right. So SQL Server, the team, they've added some in-memory features to just stay alive. And let's have a look at how B3s work. OK, this is a technical part of the talk. So B3, it's a logical tree. You see the tree on the left? It's a logical structure in all of the squares there, the green and yellow ones, or whatever color they are, are data pages. And each data page maps to a block on disk with the size of eight sheila bytes. So it's a logical structure. And on disk, it lives in the data files. We can have one more data files in SQL Server, right? So on the right, it says data, 64k blocks. Those are called extents. So when data on disk is consecutive blocks of data pages. And in between, we have a buffer manager. And the buffer manager is the component that writes the dirty pages to disk, and it reads the pages into memory, into the buffer pool. That's on the left. The tree is in the buffer pool. Some of it. There are pages. The buffer pool is like a cache. So if you have a lot of memory in for SQL Server, all the memory on the server, it will take as much memory as possible, and it will use it mainly for buffer pool. So it'll keep the data pages in memory. And that's a good thing. If you have a lot of data pages in memory, you don't have to move them back and forth between memory and disk. And that's the buffer manager's job. That's a component of SQL Server. So the buffer manager takes dirty pages from the tree and writes them during checkpoints. And that can happen every second or every five seconds or sooner. It's pretty complicated. But between checkpoints, you have dirty pages in the tree. And if the computer crashes, the server crashes, then you lose those pages. And that's why you have a transaction log to keep the durability. So if you crash, you save everything in the transaction log. So every time a transaction commits, all the information from the transaction gets written to the log. That's a transaction log. And SQL Server uses effect logging or binary logging, because it logs the data pages. We're going to look at other types of logging a bit later. Origo uses a different type of logging. VoltDB uses a different type of logging. Redis uses logging, also a different type. It uses logical logging. It remembers the information that caused the change. All right? But this is effect logging. We don't remember what the insert, update, delete things were. We remember which pages were new, which pages were modified, and which pages were deleted. And all those get written to the log. We have to write deleted and modified pages to the log because, or the original pages before they were modified, so we can roll back. If something goes wrong, you have to roll back. And then we read the pages back from the log and replace them in the tree. Got it? So that's logging. So for every single transaction in SQL Server, it has to flush one or more pages to disk. And hopefully, the heads on the spinning disk won't be doing other things. You have a dedicated disk for your log if you have spinning disks. Then the head will remain in the same position, and it can write pretty fast. So it won't be doing random access. It will be appending to the log all the time. But if you put your log and your data files on the same disk, then the disk will have to jump around. OK? So the log kind of defines how fast, how many transactions, what kind of performance you can get from SQL Server in write transactions. All right. Got it? So that's a B3. That's the old structure. And it's designed in the 70s or 80s. So that's pretty old. The implementation was designed in the 67. Yeah, but nobody does that anyway, right? That's not what we're doing. It says Al and K, right? No? Yeah. But this is kind of an implementation or architecture, but not really an idea. But if you take this idea, you can build it specifically for in-memory, then you get totally different performance and other properties. And we'll be looking at that soon. OK, so let's talk a little bit about SQL Server's OLTP. The previous slide was the traditional architecture. This is kind of like, OK, so what do they do about it? How are they going to speed things up? OK, so what do they do about it? How are they going to speed things up? Yeah, so here's a pyramid and at the bottom, that's the most, the factor with the largest impact, the Buffer Manager I.O. Right? Writing and reading to disk, Buffer Manager, at random places in the data file, that consumes a lot of time. And so on the right, we see what they did. So they kind of, oops, I'm pressing too much there. OK? Yeah, so they added features. They redesigned the B3 and just created an entirely different, a new representation of data in memory. So they keep the entire table. You can choose to create one table and make it in memory. It's on a table-based basis. So it's not the entire database or portions of it. You take one table at a time and you migrate it to in-memory. So the table is either in-memory or it's not. And the in-memory architecture is totally new. So it's redesigned, architecture optimized to work in memory. OK? So the B3 we saw on the previous page, it's gone. Right? And then we have locks. Locks are during a transaction because relational databases are so slow, transactions need to be concurrent. Because if you have to wait for the previous transactions, things, it will be chaos, right? So locks are used by the transactions to remember what kind of data, I think I have a timer on that one. OK? The locks are used to remember which data rows and data pages have been modified by transactions. So to isolate transactions so they don't step on each other's data. More on isolation later. So what did they do about that? They added a multi-version concurrency control. It's kind of like snapshot isolation. But it means that if one process, if one transaction needs to read the data, it will get a copy of it. So they don't have to block on the same data row. So you just copy the row and then both use it. And then at the end of the transaction, it's kind of optimistic. If there's no conflict, if nobody changed the two same rows, there's no conflict, then both transactions can commit. So they've eliminated locks from in-memory old TP. And that's a significant amount of time for processing the locks. Laches are for the data structures themselves to protect the B trees and other structures. And they've redesigned them as well to using lock-free data structures. All right. And logging, we saw logging. We write stuff to the log, effect logging, all the data pages that were modified and so on, written to the log, but not in-memory old TP. We just log the actual data rows, right? The data row that is being written. That's the only thing that goes in the log. And it's very small. So it's not the entire data page. It's just a tiny bit. And then the log, it doesn't use checkpointing because the data is only in memory and there's a background process that reads the log and updates the data. And the data isn't regular data files. It uses a file stream thing. So this is kind of technical. If you want to use, I'll back it up. If you want to use SQL Server in-memory old TP, you don't have to know all these things. You just start the wizard and you say, OK, next, next, next, and then you're finished to use it. It's good to understand things and why it's fast. But the main benefits come from the bottom of the pyramid and then they, yeah, diminishing returns the higher you get up the pyramid. And at the top we have a store procedure execution and query planning and that type of thing. Today you can create, in-memory old TP, you can create native compiled store procedures. So they're compiled to C. They're trans-compiled to C when you create them. And then the C compiles, we have like native code. Your procedures are native code working on these in-memory structures. So everything's pretty fast. You guys heard of Oracle x10. They bought some startup and x10, they're boasting because their goal was to make things 10 times faster. And SQL Server, the code name for SQL Server in memory old TP was Hecaton. You remember that? So Hecaton is Greek for 100. So that was the goal to make everything 100 times faster. But it's kind of silly. You can't just make everything faster. It's difficult to measure. You have to compare different workloads and it's pretty complicated, but you can probably get not really near that. So it's really transparent for the application. That's one of the strengths. You can use SQL Server Enterprise Edition. Yeah, lots of money. Okay. Then you can just convert your tables to in-memory. And then you'll benefit from the speed. And it's entirely transferred to the applications. But not really because there are limitations. You can't have foreign keys. You can't have certain types of indexes. You can't have certain types of columns are not supported. And that can be a bit disappointing because you have to do a bit of work to really get there. And then you enterprise license and some performance there. All right. Let me show you. There. SQL Server somewhere. I have a 2014 developer edition here. And Adventureworks, you've probably seen it before. So on the context menu, there's a memory optimization advisor. And I'm just picking a table at random here in Adventureworks. And it runs a checklist. It shows me all the reasons I can't convert this to in-memory. So it's a checklist. And there's information. There's a link to the help. But user defined data types, not supported. Foreign key, not supported. Unsupported indexes defined on this table. Default constraint, not supported. OK. Not supported. Let's try another one. Data's log. Oh, XML type, not supported. Large object binary, not supported. OK. Try another one. OK. Almost. Foreign keys. Yeah. There are a lot of limitations. So you have to work with that. I didn't manage to find a single table in Adventureworks that could pass the test. All right. So I did this one. Fairly simple table, right? Just to show you how it works. We don't have a lot of time, so we're not going into the details. But I think that table's right there. OK. And we'll run that one. And it passes all the tests. Amazing. All right. And some warnings. We have some procedures that access that table. So we could probably convert those also. We don't have to. But we can compile them, pre-compile them. All right. So I... Make some choices here. We can choose between a hash index. It uses a specialized hash structure. It's like a dictionary or a hash table in memory. And it's really, really fast for lookups. Or we can use a traditional tree-based index. Those are the two we can choose from. We can also create secondary indexes later. But those are never saved to disk. They're always recreated on startup. All right. So I won't do that. I'll just script it so you can see what happens. It kind of looks like this. You create a table. And you say with memory optimized on. Endurability, that schema and data. If we say just schema, then the table would be empty on startup. If we want to do some kind of caching or something. Yeah. Okay. Enough of that. And back to the slides. Okay. Okay. So let's talk about Redis. Redis is a key value store. But the values are very special. It's actually an abbreviation. Remote dictionary server. So that's the abbreviation Redis. So it's a dictionary server. So you have keys and values. You know dictionaries, keys, values. But the values are very special. You can put either strings, but you can also put. Let's have a look at that. You can put the values can be strings. It can also be lists. So it's kind of like a data structure server. So the one key can point at a set or a hash or a sorted set. So it's a limited set of collections you can use. And then there are a lot of predefined commands. And you can talk to it with Telnet. You just start up. Yeah. You just talk over TCP and you send text and you get text back. So that's the interface. And there are a lot of, a lot of drivers for a lot of languages. Redis is extremely popular used by Twitter and Facebook and all those big stuff for just keeping data and cache. Caching data, keeping it near applications. For speed. And Redis also has persistence. So you can do, there's an append only file. So every time you change something, you send a command, it will write the command to the append only file. So when you start it up again, it can reread the commands in the file and rebuild the data in memory. But most people use it. They start an up and empty node and then just start using it like a cache from the application. Okay. Written by a guy in Italy. Highly optimized. See a lot of fast algorithms. And every single command has a documentation on how it performs. So it's a great performance written in C. No garbage collection installs or that type of stuff and widespread open source. Okay. So if we do, we do something crazy like build Twitter using Redis only. What would we do? It could look like this. So here's a command increment. It means increment the key with the name next user ID. And if it doesn't exist, it will create one. So it, and it returns the incremented value. And it's an atomic operation. So it's kind of like a key with a number behind it, but we're incrementing it or creating it. So we're adding a user to Twitter. So we need the user ID, a unique user ID. So that's what we do first. HM set means where H means a hash and M means multiple. So we're creating a hash and we're setting values for it. And the key is user 1000. That's the user ID we just created. So the application sends that ID back, right? And then we're creating name bar password hash. And you could fairly easily map that to a JSON object, right? Key value, key value. So the key points at a hash and the hash in itself has keys and values. All right. Hash, we're setting the key users is a hash and we're setting the key bar to 1000 in that one. This kind of looks like an assembly language, right? It's pretty cool. Yeah. But I wouldn't want to build a big, large application where people's lives depend on this, right? Because your application has to translate this, right? Create these and send to Redis. Of course, the library will do that, but this isn't really a data model, right? Yeah. So the title of the talk was IMDB in memory databases. Redis is really a database, is it? It's kind of like a key value store on steroids. Okay. Yeah. I don't think I'm going to talk you through each command here, but you get the feel of it. If you go to redis.io, the homepage, they have a complete example built with PHP with all this thing. It's a Twitter clone and you can download it and you can play with it. But this is what the application does. It sends these commands and gets stuff back. So it's kind of like an advanced key value store. All right. So that was Redis. Very short. Okay. Let's talk about OrigoDB. I'm checking my time here. Okay. So here's our slogan. One of them, build faster systems faster. And that's not really a good slogan because it doesn't really say what we built it for. It's kind of catchy because it uses the same word twice, faster and faster. Okay. So you can build fast systems or faster systems and you can build them faster. And that's what the emphasis should be. Okay. Building stuff faster. I'm extremely impatient. I don't like wasting time. I don't like dealing with object relational mappers or relational database. Well, I don't have to. I just want my objects and I want to perform some operations on them. And then I have to write, do a bunch of modeling on the database and I have to pull the database into memory. I have to modify it and I have to keep track of concurrency and then push the data back. It feels like there should be some simpler way, right? And that's kind of the idea behind or egoDB to make it easier. Build faster systems faster. Yeah. And they are faster, but that's kind of for free. So I didn't work really hard to make this thousands of times faster than SQL Server. The engineers on the SQL Server team, they're like, yeah, every single one of them is 10 times smarter than me. But this is just, it's so simple, it's so easy. Okay. So what's the problem? I kind of mentioned it. It's a thing. Relational databases, they were kind of meant to do domain modeling when they were invented, right? The SQL was intended as an ubiquitous language, right? End users were supposed to type insert and delete and select and stuff. A select statement, that's pretty close to plain English. That was kind of the intent during the design. So why do that twice? Once with the old style, designed from the 70s and all the work that comes with that. That's the thing all the way to the right. And then we have all this mapping because we really don't want to deal with the database. So we use tools in between that kind of hide the database and those tools mess things up even more. Performance, right? The mapping. All right. And databases, they're so broken, they're so slow, so we need a cache. The kind of point of a database is I can put my information there, I can do some transactions and give me some information so I can show it to the user. But they're so slow. So we need a cache. Isn't that stupid? Yeah, so if the database was fast enough, we wouldn't need a cache. And that's kind of like the performance and what happens when you go pure in memory. And this is kind of the idea we use our philosophy. So it's not really the performance, it's kind of like we don't have to do those three things and we save a lot of time. Okay. Yeah, so we stick all the objects in the domain layer and just keep on there. And then we run the transactions in memory and then we log the commands. We write them to a file like a log. And then when the system starts, you reread the log and then you've rebuilt the system. And the only catch is it has to fit in RAM. But every single database I've worked with the past 20 years, they easily fit in the RAM we have today. So it's not really a problem. Okay. So it's a pretty simple idea. We have, this is some systems or transaction theory. We have a state, an initial state, right? And then we apply an operation to the state and then we land in a new state. And each state is consistent, you know, asset transactions. We don't get stuck between the states. And, yeah, we just apply the operations and get new states. So that's really the idea. And if you keep the state in memory and just persist the operations, then you don't have to worry about how you map your data or represent it on disk. And then you can keep it round the maxes because it doesn't have to have a specific structure. So you can use collections, you can use lists and dictionaries and queues, all the dotnet collections and types. And there's no mapping to different types. So that's a really simple idea. But I think it's pretty powerful. It's so simple, it's deceptive. People don't understand the power because you like shiny tools with lots of buttons and stuff, right? Most developers do. Okay, this isn't really a new concept. System prevalence. There's a project called Prevaler if anyone's working with Java. It's a pretty old project from 2000, I think. So they're doing exactly the same thing as ORIODB is. Yeah. And the guy who started it, he kind of, he thought he solved all the problems of a system. He solved all the problems of databases year 2000. And he showed it to the world and no one understood. And now there's a small group, a couple of hundred or a couple of thousand just using this and just benefiting from it. Okay. MongoDB uses up logging. You know what that is? The operations you do in MongoDB, it logs those. And also uses the operations for replication. So it's pretty easy to set up a slave server and you just send the operations over there and wait for them to complete. And then you have a synchronous replica. All right. Redis uses a Pandio only file. If you read Martin Fowler's blog, he has a post there on memory image. If you read that carefully, you'll see it's the same pattern. It's the name of it. VoltDB uses logical logging. So they have a log and they log the operations and they keep all the data in memory. Okay. Acca, pretty cool right now. You guys know talks about Acca at NBC. That's strange. Acca.net, Actors Model. They have persistence for the actors and it's the same principle. So you log all the operations or the events, the messages that have received the actor. And then you can rebuild the state of the actor later. Event sourcing, of course, you've all heard it. Same principle, right? Of those of you familiar with event sourcing, CQRS, but mostly event sourcing. It's the same thing. You save the events and not the data. Okay. So let's look at how it works. Let's look at how it's built. This is also pretty simple. The blue things here are the components or the parts of the framework that you interact with. And the yellow things, if that's the color, are the things that you create. Right? So you have application code and it talks to the engine. If it's in the same process, you can run ORIGODB in the same process. You can host it directly in your IIS web application. So it runs in the same process. You should turn off recycling for the IIS if you do that. Because then you have to reload from memory every recycle. Okay. So your application speaks with either the engine or a server. And the server, it's a standalone ORIGODB server with a lot of features like replication. I mentioned MongoDB replication. ORIGODB server has it as well. So you can do single, master, multiple-slave replication. You can do ad hoc queries, the web UI on the server. I'll show it to you if we don't run out of time. Okay. And we have the engine. So the server embeds an engine. And the engine has a storage module. And it's pluggable. So default, it writes the file. So if you just start up ORIGODB, it will, in the current directory, it will just start writing the events to files in the current directory. You can also write snapshots. But you can use SQL as a backing store as well. So you can put the events or the commands, write them to SQL. Or you can use event store or you can write your own. Okay. You can choose serialization. Default is binary format. You probably shouldn't use that in production. I would go for protobuf because it's flexible and fast and small. Pretty optimal. Interoperable. You can use JSON. That's pretty smart. So you can pass commands and queries from your application to the server in JSON. And that opens up for HTTP and REST APIs and stuff. And communicating with, from other platforms. Okay. So the kernel is the thing that guards the model. The model, that's the most important thing. That's your data and that's in memory. So you have an instance of the model and it lives in memory. And you don't have access to it. You don't have a reference to it. All communication goes through the engine. And what do you do? You do the two things you can do with it. You can ask it something or you can tell it something. And asking something, that's a query. So you create a query and that's just an object that you've written yourself, probably using link or something else. And you pass the query to the engine. The engine executes the query and takes care of concurrency and that kind of thing. And returns the results to your application code. If you pass a command, the command gets written to the journal. All right. It gets journal. And then it gets executed. So the kernel executes a single command at a time. So it's single threaded. So there's no concurrency issues between the commands. Because it's fast enough. I can run 20,000 transactions per second on this laptop. Commands. Sustained commands per second on my laptop. And then there's probably a room for queries as well. Because in between those commands, while the engine is waiting for them to flush the disk, we can run hundreds of thousands of queries, depending on their nature, of course. So it's really, really fast. But we didn't work hard to make it fast. We got that for free because it's in memory. Okay. So you write commands and queries in a model and then you just go. Okay. This is so important. I made a slide specific from it. When you have a complete history of events, everything that's happened, and something goes wrong at the client, I can grab their log, the command journal, and I can just load it up in the debugger. And I can go to the exact point where they wonder something or something failed. And I can start stepping through the code. Just replay the events. And that's extremely powerful. I can fix a bug. If I have a bug in the command execution code, so the thing in memory isn't really what it should be. Because there's a bug somewhere. I can fix that bug. And then you replay the events and you'll fix the bugs that are old, not just the new bugs. So if you screw up some data, you just re-project it. All right. That's pretty cool. And you can restore to a specific point in time. I want to back up to yesterday. You just tell it what time you want to start from. So that's kind of the benefits from event sourcing. Okay. And then you have an audit trail. You know who changed what and when. Every single command when it was executed and by whom. That's kind of, you get that for free. Okay. Want to say that? Okay. So let's look at some code. Here's the, an example model. Could look like this. Okay. So we write some code. We write a model and it derives from model. It's one of the framework types. And it should be serializable if it's supposed to support snapshots. If we try to take a snapshot with binary format and it's not serializable, the snapshot will fail. So that's why it's serializable. But you, you don't have to do snapshots. You can just stay with the journal. Okay. So here are three datasets, three sorted dictionaries. So the keys are grids and they're pointing to specific objects, classes, types, customer product. You see. And we're newing that up in the constructor. You see it's internal so we can manipulate these data structures from commands and queries. This is kind of anemic model. It's just a container for structures. You don't have to do it this way. You can put behavior here as well and you can test it. But this example is anemic. So the command, here's one add customer. And it drives from, you see command and we're passing the generic type commerce model and that was the model on the previous page. And it's immutable. So we have read only properties. They should be immutable. You shouldn't be mixing, changing these things after they've been logged. That's kind of like getting in trouble. All right. And then you have an execute method and it's the kernel that calls execute. You give the engine the command and the engine passes the command to the kernel and the kernel it calls execute and passing in the model. Running only a single command at a time. It's pretty simple. It's calling a board here so it's throwing an exception if there's a duplicate key and abort signals to the engine that, okay, something went wrong but we didn't touch the model. I promise we didn't screw things up. So just ignore, just continue. That's what abort means there. If we throw an exception that's unexpected, then we can tell the system, uh-oh, shut down because this is not good. This command failed and we don't know what happened to the model. Because some systems that's really important. If you work with the healthcare or you work with the medicine, how much medicine did that patient get? The last time they got medicine? We don't know. That's not a good answer, right? So some systems are really, really important. And if you need that consistency, then you can get that here. Okay. So queries, same principle. We derive from query, you pass in the model and then the return type. You see? Customer view, that's what we're returning. And the customer view is not a customer because if we return a customer and the customer has orders and each order has a reference to a product and each product has a reference to every order, you get the picture right. It's strongly related, highly connected. If we grab a customer, we'll pull that entire graph and return the entire graph back. That's not what you want. So you have to kind of break the graph and that's why we use views, though. So return customer view. Okay. Kind of a principle. You could use a lot of advanced link here as well. I'll show you that soon. Okay. So now we have a model, we have a command and we have queries. So then we start our engines. So we see engine.for as a convenience method for creating or loading an existing database. It'll look in the current directory. We can create a configuration object and pass to engine for pointing it at a remote server or pointing it at a specific directory or using a specific storage provider and so on. Yeah. So you see what's happening. It's pretty simple. We're creating a new add customer command and we're passing the command to engine, right? Engine execute. So that gets logged and the model gets transformed. And then we're asking a question also by passing a query to execute. And that's all there is to it. I build applications using this. And I've been doing that for the past 10 years. Okay. And it works great. Okay. So let's have a look. A little demo. I'm connected. Let's do that. There. Okay. This is an example. We built it just for fun just to show how fast things are. So there's no cache here. This is a, it's a kind of search engine. It's like Google, but really, really small. So I have a little process that goes fetches, a lot of blog articles, and just pushes them into an in-memory model. So when we're searching, if I do a search here. Oh, that was great. Something better. I think the JavaScript is wedged. Oh, server not found. I'm not connected. Good. That kind of explains it, right? Good. There. It's better. Okay. So 48 results in the memory database, and it took 0.07 seconds. And there's no caching. So it did a round trip to the server and searched all the data. And there's not a terrible amount of data. There are 8,153,000 connections between links from keywords to the articles that contain the keywords. And that's what we're getting in the list. You can check this out if you like. It's on GitHub. You see the address. Just Google for Geekstream. And it's built with OrigoDB and OrigoDB server. The source here is on GitHub. And it's useful as well. Okay. So let's show you a bit of code. Complicated, right? So the entire model is 216 lines of code. That's the model. And then there are some commands over here. So we have a domain with one, two, three entities, feeds and feed items. And then we have the model, the Geekstream model. And then there are query objects over here. Get feed by ID, get feeds query and so on, search query. Look at the search query. Yeah. So it's a pretty simple link extension methods. Get some items. We're searching the model over here. So the search logic is a part of the model. And then we're just skipping and taking and transforming the result in the query. Pretty simple stuff. Great. And here's the core. It's a sort of dictionary of string. So the string keys are the words that we're searching for. And they point at sorted sets of feed items. The feed items that contain that word. Okay. That was Geekstream. Okay. I'm not going to show you this one, but I actually built a Redis model because I don't like that. Redis kind of feels like assembler. So I wrote a Redis clone using origoDB. So it's a Redis model. And here's a dictionary of strings to objects. And those are just using regular types. So I implemented every single basic command for Redis using origoDB. And it took an evening or two. Pretty fun. It's not as fast. Of course, Redis is a great piece of work. But I kind of did this for fun. But you can use this and you can do it embedded. You can't run Redis embedded. You have to run it as a remote server. But now you can have the same model if you want it embedded. Okay. So that was that. Extend. Great. The demo and we've got nine minutes left. Okay. Great. So conclusion. Yeah. It's not a lot of time to look at three products. But I hope you got a little hint at what they all can do. Really different. The only thing they have in common is it's in memory. We're keeping data in memory. That's kind of the commonality here. So, yeah, you like that text, the older, the new and the ugly. Can you guess which one? Which one is which one? Yeah. Well, and I was really being nice because I thought you could say, I was thinking about saying the disease and the cure. All right. So the sequel, it's great, but it's old. It's not fast enough for us. So we add a cache. And so the relational database and the cache are kind of like partners in crime. Yeah. So I say use the OrigoDB when you can, but I'm crazy. So I use it for everything. I want to try it out. But it fits really well when your data fits in RAM, if you don't have extreme amounts of RAM. It fits really well when you have performance problems with SQL. But if you have an existing SQL, it's pretty hard to integrate. So you kind of have to go all in. It's good for green field development, I would say. But you can integrate because you can write the log to a SQL server. So if someone says, oh, everything has to be SQL server, OK, fine. So I'll create one table with events in SQL server. And then everyone's happy because the operations they have SQL and they can back it up. And that's a strength. OK. So you can combine them. So use Origo when you can. Use SQL server when you have to. Use Redis when you have to. Redis is really cool and fast, but I wouldn't use it to build applications. All right. Caching. It's good for caching. One thing about Redis is you can't really do queries across all the values or keys. You operate on one key at a time. So that's a limitation. You can't do analysis. You can do analysis with the SQL server. You can do analysis with Origo as well. All right. SQL is great if you have a lot of memory, a lot of data, more than fits in RAM, right? So Origo is when you run out of RAM, we have to pay for a lot of RAM. If you're not really benefiting from the performance, maybe it's not really a good fit. Yeah. And SQL server. OK, I'll skip that slide. There. I have a demo that shows that SQL server is not really atomic unless you really, really know what you're doing. If you're interested, you can ask me later. And it's not consistent, and it's not isolated either if you're into Asset and all those kinds of things. You know, it's read committed, isolation level by default, and there's a lot of anomalies you can get with that. OK. We can talk about that later. So, yeah. So licensing. SQL costs money. It's enterprise license. Redis is free. Origo is free. But we're trying to sell some enterprise stuff, some replication. But I think we're going to change that, change the model. So we're working on that. But the embedded engine is free at the moment. OK. So SQL, that's great writing, right? T-SQL. You like writing application logic with SQL? That's great. Testable. Great. No. So, Origo, C-sharp, link, nothing but. No mapping, nothing. Redis, you use the commands, but you can also use Lua scripting. It feels kind of glued on, but you can write transactions and do them with Lua script. I haven't really played with it. So OLTP, yes, for all of them. OLAP, analysis can't do with Redis. Origo, you can run it in process. If we were to rank performance, I'd say Redis is first. It would be silly to say claim some amount of transactions per second and stuff, but Redis is a great performance. And Origo is great performance as well, but not really near Redis. But we can do more with Origo. So, yeah, the modeling is pretty powerful. I think you understood that, right? You create a model with C-sharp, and then we do commands and queries, everything C-sharp. So the modeling is really flexible, really powerful. Relational is powerful as well, but it's kind of old. Okay, but the modeling capabilities of Redis, kind of limited and kind of fixed. But then again, SQL is extremely mature, right? Lots of connectivity, lots of operations, lots of knowledge. We know how to use it, lots of tools. Yeah, really powerful. It's hard to beat that, but I think it'll go away in 50 years, maybe. Okay, yeah, size. In-memory OLTP for SQL has a limit of 250 gigabytes per database. But then again, you don't have to put all the data in memory with SQL, right? So you can choose specific tables where you get the largest benefits. You should put those in memory and figure out what those are. So it's not really a limitation, but both Redis and Origo, this amount of available RAM on the server. But it has to stay in memory. So, little comparison. And great, I'm finished. And if you have any questions, just shoot. We have three minutes left. Any questions? What happens if the database really exhausts the memory? Exhaust the memory. Well, you have to check that. If you have a relational database, you heard the question, what happens if you run out of memory? Well, then you get an out-of-memory exception, and commands will fail. That's a simple answer. But what happens if you run out of disk? We've all done that before, right? The log is full, or the disk is full, and when we're doing SQL, you have to monitor it and just be aware of it. So you have to know when it happens, or you're not really doing your job. So it's not really a difference, I wouldn't say. Yeah? Could it be another language adapter or SDKs for other languages and C-sharp? Well, you can do JavaScript at the moment. At the server, you can run stuff. There's an API. If you run the server, there's a JSON. You can pass messages as JSON over HTTP. So in theory, you could write a client and then talk to the server of JSON and HTTP, and then the client could read any language? Exactly. And you can also write a client. If you use the protobuf formatter and describe the format, then you can write a binary client in any language as well, theoretically. Yeah? This is just basic request response. Okay? More questions? Yeah? Can you only have one or a server? Or can you scale them out in any kind of way? Well, that kind of depends on the model. It's a good question. Okay, you heard the question. Can we partition? Can we scale it, scale out? Well, if the model is partitionable, then yes. The Geekstream model we looked at, it's partitionable. We kind of designed it for that. So each feed has a bunch of feed items. And we can put those on. But we can partition those based on feed ID, because there's no relationships between feeds. And then we just have duplicate keywords. So we just parse the keywords that are on each server and let them point at the items on the server. And there's built into the client so you can send a query to multiple servers, and then you gather the results in parallel. So it's not map reduced, but it's something similar. And it's built into the client. But it's not automatically scalable. You have to design the model to work that way. Okay? Is it possible to use it for a graph structure with such ability to tracing and such operations? That's a really great question, because what is a graph? A graph is nodes and edges. And what is an object with references? It's a graph, right? So the model you saw here, it's a strongly typed graph. The other day, I created a generic graph model. It took a few hours. It's not complete. But it's just a model with a dictionary of nodes and a dictionary of edges. And it's finished. And then just some operations on that. So you can do a generic model. So you can do the type of stuff like you can do with Neo4j. Or you can do a specific model like we did here. And you can query it with strongly typing language. I found a project on GitHub where you can take link expressions. Expression trees, right? You know what that is? Okay? Expression trees and serialize them. So you can write queries for the graph model and then serialize them and pass them to the server. And I already experimented with it and it's working. Yeah? Great. Took a few hours. Next. If you want to persist it, what happens if the binary load starts to grow way too large? You can't really delete anything because you have to play back from the beginning, right? Yeah, good question. Okay, so what happens when the log grows? This will be the last question because it's 10-2. So we can continue down here later. Okay, so what happens when the log grows too big? Well, it takes a long time to load. Yeah, and depends kind of on the size and the serialization format. The binary format is pretty slow to serialize because it has to reattach all the references and it's kind of complex, right? But if you use a protobuf and you compress it, it's pretty fast. So you can load hundreds of gigabytes of log in like 10 minutes. So if you have two nodes, if you have a master and a slave and you're patching a server, then you just take down the slave, you patch the server, you bring it up, and then you sync it. Yeah, and then you take the other one down. So you always have one node up. At some point you have to scratch it. Yeah, I would, I try to avoid that. I save the data as far as possible, but you can take snapshots. Yeah, so when it loads, it loads the most recent snapshot and then the log from there. But I don't like to throw away data, so. Okay. Great. Thanks for coming. Thanks for having me. Good. Yeah. Yeah. And then some button on the way out, right? Okay. Thanks.
 Yeah.  Yeah, welcome.  Really happy to be here.  I'm Robert.  I'm from Deverex Labs.  It's a small company in Sweden.  And I'm a developer.  I've been developing since about 20 years, mainly on .NET and a lot of SQL Server and a lot of other things.  But at Deverex, we're building a OrigoDB.  It's an in-memory database.  And we're trying to make a living from it, not really  there yet.  So we support ourselves doing consulting and training in  Sweden, mainly.  So today, I'm going to show you a little bit about  OrigoDB.  I'm going to show you a little bit about Redis and a little  bit about SQL Servers in-memory OLTP function.  One hour is not a lot of time.  So it's going to be pretty brief, except we'll spend a  lot of time with Origo, of course.  Yeah.  OK, so let's get started.  So what's this in-memory thing all about?  Why is in-memory so special?  Why is there so much buzz about in-memory?  Well, it's speed, obviously, right?  In-memory is really fast.  Here's a little table with some numbers.  And in the middle, you can see RAM there.  We measure the time, access time to RAM, about in  nanoseconds, right?  You buy a module of RAM.  The access time is about 60 nanoseconds.  And if we compare that to a rotating disk, then the  access time is about 10 milliseconds.  That's about right.  You probably all have SSDs, but we can use a rotational  disk as a reference.  So 10 milliseconds.  That sounds pretty fast, but if you compare that on a scale,  that's 167,000 times slower than reading from memory.  So that's kind of a head start right there.  So you don't really have to be, you don't have to really do  perfect stuff in memory.  If you just stick your data in memory, then you can cheat  with everything.  You don't need good data structures or algorithms  because you have a head start at that speed.  So yeah, so speed is the primary driver.  And just to get some perspective, the last column  out there, it says one second for RAM.  So one second, that's a pretty short period of time.  If I'm reading some value for somewhere and it takes me one  second to get the reply.  One second is one second.  But if we compare that to rotating this, that's 46 hours.  I hope I did the math right.  I did it on an appkin during Geekbeers yesterday.  You might have to check it.  All right.  But all right, what more?  You have the distance.  If we have a distance of 240 meters, it's just down the  street, buy a beer and go back, 240 meters.  That's one trip around the earth, 40,000 kilometers.  So at the same time that I can go to the store and buy a beer  and go back, I'd have to go around the world, 40,000  kilometers.  So memory is really fast.  Memory has a different property because it's a  random access.  You can read at any position in memory.  And you pay the same price, about 60 in  nanoseconds.  And when you access disk, you need to organize your data.  So you read it sequentially.  Because if you try to do random accesses, a  transaction in a relational database, it might do 20 or  30 page modifications.  And if we had to read and write all those pages, we could  have like 100 reads and writes of a page.  And if those were random, then each transaction would take  like a second.  Then you can do a maximum of one transaction per second.  But they're not really like that.  We're going to look at the architecture a bit, the  current architecture of the relational database systems,  SQL Server in particular.  And we'll look at that soon.  Here's another factor.  So OK, so memory is fast, right?  That's great.  But if I only can fit a tiny bit of my data in RAM, then I  still have a problem, right?  I still have to keep it on disk.  And that's necessary.  But if you look at the prices, how they've developed the  last 30 years or more, one gigabyte of RAM cost $6.5  million.  That's a lot of RAM around the 80s.  And if you look at the price today, it's about $8 US  for a gigabyte of RAM.  And at the same time, the top capacity in servers for RAM  is also grown.  I remember my first PC, it was a 386S6.  You remember those?  And I got it, and I thought, well, I better double the  amount of RAM, so I got two megabytes of RAM in that  machine.  And that was awful out at that time.  But yeah, this one's got 16 gigs in it, this laptop.  And on Azure, the largest VM you can purchase on Azure has  112 gigs of data.  And on Amazon, it's twice as big, 244.  They are expensive, but you can get the  servers with RAM.  And you can buy commodity servers from Dell with about  two terabytes of RAM.  And that's a lot of data, right?  How big is your relational database?  How big is your OLTP database with your  working set of data?  Well, if I look back at my experience, the projects I've  worked with, health care and that kind of thing in Sweden,  a big database is always less than a terabyte.  Around 100 gigs or 200 gigs.  But then again, there's a lot of extra data in those gigs  because there's a lot of index and there's a lot of  overhead with the data pages.  If I represent that data in memory with an optimal  structure for memory, then it will just shrink away.  And we don't need all the indexes because we can just  scan and seek the data.  So we don't need indexes for every type of query.  So your data probably fits in RAM.  And if it does, it could be a good idea to put it in memory  and not just for speed.  All right, we'll get to that.  That's not really why we built Origo.  We didn't build it for speed because most of the time, a  relational database, unless you really mess things up, is  fast enough.  If you know what you're doing, you can make it work well  enough for what you need to do.  So we kind of built it to save time.  OK, there are a lot of in-memory stores out there.  A lot of products claiming they're in memory, kind of  riding on the buzz and trying to get a market share just by  claiming they're in memory.  Here are just some of the products.  I erased some of them.  I had a list about 40 or 50, but just to show you some of  them, you've probably seen them.  There's one group of in-memory databases called New Sequel.  You've probably heard of that, the New Sequel.  I'm going to show you the old sequel today with the relational  architecture, the B-tree structure.  We're going to look at that soon.  But some of these new vendors, they're putting all the data  in memory, and they redesigned the relational  architecture from scratch.  And VoltDB is one of them.  And another one is MemSequel.  And there are probably others as well, if you've heard of  New Sequel.  And that's pretty smart because they're utilizing the  existing knowledge that we have.  We all know how to use a relational database.  We all know how to type SQL and so on.  So they're giving us something that we can get working with  quickly, a model we're familiar with.  And then we have all these key value cache thingies.  And they're not really databases because it's just a key  value store.  You push in a piece of data and you associate it with a key.  They're not a database anymore than a dictionary is, in .NET.  Perhaps.  You heard of these, Aerospy, KSLcast?  They're used mainly for caching, kind of like  distributed caches.  Does anyone use app fabric?  The cache thing in app fabric?  Maybe?  All right.  Well, I think they're going out of business, app fabric.  They're getting rid of it, right?  I heard that the other day.  So you'll probably be choosing something like this,  Hazelcast, Aerospy, or Redis in Azure.  We're going to be talking about Redis as well.  Well, those down there to the left are key value stores.  And then on the right, we see these hybrid solutions, the  existing, the dinosaurs, the ones that are taking all your  money.  And they want to keep taking your money.  So they have to offer some kind of in-memory thing before  they go and die.  And hopefully Oracle will soon.  Yeah, and we'll be looking specifically at SQL servers  in-memory OLTP.  It's well a bit.  We'll be looking at it.  I call it a hybrid because you have all the same behavior,  all the regular stuff is there.  They just added new features that can take advantage of the  in-memory features.  SAP, when I see that in the project, I run.  But they have some cool in-memory stuff.  Have you heard of it?  Saupana, the Germans.  OK.  This is pretty impressive stuff they built there.  And then there's Origo.  I don't really know where it is, where it fits on this map.  It's not new SQL.  It's not hybrid.  It's not key value store unless you want to.  But we'll have a look at that later.  OK.  So let's start by having a look at SQL server.  I'm sure you're all familiar with it.  Use it.  Or any other relational database.  Most of the things I'm talking about here apply to most  relational databases with traditional architecture.  Well, SQL is old.  It's from the 70s.  So what else from the 70s do you use?  Languages or libraries and stuff?  Nothing, right?  OK.  And the design for the B3, the underlying architecture,  the data structure behind relational databases,  was conceived during the 70s when we had tiny amounts of  RAM and pretty small amounts of disk as well.  But our data had to be on disk.  And before that, we read from punch cards or tapes like  streaming media.  And then when disks came, we had to utilize them in some way.  Just make things go reasonably fast.  Just reading and writing data.  It was slow at that time.  So the relational database was kind of a solution to those  constraints that were really extreme.  But none of that is really true today, like I showed you a  few slides ago.  We have a lot of memory, and it's cheap.  So it's kind of old.  It's impressive, but it's obsolete.  All right.  So SQL Server, the team, they've added some in-memory  features to just stay alive.  And let's have a look at how B3s work.  OK, this is a technical part of the talk.  So B3, it's a logical tree.  You see the tree on the left?  It's a logical structure in all of the squares there, the  green and yellow ones, or whatever color they are, are  data pages.  And each data page maps to a block on disk with the size  of eight sheila bytes.  So it's a logical structure.  And on disk, it lives in the data files.  We can have one more data files in SQL Server, right?  So on the right, it says data, 64k blocks.  Those are called extents.  So when data on disk is consecutive blocks of data  pages.  And in between, we have a buffer manager.  And the buffer manager is the component that writes the dirty  pages to disk, and it reads the pages into memory, into the  buffer pool.  That's on the left.  The tree is in the buffer pool.  Some of it.  There are pages.  The buffer pool is like a cache.  So if you have a lot of memory in for SQL Server, all the  memory on the server, it will take as much memory as  possible, and it will use it mainly for buffer pool.  So it'll keep the data pages in memory.  And that's a good thing.  If you have a lot of data pages in memory, you don't have  to move them back and forth between memory and disk.  And that's the buffer manager's job.  That's a component of SQL Server.  So the buffer manager takes dirty pages from the tree and  writes them during checkpoints.  And that can happen every second or every five  seconds or sooner.  It's pretty complicated.  But between checkpoints, you have dirty pages in the tree.  And if the computer crashes, the server crashes, then you  lose those pages.  And that's why you have a transaction log to  keep the durability.  So if you crash, you save everything in the transaction  log.  So every time a transaction commits, all the information  from the transaction gets written to the log.  That's a transaction log.  And SQL Server uses effect logging or binary logging,  because it logs the data pages.  We're going to look at other types of logging a bit later.  Origo uses a different type of logging.  VoltDB uses a different type of logging.  Redis uses logging, also a different type.  It uses logical logging.  It remembers the information that caused the change.  All right?  But this is effect logging.  We don't remember what the insert, update, delete  things were.  We remember which pages were new, which pages were modified,  and which pages were deleted.  And all those get written to the log.  We have to write deleted and modified pages to the log  because, or the original pages before they were modified,  so we can roll back.  If something goes wrong, you have to roll back.  And then we read the pages back from the log and replace them  in the tree.  Got it?  So that's logging.  So for every single transaction in SQL Server,  it has to flush one or more pages to disk.  And hopefully, the heads on the spinning disk  won't be doing other things.  You have a dedicated disk for your log  if you have spinning disks.  Then the head will remain in the same position,  and it can write pretty fast.  So it won't be doing random access.  It will be appending to the log all the time.  But if you put your log and your data files on the same disk,  then the disk will have to jump around.  OK?  So the log kind of defines how fast, how many transactions,  what kind of performance you can get from SQL Server  in write transactions.  All right.  Got it?  So that's a B3.  That's the old structure.  And it's designed in the 70s or 80s.  So that's pretty old.  The implementation was designed in the 67.  Yeah, but nobody does that anyway, right?  That's not what we're doing.  It says Al and K, right?  No?  Yeah.  But this is kind of an implementation or architecture,  but not really an idea.  But if you take this idea, you can  build it specifically for in-memory,  then you get totally different performance and other properties.  And we'll be looking at that soon.  OK, so let's talk a little bit about SQL Server's OLTP.  The previous slide was the traditional architecture.  This is kind of like, OK, so what do they do about it?  How are they going to speed things up?  OK, so what do they do about it?  How are they going to speed things up?  Yeah, so here's a pyramid and at the bottom,  that's the most, the factor with the largest impact,  the Buffer Manager I.O.  Right?  Writing and reading to disk, Buffer Manager,  at random places in the data file,  that consumes a lot of time.  And so on the right, we see what they did.  So they kind of, oops, I'm pressing too much there.  OK?  Yeah, so they added features.  They redesigned the B3 and just created  an entirely different, a new representation of data in memory.  So they keep the entire table.  You can choose to create one table and make it in memory.  It's on a table-based basis.  So it's not the entire database or portions of it.  You take one table at a time and you migrate it to in-memory.  So the table is either in-memory or it's not.  And the in-memory architecture is totally new.  So it's redesigned, architecture optimized to work in memory.  OK?  So the B3 we saw on the previous page, it's gone.  Right?  And then we have locks.  Locks are during a transaction because relational databases  are so slow, transactions need to be concurrent.  Because if you have to wait for the previous transactions,  things, it will be chaos, right?  So locks are used by the transactions to remember what  kind of data, I think I have a timer on that one.  OK?  The locks are used to remember which data  rows and data pages have been modified by transactions.  So to isolate transactions so they don't step on each other's data.  More on isolation later.  So what did they do about that?  They added a multi-version concurrency control.  It's kind of like snapshot isolation.  But it means that if one process, if one transaction needs to  read the data, it will get a copy of it.  So they don't have to block on the same data row.  So you just copy the row and then both use it.  And then at the end of the transaction, it's kind of optimistic.  If there's no conflict, if nobody changed the two same rows,  there's no conflict, then both transactions can commit.  So they've eliminated locks from in-memory old TP.  And that's a significant amount of time for processing the locks.  Laches are for the data structures themselves to protect the B  trees and other structures.  And they've redesigned them as well to using lock-free data  structures.  All right.  And logging, we saw logging.  We write stuff to the log, effect logging, all the data pages  that were modified and so on, written to the log,  but not in-memory old TP.  We just log the actual data rows, right?  The data row that is being written.  That's the only thing that goes in the log.  And it's very small.  So it's not the entire data page.  It's just a tiny bit.  And then the log, it doesn't use checkpointing because the data  is only in memory and there's a background process that reads the  log and updates the data.  And the data isn't regular data files.  It uses a file stream thing.  So this is kind of technical.  If you want to use, I'll back it up.  If you want to use SQL Server in-memory old TP,  you don't have to know all these things.  You just start the wizard and you say, OK,  next, next, next, and then you're finished to use it.  It's good to understand things and why it's fast.  But the main benefits come from the bottom of the pyramid  and then they, yeah, diminishing returns the higher you get up the pyramid.  And at the top we have a store procedure execution  and query planning and that type of thing.  Today you can create, in-memory old TP,  you can create native compiled store procedures.  So they're compiled to C. They're trans-compiled to C when you create them.  And then the C compiles, we have like native code.  Your procedures are native code working on these in-memory structures.  So everything's pretty fast.  You guys heard of Oracle x10.  They bought some startup and x10, they're boasting because  their goal was to make things 10 times faster.  And SQL Server, the code name for SQL Server in memory old TP was Hecaton.  You remember that?  So Hecaton is Greek for 100.  So that was the goal to make everything 100 times faster.  But it's kind of silly. You can't just make everything faster.  It's difficult to measure. You have to compare different workloads  and it's pretty complicated, but you can probably get not really near that.  So it's really transparent for the application.  That's one of the strengths.  You can use SQL Server Enterprise Edition.  Yeah, lots of money.  Okay. Then you can just convert your tables to in-memory.  And then you'll benefit from the speed.  And it's entirely transferred to the applications.  But not really because there are limitations.  You can't have foreign keys.  You can't have certain types of indexes.  You can't have certain types of columns are not supported.  And that can be a bit disappointing because you have to do a bit of work to really get there.  And then you enterprise license and some performance there.  All right.  Let me show you.  There.  SQL Server somewhere.  I have a 2014 developer edition here.  And Adventureworks, you've probably seen it before.  So on the context menu, there's a memory optimization advisor.  And I'm just picking a table at random here in Adventureworks.  And it runs a checklist.  It shows me all the reasons I can't convert this to in-memory.  So it's a checklist.  And there's information.  There's a link to the help.  But user defined data types, not supported.  Foreign key, not supported.  Unsupported indexes defined on this table.  Default constraint, not supported.  OK.  Not supported.  Let's try another one.  Data's log.  Oh, XML type, not supported.  Large object binary, not supported.  OK.  Try another one.  OK.  Almost.  Foreign keys.  Yeah.  There are a lot of limitations.  So you have to work with that.  I didn't manage to find a single table in Adventureworks  that could pass the test.  All right.  So I did this one.  Fairly simple table, right?  Just to show you how it works.  We don't have a lot of time, so we're not going into the details.  But I think that table's right there.  OK.  And we'll run that one.  And it passes all the tests.  Amazing.  All right.  And some warnings.  We have some procedures that access that table.  So we could probably convert those also.  We don't have to.  But we can compile them, pre-compile them.  All right.  So I...  Make some choices here.  We can choose between a hash index.  It uses a specialized hash structure.  It's like a dictionary or a hash table in memory.  And it's really, really fast for lookups.  Or we can use a traditional tree-based index.  Those are the two we can choose from.  We can also create secondary indexes later.  But those are never saved to disk.  They're always recreated on startup.  All right.  So I won't do that.  I'll just script it so you can see what happens.  It kind of looks like this.  You create a table.  And you say with memory optimized on.  Endurability, that schema and data.  If we say just schema, then the table would be empty on startup.  If we want to do some kind of caching or something.  Yeah.  Okay.  Enough of that.  And back to the slides.  Okay.  Okay.  So let's talk about Redis.  Redis is a key value store.  But the values are very special.  It's actually an abbreviation.  Remote dictionary server.  So that's the abbreviation Redis.  So it's a dictionary server.  So you have keys and values.  You know dictionaries, keys, values.  But the values are very special.  You can put either strings, but you can also put.  Let's have a look at that.  You can put the values can be strings.  It can also be lists.  So it's kind of like a data structure server.  So the one key can point at a set or a hash or a sorted set.  So it's a limited set of collections you can use.  And then there are a lot of predefined commands.  And you can talk to it with Telnet.  You just start up.  Yeah.  You just talk over TCP and you send text and you get text back.  So that's the interface.  And there are a lot of, a lot of drivers for a lot of languages.  Redis is extremely popular used by Twitter and Facebook and all those big stuff for just keeping data and cache.  Caching data, keeping it near applications.  For speed.  And Redis also has persistence.  So you can do, there's an append only file.  So every time you change something, you send a command, it will write the command to the append only file.  So when you start it up again, it can reread the commands in the file and rebuild the data in memory.  But most people use it.  They start an up and empty node and then just start using it like a cache from the application.  Okay.  Written by a guy in Italy.  Highly optimized.  See a lot of fast algorithms.  And every single command has a documentation on how it performs.  So it's a great performance written in C.  No garbage collection installs or that type of stuff and widespread open source.  Okay.  So if we do, we do something crazy like build Twitter using Redis only.  What would we do?  It could look like this.  So here's a command increment.  It means increment the key with the name next user ID.  And if it doesn't exist, it will create one.  So it, and it returns the incremented value.  And it's an atomic operation.  So it's kind of like a key with a number behind it, but we're incrementing it or creating it.  So we're adding a user to Twitter.  So we need the user ID, a unique user ID.  So that's what we do first.  HM set means where H means a hash and M means multiple.  So we're creating a hash and we're setting values for it.  And the key is user 1000.  That's the user ID we just created.  So the application sends that ID back, right?  And then we're creating name bar password hash.  And you could fairly easily map that to a JSON object, right?  Key value, key value.  So the key points at a hash and the hash in itself has keys and values.  All right.  Hash, we're setting the key users is a hash and we're setting the key bar to 1000 in that one.  This kind of looks like an assembly language, right?  It's pretty cool.  Yeah.  But I wouldn't want to build a big, large application where people's lives depend on this, right?  Because your application has to translate this, right?  Create these and send to Redis.  Of course, the library will do that, but this isn't really a data model, right?  Yeah.  So the title of the talk was IMDB in memory databases.  Redis is really a database, is it?  It's kind of like a key value store on steroids.  Okay.  Yeah.  I don't think I'm going to talk you through each command here, but you get the feel of it.  If you go to redis.io, the homepage, they have a complete example built with PHP with all this thing.  It's a Twitter clone and you can download it and you can play with it.  But this is what the application does.  It sends these commands and gets stuff back.  So it's kind of like an advanced key value store.  All right.  So that was Redis.  Very short.  Okay.  Let's talk about OrigoDB.  I'm checking my time here.  Okay.  So here's our slogan.  One of them, build faster systems faster.  And that's not really a good slogan because it doesn't really say what we built it for.  It's kind of catchy because it uses the same word twice, faster and faster.  Okay.  So you can build fast systems or faster systems and you can build them faster.  And that's what the emphasis should be.  Okay.  Building stuff faster.  I'm extremely impatient.  I don't like wasting time.  I don't like dealing with object relational mappers or relational database.  Well, I don't have to.  I just want my objects and I want to perform some operations on them.  And then I have to write, do a bunch of modeling on the database and I have to pull the database into memory.  I have to modify it and I have to keep track of concurrency and then push the data back.  It feels like there should be some simpler way, right?  And that's kind of the idea behind or egoDB to make it easier.  Build faster systems faster.  Yeah.  And they are faster, but that's kind of for free.  So I didn't work really hard to make this thousands of times faster than SQL Server.  The engineers on the SQL Server team, they're like, yeah, every single one of them is 10 times smarter than me.  But this is just, it's so simple, it's so easy.  Okay.  So what's the problem?  I kind of mentioned it.  It's a thing.  Relational databases, they were kind of meant to do domain modeling when they were invented, right?  The SQL was intended as an ubiquitous language, right?  End users were supposed to type insert and delete and select and stuff.  A select statement, that's pretty close to plain English.  That was kind of the intent during the design.  So why do that twice?  Once with the old style, designed from the 70s and all the work that comes with that.  That's the thing all the way to the right.  And then we have all this mapping because we really don't want to deal with the database.  So we use tools in between that kind of hide the database and those tools mess things up even more.  Performance, right?  The mapping.  All right.  And databases, they're so broken, they're so slow, so we need a cache.  The kind of point of a database is I can put my information there, I can do some transactions and give me some information so I can show it to the user.  But they're so slow.  So we need a cache.  Isn't that stupid?  Yeah, so if the database was fast enough, we wouldn't need a cache.  And that's kind of like the performance and what happens when you go pure in memory.  And this is kind of the idea we use our philosophy.  So it's not really the performance, it's kind of like we don't have to do those three things and we save a lot of time.  Okay.  Yeah, so we stick all the objects in the domain layer and just keep on there.  And then we run the transactions in memory and then we log the commands.  We write them to a file like a log.  And then when the system starts, you reread the log and then you've rebuilt the system.  And the only catch is it has to fit in RAM.  But every single database I've worked with the past 20 years, they easily fit in the RAM we have today.  So it's not really a problem.  Okay.  So it's a pretty simple idea.  We have, this is some systems or transaction theory.  We have a state, an initial state, right?  And then we apply an operation to the state and then we land in a new state.  And each state is consistent, you know, asset transactions.  We don't get stuck between the states.  And, yeah, we just apply the operations and get new states.  So that's really the idea.  And if you keep the state in memory and just persist the operations,  then you don't have to worry about how you map your data or represent it on disk.  And then you can keep it round the maxes because it doesn't have to have a specific structure.  So you can use collections, you can use lists and dictionaries and queues,  all the dotnet collections and types.  And there's no mapping to different types.  So that's a really simple idea.  But I think it's pretty powerful.  It's so simple, it's deceptive.  People don't understand the power because you like shiny tools with lots of buttons and stuff, right?  Most developers do.  Okay, this isn't really a new concept.  System prevalence.  There's a project called Prevaler if anyone's working with Java.  It's a pretty old project from 2000, I think.  So they're doing exactly the same thing as ORIODB is.  Yeah.  And the guy who started it, he kind of, he thought he solved all the problems of a system.  He solved all the problems of databases year 2000.  And he showed it to the world and no one understood.  And now there's a small group, a couple of hundred or a couple of thousand just using this and just benefiting from it.  Okay.  MongoDB uses up logging.  You know what that is?  The operations you do in MongoDB, it logs those.  And also uses the operations for replication.  So it's pretty easy to set up a slave server and you just send the operations over there and wait for them to complete.  And then you have a synchronous replica.  All right.  Redis uses a Pandio only file.  If you read Martin Fowler's blog, he has a post there on memory image.  If you read that carefully, you'll see it's the same pattern.  It's the name of it.  VoltDB uses logical logging.  So they have a log and they log the operations and they keep all the data in memory.  Okay.  Acca, pretty cool right now.  You guys know talks about Acca at NBC.  That's strange.  Acca.net, Actors Model.  They have persistence for the actors and it's the same principle.  So you log all the operations or the events, the messages that have received the actor.  And then you can rebuild the state of the actor later.  Event sourcing, of course, you've all heard it.  Same principle, right?  Of those of you familiar with event sourcing, CQRS, but mostly event sourcing.  It's the same thing.  You save the events and not the data.  Okay.  So let's look at how it works.  Let's look at how it's built.  This is also pretty simple.  The blue things here are the components or the parts of the framework that you interact with.  And the yellow things, if that's the color, are the things that you create.  Right?  So you have application code and it talks to the engine.  If it's in the same process, you can run ORIGODB in the same process.  You can host it directly in your IIS web application.  So it runs in the same process.  You should turn off recycling for the IIS if you do that.  Because then you have to reload from memory every recycle.  Okay.  So your application speaks with either the engine or a server.  And the server, it's a standalone ORIGODB server with a lot of features like replication.  I mentioned MongoDB replication.  ORIGODB server has it as well.  So you can do single, master, multiple-slave replication.  You can do ad hoc queries, the web UI on the server.  I'll show it to you if we don't run out of time.  Okay.  And we have the engine.  So the server embeds an engine.  And the engine has a storage module.  And it's pluggable.  So default, it writes the file.  So if you just start up ORIGODB, it will, in the current directory, it will just start  writing the events to files in the current directory.  You can also write snapshots.  But you can use SQL as a backing store as well.  So you can put the events or the commands, write them to SQL.  Or you can use event store or you can write your own.  Okay.  You can choose serialization.  Default is binary format.  You probably shouldn't use that in production.  I would go for protobuf because it's flexible and fast and small.  Pretty optimal.  Interoperable.  You can use JSON.  That's pretty smart.  So you can pass commands and queries from your application to the server in JSON.  And that opens up for HTTP and REST APIs and stuff.  And communicating with, from other platforms.  Okay.  So the kernel is the thing that guards the model.  The model, that's the most important thing.  That's your data and that's in memory.  So you have an instance of the model and it lives in memory.  And you don't have access to it.  You don't have a reference to it.  All communication goes through the engine.  And what do you do?  You do the two things you can do with it.  You can ask it something or you can tell it something.  And asking something, that's a query.  So you create a query and that's just an object that you've written yourself,  probably using link or something else.  And you pass the query to the engine.  The engine executes the query and takes care of concurrency and that kind of thing.  And returns the results to your application code.  If you pass a command, the command gets written to the journal.  All right.  It gets journal.  And then it gets executed.  So the kernel executes a single command at a time.  So it's single threaded.  So there's no concurrency issues between the commands.  Because it's fast enough.  I can run 20,000 transactions per second on this laptop.  Commands.  Sustained commands per second on my laptop.  And then there's probably a room for queries as well.  Because in between those commands, while the engine is waiting for them to flush the disk,  we can run hundreds of thousands of queries, depending on their nature, of course.  So it's really, really fast.  But we didn't work hard to make it fast.  We got that for free because it's in memory.  Okay.  So you write commands and queries in a model and then you just go.  Okay.  This is so important.  I made a slide specific from it.  When you have a complete history of events, everything that's happened,  and something goes wrong at the client,  I can grab their log, the command journal,  and I can just load it up in the debugger.  And I can go to the exact point where they wonder something or something failed.  And I can start stepping through the code.  Just replay the events.  And that's extremely powerful.  I can fix a bug.  If I have a bug in the command execution code,  so the thing in memory isn't really what it should be.  Because there's a bug somewhere.  I can fix that bug.  And then you replay the events and you'll fix the bugs that are old,  not just the new bugs.  So if you screw up some data, you just re-project it.  All right.  That's pretty cool.  And you can restore to a specific point in time.  I want to back up to yesterday.  You just tell it what time you want to start from.  So that's kind of the benefits from event sourcing.  Okay.  And then you have an audit trail.  You know who changed what and when.  Every single command when it was executed and by whom.  That's kind of, you get that for free.  Okay.  Want to say that?  Okay.  So let's look at some code.  Here's the, an example model.  Could look like this.  Okay.  So we write some code.  We write a model and it derives from model.  It's one of the framework types.  And it should be serializable if it's supposed to support snapshots.  If we try to take a snapshot with binary format and it's not serializable, the snapshot will  fail.  So that's why it's serializable.  But you, you don't have to do snapshots.  You can just stay with the journal.  Okay.  So here are three datasets, three sorted dictionaries.  So the keys are grids and they're pointing to specific objects, classes, types, customer  product.  You see.  And we're newing that up in the constructor.  You see it's internal so we can manipulate these data structures from commands and queries.  This is kind of anemic model.  It's just a container for structures.  You don't have to do it this way.  You can put behavior here as well and you can test it.  But this example is anemic.  So the command, here's one add customer.  And it drives from, you see command and we're passing the generic type commerce model and  that was the model on the previous page.  And it's immutable.  So we have read only properties.  They should be immutable.  You shouldn't be mixing, changing these things after they've been logged.  That's kind of like getting in trouble.  All right.  And then you have an execute method and it's the kernel that calls execute.  You give the engine the command and the engine passes the command to the kernel and the kernel  it calls execute and passing in the model.  Running only a single command at a time.  It's pretty simple.  It's calling a board here so it's throwing an exception if there's a duplicate key and  abort signals to the engine that, okay, something went wrong but we didn't touch the model.  I promise we didn't screw things up.  So just ignore, just continue.  That's what abort means there.  If we throw an exception that's unexpected, then we can tell the system, uh-oh, shut down  because this is not good.  This command failed and we don't know what happened to the model.  Because some systems that's really important.  If you work with the healthcare or you work with the medicine, how much medicine did that patient get?  The last time they got medicine?  We don't know.  That's not a good answer, right?  So some systems are really, really important.  And if you need that consistency, then you can get that here.  Okay.  So queries, same principle.  We derive from query, you pass in the model and then the return type.  You see? Customer view, that's what we're returning.  And the customer view is not a customer because if we return a customer and the customer has orders  and each order has a reference to a product and each product has a reference to every order,  you get the picture right.  It's strongly related, highly connected.  If we grab a customer, we'll pull that entire graph and return the entire graph back.  That's not what you want.  So you have to kind of break the graph and that's why we use views, though.  So return customer view.  Okay.  Kind of a principle.  You could use a lot of advanced link here as well.  I'll show you that soon.  Okay.  So now we have a model, we have a command and we have queries.  So then we start our engines.  So we see engine.for as a convenience method for creating or loading an existing database.  It'll look in the current directory.  We can create a configuration object and pass to engine for pointing it at a remote server or pointing it at a specific directory  or using a specific storage provider and so on.  Yeah.  So you see what's happening.  It's pretty simple.  We're creating a new add customer command and we're passing the command to engine, right?  Engine execute.  So that gets logged and the model gets transformed.  And then we're asking a question also by passing a query to execute.  And that's all there is to it.  I build applications using this.  And I've been doing that for the past 10 years.  Okay.  And it works great.  Okay.  So let's have a look.  A little demo.  I'm connected.  Let's do that.  There.  Okay.  This is an example.  We built it just for fun just to show how fast things are.  So there's no cache here.  This is a, it's a kind of search engine.  It's like Google, but really, really small.  So I have a little process that goes fetches, a lot of blog articles, and just pushes them  into an in-memory model.  So when we're searching, if I do a search here.  Oh, that was great.  Something better.  I think the JavaScript is wedged.  Oh, server not found.  I'm not connected.  Good.  That kind of explains it, right?  Good.  There.  It's better.  Okay.  So 48 results in the memory database, and it took 0.07 seconds.  And there's no caching.  So it did a round trip to the server and searched all the data.  And there's not a terrible amount of data.  There are 8,153,000 connections between links from keywords to the articles that contain  the keywords.  And that's what we're getting in the list.  You can check this out if you like.  It's on GitHub.  You see the address.  Just Google for Geekstream.  And it's built with OrigoDB and OrigoDB server.  The source here is on GitHub.  And it's useful as well.  Okay.  So let's show you a bit of code.  Complicated, right?  So the entire model is 216 lines of code.  That's the model.  And then there are some commands over here.  So we have a domain with one, two, three entities, feeds and feed items.  And then we have the model, the Geekstream model.  And then there are query objects over here.  Get feed by ID, get feeds query and so on, search query.  Look at the search query.  Yeah.  So it's a pretty simple link extension methods.  Get some items.  We're searching the model over here.  So the search logic is a part of the model.  And then we're just skipping and taking and transforming the result in the query.  Pretty simple stuff.  Great.  And here's the core.  It's a sort of dictionary of string.  So the string keys are the words that we're searching for.  And they point at sorted sets of feed items.  The feed items that contain that word.  Okay.  That was Geekstream.  Okay.  I'm not going to show you this one, but I actually built a Redis model because I don't like that.  Redis kind of feels like assembler.  So I wrote a Redis clone using origoDB.  So it's a Redis model.  And here's a dictionary of strings to objects.  And those are just using regular types.  So I implemented every single basic command for Redis using origoDB.  And it took an evening or two.  Pretty fun.  It's not as fast.  Of course, Redis is a great piece of work.  But I kind of did this for fun.  But you can use this and you can do it embedded.  You can't run Redis embedded.  You have to run it as a remote server.  But now you can have the same model if you want it embedded.  Okay.  So that was that.  Extend.  Great.  The demo and we've got nine minutes left.  Okay.  Great.  So conclusion.  Yeah.  It's not a lot of time to look at three products.  But I hope you got a little hint at what they all can do.  Really different.  The only thing they have in common is it's in memory.  We're keeping data in memory.  That's kind of the commonality here.  So, yeah, you like that text, the older, the new and the ugly.  Can you guess which one?  Which one is which one?  Yeah.  Well, and I was really being nice because I thought you could say, I was thinking about  saying the disease and the cure.  All right.  So the sequel, it's great, but it's old.  It's not fast enough for us.  So we add a cache.  And so the relational database and the cache are kind of like partners in crime.  Yeah.  So I say use the OrigoDB when you can, but I'm crazy.  So I use it for everything.  I want to try it out.  But it fits really well when your data fits in RAM, if you don't have extreme amounts  of RAM.  It fits really well when you have performance problems with SQL.  But if you have an existing SQL, it's pretty hard to integrate.  So you kind of have to go all in.  It's good for green field development, I would say.  But you can integrate because you can write the log to a SQL server.  So if someone says, oh, everything has to be SQL server, OK, fine.  So I'll create one table with events in SQL server.  And then everyone's happy because the operations they have SQL and they can back it up.  And that's a strength.  OK.  So you can combine them.  So use Origo when you can.  Use SQL server when you have to.  Use Redis when you have to.  Redis is really cool and fast, but I wouldn't use it to build applications.  All right.  Caching.  It's good for caching.  One thing about Redis is you can't really do queries across all the values or keys.  You operate on one key at a time.  So that's a limitation.  You can't do analysis.  You can do analysis with the SQL server.  You can do analysis with Origo as well.  All right.  SQL is great if you have a lot of memory, a lot of data, more than fits in RAM, right?  So Origo is when you run out of RAM, we have to pay for a lot of RAM.  If you're not really benefiting from the performance, maybe it's not really a good fit.  Yeah.  And SQL server.  OK, I'll skip that slide.  There.  I have a demo that shows that SQL server is not really atomic unless you really, really  know what you're doing.  If you're interested, you can ask me later.  And it's not consistent, and it's not isolated either if you're into Asset and all those  kinds of things.  You know, it's read committed, isolation level by default, and there's a lot of anomalies  you can get with that.  OK.  We can talk about that later.  So, yeah.  So licensing.  SQL costs money.  It's enterprise license.  Redis is free.  Origo is free.  But we're trying to sell some enterprise stuff, some replication.  But I think we're going to change that, change the model.  So we're working on that.  But the embedded engine is free at the moment.  OK.  So SQL, that's great writing, right?  T-SQL.  You like writing application logic with SQL?  That's great.  Testable.  Great.  No.  So, Origo, C-sharp, link, nothing but.  No mapping, nothing.  Redis, you use the commands, but you can also use Lua scripting.  It feels kind of glued on, but you can write transactions and do them with Lua script.  I haven't really played with it.  So OLTP, yes, for all of them.  OLAP, analysis can't do with Redis.  Origo, you can run it in process.  If we were to rank performance, I'd say Redis is first.  It would be silly to say claim some amount of transactions per second and stuff, but Redis is a great performance.  And Origo is great performance as well, but not really near Redis.  But we can do more with Origo.  So, yeah, the modeling is pretty powerful.  I think you understood that, right?  You create a model with C-sharp, and then we do commands and queries, everything C-sharp.  So the modeling is really flexible, really powerful.  Relational is powerful as well, but it's kind of old.  Okay, but the modeling capabilities of Redis, kind of limited and kind of fixed.  But then again, SQL is extremely mature, right?  Lots of connectivity, lots of operations, lots of knowledge.  We know how to use it, lots of tools.  Yeah, really powerful.  It's hard to beat that, but I think it'll go away in 50 years, maybe.  Okay, yeah, size.  In-memory OLTP for SQL has a limit of 250 gigabytes per database.  But then again, you don't have to put all the data in memory with SQL, right?  So you can choose specific tables where you get the largest benefits.  You should put those in memory and figure out what those are.  So it's not really a limitation, but both Redis and Origo,  this amount of available RAM on the server.  But it has to stay in memory.  So, little comparison.  And great, I'm finished.  And if you have any questions, just shoot.  We have three minutes left.  Any questions?  What happens if the database really exhausts the memory?  Exhaust the memory.  Well, you have to check that.  If you have a relational database, you heard the question,  what happens if you run out of memory?  Well, then you get an out-of-memory exception, and commands will fail.  That's a simple answer.  But what happens if you run out of disk?  We've all done that before, right?  The log is full, or the disk is full, and when we're doing SQL,  you have to monitor it and just be aware of it.  So you have to know when it happens, or you're not really doing your job.  So it's not really a difference, I wouldn't say.  Yeah?  Could it be another language adapter or SDKs for other languages and C-sharp?  Well, you can do JavaScript at the moment.  At the server, you can run stuff.  There's an API.  If you run the server, there's a JSON.  You can pass messages as JSON over HTTP.  So in theory, you could write a client and then talk to the server of JSON and HTTP,  and then the client could read any language?  Exactly.  And you can also write a client.  If you use the protobuf formatter and describe the format,  then you can write a binary client in any language as well, theoretically.  Yeah?  This is just basic request response.  Okay?  More questions?  Yeah?  Can you only have one or a server?  Or can you scale them out in any kind of way?  Well, that kind of depends on the model.  It's a good question.  Okay, you heard the question.  Can we partition?  Can we scale it, scale out?  Well, if the model is partitionable, then yes.  The Geekstream model we looked at, it's partitionable.  We kind of designed it for that.  So each feed has a bunch of feed items.  And we can put those on.  But we can partition those based on feed ID,  because there's no relationships between feeds.  And then we just have duplicate keywords.  So we just parse the keywords that are on each server  and let them point at the items on the server.  And there's built into the client so you can send a query to multiple servers,  and then you gather the results in parallel.  So it's not map reduced, but it's something similar.  And it's built into the client.  But it's not automatically scalable.  You have to design the model to work that way.  Okay?  Is it possible to use it for a graph structure with such ability to tracing and such operations?  That's a really great question, because what is a graph?  A graph is nodes and edges.  And what is an object with references?  It's a graph, right?  So the model you saw here, it's a strongly typed graph.  The other day, I created a generic graph model.  It took a few hours. It's not complete.  But it's just a model with a dictionary of nodes and a dictionary of edges.  And it's finished.  And then just some operations on that.  So you can do a generic model.  So you can do the type of stuff like you can do with Neo4j.  Or you can do a specific model like we did here.  And you can query it with strongly typing language.  I found a project on GitHub where you can take link expressions.  Expression trees, right?  You know what that is?  Okay?  Expression trees and serialize them.  So you can write queries for the graph model and then serialize them and pass them to the server.  And I already experimented with it and it's working.  Yeah?  Great.  Took a few hours.  Next.  If you want to persist it, what happens if the binary load starts to grow way too large?  You can't really delete anything because you have to play back from the beginning, right?  Yeah, good question.  Okay, so what happens when the log grows?  This will be the last question because it's 10-2.  So we can continue down here later.  Okay, so what happens when the log grows too big?  Well, it takes a long time to load.  Yeah, and depends kind of on the size and the serialization format.  The binary format is pretty slow to serialize because it has to reattach all the references and it's kind of complex, right?  But if you use a protobuf and you compress it, it's pretty fast.  So you can load hundreds of gigabytes of log in like 10 minutes.  So if you have two nodes, if you have a master and a slave and you're patching a server, then you just take down the slave, you patch the server, you bring it up, and then you sync it.  Yeah, and then you take the other one down.  So you always have one node up.  At some point you have to scratch it.  Yeah, I would, I try to avoid that.  I save the data as far as possible, but you can take snapshots.  Yeah, so when it loads, it loads the most recent snapshot and then the log from there.  But I don't like to throw away data, so.  Okay.  Great.  Thanks for coming.  Thanks for having me.  Good.  Yeah.  Yeah.  And then some button on the way out, right?  Okay.  Thanks.
IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_0_1391.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_1392_6491.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_6492_10583.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_10584_15635.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_15636_18203.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_18204_25127.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_25128_25379.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_25380_25643.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_25644_26159.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_26160_26207.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_26208_27263.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_27264_27863.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_27864_28259.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_28260_30719.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_30720_30815.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_30816_32099.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_32100_33983.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_33984_34151.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_34152_34211.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_34212_34283.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_34284_34367.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_34368_34751.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_34752_35399.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_35400_35507.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_35508_35687.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_35688_35747.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_35748_35927.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_35928_36491.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_36492_36911.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_36912_37055.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_37056_37667.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_37668_37751.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_37752_37799.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_37800_38387.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_38388_38495.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_38496_38555.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_38556_38615.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_38616_38711.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_38712_38807.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_38808_39491.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_39492_41663.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_41664_42383.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_42384_43535.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_43536_45551.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_45552_48071.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_48072_51491.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_51492_53291.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_53292_53639.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_53640_54791.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_54792_56231.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_56232_61691.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_61692_63599.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_63600_65051.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_65052_68639.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_68640_70031.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_70032_70223.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_70224_70463.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_70464_70523.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_70524_71291.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_71292_71699.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_71700_71987.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_71988_72035.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_72036_72347.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_72348_72695.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_72696_72743.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_72744_72815.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_72816_72947.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_72948_73079.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_73080_73163.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_73164_73271.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_73272_73451.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_73452_74099.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_74100_75119.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_75120_75215.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_75216_75275.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_75276_75323.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_75324_75443.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_75444_75503.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_75504_75563.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_75564_76367.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_76368_76451.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_76452_76535.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_76536_76679.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_76680_80447.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_80448_80555.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_80556_85031.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_85032_93143.png IMDB_showdown_-_comparing_OrigoDB__Redis_and_SQL_Server_Hekaton_-_Robert_Friberg_93144_93326.png