How Microsoft SQL Server Went Multi-Platform: SQLPAL
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Fernandez, Argenis
Gianforcaro, Brian
Birukov, Eugene
How did Microsoft made SQL Server available on Linux, Containers and ARM CPUs? Come hear the story from the SQL Server engineering team. We'd love to tell the story on how we made SQL Server available to ecosystems outside of Windows in this talk. It's a great story that involves quite a bit of interesting technologies and we'd like to share that with everyone!
 So first of all, we're not paid actors. We actually work for Microsoft. And yes, we're here. I've been getting a lot of strange looks from you guys because of my T-shirt. And trust me, it's fine. It's a different world now. So we're here to tell you guys a story on how we started on a journey to move SQL Server to other platforms. And this basically started back in 2015 when our executives decided that it would be a good idea to explore how to learn SQL Server on other platforms. And Linux was obviously the natural first choice. So we did this using a technology called SQLPOW, which is based on a project from Microsoft Research called Drawbridge. There's tons more information about the stuff out there, but we'll give you links and everything in the deck. So this is kind of how we went from, you know, this was basically the state of the world in 2016. We only ran on Intel processors. We only ran on Windows. We only supported officially an application fabric, service fabric in Azure. And we could run on-prem and on-clouds, but we could only do so on virtual machines. And when the decision was made to switch into other platforms, by using SQLPOW, we enabled all of these following use cases now. We run on top of ARM, which I know may be a shocker to a lot of you. Well, the fact that we run on Linux is already a shocker to a lot of you. We're naturally now able to run not just on Linux, but Mac OS on top of Docker. So if you guys are Mac users, you can just download SQL Server container and hack away. We also support multiple application fabrics, including Kubernetes, OpenShift, and all the Kubernetes offerings by the cloud providers, including our own, a.k.a. Azure Kubernetes service, the ones in Amazon and the ones in Google as well. And containers is something that we're really embracing fully. There's a new feature of SQL Server 2019 that's called Big Data Clusters that only runs on Kubernetes on Linux. We don't even have support for Kubernetes on Windows Notes yet. So it isn't that interesting for a change of mind on how we do things with Microsoft. We also support secure enclaves. That's another technology. If you guys don't know what a secure enclave is, it's a secure area of memory, protected area of memory within your computer or computing node that enables you to perform secure operations within that area so that no high-privileged user has access to them. SQLPAL also uses this. This is a screenshot of SQL Server running on ARM. We actually call this product Azure SQL Database Edge. So it's the same database engine as SQL Server, the traditional SQL Server that we ship out there. It's just that it's packaged in a different way because we bundle a lot of other things. And we tailor it to Edge use cases. So this was one of the first builds that we did for SQL Server on ARM. If you look at the date there, September 14, 2018. And that's it right there. It's running on top of ARM 64. So not only are we able to run on top of other platforms like Linux, but we're also enabling switching into other instruction sets like ARM. And we do this by leveraging, guess what? LLVM. So we leverage a little bit of magic there. There's some binary translation sprinkled on top. And things just work. So if you want to run SQL Server on a Raspberry Pi, you're able to. So it's a very, very, very different world for us. Anyway, when we started exploring what it would take for us to move SQL Server to Linux, the first thing that came up was, well, what are the possible ways through which we could achieve this? And we thought, well, let's explore how much it would actually take us to get there. If we did a native port of SQL Server to Linux, it would take us five years. We will still be working on this. We started this effort in early 2016. If we had started back then, we will still be working on it, and we would have not released yet. It would have taken us five years to complete. Using SQLPow, it only took us three weeks to get a working prototype. And that working prototype was enough to convince everyone around Microsoft that this was possible, we could leverage this technology for this purpose, but not just that, that we can enable a whole lot of other things using this tech. The rest of the time that we spent building SQL Server on Linux was basically to make sure that we had a mature product, something that we could release to our customers and do feel comfortable in using it. That last number that I have right there, that is a soft three weeks ago, 25 million Docker pools. SQL Server is actually very, very popular in containers. A lot of our customers are using the containers on their CI CD pipelines, and we ourselves are using containers on our functional tests. So instead of spinning up VMs running Windows for SQL Server, that is actually a lengthy process, even if you use tricks like VM images and snapshots and bring things up from that and that sort of thing, that takes a lot more work than it actually takes to just spin up a container with the latest build from the build pipeline and just run our millions of tests against that stuff. So it makes sense for us to use or leverage our own technology to make our build process a lot better. So that number, 25 million Docker container pools grows at about 60,000 pools per day. So we're having a significant use base out there. So this is the secret sauce that enabled us to move SQL Server for Windows on Linux. And again, it's not like we actually compile SQL Server for Linux. We compile SQL Server for Windows and then we package it on top of SQL Pal. And this is how we ship it so you can run it natively on Linux as if it were just a package or on a container by just doing a Docker pull on Linux today. So the architecture goes like this. From the top, you have applications that are running within a sandbox of sorts created by SQL Pal. You have a SQL Server process. You have some DLLs for every process. This host is the one that actually handles services within Windows. We have an infrastructure around services on Windows. We run that stuff within the SQL Pal sandbox as well. Then you have SQL Pal, which is a PE executable. It's actually running within Windows. So it really thinks that it's, SQL Server really thinks that it's running on the Windows. It's just happened. We're tricking it into providing a Windows-like infrastructure within Linux itself. And so you can see how SQL Pal actually makes something like 400 NT calls down to Win32K CIS, which is another PE executable running within Linux. And then from there, there's about 50 calls that actually make it down to the Linux host extension, which is actually a NELF executable. So when you run SQL Server on Linux, the host extension is the first thing that kicks off and actually generates all this environment for SQL Server to run inside of. It's a pretty neat technology, but it also enables us to do a lot of really interesting things. For example, we built support for persistent memory on Linux. So if you guys are not familiar with it, it's basically sticks of RAM that actually persists your data in them. With persistent memory, you can do interesting things like avoiding the entire operating system stack for IO and just keep the IO in user mode. So this is sort of one of the things that we take advantage of. Most of this stuff is running in user mode. Hence why we are at this conference. For the use of B-MEM, specifically for persistent memory, we can actually perform mapping of the entire database set of database files that you place on the persistent memory device and only do MEM copy operations in user mode to access them. So the latency for those operations is extremely fast, extremely low, I should say. Anyway, just wanted to give you a brief introduction to the SQL Pal architecture. But I brought two really smart guys that are actually going to talk in depth about this stuff, make it a little more interesting for you guys. And I'm going to hand it over to Brian now. Hello. So our DNS just painted a pretty beautiful picture of like, oh, this is amazing technology, so how does it really work? It's really terrible. There's a lot of things we have to do that are nasty and complicated to get this to actually work. So think about how a normal Linux process works. So like time, for instance, you have the VDSO where you can go directly into get time from user mode via shared map page. Windows has the same thing. They have a user shared page that's mapped into every process. You can get time, you can get all kinds of performance counters, and it's like a strict known address. So now we have to have that same address mapped into every Linux process. And how do we plumb Linux time into this? Like we can't go load or call the VDSO straight from Windows, right? And it's pretty costly to go straight down to Linux every time. Like from SQL Pal or from SQL Server, and by the way, you can go in through all these layers down to the host extension to get the current time, right? And that's pretty often in SQL Server. So that's complicated. Time zones are also complicated. So Windows time zones are very different than how Linux time zones works. We have to map them. They're all stored in the registry. It's very complicated. Windows manages memory different than Linux. So you can separately commit. You can separately reserve in Windows. There's not really a concept of that in Linux. So we actually had to build our own virtual memory manager, Eugene built it, inside SQL Pal. So it's kind of weird where these SQL Server process and any other virtual process that runs inside SQL Pal, it's running in one giant shared address space. So we have to manage that address space. So what if one of the processes dies? We have to make sure that we clean up any mapped memory, unmap it, manage protections, all that kind of stuff. File systems are another interesting problem. No, I know. Sorry. So Linux file systems are case sensitive. Windows file systems are not case sensitive. So if you break this assumption, lots of Windows paradigms don't work. So now we have to go do case insensitive and sensitive look up at the same time and go kind of try to figure out what the program is trying to do on the covers. And so there's kind of this endless stream of things we had to go think about and implement at the right layer, at the right place. So we actually have thunks that switch calling conventions. So PE has one calling convention, like where to put registers, right, and where arguments are expected to be. ELF has a different calling convention. So we have these thunks. So at that ABI layer, we have these thunks that say, hey, switch from Linux calling convention to Windows calling convention. So if you're doing up call, you switch it one way. If you're calling down, you switch it the other way, right? So those aren't extremely performant. So we want to make sure we do as few of those as possible. And so we purposely kind of keep this layer very thin. We want to only expose things that are actually needed and are very important. So we try to do as much as inside the SQLPow.dll, our version of the anti-kernel in user mode. And leave only the strict, actually required things that you can only implement via the Linux kernel API. So I'm going to go deep dive into some of the actual, like, MIDI challenges after I gave you a small sampling of the things we had to do. So one of the big problems that we have is asynchronous IO. So SQL Server is a database, of course. Databases rely on asynchronous IO. SQL Server is no exception. So we actually have our own user mode scheduler inside SQL Server. So we have our own events. We have our own semaphores. We have our own mutexes, pretty much everything, right? And so that entire user mode scheduler is tied very closely to how we do asynchronous IO via the Windows model. And so if you're not familiar with how synchronous via asynchronous IO works, kind of the traditional model that you would do in a blocking application is you'd issue an IO to the IO subsystem. Sorry. You'd issue the IO, you'd wait for it to complete, and then you'd process the completion. It's obviously not very scalable, not very performant. So most databases try to use an asynchronous IO model where you batch as many requests as possible. You issue them all and then wait for them to complete in unison and then process all those completions as they come in. You get a lot more throughput, a lot more scalability, and a lot more flexibility. So when we go try to map these primitives from Windows to Linux, the problem is that Windows has a common view over asynchronous IO, whereas Linux really has no high performance abstraction over a network in disk IO. So people would probably think that the G-Lib C-AIO is a solution. In reality, it's not really a solution because it has a lot of issues. It's very slow in our experience and it has its own thread pool. So we have a lot of threads in our process and we don't want to add any additional overhead that we don't need. So G-Lib C-AIO was not a candidate for us. So of course, Linux has E-Pool wait and IO get events for processing completions. So E-Pool for network IO, IO get events for disk IO. And so there's a small caveat that this was circa 2016 when we were building this. So there's kind of some new stuff that I'll talk about later, but this was the kind of template at that time that we'd go by. And so Windows exposes a single mechanism called IO completion ports. And so via disk IO, via network IO, you can use kind of the same primitives to manage completions abstractly of what actually issued them. So how do we model the Linux primitives to Windows IO CP? So there's actually multiple ways to use IO completion ports, but one common pattern is that you have IO is bound to completion port. So you call these kind of extension methods read file EX, write file EX, and then there's a socket version of Windows, WSA is WinSock. So Windows socket send, WinSocket receive. There was also crazy stuff like asynchronous except in WinSock. And then so you would issue those IOs and then you can either wait on the completion port or you can call get queue completion status. So do a blocking call or a polling call to retrieve the completions and then you would process the completion packet and do whatever application specifically you need. So when we were looking at this, one of the interesting things we saw was that get queue completion status is kind of, you can think about it as a polling mechanism, right? And so that kind of maps to EpoL and IO get events. And so the nice thing about get queue completion status is that it exposes a timeout. So as long as we honor that timeout, we can do whatever we want inside that call, right? We just have to make sure we get back to when the application gave us the deadline, right? So luckily EpoL await and IO get events both expose a timeout. So we can kind of say for this completion port, we know that it's actually bound internally to a socket or a network, a socket or a file descriptor, right? So we know which one to call when we call get IO completion status. And we can use get IO completion status to actually pump them from the application. So when you call get queue completion status, we'll actually go honor that timeout, plumb the timeout all the way down into EpoL or IO get events, and actually use that blocking cause and method to pump these IOs back up into your application and surface them that way. So this is a general model that works very well for SQL server or applications that use asynchronous IO heavily. It kind of breaks down though when you go to applications that are, that use blocking IO. So like very simple programs. So we actually have, we use the same model, but we do it on background threads so that you don't actually have to use the completion API if you, your application doesn't, will still surface the IOs, but that's kind of an implementation detail in a caveat. So this works. There's one complication though that gave us a lot of trouble. So Windows has this thing called APCs, asynchronous procedure calls. And so when you say, hey, suspend that thread, Windows will actually send an APC to that thread and then ask it to a suspend itself. You can suspend a thread, you can resume a thread, you can actually execute callbacks on a thread. So you can say, execute this APC routine on this thread remotely. And so this is a very common paradigm in Windows. So we had to honor that as well. But what happens if you try to issue an APC and you're blocked with infinite weight inside E-Poll? So our Linux, our Windows stuff doesn't know anything about the Linux user space. So how do we get that guy out of the way? So we actually inject an event FD into every single one of our weights to make sure that we can wake it up externally. So we can go process that APC and then go back down into the weight if we need to. So there's all kinds of little corner cases that you have to think about when you're modeling these Windows things in Linux. So what would we do in the future, maybe? So the gens at Facebook has been working on this awesome IOU ring technology. So it's a new sys call in Linux as of 5.1. And so you have two ring buffers that are mapped in the user space. And so you have a submission queue and a completion queue. So without actually entering the kernel, you can go say, hey, I want to do these IOs. And the kernel will go pull that ring buffer because the kernel maps it into your process. So you both have a view of the ring buffer. So it takes the submissions out, will go process them, and then you can go pull them without entering the kernel as well. So we can remove a whole bunch of context switches from our IO path. And yeah, I'm excited to go play with it. It's in relatively new kernels though, so it's not like we could use it in production everywhere yet. But it's interesting technology. So another problem we have is synchronization. So pretty much there's like one single paradigm for all synchronization primitive surfaced by the NT kernel. And that's a wait for multiple objects or there's a variant of it called wait for a single object. So on Windows, practically everything is a waitable handle. So you have processes, you have events, you have mutexes, threads, semaphores, the list goes on. You can wait on all of these via this one API. So there's no real direct corollary in Linux. There's stuff that if you squint, it kind of looks similar or you could maybe model it, but when you go look at the details, it kind of breaks down. So we had to do a bunch of work to actually implement wait for multiple objects on top of Linux primitives. So we built this kind of class hierarchy where you have a thread, and every thread has a thread wait context. And so that thread wait context has a condition variable embedded in it. And so obviously as a mutex for that condition variable as well. And then has it an array of these wait infos. And so these wait infos are kind of the pair between the waitable object and the thread wait context. So when you go and start a wait, you'll pass in five handles. And so those handles will actually be backed by this abstract class called a waitable object. And so the thread wait context will go and queue itself into a wait queue that every waitable object has. And so when that waitable object becomes signaled, it'll go process the queue and depending on what kind of object it is. So if it's a auto reset event, it'll go take everybody that's waiting and then reset itself. If it's a semi-four, it can count as semi-four, it might take one of it, one waiter. And so it'll go resume that waiter by calling back through the wait info to the thread wait context and then posting on that condition variable. And so you kind of have this very simple paradigm that lets you abstractly model wait for multiple using just the primitive condition variable. Yeah. And so for the future, there was a patch to the next kernel about June 2019 from the Wine people. And so they have a similar problem. So they have an interesting idea where they could model this problem using few texas. So if you expand the few tex API in the kernel to allow to wait for multiple few texas, then you could model the same problem using Linux's calls directly pretty much. So if we did that, we could get rid of a bunch of our code, let the kernel do the heavy lifting and it's a win. So if you're really interested in how all the internals work, one of our colleagues, Bob Doerr, has this awesome blog where he goes through a bunch of internals and then we have a blog that goes kind of high level overview again that we posted when we released the project. So Eugene is going to talk to you about our debugging story. Hello. One of the challenges that Brian did not mention is how to debug that thing. So as our genius told you, when we run SQLPAL process, it is elf process. And Linux host extension, which directly interacts with Linux operating system, is pure elf process. And we can use any debugger, GDB, LLDB, whatever, just to debug it. But that guy loads SQLPAL DLL, which we internally call library.js, and this is PE binary. This is Windows binary. So inside Linux host extension, we implemented PE binary parser, which reads file and uploads it exactly the same way as Windows kernel does. And now on top of that, it's SQL server and tons of other DLLs like NTDLL, kernel-based DLL, and probably some other process as well. And all these guys, pretty much 99% of the code, if you start counting the lines of the code, it will be that much. That's all in Windows format, in PE format. Of course, we have PDB files for them, but what use of those PDB files if neither of Linux debugger can deal with that? So the compelling idea is to just launch, just have separate Windows VM or computer, and launch VindBG on that. VindBG is absolutely capable of dealing with PE binaries and with PDB files with everything. And so VindBG will be able to give us insight into that part of the system, but unfortunately VindBG has no way to connect to Linux machine and to break into Linux process. So we have a cousin, and if we have a cousin, people usually build bridges. So that's what we did. We created a program which is called dbgbridge, debugger bridge. This is also Linux executable, which uses LLDB library, and that using LLDB class library, which connects to SQL Pult process and can manipulate it. Or it either uses a live process or it can load core dump from the file system. That guy pretends that this is a remote debugger server for VindBG. VindBG has well-known protocol, a user debug services protocol, which talks over networked and can connect to Windows debugger running on different machine. So dbgbridge pretends to be just normal Windows debugger service running on this machine and gives access to that part of the system. This is fine, but we want to debug both parts of the system. We want to debug both SQL, both P binaries, and we want to get insight into what Linux first extension is doing at the same time because they tightly interact. Fortunately for us, VindBG is an extensible debugger. So we wrote extension for that guy who directly connects to LLDB library, we have an intermediate code here, and controls this portion of the system. Now, what Windows, what VindBG needs to know about the process? VindBG, it turns out that VindBG has very little, it needs very little to start debugging the process. It needs to be able to read memory, it needs to be able to write memory to set breakpoints, and it also needs to know about two lists. It needs to know about how many P binaries we loaded in the system and where they are, and it needs to know how many threads are there in the system and where they are. Of course, LLDB library does not know that, but every time we load any P binary, we go to Linux host extension and tell it, hey, please, here is the file, and tdll.dll, please load into memory. At that time, Linux host extension says, ah, here is some module loaded, let me put it in the list. And the same thing happens when thread starts. Every thread in SQL has four stacks. It has normal Linux stack where it starts executing, it has Linux signal stack where it processes all signals like Trap, CQB, whatever. It also has normal Windows stack where it executes Windows code, and it has Windows exception stack. Pretty much, signal stack in Windows world. So every time thread starts, which belongs to that world, it also goes through Linux host extension and all those stacks are registered there in Linux host extension. And it also is put in the list. And now LLDB library can examine the memory, can find those data structures, and we can report them back to VindBG. Now VindBG also has ability to stop when we load a new module or we launch a new thread or we thread access. To implement this functionality, we just put breakpoints in Linux host extension. Every time a new module is created, we put breakpoints onto that function and when breakpoints hits, we stop the execution. Let me show some, quickly show some demo for you. I'm connecting to my machine and Redmond. So that debugger bridge is absolutely not an interactive program. I'll start it once and after that it just goes. Come on. Yes. Here, I don't know if you see it. Here we start, I started that program. Yeah, I started the program, you see the bridge SH that's shell script which contains all the incontentions and it opens the core dump which I got in our lab. It's real core dump as the bug I'm executing, even restricting now. And after it started, it says, hey, if you want to connect me, go to Windows box and launch that command. After that, it just brings some warning messages that it did not like something. And all the debugging going through WinDBG. This is WinDBG. And you see, WinDBG says, hey, there was thread created. That means we read the list of threads and tell WinDBG that the thread is created. WinDBG really thinks it's connected to live process. And the same thing goes for modules. It says modules loaded. And then WinDBG, we can just go and just inspect memory here, inspect locals. But this is only for Windows part. Because if you do stack here, say K, it shows the stack ends. It's a little bit stack ends right here. You can call stream right. That's the way how we tell Linux host extension, please take core dump. I'm damaged, Bankam. But if you do Banka here, in working the debugger extension, now we see Linux stack here. And if you click on Linux frame, you see here is the Linux sources and where the execution is. The same way we can do, say, Banktl. And it gives me a list of, that's probably a bad idea. About a couple of thousands of threads there. It gives me a list of all threads which run in the system. And some of the threads are not visible to WinDBG. Some threads are launched specifically for host extension needs. And WinDBG cannot see them. But through that host extension, I see them and I can also inspect the memory or whatever. And we also have commands to do single step debugging, set breakpoints inside Linux host extension using extensions. And we set breakpoints and do single step whatever, debugging in WinDBG world, in P world using normal Windows commands. Pretty much if you single step through the program, in WinDBG world, it will skip over the Linux host extension calls. All right, guys. Questions? Hi, so say I have an arm stop or a left. What's the actual path to run this on ARM systems? Is it released yet? I know you showed it, so I believe it works. So yeah, like I said before, on ARM, this is a new product. It's not called SQL server, it's called Azure SQL database edge. We announced it at build. It's not GA yet. We have a private preview of it. It's not going to be released the same way that we have released SQL server for box. It's going to be tied to an OEM. But for testing and validation and all these things in development, we're going to have containers available for it as well. We do have an EAP, so if you're interested, you can search for Azure SQL database edge EAP, and it will actually take you there. Early adoption program, that's what it means. I can give you more information offline. Thank you. Good question. In the back. So with all of this increased abstraction layers to map between Windows constructs and Linux constructs, and SQL being a performance-based product, how do you guys overcome all of the additional latencies in this process? It seems like there's a lot of layers here. And I mean, if we're trying to query something in a reasonable amount of time, this has to be fast. Right. So as you can imagine, this is a generic concern from our customers. I can tell you that we have really smart people, not me, working on the product. And we have basically established performance baselines for SQL server Linux, and they're basically on par with Windows these days. As a matter of fact, one of our bosses thinks that we can go faster than Windows. We haven't gotten there yet. We're basically par on the same performance on Windows and Linux. Are those public metrics? Are those public published metrics? So you're going to see some TPC-E. If they're not public out yet, there are some TPC-H benchmarks that we have comparing SQL server on Linux to SQL server on Windows. That's fairly similar. And we're also going to have TPC-E. If it hasn't been published yet, it will come out soon. Are there any use cases of SQL server that are slower on Linux than on Windows? I can't think of any. So there have been regressions and bugs that we have fixed were, like I mentioned, time. So querying down a time path. So that user shared page that's mapped into a process that has a little bit that says, hey, can I go fast when I query time? And so we weren't exposing that correctly. So you could query time in a loop on Windows versus Linux. And it would be like 10 times slower on Linux. But that should be now resolved. So there are bugs, right? But most of them we can fix. So it's not a performance question. I'm just curious about your user links. So you've mentioned that I understand it's like a plans for the future. But then I guess you already more or less realize how can it work from the design perspective. And two most important parts there about, well, at least what I'm aware of, it's this balance between completion and submission gives. Can SQL server by itself already handle this situation, the balance between those two? So one will not, for example, oversaturate the system by just submission. Can SQL server do this? Sure. So I have a completion part that have a concept of parallelism. So you can say how much parallelism do I have my IO completion part? And there is internal rate limiting. But yeah, it's something we have to design. We haven't thought about it fully yet. But the IO model of LibOs and SQLPowl is, it matches the pattern of IOU ring pretty closely. Where you go issue an async IO and then you immediately pull it for completion. And then you can have this background guy that can complete it in the background. And so we're already having these dedicated pump threads. So theoretically those guys can just go pump out of your ring instead of the current APIs that they're using. Yeah. And then second related question. It's also, well, there are some limitations. For example, the memory for those IO rings should be locable. So it's like up to, I don't remember, there's some limit. So there is also a good question, how many of those rings you want to have. So any ideas like pet thread process? Sure. I mean, I don't have any numbers. I'll tell you. Well, there's a similar problem with kernel AIO as well, right? Where you can only have so many kernel AIO descriptors or whatever they're called in flight on the process. And so that's something that they manage as well right now. Okay. Thanks. Yeah. Any other questions? One more question? All right. So the alvin in the room, you did mention wine. So why not just extend wine with the API calls that you needed? Sure. So there's a lot of reasons. One of the main reasons was, so before SQL Surrealinix, Eugene and our boss worked on a research operating system inside Microsoft called Midori. And so Midori actually used the MSR research project drawbridge to get Windows compatibility on Midori because Midori didn't run normal Windows applications. So he was pretty intimately familiar with the technology. We already knew it worked. It was proven. And also there's like licensing issues. We can make SQL Pal as optimized as we want for SQL server rather than fork wine and do custom changes. And there's lots of reasons. Another reason is that, you know, on the original slide with architecture, there was Win32K C's Midori. We take it from Windows, unmodified. We patch it. We remove privileged instruction after we load it. But on the disk it is unmodified. So we can keep in sync with new releases of Windows. Really easy. We just grab new Win32K C's, new HTTP C's, new, and with wine, you have what you have. It's adoption of new APIs and security fixes for different cadents. I think the GPL part is the most concerning one for us for wine, right? It's not really very friendly for two commercial software. You also distribute Linux kernel. But anyway, thank you for that. Brian, Eugene, and Arjunis. Thanks, guys.
 So first of all, we're not paid actors.  We actually work for Microsoft.  And yes, we're here.  I've been getting a lot of strange looks from you guys because of my T-shirt.  And trust me, it's fine.  It's a different world now.  So we're here to tell you guys a story on how we started on a journey to move SQL Server  to other platforms.  And this basically started back in 2015 when our executives decided that it would be a  good idea to explore how to learn SQL Server on other platforms.  And Linux was obviously the natural first choice.  So we did this using a technology called SQLPOW, which is based on a project from Microsoft  Research called Drawbridge.  There's tons more information about the stuff out there, but we'll give you links and everything  in the deck.  So this is kind of how we went from, you know, this was basically the state of the world  in 2016.  We only ran on Intel processors.  We only ran on Windows.  We only supported officially an application fabric, service fabric in Azure.  And we could run on-prem and on-clouds, but we could only do so on virtual machines.  And when the decision was made to switch into other platforms, by using SQLPOW, we enabled  all of these following use cases now.  We run on top of ARM, which I know may be a shocker to a lot of you.  Well, the fact that we run on Linux is already a shocker to a lot of you.  We're naturally now able to run not just on Linux, but Mac OS on top of Docker.  So if you guys are Mac users, you can just download SQL Server container and hack away.  We also support multiple application fabrics, including Kubernetes, OpenShift, and all the  Kubernetes offerings by the cloud providers, including our own, a.k.a. Azure Kubernetes  service, the ones in Amazon and the ones in Google as well.  And containers is something that we're really embracing fully.  There's a new feature of SQL Server 2019 that's called Big Data Clusters that only runs on  Kubernetes on Linux.  We don't even have support for Kubernetes on Windows Notes yet.  So it isn't that interesting for a change of mind on how we do things with Microsoft.  We also support secure enclaves.  That's another technology.  If you guys don't know what a secure enclave is, it's a secure area of memory, protected  area of memory within your computer or computing node that enables you to perform secure operations  within that area so that no high-privileged user has access to them.  SQLPAL also uses this.  This is a screenshot of SQL Server running on ARM.  We actually call this product Azure SQL Database Edge.  So it's the same database engine as SQL Server, the traditional SQL Server that we ship out  there.  It's just that it's packaged in a different way because we bundle a lot of other things.  And we tailor it to Edge use cases.  So this was one of the first builds that we did for SQL Server on ARM.  If you look at the date there, September 14, 2018.  And that's it right there.  It's running on top of ARM 64.  So not only are we able to run on top of other platforms like Linux, but we're also enabling  switching into other instruction sets like ARM.  And we do this by leveraging, guess what?  LLVM.  So we leverage a little bit of magic there.  There's some binary translation sprinkled on top.  And things just work.  So if you want to run SQL Server on a Raspberry Pi, you're able to.  So it's a very, very, very different world for us.  Anyway, when we started exploring what it would take for us to move SQL Server to Linux,  the first thing that came up was, well, what are the possible ways through which we could  achieve this?  And we thought, well, let's explore how much it would actually take us to get there.  If we did a native port of SQL Server to Linux, it would take us five years.  We will still be working on this.  We started this effort in early 2016.  If we had started back then, we will still be working on it, and we would have not released  yet.  It would have taken us five years to complete.  Using SQLPow, it only took us three weeks to get a working prototype.  And that working prototype was enough to convince everyone around Microsoft that this was possible,  we could leverage this technology for this purpose, but not just that, that we can enable  a whole lot of other things using this tech.  The rest of the time that we spent building SQL Server on Linux was basically to make  sure that we had a mature product, something that we could release to our customers and  do feel comfortable in using it.  That last number that I have right there, that is a soft three weeks ago, 25 million  Docker pools.  SQL Server is actually very, very popular in containers.  A lot of our customers are using the containers on their CI CD pipelines, and we ourselves  are using containers on our functional tests.  So instead of spinning up VMs running Windows for SQL Server, that is actually a lengthy  process, even if you use tricks like VM images and snapshots and bring things up from that  and that sort of thing, that takes a lot more work than it actually takes to just spin up  a container with the latest build from the build pipeline and just run our millions of  tests against that stuff.  So it makes sense for us to use or leverage our own technology to make our build process  a lot better.  So that number, 25 million Docker container pools grows at about 60,000 pools per day.  So we're having a significant use base out there.  So this is the secret sauce that enabled us to move SQL Server for Windows on Linux.  And again, it's not like we actually compile SQL Server for Linux.  We compile SQL Server for Windows and then we package it on top of SQL Pal.  And this is how we ship it so you can run it natively on Linux as if it were just a  package or on a container by just doing a Docker pull on Linux today.  So the architecture goes like this.  From the top, you have applications that are running within a sandbox of sorts created  by SQL Pal.  You have a SQL Server process.  You have some DLLs for every process.  This host is the one that actually handles services within Windows.  We have an infrastructure around services on Windows.  We run that stuff within the SQL Pal sandbox as well.  Then you have SQL Pal, which is a PE executable.  It's actually running within Windows.  So it really thinks that it's, SQL Server really thinks that it's running on the Windows.  It's just happened.  We're tricking it into providing a Windows-like infrastructure within Linux itself.  And so you can see how SQL Pal actually makes something like 400 NT calls down to Win32K  CIS, which is another PE executable running within Linux.  And then from there, there's about 50 calls that actually make it down to the Linux host  extension, which is actually a NELF executable.  So when you run SQL Server on Linux, the host extension is the first thing that kicks off  and actually generates all this environment for SQL Server to run inside of.  It's a pretty neat technology, but it also enables us to do a lot of really interesting  things.  For example, we built support for persistent memory on Linux.  So if you guys are not familiar with it, it's basically sticks of RAM that actually  persists your data in them.  With persistent memory, you can do interesting things like avoiding the entire operating  system stack for IO and just keep the IO in user mode.  So this is sort of one of the things that we take advantage of.  Most of this stuff is running in user mode.  Hence why we are at this conference.  For the use of B-MEM, specifically for persistent memory, we can actually perform mapping of  the entire database set of database files that you place on the persistent memory device  and only do MEM copy operations in user mode to access them.  So the latency for those operations is extremely fast, extremely low, I should say.  Anyway, just wanted to give you a brief introduction to the SQL Pal architecture.  But I brought two really smart guys that are actually going to talk in depth about this  stuff, make it a little more interesting for you guys.  And I'm going to hand it over to Brian now.  Hello.  So our DNS just painted a pretty beautiful picture of like, oh, this is amazing technology,  so how does it really work?  It's really terrible.  There's a lot of things we have to do that are nasty and complicated to get this to actually  work.  So think about how a normal Linux process works.  So like time, for instance, you have the VDSO where you can go directly into get time  from user mode via shared map page.  Windows has the same thing.  They have a user shared page that's mapped into every process.  You can get time, you can get all kinds of performance counters, and it's like a strict  known address.  So now we have to have that same address mapped into every Linux process.  And how do we plumb Linux time into this?  Like we can't go load or call the VDSO straight from Windows, right?  And it's pretty costly to go straight down to Linux every time.  Like from SQL Pal or from SQL Server, and by the way, you can go in through all these  layers down to the host extension to get the current time, right?  And that's pretty often in SQL Server.  So that's complicated.  Time zones are also complicated.  So Windows time zones are very different than how Linux time zones works.  We have to map them.  They're all stored in the registry.  It's very complicated.  Windows manages memory different than Linux.  So you can separately commit.  You can separately reserve in Windows.  There's not really a concept of that in Linux.  So we actually had to build our own virtual memory manager, Eugene built it, inside SQL  Pal.  So it's kind of weird where these SQL Server process and any other virtual process that  runs inside SQL Pal, it's running in one giant shared address space.  So we have to manage that address space.  So what if one of the processes dies?  We have to make sure that we clean up any mapped memory, unmap it, manage protections,  all that kind of stuff.  File systems are another interesting problem.  No, I know.  Sorry.  So Linux file systems are case sensitive.  Windows file systems are not case sensitive.  So if you break this assumption, lots of Windows paradigms don't work.  So now we have to go do case insensitive and sensitive look up at the same time and go  kind of try to figure out what the program is trying to do on the covers.  And so there's kind of this endless stream of things we had to go think about and implement  at the right layer, at the right place.  So we actually have thunks that switch calling conventions.  So PE has one calling convention, like where to put registers, right, and where arguments  are expected to be.  ELF has a different calling convention.  So we have these thunks.  So at that ABI layer, we have these thunks that say, hey, switch from Linux calling convention  to Windows calling convention.  So if you're doing up call, you switch it one way.  If you're calling down, you switch it the other way, right?  So those aren't extremely performant.  So we want to make sure we do as few of those as possible.  And so we purposely kind of keep this layer very thin.  We want to only expose things that are actually needed and are very important.  So we try to do as much as inside the SQLPow.dll, our version of the anti-kernel in user mode.  And leave only the strict, actually required things that you can only implement via the  Linux kernel API.  So I'm going to go deep dive into some of the actual, like, MIDI challenges after I  gave you a small sampling of the things we had to do.  So one of the big problems that we have is asynchronous IO.  So SQL Server is a database, of course.  Databases rely on asynchronous IO.  SQL Server is no exception.  So we actually have our own user mode scheduler inside SQL Server.  So we have our own events.  We have our own semaphores.  We have our own mutexes, pretty much everything, right?  And so that entire user mode scheduler is tied very closely to how we do asynchronous  IO via the Windows model.  And so if you're not familiar with how synchronous via asynchronous IO works, kind of the traditional  model that you would do in a blocking application is you'd issue an IO to the IO subsystem.  Sorry.  You'd issue the IO, you'd wait for it to complete, and then you'd process the completion.  It's obviously not very scalable, not very performant.  So most databases try to use an asynchronous IO model where you batch as many requests  as possible.  You issue them all and then wait for them to complete in unison and then process all  those completions as they come in.  You get a lot more throughput, a lot more scalability, and a lot more flexibility.  So when we go try to map these primitives from Windows to Linux, the problem is that  Windows has a common view over asynchronous IO, whereas Linux really has no high performance  abstraction over a network in disk IO.  So people would probably think that the G-Lib C-AIO is a solution.  In reality, it's not really a solution because it has a lot of issues.  It's very slow in our experience and it has its own thread pool.  So we have a lot of threads in our process and we don't want to add any additional overhead  that we don't need.  So G-Lib C-AIO was not a candidate for us.  So of course, Linux has E-Pool wait and IO get events for processing completions.  So E-Pool for network IO, IO get events for disk IO.  And so there's a small caveat that this was circa 2016 when we were building this.  So there's kind of some new stuff that I'll talk about later, but this was the kind of  template at that time that we'd go by.  And so Windows exposes a single mechanism called IO completion ports.  And so via disk IO, via network IO, you can use kind of the same primitives to manage  completions abstractly of what actually issued them.  So how do we model the Linux primitives to Windows IO CP?  So there's actually multiple ways to use IO completion ports, but one common pattern  is that you have IO is bound to completion port.  So you call these kind of extension methods read file EX, write file EX, and then there's  a socket version of Windows, WSA is WinSock.  So Windows socket send, WinSocket receive.  There was also crazy stuff like asynchronous except in WinSock.  And then so you would issue those IOs and then you can either wait on the completion  port or you can call get queue completion status.  So do a blocking call or a polling call to retrieve the completions and then you would  process the completion packet and do whatever application specifically you need.  So when we were looking at this, one of the interesting things we saw was that get queue  completion status is kind of, you can think about it as a polling mechanism, right?  And so that kind of maps to EpoL and IO get events.  And so the nice thing about get queue completion status is that it exposes a timeout.  So as long as we honor that timeout, we can do whatever we want inside that call, right?  We just have to make sure we get back to when the application gave us the deadline, right?  So luckily EpoL await and IO get events both expose a timeout.  So we can kind of say for this completion port, we know that it's actually bound internally  to a socket or a network, a socket or a file descriptor, right?  So we know which one to call when we call get IO completion status.  And we can use get IO completion status to actually pump them from the application.  So when you call get queue completion status, we'll actually go honor that timeout, plumb  the timeout all the way down into EpoL or IO get events, and actually use that blocking  cause and method to pump these IOs back up into your application and surface them that  way.  So this is a general model that works very well for SQL server or applications that use  asynchronous IO heavily.  It kind of breaks down though when you go to applications that are, that use blocking  IO.  So like very simple programs.  So we actually have, we use the same model, but we do it on background threads so that  you don't actually have to use the completion API if you, your application doesn't, will  still surface the IOs, but that's kind of an implementation detail in a caveat.  So this works.  There's one complication though that gave us a lot of trouble.  So Windows has this thing called APCs, asynchronous procedure calls.  And so when you say, hey, suspend that thread, Windows will actually send an APC to that  thread and then ask it to a suspend itself.  You can suspend a thread, you can resume a thread, you can actually execute callbacks  on a thread.  So you can say, execute this APC routine on this thread remotely.  And so this is a very common paradigm in Windows.  So we had to honor that as well.  But what happens if you try to issue an APC and you're blocked with infinite weight inside  E-Poll?  So our Linux, our Windows stuff doesn't know anything about the Linux user space.  So how do we get that guy out of the way?  So we actually inject an event FD into every single one of our weights to make sure that  we can wake it up externally.  So we can go process that APC and then go back down into the weight if we need to.  So there's all kinds of little corner cases that you have to think about when you're modeling  these Windows things in Linux.  So what would we do in the future, maybe?  So the gens at Facebook has been working on this awesome IOU ring technology.  So it's a new sys call in Linux as of 5.1.  And so you have two ring buffers that are mapped in the user space.  And so you have a submission queue and a completion queue.  So without actually entering the kernel, you can go say, hey, I want to do these IOs.  And the kernel will go pull that ring buffer because the kernel maps it into your process.  So you both have a view of the ring buffer.  So it takes the submissions out, will go process them, and then you can go pull them  without entering the kernel as well.  So we can remove a whole bunch of context switches from our IO path.  And yeah, I'm excited to go play with it.  It's in relatively new kernels though, so it's not like we could use it in production  everywhere yet.  But it's interesting technology.  So another problem we have is synchronization.  So pretty much there's like one single paradigm for all synchronization primitive surfaced  by the NT kernel.  And that's a wait for multiple objects or there's a variant of it called wait for a  single object.  So on Windows, practically everything is a waitable handle.  So you have processes, you have events, you have mutexes, threads, semaphores, the list  goes on.  You can wait on all of these via this one API.  So there's no real direct corollary in Linux.  There's stuff that if you squint, it kind of looks similar or you could maybe model  it, but when you go look at the details, it kind of breaks down.  So we had to do a bunch of work to actually implement wait for multiple objects on top  of Linux primitives.  So we built this kind of class hierarchy where you have a thread, and every thread has a  thread wait context.  And so that thread wait context has a condition variable embedded in it.  And so obviously as a mutex for that condition variable as well.  And then has it an array of these wait infos.  And so these wait infos are kind of the pair between the waitable object and the thread  wait context.  So when you go and start a wait, you'll pass in five handles.  And so those handles will actually be backed by this abstract class called a waitable object.  And so the thread wait context will go and queue itself into a wait queue that every  waitable object has.  And so when that waitable object becomes signaled, it'll go process the queue and depending on  what kind of object it is.  So if it's a auto reset event, it'll go take everybody that's waiting and then reset  itself.  If it's a semi-four, it can count as semi-four, it might take one of it, one waiter.  And so it'll go resume that waiter by calling back through the wait info to the thread wait  context and then posting on that condition variable.  And so you kind of have this very simple paradigm that lets you abstractly model wait for multiple  using just the primitive condition variable.  Yeah.  And so for the future, there was a patch to the next kernel about June 2019 from the  Wine people.  And so they have a similar problem.  So they have an interesting idea where they could model this problem using few texas.  So if you expand the few tex API in the kernel to allow to wait for multiple few texas, then  you could model the same problem using Linux's calls directly pretty much.  So if we did that, we could get rid of a bunch of our code, let the kernel do the heavy  lifting and it's a win.  So if you're really interested in how all the internals work, one of our colleagues,  Bob Doerr, has this awesome blog where he goes through a bunch of internals and then  we have a blog that goes kind of high level overview again that we posted when we released  the project.  So Eugene is going to talk to you about our debugging story.  Hello.  One of the challenges that Brian did not mention is how to debug that thing.  So as our genius told you, when we run SQLPAL process, it is elf process.  And Linux host extension, which directly interacts with Linux operating system, is pure elf process.  And we can use any debugger, GDB, LLDB, whatever, just to debug it.  But that guy loads SQLPAL DLL, which we internally call library.js, and this is PE binary.  This is Windows binary.  So inside Linux host extension, we implemented PE binary parser, which reads file and uploads  it exactly the same way as Windows kernel does.  And now on top of that, it's SQL server and tons of other DLLs like NTDLL, kernel-based  DLL, and probably some other process as well.  And all these guys, pretty much 99% of the code, if you start counting the lines of the  code, it will be that much.  That's all in Windows format, in PE format.  Of course, we have PDB files for them, but what use of those PDB files if neither of  Linux debugger can deal with that?  So the compelling idea is to just launch, just have separate Windows VM or computer,  and launch VindBG on that.  VindBG is absolutely capable of dealing with PE binaries and with PDB files with everything.  And so VindBG will be able to give us insight into that part of the system, but unfortunately  VindBG has no way to connect to Linux machine and to break into Linux process.  So we have a cousin, and if we have a cousin, people usually build bridges.  So that's what we did.  We created a program which is called dbgbridge, debugger bridge.  This is also Linux executable, which uses LLDB library, and that using LLDB class library,  which connects to SQL Pult process and can manipulate it.  Or it either uses a live process or it can load core dump from the file system.  That guy pretends that this is a remote debugger server for VindBG.  VindBG has well-known protocol, a user debug services protocol, which talks over networked  and can connect to Windows debugger running on different machine.  So dbgbridge pretends to be just normal Windows debugger service running on this machine  and gives access to that part of the system.  This is fine, but we want to debug both parts of the system.  We want to debug both SQL, both P binaries, and we want to get insight into what Linux  first extension is doing at the same time because they tightly interact.  Fortunately for us, VindBG is an extensible debugger.  So we wrote extension for that guy who directly connects to LLDB library, we have an intermediate  code here, and controls this portion of the system.  Now, what Windows, what VindBG needs to know about the process?  VindBG, it turns out that VindBG has very little, it needs very little to start debugging  the process.  It needs to be able to read memory, it needs to be able to write memory to set breakpoints,  and it also needs to know about two lists.  It needs to know about how many P binaries we loaded in the system and where they are,  and it needs to know how many threads are there in the system and where they are.  Of course, LLDB library does not know that, but every time we load any P binary, we go  to Linux host extension and tell it, hey, please, here is the file, and tdll.dll, please  load into memory.  At that time, Linux host extension says, ah, here is some module loaded, let me put it  in the list.  And the same thing happens when thread starts.  Every thread in SQL has four stacks.  It has normal Linux stack where it starts executing, it has Linux signal stack where  it processes all signals like Trap, CQB, whatever.  It also has normal Windows stack where it executes Windows code, and it has Windows exception  stack.  Pretty much, signal stack in Windows world.  So every time thread starts, which belongs to that world, it also goes through Linux  host extension and all those stacks are registered there in Linux host extension.  And it also is put in the list.  And now LLDB library can examine the memory, can find those data structures, and we can  report them back to VindBG.  Now VindBG also has ability to stop when we load a new module or we launch a new thread  or we thread access.  To implement this functionality, we just put breakpoints in Linux host extension.  Every time a new module is created, we put breakpoints onto that function and when breakpoints  hits, we stop the execution.  Let me show some, quickly show some demo for you.  I'm connecting to my machine and Redmond.  So that debugger bridge is absolutely not an interactive program.  I'll start it once and after that it just goes.  Come on.  Yes.  Here, I don't know if you see it.  Here we start, I started that program.  Yeah, I started the program, you see the bridge SH that's shell script which contains all the  incontentions and it opens the core dump which I got in our lab.  It's real core dump as the bug I'm executing, even restricting now.  And after it started, it says, hey, if you want to connect me, go to Windows box and  launch that command.  After that, it just brings some warning messages that it did not like something.  And all the debugging going through WinDBG.  This is WinDBG.  And you see, WinDBG says, hey, there was thread created.  That means we read the list of threads and tell WinDBG that the thread is created.  WinDBG really thinks it's connected to live process.  And the same thing goes for modules.  It says modules loaded.  And then WinDBG, we can just go and just inspect memory here, inspect locals.  But this is only for Windows part.  Because if you do stack here, say K, it shows the stack ends.  It's a little bit stack ends right here.  You can call stream right.  That's the way how we tell Linux host extension, please take core dump.  I'm damaged, Bankam.  But if you do Banka here, in working the debugger extension, now we see Linux stack here.  And if you click on Linux frame, you see here is the Linux sources and where the execution  is.  The same way we can do, say, Banktl.  And it gives me a list of, that's probably a bad idea.  About a couple of thousands of threads there.  It gives me a list of all threads which run in the system.  And some of the threads are not visible to WinDBG.  Some threads are launched specifically for host extension needs.  And WinDBG cannot see them.  But through that host extension, I see them and I can also inspect the memory or whatever.  And we also have commands to do single step debugging, set breakpoints inside Linux host  extension using extensions.  And we set breakpoints and do single step whatever, debugging in WinDBG world, in P world  using normal Windows commands.  Pretty much if you single step through the program, in WinDBG world, it will skip over  the Linux host extension calls.  All right, guys.  Questions?  Hi, so say I have an arm stop or a left.  What's the actual path to run this on ARM systems?  Is it released yet?  I know you showed it, so I believe it works.  So yeah, like I said before, on ARM, this is a new product.  It's not called SQL server, it's called Azure SQL database edge.  We announced it at build.  It's not GA yet.  We have a private preview of it.  It's not going to be released the same way that we have released SQL server for box.  It's going to be tied to an OEM.  But for testing and validation and all these things in development, we're going to have  containers available for it as well.  We do have an EAP, so if you're interested, you can search for Azure SQL database edge  EAP, and it will actually take you there.  Early adoption program, that's what it means.  I can give you more information offline.  Thank you.  Good question.  In the back.  So with all of this increased abstraction layers to map between Windows constructs and  Linux constructs, and SQL being a performance-based product, how do you guys overcome all of the  additional latencies in this process?  It seems like there's a lot of layers here.  And I mean, if we're trying to query something in a reasonable amount of time, this has to  be fast.  Right.  So as you can imagine, this is a generic concern from our customers.  I can tell you that we have really smart people, not me, working on the product.  And we have basically established performance baselines for SQL server Linux, and they're  basically on par with Windows these days.  As a matter of fact, one of our bosses thinks that we can go faster than Windows.  We haven't gotten there yet.  We're basically par on the same performance on Windows and Linux.  Are those public metrics?  Are those public published metrics?  So you're going to see some TPC-E.  If they're not public out yet, there are some TPC-H benchmarks that we have comparing  SQL server on Linux to SQL server on Windows.  That's fairly similar.  And we're also going to have TPC-E.  If it hasn't been published yet, it will come out soon.  Are there any use cases of SQL server that are slower on Linux than on Windows?  I can't think of any.  So there have been regressions and bugs that we have fixed were, like I mentioned, time.  So querying down a time path.  So that user shared page that's mapped into a process that has a little bit that says,  hey, can I go fast when I query time?  And so we weren't exposing that correctly.  So you could query time in a loop on Windows versus Linux.  And it would be like 10 times slower on Linux.  But that should be now resolved.  So there are bugs, right?  But most of them we can fix.  So it's not a performance question.  I'm just curious about your user links.  So you've mentioned that I understand it's like a plans for the future.  But then I guess you already more or less realize how can it work from the design perspective.  And two most important parts there about, well, at least what I'm aware of, it's this  balance between completion and submission gives.  Can SQL server by itself already handle this situation, the balance between those two?  So one will not, for example, oversaturate the system by just submission.  Can SQL server do this?  Sure.  So I have a completion part that have a concept of parallelism.  So you can say how much parallelism do I have my IO completion part?  And there is internal rate limiting.  But yeah, it's something we have to design.  We haven't thought about it fully yet.  But the IO model of LibOs and SQLPowl is, it matches the pattern of IOU ring pretty  closely.  Where you go issue an async IO and then you immediately pull it for completion.  And then you can have this background guy that can complete it in the background.  And so we're already having these dedicated pump threads.  So theoretically those guys can just go pump out of your ring instead of the current APIs  that they're using.  Yeah.  And then second related question.  It's also, well, there are some limitations.  For example, the memory for those IO rings should be locable.  So it's like up to, I don't remember, there's some limit.  So there is also a good question, how many of those rings you want to have.  So any ideas like pet thread process?  Sure.  I mean, I don't have any numbers.  I'll tell you.  Well, there's a similar problem with kernel AIO as well, right?  Where you can only have so many kernel AIO descriptors or whatever they're called in  flight on the process.  And so that's something that they manage as well right now.  Okay.  Thanks.  Yeah.  Any other questions?  One more question?  All right.  So the alvin in the room, you did mention wine.  So why not just extend wine with the API calls that you needed?  Sure.  So there's a lot of reasons.  One of the main reasons was, so before SQL Surrealinix, Eugene and our boss worked on  a research operating system inside Microsoft called Midori.  And so Midori actually used the MSR research project drawbridge to get Windows compatibility  on Midori because Midori didn't run normal Windows applications.  So he was pretty intimately familiar with the technology.  We already knew it worked.  It was proven.  And also there's like licensing issues.  We can make SQL Pal as optimized as we want for SQL server rather than fork wine and do  custom changes.  And there's lots of reasons.  Another reason is that, you know, on the original slide with architecture, there was Win32K  C's Midori.  We take it from Windows, unmodified.  We patch it.  We remove privileged instruction after we load it.  But on the disk it is unmodified.  So we can keep in sync with new releases of Windows.  Really easy.  We just grab new Win32K C's, new HTTP C's, new, and with wine, you have what you have.  It's adoption of new APIs and security fixes for different cadents.  I think the GPL part is the most concerning one for us for wine, right?  It's not really very friendly for two commercial software.  You also distribute Linux kernel.  But anyway, thank you for that.  Brian, Eugene, and Arjunis.  Thanks, guys.
How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_0_131.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_132_1307.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_1308_4163.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_4164_5783.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_5784_8939.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_8940_18875.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_18876_20735.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_20736_20783.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_20784_22811.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_22812_24443.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_24444_24887.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_24888_29687.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_29688_30743.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_30744_33251.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_33252_34139.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_34140_34607.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_34608_44939.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_44940_44999.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_45000_45047.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_45048_45191.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_45192_45239.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_45240_45335.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_45336_45407.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_45408_45683.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_45684_45731.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_45732_45827.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_45828_45875.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_45876_46247.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_46248_47291.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_47292_48071.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_48072_52319.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_52320_52367.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_52368_52487.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_52488_53171.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_53172_53339.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_53340_53399.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_53400_53459.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_53460_53675.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_53676_53771.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_53772_53831.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_53832_53915.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_53916_60335.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_60336_60395.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_60396_60875.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_60876_60959.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_60960_61007.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_61008_61055.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_61056_61103.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_61104_61151.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_61152_61199.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_61200_61247.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_61248_61295.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_61296_61367.png How_Microsoft_SQL_Server_Went_Multi-Platform__SQLP_zq1WTLnntIg_61368_61536.png