High Performance Network Function Virtualization with ClickOS
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Martins, Joao
Middleboxes are both crucial to today's networks and ubiquitous, but embed knowledge of today's protocols and applications to the detriment of those of tomorrow, making the network harder to evolve. While virtualization technologies like Xen have been around for a long time, it is only in recent years that they have started to be targeted as viable systems for implementing middlebox processing (e.g., firewalls, NATs). Can they provide this functionality while yielding the high performance expected from hardware-based middlebox offerings? In this talk Joao Martins will introduce ClickOS, a tiny, MiniOS-based virtual machine tailored for network processing. In addition to the vm itself, this talk will hopefully help to clarify where some of the bottlenecks in Xen's network I/O pipe are, and describe performance improvements done to the entire system. Finally, Joao Martins will discuss an evaluation showing that ClickOS can be instantiated in 30 msecs, can process traffic at 10Gb/s for almost all packet sizes, introduces delay of only 40 microseconds and can run middleboxes at rates of 5 Mp/s. The audience is anyone interested in improving the network performance of Xen, including improvements to the MiniOS and Linux netfront drivers. In addition, the talk should interest people working towards running large numbers of small virtual machines for network processing, as well as those involved with the recent network function virtualization trend
 I come from the NEC Labs research, the research labs in Germany. And this I'll be presenting in Clic OS and this is work done by me and my folks at NEC, as well as some other people at the University of Bucharest. So in one sentence, Clic OS is a VM to do network processing. So what's the motivation behind it? A long time ago, some folks designed the Internet and how does it look like as we are thought in schools? And those with a natural routine containing switches and routers. Pretty simple. But in reality, middle boxes are a commonplace in today's networks. They are very useful for a high number of reasons and they extend operators' sniper for a lot of futures. Security, such as firewalls, intrusion and text systems. Monitoring, such as DPI boxes. Load balancers, dealing with the addresses, exhaustion issues. Performance, such as media gateways, transcorders, wide air accelerators. And some others of more dubious nature, like the advertisement session box over there. I'm having some issues with the display port so you cannot see exactly how this looks like. But these middle boxes are really useful. But there are a lot of disadvantages at the same time. First of all, they are really, really expensive. One of the boxes I showed you, like the media gateway, costs around 200k. It's really difficult to add new features. Often you get locked in to vendors to upgrade your firmware, add new features, and so on and so forth. They are difficult to manage. It often requires specialized personnel to manage the box. And normally you deploy these boxes to keep with the peak rate, so they cannot be scaled like you add more like you want. And it's difficult to share these devices among other parties. And since it's hardware market, it's hard to... new players come along. You know, it's hardware. So clearly if we had some way to virtualize these middle boxes or shift these middle boxes to be software instead of expensive boxes, we would have... we mostly would solve most of these issues we have here. But it is still not clear if we can do it in software and really coping with the performance that the hardware middle boxes can achieve. So we believe so, and that's what I'm talking about here today, and that's what we propose with ClickOS. ClickOS is a mini-OS-based Zen virtual machine that runs on top... that runs ClickModeralRouter. So let me first talk a little bit about Click. Click is a framework to design routers and process packets, and has this concept of elements. Each of these elements does an operation to the packets or receive packets. For example, the configuration on the side basically decreases CTL and forwards back the packet to a different interface. These elements are contained in a configuration, and basically you install it either as a user space application or as a kernel module. These elements are exposed to their counters. If we have a counter on an interface, for example, via variables on the slash proc file system, and for example, the counter, you have a count element to just read only and to reset the counter, you just write one. For example, for ClickOS, we compile 262 of these elements, excluding the ones that deal with the file system. And if you want to further extend the click, you can just write new elements. It's quite easy. So an example of a click configuration looks something like that. You have from Netron to receive packets, into Netron to send packets out. You have an IP filter element that allows packets from 10.01 in destination 10.01, UDP packets, otherwise drops it all. So you can see that it's quite easy just to filter packets, for example. So what's ClickOS? Typically, we have in Xen, our per-visualized operating system is slightly modified to work on top of Xen, and the kernel and the applications are in on top. In the case of ClickOS, we use Minus as our best operating system, and we just run click, and we call the whole thing ClickOS. So it kind of follows the trend of OSV in the previous presentation, or Erlang on Xen, Mirage, so that it's application-specific operating systems. So the work we did was having a build system to build kind of these Minus apps that we used to build ClickOS, and what we get is a 5 megabytes image in total, in size, and we had to emulate the control plane, so we don't have a slash-proc file system between VMs, so we had to emulate this over Xen via the Xen store. And we further reduced boot times as well. We started with something like one second, and we get it down to 30 milliseconds per time. And the most important of the contributions are actually driving 10 gigabytes for almost all packet sizes. So in these VMs, what matters is not bandwidth, but the ability to process really, really high packet rates. So the talk will be focused on this last bullet point, but let me just give you one slide of what consists of the toolchain and the tool stack. We compile a new lib, which is a slightly modest libc. It's normally used for embedded systems, and it's already quite old, that version, and we compile as well LDOIP as our networking stack, and some other utilities in order to compile click to Minimus. We also use this toolchain to build, for example, IPerf and the fast-package generator based on Netmap called Packagen. We expect to actually port other applications on top of it. Our tool stack is based... We have our own tool stack. We use XCP, Excel, or SAPI on Zen. We design our own Cosmos tool stack, which is basically using the Zen libraries, and we use SWEAK to generate the bindings for our tool stack. For now, we support Python, JavaScript, and OCaml. It just compiles for now. The optimizations were... It was not exactly... Optimization was like a learning process, but most of the contributions of having slow boot times were preferring OZM-STOR-D, the new OZM-STOR implementation shipped in 4.2. I'm not wrong. A different network attach. The hot-blocks scripts really slow things down, like 200 milliseconds, and further booting uncompressed images as opposed to gizypt image. Actually, uncompressing the image takes longer than booting the VM itself. So that's it for the control plane. For this talk, it's about how we actually improve this, and our main objective is to drive these packet rates. So 14.88 million packets per second for minimum-sized packets. For reference, TCP-Hack is a minimum-sized packet. Or for bigger-sized packets, 810,000 packets per second. So we did the performance analysis of this, the whole pipe. And normally we have our network driver attached to an open-v switch, all in a spridge, and a VIF for each VM, which is managed by NetBack. Then we have the shared memory, the fold used with the ZenRing API in event channels to exchange notifications, our net front, and our quick environment. Additionally, we add two elements, so for our network interface, which is net front, and so that's the configurations that actually full packets and so on. So when we plug all of this together, there were several bottlenecks. First, the Linux bridge, just the disclaimer, numbers change all the time, so these are actually a bit better in open-v switch, but a long time ago when we did this analysis and started working on improvements, it was something like this. For maximum-sized packets, we were able to forward more than 300,000 packets per second when we placed NetBack in 350. And with the whole thing together, just roughly 225,000 packets per second, which corresponds to two to three gigs of throughput. So there are a lot of issues on this pipe, and first of them all is the bridge that we use. Afterwards, it relies on TCP, the Linux was stuck, which is not exactly... It drives 10 gigs, but not exactly with really high packet rates. Each time you want to send a packet, you copy that. You need to copy the page to the backend domain, and this copy, although it's the net batch, is really, really expensive. Relying on SK Buffs, it uses the Linux stack, the allocation and manipulation of these socket buffers are really expensive. And also, Minio S backend and front-end driver is not exactly as major as Linux, it's pretty functional, so it's also slow. Roughly two times as performance than Linux. And Rx was a mere 100 megabits of performance when we get it. So the starters, we started not healing the whole backend and front-end, and started doing small optimizations on the backend driver, and we started by replacing the switch called Valle. I will talk a bit about Valle and Netmap right after. We did slight modifications to ZenNetback to support multi-page rings. This is work basically apported to Netback, but it's work from Wangliu. And because we are relying on Valle, it doesn't use STVs. We removed that packet metadata manipulation, and overall, the results were actually nice. We get the three times better performance and 1.2 million packets per second for minimum-sized packets. So before telling the very last piece, let me give you just a background on Netmap. Netmap, available in free BST and Linux as an adopted rebuild, is a fast packet-tile framework that drives line rates for 9 MHz CPU downclocked to 9 MHz. It roughly corresponds to 67 nanoseconds per packet to fill up the 10 gigabits with minimum-sized packets. It requires changes on the device or on the device driver, but these are minimal changes as to support the Netmap mode. But all the NIC registers and physical memory addresses and the packet descriptors are not exposed to user space in any way. These must be validated by the kernel when you pull for packets and thus not accessed by user space. It's similar to what Netchannel, in the previous presentation, where you map the software ring into user space and you use all the data structures, copy packets, and basically pull and flushes out, bypassing the Linux host stack. Vale on the other side was born like a small extension of the Netmap and its software switch that drives between virtual ports, around 18 mpps. Lately, the latest release is like 22 mpps. The graph on top shows a comparison of all the three switches, the free BST bridge, open-v-switch, and Vale. But just a small detail, the Vale case is between virtual ports, whereas the open-v-switch and bridge are between two NICs. So we did a number of extensions to Vale to let us attach NICs to the switch, to modularize the switch, which means you could implement your own switching function and having your Rokinet modules that extends the switch. And basically the way you use the switch is like a normal Netmap API, not a normal application using Netmap API. So plugging all of this together, we got free of the Netpack in Netfront and we implemented one of our own, whereas it's a really small Netpack that tries to mimic what the kernel does to the application. We removed the extra copy, which means the buffers are granted to the VM. The ring protocol is not the same as in Xen, like you have a request and expect an IO response. You copy back to the ring, notify the backends and you forget. Event channels are used to proxy the poll. So when you send a packet, you need to poll in the user space, you need to poll the ring to send the packets out. And in this case, we emulate this with event channels. But the biggest problem with this is that we break Linux and other frontends. But to show that these optimizations are viable, we also implemented Linux as a compliance frontend. We eliminated the copy, so I thought it would be good to mention how much memory do we share. Netmap buffers are allocated in contiguous pages, and buffers are around 2K, which means for the normal ring size of Netmap, which is 1024, you share a total of 2MB per ring, as of Netmap API version 3. We provide different ring sizes, depending on the throughput our CLICK or SVM or Linux requires, you require a smaller batch. So basically, the way Netmap works, you basically issue IOCTL to open the Netmap device. You open the Netmap device and you do an IOCTL to register the interface to Netmap, which detaches the Linux from the Linux host stack. Then our backend will grant the frontend the ring and the buffers, including the grant references of the ring, of the buffers inside the ring slots. The frontend on the other side grabs the grants regarding the ring and checks the reach of the slots in order to map the rest of the buffers. So to run our CLICK or SVM, the minimum memory requirement is for 64 ring slots. It's 6MB in total. And we drive 10 gigs with this ring size as well. So let me just show you some performance evaluation of our system. I told you that our VMs boot in 30 milliseconds, and this is when we boot for 100, how does the boot times evolve. So you ask yourselves why we want 300 milliseconds boot time. The main motivation is that you are able to react as the traffic which is reached a peak. And you can expand the number of filter machines in other machines in order to cope with higher rates. When I say higher rates, like 10, 20, 30, 40 gigs, for example. So the time to create the VM evolves from 22 milliseconds up to 220 milliseconds. The time to instantiate CLICK configuration takes around between 5 to 20 milliseconds. Regarding the packet performance, we got huge improvements. And we basically achieved around 9 million packets per second for receive sites. And for TX, we achieved 1438 million packets per second. Which is 95% of the line-line from VM size packets. This all the way changed a little bit on the latest Netmap API version, which we are down to 11, 12. And you can see that as we decrease the ring size, we decrease the throughput as expected. The cost of event channel starts to be a bit more visible. But as you can see, all the other packet sizes, we fill up the pipe. This was tested on a low-end server of 4 cores at 3.2 gigahertz. We do a port 10 gigabit nicks. Each core is assigned one CPU core for the VM, the rest for DOM0. So after getting 10 gigs, we ask ourselves how we would scale for more nicks. So we actually run some experiments to see if we were able to drive 40 up to 80 gigs of throughput. And we actually did. So the bars are TX only, just VMs doing a packet generation. And the lines represent forwarding. So VMs actually processing packets and forwarding back to the outside world. And we achieved roughly 30 gigs for 3 ports. And this is only the rate you receive. So remind that it fits full duplex. It's the double of the performance, roughly. So if we manage TX with RX, it's actually 60 gigs around it. And we don't exactly scale that well for minimum size packets over multiple nicks. Again, this was one of the same setup as the previous experiments. We assigned 3 cores for DOM0 and 3 cores for the VMs. Each of the cores has all the nicks assigned to it. Otherwise, the nicks starts to starve and they're not able to process all the packets. So there was a bit fine grained tuning on the interrupts. So the Linux guest performance also gains some huge improvements. But the disclaimer, this is not exactly a fair comparison. The highest bars are using Netmap API, which Xan and KVM don't support it. And you still get all the bottlenecks of the host TCP IPO stack. So this is actually my favorite graph, when we actually run middleboxes on ClickOS. So the first one is just packet forwarding. So it doesn't touch the packets. The second group of bars is an Ethernet mirror. Basically, it changes source and destination MAC address, forwards back the packet. The third one is a standard compliant IP router. The fourth is a firewall loaded with 10 rules. The card grade nut, which has, I think, 10 flows with randomized ports. Software BRAS, which for those who don't know, it's the first hop of the DSL subscriber. And it does the PPP termination, IPLCP handling, and session management as well. A load balancer, a flow monitor that gather statistics about the flow, and intrusion detection system, just checking the contents of the packet with just five rules. After that, since this is a middlebox, it's not a hand system, it's actually important to have really low delay. And our ClickOS machines adds just 40 microseconds of delay to the whole traffic. And we compare it with other systems, and we almost cope with DOM zeroes delay. And we did the best of our efforts to put the KVM setup the lowest possible. And we got also huge improvement compared to the Linux DOM new delay. So this is ClickOS. A really tiny virtual machine to drive to do packet processing. We drive 10 gigs. I will show you a demo just right after the presentation. I will be at the Zen booth tomorrow between 10 and 11. So feel free to go there, and I ask any questions. I will show you more demos. And in the future work, we are exploring the performance on numerous systems, which is not exactly that good. Doing high consolidation of these VMs, something like running 1000, 2000, 2000 VMs. We did some experiments in already two 2K VMs on a single machine. And doing service training of these virtual machines, like training these network functions with each other to do more packet operations. Last and not least, in the Zen Summit, we thought that we would be open source, but we weren't clear how it would take. There is a presentation on NSDI 2014 about ClickOS, and we will be open sourcing on that time, which is April 4th. And that comprises patches to MinioS, requests for comments on the backend and front end drivers, and more things to come. So that's it. Hope you guys like it. Any questions? Thank you. We had a picture with the queue from different user nets and so on. The configuration is pretty simple. It just decreases the TTL on the ICMP request. It's just to visualize how the configuration works. It just decreases TTL and forwards back the packet. Paint is just a click internal thing to mark the packet, if you see it said again. Yeah, yeah, yeah, yeah, yeah. Not actually, it's something on the packet. It's like a drawback for this. Okay, he was asking what the configuration does with TTL and what paint means on Click. Excuse me. Yeah, yeah. This is for just one VM. Yeah. The modifications that you made for the send, for example, the switch, how hard it is for someone to run that and put it into the new standardization. Yeah, you run on patching the send or it's going to be out from practical point of view, how can you start using this? Actually, right now, it's on Phoebe S10. It's already on Phoebe S10. Yeah, exactly. So Netmap is normally referred to next. Vale is a switch, so they'll kind of work together and do the same API. So the API is really quite simple. If you go to, so this is from Luigi Rizzo. And we made some contributions which are already on Phoebe S10, so you can attach a nick, you can attach the host stack, you can extend the switch with your own switching functions, and all of that is already on Phoebe S10. Any more questions? Okay. Yeah, yeah, yeah, yeah. The rates I'm showing here, so for example, the middle boxes, it's actually, for example, that's a top. So the middle boxes on one, the send machine, one of them and two endos with normal TCP, Linux tags in order. All right. Yeah, yeah, yeah. So I'm not sure if I still have nine minutes, and I would like to show you a demo. So this demo, it's, so the resolution is not exactly quite good. But this is just to show how the, just to show the boot times. And we, so basically you have the create VM over there and Cosmos is very similar to itself, for example. And basically create the VM over here, and we start the click configuration. And let's see how it does. It basically will boot 100 VMs. Now on the other machine, they are connected back to back. We just think the VMs to see if they're actually working. So in Sennar thing, so it goes to every single VM and every single VM runs a single iteration, answers the same thing. So that's it. There are more demos like on demand 10 gigs, single HTTP transfer with the VM booting on demand and can seal all that. Just go to the booth. I'll be there at lunchtime in 10 to 11. So that's it. My name is John. I have my machine and they have like class and they have this class. I'm not exactly sure. Okay, you put it on when you speak in, I'll go up. I think I configured it already. Okay, no problem. What I need to do? Just talk it. I'll go up. Okay. Start talking. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. I do the timing for this one. If you occasionally look up, I'll give you 10 minutes. Okay. I have a clock. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Okay. Can you hear me? Okay. We'll talk about the problem of scheduling a little bit. Then we'll go ahead and talk about Nova filter schedule concepts. And then we'll deep dive into overt scheduling, giving a lot of samples. Think the best way to understand stuff are through samples. OK, can you raise your hand if you heard about overt? Yeah, I guess. A lot of you are. It's a home crowd. So for those of you who don't know what is overt, overt is a management platform for VMs based on KVMI providers. Can handle thousands of VMs, has live snapshot, live storage migration, live VM migration, everything is live, basically. And sorry. We support also advanced network configuration for those those and a lot of storage connections, sun, NFS, cluster, et cetera. Basically, let's see what we've got a long time ago. We got the questions from the user's list. I urge you to use that list. If you have any questions about overt, it's quite active. How can I define a maximum number of running VMs per host? It's pretty trivial, but we didn't support it that yet then. So we get back to it later. I just wanted to say that you should use the user's list. OK, what we had in overt a long time ago, basically we had two distribution algorithms for running and migrating VMs. When we selected a VM, when we ran a VM, then we selected a host based on CPU load policies, either even distribution CPU load or power saving. And then it's pretty limited. We have only two distribution algorithms, and we can't construct a user defined one. Basically, that is it. Then we take a look at the Nova scheduler, which brought us filters and weights. Very simple and easy way to schedule VM on hosts. Basically, the filters are very cut and clear logic that gets a set of hosts on the left-hand side. Then we run a filter on them. Then we apply weights on top of them. Another cool slide from Nova scheduler. We collect a set of weights, then we sum them up, and then we order the host that we got. Let's see a simple sample for Nova scheduler. This is a RAM filter written in Python. Basically, it's a very simple method that gets a single host data and then a set of properties, and then run a really simple code that either, decide if this host is in the scheduling process or excluding it from the scheduling process according to the requested RAM for that VM. When we looked at that and took it to the data center virtualization, which is over it, we saw that we have a little bit of problem with that concept. Because in Nova, each filter and weight is applied on a single host, and we have a larger concept of clustering in over it, which is called migration domain. So in the migration domain, each VM on a host can be migrated to a separate host on that migration domain, that cluster. And we have another concept of load balancing for that cluster. And also a policy container, which a user can decide and create some policy and apply that on that cluster. So let's take a look what we had in over it. Basically, we can create internal and external filter and weights. The internal part is used for better performance. We are within the server, so we get a quick access to the DB. And originally, all the filter and weights from the previous over back then when we didn't have filters and weight were migrated into that internal scheduler. And the external, basically, all the users can support and extend. We apply all the filters, we apply the filter and weights on all the hosts in the cluster for better performance. And we want to have a better grasp of how the cluster behaves. We have containers called cluster policies. For each cluster, we can define the set of filters, set of weights, and a single load balancing for each cluster. And then we support custom properties that you can pass through to the filter and weight. This is a diagram of the new model. On the left-hand side, we can see a set of hosts within the cluster. Then we apply each filter, chaining each filter, one on top of the other. And then we construct a weight table, which gets us the result of the selected host that we want to schedule the VM on. We had the same concept as Nova and filters. The existing logic that we had previously, which were basically validation, was emigrated into internal filters. And we can extend it in Python using the external scheduler that I will get into later. One tissue show you a really easy sample of how we can use filters. Basically, this is a filter in Python. The name of the class will be the name that the server gets. The properties validation are basically a set of property that you can add within the server. And then the filter, we get it. This is the name of the filter. And this is actually the signature that you need to override in order to extend a filter, to add a filter to the system. This is how you get the custom properties within the filter. Basically, I didn't tell you what the filter is all about. But here you can see we get the time. And if the time is within the interval that we get from the user, then we print the host. We turned all the hosts that we got. If not, we just remove, extrude all the hosts from the filter. That's kind of a bank example. Let's talk about the weights. It weights all the hosts that pass through the filters. We have the predefined weights that first two are CPU load filters. And then in 3.4, we added a lot more weight models. I said filter weight models. It's kind of easy now to add because we have the new architecture. Each filter can have factors. We can prioritize the filters. Each weight can have factors. We can prioritize the weights using factors. We can also add external weights. Let's see another sample. In this sample, we use a connection to the server using Python SDK that we have. This SDK is backward compatible and stable. We connect to the SDK. And then the logic of the weight is basically we iterate over all the hosts. And we append to a list a tuple of the host ID and the weight of that host. Here it's a little bit cut, but the weight of the host is the number of active VMs on that host. So if we have a host, basically it would be ordered by number of running VMs within that host. Let's continue. Talk about the load balancing. That's the third model we have in the cluster policy. Each cluster policy can have only one load balancing logic. Basically, you can do whatever you want within the load balancing, connect to the SDK, and you can shut down everything basically. But what we support internally is that load balancing logic will return a VM and a set of hosts. I will show it in a sample later on. And we will migrate that VM, a single VM, within the server. It's basically more safe to migrate a single VM within a period of time, not to cause a migration rush. And I don't know where. Use all our resources for that. It's pretty not safe to do that. We also have a pretty fine load balancing within Overt. The two legacy ones on CPU and now added in 3.4, even distribution one, even VM distribution that we didn't have. The same balancing sample that I want to show, as I showed earlier, is we get the same numbers that we want to shut down all the VMs. But in this example, we will actually do that and not exclude hosts that prevent users from running hosts, VMs like in the filter example. Here we will show how in that hour in the wake up hour, we will basically get all the hosts. And if it's wake up hour, we will activate all the hosts. If we need to sleep, then we will connect using our SDK and get all the VMs from that host. We will stop all the VMs and we'll deactivate the hosts. Same bank example. This is how we use it internally to migrate VMs. You get, according to some logic, you get the overloaded host. It's called snippet. Then you select here it's random, the first VM on that host. Then we actually print it because we're using a CDIO to get the data from the model. And we return it and we return a right listed host, which is kind of a filter. The first filtering we do before we activate the filter and weights and the normal process that we do. Basically, as I said, we have a cluster policy, which is a container for all the filters and weights and a single load balancing logic. And we have two optimizations for a cluster policy, speed and overbooking. Basically, we run each time we schedule a VM, we run it one by one because we want to prevent overbooking. We want to guarantee the same resources for each VM. If we will try to schedule two VMs together, we can fail because they both see the same resources. So basically, the speed optimization is to exclude the weight of the host. So later on, the load balancing will do the weighing for us and balance out the cluster for us. And the overbooking is to emit the, I don't know, just be able to paralyze the scheduling process. Let's see if we can show those pictures within Overt. OK. OK, this is Overt. The VM is because of Wi-Fi and VPN. The VM is unknown. It's running somewhere. Here, I configure a new cluster policy. Can give it a name, like the shutdown one. OK. I can add the external filter that I've added to the system, the shutdown host filter to the enabled filters. As a weight, I will give optimal for power saving. Let's try to aggregate all the VMs within a single host as much as possible. Then I will select the balance that I created earlier. I can give it wake up power, like 8 AM. And shutdown our 8. Press OK. Should be created. OK, I don't know what happened. Second. What? OK, let's go back to the. No, no. It shouldn't take some time. What? Yeah, I'm not connected to the VPN. No, no. Something happened to the VPN. It doesn't like me. Yeah, I have screenshots. But you know, I don't know. But you know, maybe it will work second. OK, let's take a look at the screen. It works, believe me. OK, we were. We were here. Wait a second. No, still no. Never mind. We'll show you that one. Then you go to cluster. No, still not working. OK, forget about it. OK, I create the cluster policy that I've showed you. Then I created to a cluster that I already defined. So all the hosts within that cluster, we are covering to that cluster policy. OK, let's talk about how it's implemented in the back. It's disabled by default. Whoever wants to extend to add filters should be able to know how to install the external scheduler. The external scheduler is a separate process written in Python. We externalize it because we want to guarantee the engine's safety. You know, if a user writes a code, it can be dangerous to the system. We want to allow other languages as well. If you know, the end is written in Java. So in this model is written in Python. And going forward, we want to support SAS, which is kind of a scheduling as a service. It's a separate RPM. You need to manually install it. How it works. Basically, it's initialized. It reads from a local directory all the filters, weights, and balancing logic that you wrote. Then it's published in an internal API. The engine, the server reads it. And then it waits for calls from the engine for filtering, weighing, and balancing. This is how it looks like when it's loaded, the filter, and load balancing here. Back to the users list. Now we can easily write a filter that maximizes the number of running VMs per host. Early. To sum it up, we support easy Python plugins for you to use for VM scheduling. You can manage a separate policy for each cluster, for each group of hosts. And every version that comes out for over gets new models for scheduling. Question? Yes. So the question is, I have the possibility to read from what the hypervisor provides, memory utilization, CPU utilization. Since you have a Python extensive API, if I got it right, you may have the possibility to know if the storage behind or the storage framework behind is constrained. So you have the possibility to say, all right, do a storage migration. This virtual machine is to the other storage, which is SSD base or whatever. So you just need, for example, to have a good talk with all the storage big guys and find more things about their IOPs and what we are doing for their storage to get that information. Basically, because we have the memory, we have the CPU, maybe we have some network information about throughputs and stuff. So what's left behind is the IOPs and the quality of the storage. You can think of whatever. Did you all hear the question? I think within when you extend, you can do whatever you want when you extend filter or something like that. So what you can do is you can extend whatever you want when you extend filter or when you basically can connect to every provider or use any SDK that you like. So what you can provide within the engine is all that you ask, memory and CPU load. It's your own choice. Do you have an answer? We have a sample of guys in the bar below talking about connecting X. Hi. So we had a few guys in the overt workshop in Bagelot, and they were asking us to connect the scheduler to a BMC system that is monitoring extra parameters. For example, they can monitor the CPU temperature and the fan speed, and they can actually predict that if the fan speed is constant or zero and the CPU temperature is high, that host is going to crash and burn in a few minutes. So what they asked from us is to give them a list of hosts, and they can actually blacklist some of them because they are aware of more information than what OVIR should have. And there are so many other examples which are very similar. For example, Cisco has very similar concepts. They actually want to blacklist some of the hosts because the networking is going to go down there. There are a lot of very similar scenarios in very big enterprises. This is actually highly wanted. Actually, it sounds promising. Yeah. So now you have the power. You can actually do it yourself. Well, my first single question was, all right, I know about my CPU, my memory. Maybe I know about my networking because we do the networking. So the next good thing is if I had the storage IEO information or information that had to do with the quality of my storage, perhaps I could utilize multi-tiering in a storage or storage in a virtual machine from one place to the other. That could be nice with this engine. So the endless space of this information is not a big deal. That's a lot of cool. There's no properly made for that. So everyone can actually interact with these old new space and have a good experience. So I would like to add that sometimes some of you have to buy the network. You can't really do that. No questions? Thank you, Danali. Go drink a beer, Anishin. Thank you. Thank you. Thank you.
 I come from the NEC Labs research, the research labs in Germany.  And this I'll be presenting in Clic OS and this is work done by me and my folks at NEC,  as well as some other people at the University of Bucharest.  So in one sentence, Clic OS is a VM to do network processing.  So what's the motivation behind it?  A long time ago, some folks designed the Internet and how does it look like as we are thought in schools?  And those with a natural routine containing switches and routers. Pretty simple.  But in reality, middle boxes are a commonplace in today's networks.  They are very useful for a high number of reasons and they extend operators' sniper for a lot of futures.  Security, such as firewalls, intrusion and text systems.  Monitoring, such as DPI boxes.  Load balancers, dealing with the addresses, exhaustion issues.  Performance, such as media gateways, transcorders, wide air accelerators.  And some others of more dubious nature, like the advertisement session box over there.  I'm having some issues with the display port so you cannot see exactly how this looks like.  But these middle boxes are really useful.  But there are a lot of disadvantages at the same time.  First of all, they are really, really expensive.  One of the boxes I showed you, like the media gateway, costs around 200k.  It's really difficult to add new features.  Often you get locked in to vendors to upgrade your firmware, add new features, and so on and so forth.  They are difficult to manage. It often requires specialized personnel to manage the box.  And normally you deploy these boxes to keep with the peak rate,  so they cannot be scaled like you add more like you want.  And it's difficult to share these devices among other parties.  And since it's hardware market, it's hard to... new players come along.  You know, it's hardware.  So clearly if we had some way to virtualize these middle boxes or  shift these middle boxes to be software instead of expensive boxes,  we would have... we mostly would solve most of these issues we have here.  But it is still not clear if we can do it in software  and really coping with the performance that the hardware middle boxes can achieve.  So we believe so, and that's what I'm talking about here today,  and that's what we propose with ClickOS.  ClickOS is a mini-OS-based Zen virtual machine that runs on top...  that runs ClickModeralRouter.  So let me first talk a little bit about Click.  Click is a framework to design routers and process packets,  and has this concept of elements.  Each of these elements does an operation to the packets or receive packets.  For example, the configuration on the side basically decreases CTL  and forwards back the packet to a different interface.  These elements are contained in a configuration,  and basically you install it either as a user space application  or as a kernel module.  These elements are exposed to their counters.  If we have a counter on an interface, for example,  via variables on the slash proc file system,  and for example, the counter, you have a count element to just read only  and to reset the counter, you just write one.  For example, for ClickOS, we compile 262 of these elements,  excluding the ones that deal with the file system.  And if you want to further extend the click,  you can just write new elements.  It's quite easy.  So an example of a click configuration looks something like that.  You have from Netron to receive packets,  into Netron to send packets out.  You have an IP filter element that allows packets from 10.01  in destination 10.01, UDP packets, otherwise drops it all.  So you can see that it's quite easy just to filter packets, for example.  So what's ClickOS?  Typically, we have in Xen, our per-visualized operating system  is slightly modified to work on top of Xen,  and the kernel and the applications are in on top.  In the case of ClickOS, we use Minus as our best operating system,  and we just run click, and we call the whole thing ClickOS.  So it kind of follows the trend of OSV in the previous presentation,  or Erlang on Xen, Mirage,  so that it's application-specific operating systems.  So the work we did was having a build system to build kind of these Minus apps  that we used to build ClickOS,  and what we get is a 5 megabytes image in total, in size,  and we had to emulate the control plane,  so we don't have a slash-proc file system between VMs,  so we had to emulate this over Xen via the Xen store.  And we further reduced boot times as well.  We started with something like one second,  and we get it down to 30 milliseconds per time.  And the most important of the contributions  are actually driving 10 gigabytes for almost all packet sizes.  So in these VMs, what matters is not bandwidth,  but the ability to process really, really high packet rates.  So the talk will be focused on this last bullet point,  but let me just give you one slide of what consists of the toolchain and the tool stack.  We compile a new lib, which is a slightly modest libc.  It's normally used for embedded systems,  and it's already quite old, that version,  and we compile as well LDOIP as our networking stack,  and some other utilities in order to compile click to Minimus.  We also use this toolchain to build, for example,  IPerf and the fast-package generator based on Netmap called Packagen.  We expect to actually port other applications on top of it.  Our tool stack is based...  We have our own tool stack.  We use XCP, Excel, or SAPI on Zen.  We design our own Cosmos tool stack,  which is basically using the Zen libraries,  and we use SWEAK to generate the bindings for our tool stack.  For now, we support Python, JavaScript, and OCaml.  It just compiles for now.  The optimizations were...  It was not exactly... Optimization was like a learning process,  but most of the contributions of having slow boot times  were preferring OZM-STOR-D,  the new OZM-STOR implementation shipped in 4.2.  I'm not wrong.  A different network attach.  The hot-blocks scripts really slow things down,  like 200 milliseconds,  and further booting uncompressed images as opposed to gizypt image.  Actually, uncompressing the image takes longer than booting the VM itself.  So that's it for the control plane.  For this talk, it's about how we actually improve this,  and our main objective is to drive these packet rates.  So 14.88 million packets per second for minimum-sized packets.  For reference, TCP-Hack is a minimum-sized packet.  Or for bigger-sized packets, 810,000 packets per second.  So we did the performance analysis of this, the whole pipe.  And normally we have our network driver attached to an open-v switch,  all in a spridge, and a VIF for each VM, which is managed by NetBack.  Then we have the shared memory, the fold used with the ZenRing API  in event channels to exchange notifications, our net front,  and our quick environment.  Additionally, we add two elements, so for our network interface,  which is net front,  and so that's the configurations that actually full packets and so on.  So when we plug all of this together, there were several bottlenecks.  First, the Linux bridge, just the disclaimer, numbers change all the time,  so these are actually a bit better in open-v switch,  but a long time ago when we did this analysis and started working on improvements,  it was something like this.  For maximum-sized packets, we were able to forward more than 300,000 packets per second  when we placed NetBack in 350.  And with the whole thing together, just roughly 225,000 packets per second,  which corresponds to two to three gigs of throughput.  So there are a lot of issues on this pipe,  and first of them all is the bridge that we use.  Afterwards, it relies on TCP, the Linux was stuck, which is not exactly...  It drives 10 gigs, but not exactly with really high packet rates.  Each time you want to send a packet, you copy that.  You need to copy the page to the backend domain,  and this copy, although it's the net batch, is really, really expensive.  Relying on SK Buffs, it uses the Linux stack,  the allocation and manipulation of these socket buffers are really expensive.  And also, Minio S backend and front-end driver is not exactly as major as Linux,  it's pretty functional, so it's also slow.  Roughly two times as performance than Linux.  And Rx was a mere 100 megabits of performance when we get it.  So the starters, we started not healing the whole backend and front-end,  and started doing small optimizations on the backend driver,  and we started by replacing the switch called Valle.  I will talk a bit about Valle and Netmap right after.  We did slight modifications to ZenNetback to support multi-page rings.  This is work basically apported to Netback,  but it's work from Wangliu.  And because we are relying on Valle, it doesn't use STVs.  We removed that packet metadata manipulation,  and overall, the results were actually nice.  We get the three times better performance  and 1.2 million packets per second for minimum-sized packets.  So before telling the very last piece,  let me give you just a background on Netmap.  Netmap, available in free BST and Linux as an adopted rebuild,  is a fast packet-tile framework that drives line rates  for 9 MHz CPU downclocked to 9 MHz.  It roughly corresponds to 67 nanoseconds per packet  to fill up the 10 gigabits with minimum-sized packets.  It requires changes on the device or on the device driver,  but these are minimal changes as to support the Netmap mode.  But all the NIC registers and physical memory addresses  and the packet descriptors are not exposed to user space in any way.  These must be validated by the kernel when you pull for packets  and thus not accessed by user space.  It's similar to what Netchannel, in the previous presentation,  where you map the software ring into user space  and you use all the data structures, copy packets,  and basically pull and flushes out, bypassing the Linux host stack.  Vale on the other side was born like a small extension  of the Netmap and its software switch that drives between virtual ports,  around 18 mpps.  Lately, the latest release is like 22 mpps.  The graph on top shows a comparison of all the three switches,  the free BST bridge, open-v-switch, and Vale.  But just a small detail, the Vale case is between virtual ports,  whereas the open-v-switch and bridge are between two NICs.  So we did a number of extensions to Vale  to let us attach NICs to the switch,  to modularize the switch, which means you could implement  your own switching function  and having your Rokinet modules that extends the switch.  And basically the way you use the switch is like a normal Netmap API,  not a normal application using Netmap API.  So plugging all of this together,  we got free of the Netpack in Netfront  and we implemented one of our own,  whereas it's a really small Netpack  that tries to mimic what the kernel does to the application.  We removed the extra copy,  which means the buffers are granted to the VM.  The ring protocol is not the same as in Xen,  like you have a request and expect an IO response.  You copy back to the ring,  notify the backends and you forget.  Event channels are used to proxy the poll.  So when you send a packet, you need to poll in the user space,  you need to poll the ring to send the packets out.  And in this case, we emulate this with event channels.  But the biggest problem with this is that we break Linux and other frontends.  But to show that these optimizations are viable,  we also implemented Linux as a compliance frontend.  We eliminated the copy,  so I thought it would be good to mention how much memory do we share.  Netmap buffers are allocated in contiguous pages,  and buffers are around 2K,  which means for the normal ring size of Netmap,  which is 1024, you share a total of 2MB per ring,  as of Netmap API version 3.  We provide different ring sizes,  depending on the throughput our CLICK or SVM or Linux requires,  you require a smaller batch.  So basically, the way Netmap works,  you basically issue IOCTL to open the Netmap device.  You open the Netmap device and you do an IOCTL to register the interface  to Netmap, which detaches the Linux from the Linux host stack.  Then our backend will grant the frontend the ring and the buffers,  including the grant references of the ring,  of the buffers inside the ring slots.  The frontend on the other side grabs the grants regarding the ring  and checks the reach of the slots in order to map the rest of the buffers.  So to run our CLICK or SVM,  the minimum memory requirement is for 64 ring slots.  It's 6MB in total.  And we drive 10 gigs with this ring size as well.  So let me just show you some performance evaluation of our system.  I told you that our VMs boot in 30 milliseconds,  and this is when we boot for 100, how does the boot times evolve.  So you ask yourselves why we want 300 milliseconds boot time.  The main motivation is that you are able to react as the traffic  which is reached a peak.  And you can expand the number of filter machines in other machines  in order to cope with higher rates.  When I say higher rates, like 10, 20, 30, 40 gigs, for example.  So the time to create the VM evolves from 22 milliseconds up to 220 milliseconds.  The time to instantiate CLICK configuration takes around between 5 to 20 milliseconds.  Regarding the packet performance, we got huge improvements.  And we basically achieved around 9 million packets per second for receive sites.  And for TX, we achieved 1438 million packets per second.  Which is 95% of the line-line from VM size packets.  This all the way changed a little bit on the latest Netmap API version,  which we are down to 11, 12.  And you can see that as we decrease the ring size, we decrease the throughput as expected.  The cost of event channel starts to be a bit more visible.  But as you can see, all the other packet sizes, we fill up the pipe.  This was tested on a low-end server of 4 cores at 3.2 gigahertz.  We do a port 10 gigabit nicks.  Each core is assigned one CPU core for the VM, the rest for DOM0.  So after getting 10 gigs, we ask ourselves how we would scale for more nicks.  So we actually run some experiments to see if we were able to drive 40 up to 80 gigs of throughput.  And we actually did.  So the bars are TX only, just VMs doing a packet generation.  And the lines represent forwarding.  So VMs actually processing packets and forwarding back to the outside world.  And we achieved roughly 30 gigs for 3 ports.  And this is only the rate you receive.  So remind that it fits full duplex.  It's the double of the performance, roughly.  So if we manage TX with RX, it's actually 60 gigs around it.  And we don't exactly scale that well for minimum size packets over multiple nicks.  Again, this was one of the same setup as the previous experiments.  We assigned 3 cores for DOM0 and 3 cores for the VMs.  Each of the cores has all the nicks assigned to it.  Otherwise, the nicks starts to starve and they're not able to process all the packets.  So there was a bit fine grained tuning on the interrupts.  So the Linux guest performance also gains some huge improvements.  But the disclaimer, this is not exactly a fair comparison.  The highest bars are using Netmap API, which Xan and KVM don't support it.  And you still get all the bottlenecks of the host TCP IPO stack.  So this is actually my favorite graph, when we actually run middleboxes on ClickOS.  So the first one is just packet forwarding.  So it doesn't touch the packets.  The second group of bars is an Ethernet mirror.  Basically, it changes source and destination MAC address, forwards back the packet.  The third one is a standard compliant IP router.  The fourth is a firewall loaded with 10 rules.  The card grade nut, which has, I think, 10 flows with randomized ports.  Software BRAS, which for those who don't know, it's the first hop of the DSL subscriber.  And it does the PPP termination, IPLCP handling, and session management as well.  A load balancer, a flow monitor that gather statistics about the flow,  and intrusion detection system, just checking the contents of the packet with just five rules.  After that, since this is a middlebox, it's not a hand system,  it's actually important to have really low delay.  And our ClickOS machines adds just 40 microseconds of delay to the whole traffic.  And we compare it with other systems, and we almost cope with DOM zeroes delay.  And we did the best of our efforts to put the KVM setup the lowest possible.  And we got also huge improvement compared to the Linux DOM new delay.  So this is ClickOS.  A really tiny virtual machine to drive to do packet processing.  We drive 10 gigs.  I will show you a demo just right after the presentation.  I will be at the Zen booth tomorrow between 10 and 11.  So feel free to go there, and I ask any questions.  I will show you more demos.  And in the future work, we are exploring the performance on numerous systems,  which is not exactly that good.  Doing high consolidation of these VMs, something like running 1000, 2000, 2000 VMs.  We did some experiments in already two 2K VMs on a single machine.  And doing service training of these virtual machines, like training these network functions with each other  to do more packet operations.  Last and not least, in the Zen Summit, we thought that we would be open source,  but we weren't clear how it would take.  There is a presentation on NSDI 2014 about ClickOS,  and we will be open sourcing on that time, which is April 4th.  And that comprises patches to MinioS, requests for comments on the backend and front end drivers,  and more things to come.  So that's it. Hope you guys like it. Any questions?  Thank you.  We had a picture with the queue from different user nets and so on.  The configuration is pretty simple.  It just decreases the TTL on the ICMP request.  It's just to visualize how the configuration works.  It just decreases TTL and forwards back the packet.  Paint is just a click internal thing to mark the packet, if you see it said again.  Yeah, yeah, yeah, yeah, yeah.  Not actually, it's something on the packet.  It's like a drawback for this.  Okay, he was asking what the configuration does with TTL and what paint means on Click.  Excuse me.  Yeah, yeah.  This is for just one VM. Yeah.  The modifications that you made for the send, for example, the switch, how hard it is for someone to run that  and put it into the new standardization.  Yeah, you run on patching the send or it's going to be out from practical point of view, how can you start using this?  Actually, right now, it's on Phoebe S10.  It's already on Phoebe S10.  Yeah, exactly.  So Netmap is normally referred to next.  Vale is a switch, so they'll kind of work together and do the same API.  So the API is really quite simple.  If you go to, so this is from Luigi Rizzo.  And we made some contributions which are already on Phoebe S10, so you can attach a nick, you can attach the host stack,  you can extend the switch with your own switching functions, and all of that is already on Phoebe S10.  Any more questions?  Okay.  Yeah, yeah, yeah, yeah.  The rates I'm showing here, so for example, the middle boxes, it's actually, for example, that's a top.  So the middle boxes on one, the send machine, one of them and two endos with normal TCP, Linux tags in order.  All right.  Yeah, yeah, yeah.  So I'm not sure if I still have nine minutes, and I would like to show you a demo.  So this demo, it's, so the resolution is not exactly quite good.  But this is just to show how the, just to show the boot times.  And we, so basically you have the create VM over there and Cosmos is very similar to itself, for example.  And basically create the VM over here, and we start the click configuration.  And let's see how it does.  It basically will boot 100 VMs.  Now on the other machine, they are connected back to back.  We just think the VMs to see if they're actually working.  So in Sennar thing, so it goes to every single VM and every single VM runs a single iteration, answers the same thing.  So that's it.  There are more demos like on demand 10 gigs, single HTTP transfer with the VM booting on demand and can seal all that.  Just go to the booth.  I'll be there at lunchtime in 10 to 11.  So that's it.  My name is John.  I have my machine and they have like class and they have this class.  I'm not exactly sure.  Okay, you put it on when you speak in, I'll go up.  I think I configured it already.  Okay, no problem.  What I need to do?  Just talk it.  I'll go up.  Okay.  Start talking.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  I do the timing for this one.  If you occasionally look up, I'll give you 10 minutes.  Okay.  I have a clock.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Okay.  Can you hear me?  Okay.  We'll talk about the problem of scheduling a little bit.  Then we'll go ahead and talk about Nova filter schedule  concepts.  And then we'll deep dive into overt scheduling,  giving a lot of samples.  Think the best way to understand stuff  are through samples.  OK, can you raise your hand if you heard about overt?  Yeah, I guess.  A lot of you are.  It's a home crowd.  So for those of you who don't know what is overt,  overt is a management platform for VMs based on KVMI  providers.  Can handle thousands of VMs, has live snapshot,  live storage migration, live VM migration,  everything is live, basically.  And sorry.  We support also advanced network configuration  for those those and a lot of storage connections,  sun, NFS, cluster, et cetera.  Basically, let's see what we've got a long time ago.  We got the questions from the user's list.  I urge you to use that list.  If you have any questions about overt, it's quite active.  How can I define a maximum number of running VMs per host?  It's pretty trivial, but we didn't support it that yet then.  So we get back to it later.  I just wanted to say that you should use the user's list.  OK, what we had in overt a long time ago,  basically we had two distribution algorithms  for running and migrating VMs.  When we selected a VM, when we ran a VM,  then we selected a host based on CPU load policies,  either even distribution CPU load or power saving.  And then it's pretty limited.  We have only two distribution algorithms,  and we can't construct a user defined one.  Basically, that is it.  Then we take a look at the Nova scheduler, which  brought us filters and weights.  Very simple and easy way to schedule VM on hosts.  Basically, the filters are very cut and clear logic  that gets a set of hosts on the left-hand side.  Then we run a filter on them.  Then we apply weights on top of them.  Another cool slide from Nova scheduler.  We collect a set of weights, then we sum them up,  and then we order the host that we got.  Let's see a simple sample for Nova scheduler.  This is a RAM filter written in Python.  Basically, it's a very simple method  that gets a single host data and then a set of properties,  and then run a really simple code that either,  decide if this host is in the scheduling process  or excluding it from the scheduling process  according to the requested RAM for that VM.  When we looked at that and took it to the data center  virtualization, which is over it,  we saw that we have a little bit of problem with that concept.  Because in Nova, each filter and weight  is applied on a single host, and we  have a larger concept of clustering  in over it, which is called migration domain.  So in the migration domain, each VM on a host  can be migrated to a separate host on that migration domain,  that cluster.  And we have another concept of load balancing for that cluster.  And also a policy container, which a user can decide  and create some policy and apply that on that cluster.  So let's take a look what we had in over it.  Basically, we can create internal and external filter  and weights.  The internal part is used for better performance.  We are within the server, so we get a quick access to the DB.  And originally, all the filter and weights  from the previous over back then when we didn't have filters  and weight were migrated into that internal scheduler.  And the external, basically, all the users  can support and extend.  We apply all the filters, we apply the filter and weights  on all the hosts in the cluster for better performance.  And we want to have a better grasp of how the cluster behaves.  We have containers called cluster policies.  For each cluster, we can define the set of filters,  set of weights, and a single load balancing for each cluster.  And then we support custom properties  that you can pass through to the filter and weight.  This is a diagram of the new model.  On the left-hand side, we can see a set of hosts  within the cluster.  Then we apply each filter, chaining each filter, one  on top of the other.  And then we construct a weight table,  which gets us the result of the selected host  that we want to schedule the VM on.  We had the same concept as Nova and filters.  The existing logic that we had previously,  which were basically validation, was  emigrated into internal filters.  And we can extend it in Python using the external scheduler  that I will get into later.  One tissue show you a really easy sample  of how we can use filters.  Basically, this is a filter in Python.  The name of the class will be the name  that the server gets.  The properties validation are basically  a set of property that you can add within the server.  And then the filter, we get it.  This is the name of the filter.  And this is actually the signature that you  need to override in order to extend a filter,  to add a filter to the system.  This is how you get the custom properties  within the filter.  Basically, I didn't tell you what the filter is all about.  But here you can see we get the time.  And if the time is within the interval  that we get from the user, then we print the host.  We turned all the hosts that we got.  If not, we just remove, extrude all the hosts from the filter.  That's kind of a bank example.  Let's talk about the weights.  It weights all the hosts that pass through the filters.  We have the predefined weights that first two are CPU load  filters.  And then in 3.4, we added a lot more weight models.  I said filter weight models.  It's kind of easy now to add because we  have the new architecture.  Each filter can have factors.  We can prioritize the filters.  Each weight can have factors.  We can prioritize the weights using factors.  We can also add external weights.  Let's see another sample.  In this sample, we use a connection  to the server using Python SDK that we have.  This SDK is backward compatible and stable.  We connect to the SDK.  And then the logic of the weight is basically  we iterate over all the hosts.  And we append to a list a tuple of the host ID  and the weight of that host.  Here it's a little bit cut, but the weight of the host  is the number of active VMs on that host.  So if we have a host, basically it  would be ordered by number of running VMs within that host.  Let's continue.  Talk about the load balancing.  That's the third model we have in the cluster policy.  Each cluster policy can have only one load balancing logic.  Basically, you can do whatever you want  within the load balancing, connect to the SDK,  and you can shut down everything basically.  But what we support internally is  that load balancing logic will return a VM and a set of hosts.  I will show it in a sample later on.  And we will migrate that VM, a single VM,  within the server.  It's basically more safe to migrate a single VM  within a period of time, not to cause a migration rush.  And I don't know where.  Use all our resources for that.  It's pretty not safe to do that.  We also have a pretty fine load balancing within Overt.  The two legacy ones on CPU and now added in 3.4,  even distribution one, even VM distribution  that we didn't have.  The same balancing sample that I want to show,  as I showed earlier, is we get the same numbers  that we want to shut down all the VMs.  But in this example, we will actually  do that and not exclude hosts that prevent users  from running hosts, VMs like in the filter example.  Here we will show how in that hour in the wake up hour,  we will basically get all the hosts.  And if it's wake up hour, we will activate all the hosts.  If we need to sleep, then we will connect using our SDK  and get all the VMs from that host.  We will stop all the VMs and we'll deactivate the hosts.  Same bank example.  This is how we use it internally to migrate VMs.  You get, according to some logic,  you get the overloaded host.  It's called snippet.  Then you select here it's random, the first VM on that host.  Then we actually print it because we're using a CDIO  to get the data from the model.  And we return it and we return a right listed host, which  is kind of a filter.  The first filtering we do before we  activate the filter and weights and the normal process  that we do.  Basically, as I said, we have a cluster policy,  which is a container for all the filters and weights  and a single load balancing logic.  And we have two optimizations for a cluster policy,  speed and overbooking.  Basically, we run each time we schedule a VM,  we run it one by one because we want  to prevent overbooking.  We want to guarantee the same resources for each VM.  If we will try to schedule two VMs together,  we can fail because they both see the same resources.  So basically, the speed optimization  is to exclude the weight of the host.  So later on, the load balancing will do the weighing for us  and balance out the cluster for us.  And the overbooking is to emit the, I don't know,  just be able to paralyze the scheduling process.  Let's see if we can show those pictures within Overt.  OK.  OK, this is Overt.  The VM is because of Wi-Fi and VPN.  The VM is unknown.  It's running somewhere.  Here, I configure a new cluster policy.  Can give it a name, like the shutdown one.  OK.  I can add the external filter that I've added to the system,  the shutdown host filter to the enabled filters.  As a weight, I will give optimal for power saving.  Let's try to aggregate all the VMs within a single host  as much as possible.  Then I will select the balance that I created earlier.  I can give it wake up power, like 8 AM.  And shutdown our 8.  Press OK.  Should be created.  OK, I don't know what happened.  Second.  What?  OK, let's go back to the.  No, no.  It shouldn't take some time.  What?  Yeah, I'm not connected to the VPN.  No, no.  Something happened to the VPN.  It doesn't like me.  Yeah, I have screenshots.  But you know, I don't know.  But you know, maybe it will work second.  OK, let's take a look at the screen.  It works, believe me.  OK, we were.  We were here.  Wait a second.  No, still no.  Never mind.  We'll show you that one.  Then you go to cluster.  No, still not working.  OK, forget about it.  OK, I create the cluster policy that I've showed you.  Then I created to a cluster that I already defined.  So all the hosts within that cluster,  we are covering to that cluster policy.  OK, let's talk about how it's implemented in the back.  It's disabled by default.  Whoever wants to extend to add filters  should be able to know how to install the external scheduler.  The external scheduler is a separate process written in Python.  We externalize it because we want to guarantee  the engine's safety.  You know, if a user writes a code,  it can be dangerous to the system.  We want to allow other languages as well.  If you know, the end is written in Java.  So in this model is written in Python.  And going forward, we want to support SAS, which  is kind of a scheduling as a service.  It's a separate RPM.  You need to manually install it.  How it works.  Basically, it's initialized.  It reads from a local directory all the filters, weights,  and balancing logic that you wrote.  Then it's published in an internal API.  The engine, the server reads it.  And then it waits for calls from the engine  for filtering, weighing, and balancing.  This is how it looks like when it's loaded, the filter,  and load balancing here.  Back to the users list.  Now we can easily write a filter that  maximizes the number of running VMs per host.  Early.  To sum it up, we support easy Python plugins for you  to use for VM scheduling.  You can manage a separate policy for each cluster,  for each group of hosts.  And every version that comes out for over  gets new models for scheduling.  Question?  Yes.  So the question is, I have the possibility  to read from what the hypervisor provides, memory  utilization, CPU utilization.  Since you have a Python extensive API, if I got it right,  you may have the possibility to know if the storage  behind or the storage framework behind is constrained.  So you have the possibility to say, all right,  do a storage migration.  This virtual machine is to the other storage, which is SSD  base or whatever.  So you just need, for example, to have a good talk  with all the storage big guys and find more things about  their IOPs and what we are doing for their storage  to get that information.  Basically, because we have the memory, we have the CPU,  maybe we have some network information about throughputs  and stuff.  So what's left behind is the IOPs and the quality of the storage.  You can think of whatever.  Did you all hear the question?  I think within when you extend, you can do whatever you want  when you extend filter or something like that.  So what you can do is you can extend whatever you want  when you extend filter or when you basically can connect  to every provider or use any SDK that you like.  So what you can provide within the engine is all that you  ask, memory and CPU load.  It's your own choice.  Do you have an answer?  We have a sample of guys in the bar below talking about  connecting X.  Hi.  So we had a few guys in the overt workshop in Bagelot,  and they were asking us to connect the scheduler to a BMC  system that is monitoring extra parameters.  For example, they can monitor the CPU temperature and the fan  speed, and they can actually predict that if the fan speed  is constant or zero and the CPU temperature is high,  that host is going to crash and burn in a few minutes.  So what they asked from us is to give them a list of hosts,  and they can actually blacklist some of them because they are  aware of more information than what OVIR should have.  And there are so many other examples which are very similar.  For example, Cisco has very similar concepts.  They actually want to blacklist some of the hosts because the  networking is going to go down there.  There are a lot of very similar scenarios in very big  enterprises.  This is actually highly wanted.  Actually, it sounds promising.  Yeah.  So now you have the power.  You can actually do it yourself.  Well, my first single question was, all right, I know about my  CPU, my memory.  Maybe I know about my networking because we do the networking.  So the next good thing is if I had the storage IEO information  or information that had to do with the quality of my storage,  perhaps I could utilize multi-tiering in a storage or storage  in a virtual machine from one place to the other.  That could be nice with this engine.  So the endless space of this information is not a big deal.  That's a lot of cool.  There's no properly made for that.  So everyone can actually interact with these old new space  and have a good experience.  So I would like to add that sometimes some of you have  to buy the network.  You can't really do that.  No questions?  Thank you, Danali.  Go drink a beer, Anishin.  Thank you.  Thank you.  Thank you.
_FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_0_755.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_756_1607.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_1608_2615.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_2616_3791.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_3792_4619.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_4620_4907.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_4908_6467.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_6468_6731.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_6732_7043.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_7044_7475.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_7476_8183.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_8184_11423.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_11424_11519.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_11520_12707.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_12708_13259.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_13260_15899.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_15900_17711.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_17712_20555.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_20556_20903.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_20904_24311.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_24312_24647.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_24648_25271.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_25272_25883.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_25884_27059.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_27060_28271.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_28272_28991.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_28992_29507.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_29508_30179.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_30180_31823.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_31824_36407.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_36408_39227.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_39228_39611.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_39612_39803.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_39804_40595.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_40596_43043.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_43044_43895.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_43896_44123.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_44124_44327.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_44328_44495.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_44496_45755.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_45756_46031.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_46032_46259.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_46260_47039.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_47040_47819.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_47820_48191.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_48192_48407.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_48408_48791.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_48792_49163.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_49164_49319.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_49320_49535.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_49536_49907.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_49908_50291.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_50292_50819.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_50820_51035.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_51036_51407.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_51408_51563.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_51564_51791.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_51792_52175.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_52176_53063.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_53064_53291.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_53292_53375.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_53376_53591.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_53592_54251.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_54252_54407.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_54408_54791.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_54792_55319.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_55320_55907.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_55908_56663.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_56664_56819.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_56820_57035.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_57036_57191.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_57192_57791.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_57792_58199.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_58200_58859.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_58860_59255.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_59256_62555.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_62556_63803.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_63804_64103.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_64104_64391.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_64392_67199.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_67200_67715.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_67716_67931.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_67932_68615.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_68616_69155.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_69156_70499.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_70500_72515.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_72516_73727.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_73728_76067.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_76068_81059.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_81060_82247.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_82248_83351.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_83352_86051.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_86052_93131.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_93132_99023.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_99024_99407.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_99408_99503.png _FOSDEM_2014__High_Performance_Network_Function_Virtualization_with_ClickOS_99504_99569.png