Incremental Backups Good things come in small packages!
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Snow, John
Full backups of large storage devices are expensive, slow, and waste a lot ofspace. Incremental and differential backups are an oft-requested feature invirtualization stacks to help eliminate the redundant copying of backup data. This presentation will cover recent developments in related delta-backuptechnologies, covering incremental and differential backups, image fleecing,dirty bitmap management, and different paradigms of consuming this data toproduce reliable backups both through QEMU as an agent or via an externalbackup appliance. # Style This talk will cover the incremental backup functionality in QEMU anddevelopments since its first introduction in 2015. The talk is intended to be a fairly light-weight, quick trip through thefeature; explaining the mechanics behind both the implementation and the API.I do not intend to cover in incredible detail the algorithmic minutiae ofincremental backup techniques, but 'dirty bitmaps' as a fundamental technologyenabling the technique will feature quite heavily. A theme of the talk will behow these features were implemented using targeted modifications to existingstable API to unlock rich functionality while maintaining backwardscompatibility - and the difficulties faced while doing so in a diverse opensource environment. Current/Future developments are intended to be a highlightof the talk. # Audience First and foremost, students who are interested in how large-scale projectstackle problems and would like some insight to how components in "the cloud"actually manage data on a technical level would find this talk interesting. Atthe very least, hopefully they find it entertaining. Secondly, any developers or hobbyists who rely on virtualization in theirworkflow, but may not be heavily invested or well-versed in QEMU/KVM/libvirtet al specifically could benefit from an expanded knowledge of how this FOSSvirtualization stack manages data. These users may be keen to gain insightinto newer, more flexible storage and backup options. Third, Upper-layer stack developers or users who script against upper-layervirtualization stack components may be interested to hear what we're cookingup - this includes libvirt, virsh, openstack, oVirt or any projects thatcount these projects as dependencies. And lastly, anyone who has a keen appreciation for bad puns on slide captionsmight enjoy stopping by. # Takeaways Attendees can expect to leave with a basic outline of the storage managementsolutions as they exist in FOSS virtualization today, and how we'd like toimprove on them. With a copy of the slides and footnotes, attendees will bewell equipped to author power-user scripts that take advantage of someadvanced functionality in QEMU-kvm today that may help them eke out moremileage from their existing FOSS virtualization stack. The talk will covergeneral design, usage, current caveats and expected next steps for relatedbackup features; this will hopefully provide generally relevant informationfor anyone interested in or contributing to the field in general.
 Okay, sure. My name is John Snow. I work for Red Hat. I am a developer on the QEMU project. I've been working on backup solutions for the past two years alongside my other project, which is making sure that ATA drives work well, but you guys are here for the backup part. So before I jump into it, can I ask how many people are familiar with QEMU or even know what it is? Just raise your hand if you... Okay, good, good. Tons of people. Good. Sometimes I worry that anybody only knows what KVM means, and then I'm like, oh no, there's this other part that does all the work, but anyway. So before I jump into it, I would like to give my acknowledgments. This is open source, of course, so this is not just my work. This is the work of dozens of individuals. So the feature proposal for incremental backups in QEMU started with Ja'gan Sundar, who gave a talk at KVM Forum 2011 on wanting to get this rolling. And here we are in 2017, so it's been a work in progress for a while. Femme Geng, a co-worker from Beijing, submitted basically the drafts that became the current version in 2014. Stefan Hoinazi and Max Reitz are block-layer maintainers, and they've been doing a lot of the review work, and they've been very patient with everybody. We've also had a lot of involvement from Virtuoso, which used to be OpenVZ, or what was their other company named Parallels, and a few others, and recently they've split off, but Virtuoso has been doing an incredible amount of work for us, so it would be upsetting if I did not give them a shout out here for all of their incredible work. So just as an overview, just so we know where we're going, I want to talk about kind of the problems with QEMU hacking incremental backup, which is probably obvious to everybody in here. I want to talk about some of the primitives we use to implement incremental backup in QEMU and some of the interfaces and how that works. I'm going to go through some examples showing kind of the life cycle and management of incremental backups. I'll get into some more advanced applications of this, things that we've been working on the last two years, and then to close out, I'll talk about some of the features that we're currently working on, things that we're hoping to push upstream soon, some status, and you can grill me with questions if I stand me too much. So, to start, the problem, of course, I mean the problem with daily backups, especially if you have dozens and dozens of virtual machines, this is gross. It's a waste of space, it's way too much. The storage efficiency is disgusting, it is clunky, it is slow, you're going to tie up your network, but it is simple and convenient. If you are doing full backups, you know exactly what you have. It is a full backup, it is stored on another desk, it is easy to restore, it's not too hard to think about. So, what we've done, of course, as you might expect, is we've added incremental backups to QEMU. The storage efficiency, of course, is much better. It's only going to copy the modified data, it's a lot fast, but maybe it's more complicated, but I would like to convince you it's worth the extra complication, of course. QEMU added preliminary support for incremental backup in 2.4, so about two years ago now. Good luck with that. Development is ongoing as of 2.8, we're working on 2.9 right now. It's not included as supported in a Red Hat product yet, not that that matters for the open stream version, but if you were to come to the upstream list and ask us, we'd say, well, we're still working on it, we don't recommend this for data center usage yet, but it's mostly for the brave as of yet, but we are nearing feature completion. So, the approach to incremental backup in QEMU. Originally, it was Jagan Sundar's patches, which were a whole series of separate command line tools. It wasn't baked into the main QEMU binary. It used an entirely new network protocol, so all the existing management protocols we had, this tapped another one on top of it. It ran as an independent thread, so it didn't respect the QEMU state machine at all, or the main loop, it added its own complete separate thing, which made it a little dangerous for making sure that the snapshots were safe, because we had race conditions with this. It utilized temporary snapshots to make sure that the backups were perfectly atomic, and it implemented the tracking with an in-memory dirty bitmap. And it was ultimately not merged due to some of the complexity problems and how it didn't play nicely with QEMU as a whole. So, Pham Chiang took another shot at this in 2014. This was also a dirty sector bitmap-based approach, but it used a lot of our existing primitives that we already had in QEMU. So, it was not a separate thread. It behaved in the main loop. It used all of our AIO scheduling. It didn't use any external tooling, no new protocols. It was managed via our existing protocol, which Libvert uses to talk to QEMU and make sure we're behaving. We implemented this simply as just a new backup mode. We have an existing backup feature, and we just added a new argument to it. We can use this with any image format, so QCOW 2 is my personal favorite. Everybody else likes raw, that's okay. And we're hoping that because this is going to use all of the existing tools that QEMU has had for a while, that we're going to maximize compatibility with people's existing scripts, and hopefully it won't be too hard for people to add incremental support to their management solutions. So, yeah, design goals. Already touching on this a little bit, we wanted to make sure that we were using as much as possible. QEMU is a frighteningly large project at times, and every time we reimplement the wheel, it grows and grows and grows. So, for this task, we wanted to make sure that we were using everything we already had. So, we have this type of, good, glad that made it in. So, key structure is the block drive dirty bitmap, and we were already using this to track dirty sectors for our block migration and drive mirroring operations. So, when you do a live migration in QEMU and you want to migrate from one physical host to another, QEMU already needs to know which parts of the disk are being dirty so that it can make sure that it migrates those as it pivots to the new host. So, we already had something that was kind of doing what we want. It had a configurable granularity to depend on the workload. Some smaller or larger granularities may help the migration pivot sooner, and we could already hook up any number of bitmaps we wanted per drive. So, it looks like this was going to be pretty much exactly what we wanted. We wanted to use the drive backup interface. I call the QMP protocol well-known. It's well-known to us, hopefully it's well-known to people using QEMU. We already have this command creating full backups for us. It was already capable of point-in-time live backups. We could already export data to arbitrary destinations via the NBD protocol. So, like I said, we just simply added a new argument and, well, a couple of bitmap management commands as well. Another goal for us of coherency, we wanted to make sure that if you were backing up multiple drives, if you have 10, 20 drives attached to your virtual machine, we wanted to make sure that the incremental backup across all 20 was point-in-time coherent across all of them. And we wanted to use an existing QMP transaction feature to accomplish this. We want to make sure that the bitmaps are persistent. So, even if the virtual machine is shut down, shuffled around, rebooted, what have you, we want to make sure that this incremental data, which is kind of precious because it allows us to make very small backups in the future, we want to make sure that stuff survives. And we didn't want to depend on the drive data format because everybody has their pet favorites on exactly what kind of data format they're going to be using. And we didn't want to depend on the backup target format. So, we wanted to be able to go from whatever format to whatever format, and trying to make this as kind of a data storage agnostic solution as possible. Again, we want to make sure that this is migration safe, and we wanted to make sure that on error, which, you know, hopefully never going to happen, but of course we know that it always will. So, we want to make sure that we have the ability to resume incremental backups without necessarily having to rely on a brand to do full backup chain, if we can help it. And for the persistence, we have to make sure that we're not relying on outdated persistence data, which can lead to lost backups. So, those are the design goals of what we're going for. For people who are familiar with snapshots, this question comes up a lot, why aren't you just using snapshots for this? Snapshots and backups have kind of different properties. Backups are, you know, we'd like those to be off-site and in an area if possible, we would like the leading backup to be free, for instance. The snapshot mechanisms we often use in FEMU or format specific, so we wanted to avoid using that. Some people do use various kinds of snapshots and then they kind of parse the snapshot data to pull out kind of a point in time backup. But we wanted to skip all that mess and just go straight to an incremental backup. So, the incremental backups we have, they're inert, they don't grow, they're not attached to the running system. You can do whatever you want with them once they can be used in the system. Unlike snapshots where you're using a copy-on-write kind of solution, the leading copy-on-write snapshot may incur an IO penalty and maybe you don't know how much, depending on how much you've been using or using the feature. So, this is kind of why we don't use snapshots, but it's going to depend on everybody's use case, because this is another option that we will offer better for today and for off-site, like that, that's in particular. Okay, so building blocks. I'm going to talk about the bitmaps and their management just a bit. So, this is the existing structure we have to track writes to our drives. Again, I already used the drive mirror and my block migration, and it's implemented using a higher-archicled bitmap, which is just a multi-level data structure to track writes to sectors. And any number of them can be attached to your drives, which is important because you can have multiple independent backup regimes occurring simultaneously if you wish. So, if you want to do both dailies and weekends and monthlys, it's just three bitmaps. You have one for each backup regime. So, the hierarchical bitmap, I'm sure you can visualize this in your mind, but just because I like making graphs, it's actually a seven-layer data structure, but two is good enough. But this structure allows us to iterate over the data very quickly, especially if it's sparse. We're able to skip large sections of the disk, so incremental backup should proceed quite fast. And as for multiple per bitmap, you can create an arbitrary number. I think we do cap it at some very large number, like 65,000. Some such thing, but you can attach any number of these bitmaps to any of your virtual drives. As you saw in the previous slide, they have names. So, the existing usages are anonymous. We call them anonymous bitmaps. But if you are intending to use these for backups, they're named so that we can interface with them from the external API. And the name becomes the unique ID. But each drive can have the same name of a bitmap, so you need the pair of the drive name and the bitmap name to address this backup sequence. So just to illustrate, you can absolutely have something named bitmap zero on both drives. I don't necessarily recommend it, but they drive you insane, but you can. And then, of course, we have internally used anonymous bitmaps for mirroring and migration. So I mentioned that we have configurable granularity on this. The smaller the granularity, this is a block-based backup, not files. So we have no idea where files begin and end necessarily. Maybe we could look in and check, but we're not going to. So we have to go by sector at some point. And so you can configure the granularity. So one byte being dirty may incur you a backup size of 64 kilobytes, but you can configure it larger or smaller depending on your requirements or your expected usages. Or you could even, like people never do, try to profile it and then use the one that actually works best for them. But I won't tell you guys what to do. So we default to 64 kilobytes, but it will attempt to match the cluster size of the actual disk you provide. And 64K of the default for QCaul 2. I was kind of getting ahead of myself here. So we're tracking per sector, but the granularity is configured in bytes because we're trying to get rid of the concept of a sector in a VM. So sometimes we run into some interesting cases like this. So the backup engine itself is going to copy out things per cluster. It's 64K at the moment. It's non-configurable at the second, but it's almost for sure going to be something that you can tweak in the future. So given that QCaul 2 defaults to 64K and the backup engine defaults to 64K, 64K is probably the best right now. So the management for these things. Everything is done via QMP, which is really good news if you're a computer. Not so great news if you like doing everything yourself by hand. So QMP, for people who aren't maybe super familiar with it, it's a JSON-based protocol. It's like JSON-ish. I think we changed maybe the type of quotes you can use or something like that. It's almost quite nearly JSON, and it is what Lidvert and everything else will use to be talking to Q me. So it can be a little cumbersome by hand, but we do have little Python tools and things you can use if you want to play with it manually. But there are four commands here. So for the management of dirty bitmaps, there's add, remove, clear, and then there's the more generic query block command, which will show you status in general about all block devices, but it includes some interesting information about the bitmaps attached to each drive. So creation. Creation is pretty simple. Bitmaps can be created at any time on any note, and for any reason, because we're not watching you. Bitmaps begin recording writes immediately as soon as you add them, and the granularity is optional. If you omit it, it's going back to 64K, but here you can see this kind of JSON-ish command, and we execute the block dirty bitmap add command on drive zero. We name it bitmap zero, and we've decided that we want a granularity of 128K this time around. Deletion. Similarly simple. The bitmaps can only be deleted when they're not in use, and what counts as in use in this case for us is not actually being used to make a backup when you type it. But otherwise you can delete them whenever you want. Bitmaps are addressed by the node name here, and if you delete the bitmap, it has absolutely no effect on any backups you've already made, it has no effect on any backups that you're still hoping to make from other bitmaps. So this is a fairly independent kind of an action. If you wish to clear a bitmap for convenience instead of deleting the bitmap and then re-adding it, we do have a clear command. Just a simple... Oh no! I lost my thing. Thank you. As for querying the bitmaps, this is the query block command, which I've truncated a lot of the output here just to show what's relevant to us, but when you query the block you'll see device drive 0, and it will give you a list of dirty bitmaps. You can see this one is active. The count will show you how many sectors are dirty, so you can get some idea of how big the backup's going to be based on how many bits are dirty in the bitmap. You'll get the name and the granularity, some things as well. The status can also indicate frozen, and that's... Ooh! Sorry about this. The frozen status indicates that the bitmap is not allowed to be interacted with, because it is currently in use for a backup operation. Yes, again, this is the sector count and the granularity, so it can be hard to convert in your head exactly all these different formats, between bytes, sectors, clusters, and so on, but the computer will do it for you, because the computer is good at math. So the QMP commands are not super useful alone. They're not atomic necessarily. They're only safe when the VM is offline, and there's no cross-drive coherence guarantee, and that's something that I specifically mentioned we didn't want to have happen. So we do have these transaction commands that are going to allow us to create full backups alongside a new bitmap, like immediately, so that every single write is going to be tracked accordingly. We're going to be able to reset bitmaps simultaneously across any number of drives, and we're going to be able to issue any number of backups across as many drives as we have. The problem with the individual commands, for instance, is if you try to reset or add a bitmap and then no matter how fast you are and you try to send the other command, oh, by the way, I'd like to start a new backup now. You have no idea. You may have gotten a write that happened in the interim, and we like everything to be live and fast and QMU, so pausing the VM to do these operations is not really a great solution for us. So we do have these transaction commands that allow us to batch the commands together to make sure that the operations are data safe. So the transaction actions that interface with bitmaps, we have the version of dirty bitmap add and dirty bitmap clear. Don't really need a transaction from a move because we're getting rid of it. This works in conjunction with the type drive backup for the transactions, and this can be used for everything I said, the multi-drive coherency, full backups, new incremental chain sync points. It's kind of the major interface you'll be using to do anything with more than one drive, really. So that's a lot of talking. Let's show some pictures. So this is the life cycle for an incremental backup chain in QMU. Hopefully it's exactly as simple as you would hope it to be. You're going to, in general, either create a new backup chain or you're going to take an existing one and synchronize it with a new full backup. And then from then on, you can make as many incremental backups as often as you want, according to whatever policy you may have. So for an example showing the QMP behind this, this is a transaction. It's going to batch the add command and the drive backup command together, such that the bitmap will be cleared exactly in time with the drive backup. Here I've also highlighted, it doesn't come across so well on the projector, but this is sync full as the argument we're giving. So this is a full backup. And the backup will start at this point in time and it will clear the bitmap at this point in time. But the backup will still take some time to complete and it will finish later. And this bitmap is going to be kind of in use until that finishes. So to show you what will happen, we have this ID drive zero and we're going to run this transaction and then in one shot, we'll get a full backup and a new bitmap with no dirty sectors in it. So from this point forward, we can use it for incrementals. For a different example, if you didn't want to add a bitmap, if you want to clear it, just kind of like reset, you can use the clear and the drive backup command at the same time together. And that's how we can make it clear. We can now use it to go back to the first backup. And similarly, the bitmap's dirty sector count has been reset for us. And then finally, we get to creating an incremental backup. So unlike the other commands with the full backup, we're going to create an image separately here. We're going to tell this image that the backing file is the last full backup we made so that when we create this sparse image of just the sectors that have changed since the last usage, if you were to mount this incremental backup, you would be able to see a full and complete image of the hard drive. It's just each is stored in a separate layer. So we're going to use an external QMU image tool to create an empty QCOW2 for us. We're going to set the backing file with the dash B command and the format is QCOW2. And then at any point we wish, we can issue this drive backup command. And the only difference here, you can see it's highlighted, is sync is incremental, and we name the bitmap and the device, and the effect is this. So if we have this many dirty sectors, we issue the backup command, and now we have a new QCOW2 file using the backup as a backing file. And then the count is reset. And then from here on out, as many times as you wish, you can keep creating incremental backups. Just keep creating new top layers. So the problem here, perhaps in the management aspect, is of course, if you decide to have a backup every single hour or something like this, you're quickly going to have a chain of several thousand of these images. But hopefully, if you've been copying them offsite, like I'm asking nicely, you can use tools at any time to condense these images. So the management layer on the other side of this can take these long backup chains and periodically condense them or copy them or move them into whatever other dedupe situation that you have. So hopefully, that is not too complicated for the management side of things. So brief break, I'm just going to... No, I'm kidding. No, I'm kidding. I'm going to talk about more stuff. Anyway, so I'd like to talk just a bit about what are jobs. So QMP commands are synchronous, and the QMP socket is going to block each time you send one. So for long running commands, we don't really have the ability to block the socket for that entire time. So we have this asynchronous task API that allows us to submit a QMP command and get some kind of a long running job that we can manage, we can cancel, and all sorts of things. For more information on this, you can see literally any talk from KVM Form 2016 because three of the block maintainers all gave talks on exactly how block management works. And figuratively, only three. It was Kashyap Chamarthi, Max Wrights, and myself all gave talks on this. And there's a link in the end if you want to know how to manage these things. So transactions in detail. We're going to batch some of the QMP commands. Each individual item is an action. The transaction succeeds only if all of the actions do. But the problem is that some actions are going to launch jobs synchronously and some don't. And I hope that doesn't cause any problems. But of course it did. The problem was that if you have a transaction and half of the items are launching these long running jobs and half aren't, the concept of a transaction failure became really muddied because normally transactions fail if any component action fails. But the problem here is that as long as the job launched, we considered that a success. So then later on some of the jobs could fail. So say if you start a two terabyte backup job because you really hate your network infrastructure. And then it failed at 1.8 terabytes or something. The transaction still succeeded. So the management client now is really confused about what exactly just happened because the transaction, oh, so what's good and what is correlated at what point of time to... So this was a bit of a problem for us. So yes, before 2.5, the action succeeds if the job is started. Jobs failing later have no effect on other jobs. So some backup succeeded and some would fail. And this wasn't quite so great. So we added a new parameter to the transaction command as a whole where there was no change from the original behavior where the action succeeds if the job is started. But any jobs launched by a transaction now will wait for completion from all other jobs before they finish. So for instance, if you try to do a backup across a two terabyte drive and like a one gig drive, clearly that one gig drive is going to finish way sooner than that two terabyte behemoth. But it's going to copy all of its data first, but then it's just going to wait in a completion mode, waiting for the other job to finish. And if the two terabyte job should fail, the one gigabyte job knows that it's not safe to clear out the bits in the bitmap and it will roll back to make sure that nothing is lost. This isn't necessarily what you want, but it's up to the management client. Some management clients assume a transaction will be fully all or nothing, and this will give them that behavior so you can avoid keeping partial completion state in memory or having to worry about that sort of thing. But it's up to you. There's no problem technically if one job fails and another doesn't. You don't have to cancel or delete the other backup. You could just retry the incremental on both to get a point in time across both. But this was a hotly requested feature for people to avoid having to think about partial completion and failure and so on. So this is an example of a multi-drive backup doing incremental across two drives. There's really no difference other than this thing is all wrapped in a transaction command. And here I have actually omitted the grouped completion mode that I was talking about, but it would be an argument below action. So in addition to the actions array, there would be like a completion mode grouped parameter here. But this is just to illustrate what happens when we want backup across two drives at the same time. So you may have something like this, and then upon issuing the transaction point in time, we get two backups at exactly the same time. Yeah, just pointing out that those two are absolutely correlated and should be safe to use together. So for partial failures, again, if one failed and the other does not, we're going to have to delete one. But then the other backup is still good. And the way to recover from this would be to add another incremental backup. So you'd have two incremental backups for drive zero and just one incremental backup for drive one at the end of that to recover from like a partial failure. But if that is untenable for you, the other solution is if one fails, we're actually going to delete both of them. So it is a bit of a waste, but it does save you the trouble of having to think about exactly how to manage the recovery situation for man. Let's see. So we have a few more features. These are kind of things that were a few more features that we're trying to work on. So bitmap migration at the moment, the initial version of this did not actually support migrating across live hosts. So the initial version of migration for us was to split these data, the dirty bitmap into one kilobyte chunks, and then serialize it piece by piece. And then using the bitmaps themselves, we could kind of copy as the drive got dirtied. And for things that were small enough, we could just kind of skip the live phase and send it wholesale. But maybe this doesn't scale so well, depending on how many bitmaps or how many drives you have. This could really add some time to the migration stream. So the bitmaps weren't transferred alongside the data. So the data would go in its own loop, and the bitmaps would go in its own loop. And then we had this dirty, dirty bitmap kind of solution that was kind of really confusing to name all the variables for. But this captures changes during live migration, and we could resend the pieces if we needed to. And it didn't use a lot of memory, but it kind of added some complexity to the migration itself. So we have a second approach now using a post-copy technique. If you're familiar with post-copy in general, post-copy migrations are the concept that you could migrate a VM from one machine to another and then immediately start running the VM on the new machine, and then as it needs the memory, it would false and fetch the memory from the previous host. And the reason this is useful is because it allows you to guarantee when the machine will pivot, because if you are not worrying about the rate that the memory is being dirty, it's possible that the VM will be so active, so busy, so loaded that it will dirty the memory faster than we can migrate it across the stream. And the same kind of problem happens for disk. But in post-copy, you just migrate the VM first with an initial set, and then as it needs memory, it fetches it from the old host. So we figured that we could probably do something like that for bitmaps. So we're just going to send the whole bitmap post-pivot, and then before the data has fully converged, we're going to record the new writes on the target with a new bitmap. We're going to disallow backups until all of the bitmaps have come back to us, and then we're just going to merge the bitmaps on the target. So what this gets us is the ability to guarantee that the bitmaps aren't going to be stalling the migration, and then the price we pay for that is you may not be able to issue a new incremental backup for the next minute or something after you migrate as everything catches up. This does add a new failure point. So this is the unfortunate part. We do consider this kind of a non-critical loss because you've just lost metadata, but the bitmap chains can be restarted using the clear commands and so on, but we are looking at ways perhaps to mitigate this kind of a data loss event. I shouldn't say data loss, I should say metadata loss. There are other options. If this really spooks you out, you can always use shared storage migration and flush the bitmaps to the disk, migrate and then load the bitmaps from the disk again, and that's probably a lot safer. So for the persistence, like I'm hinting at, we were going to do a format agnostic solution, but we were going to use QCALT 2 as a generic container to store bitmaps for arbitrary other drives, but the problem we had with that is that QEMU is not in the business of managing file names or hierarchies for where things are stored. We leave that to Libvert and other management tools. So when we were trying to add a spec that allowed us to use QCALT 2 to describe the dirty state of other files and formats, it quickly became obvious that we were trying to replicate too much from the layer above us. So now we're just targeting the QCALT 2 format, which some people might be sad to hear, but I'll get to that in a bit. So for right now, it's a QCALT 2 format. Multiple bitmaps can be stored per file. The bitmaps have types. There's a full spec that was written kind of independent of this feature that allowed us to store these dirty bitmaps in a QCALT 2 file. So this is extensible and we may use it in the future to restart migrations or things, but for now, it's incremental backup related. The bitmaps can auto load in QEMU, like if there's a persistent bitmap associated with a storage file. Once you give that to QEMU, the next time it boots, it sees, ah, yes, good, and it will start up for you. The spec amendment is already merged and there are patches ready on list from Virtuovo, and hopefully we're going to get these into 2.9. So for non-QCALT 2 files, there are some options. Some formats are going to add their own primary support for it. I know Virtuovo wants to make sure that they have support in their parallels format, but we may extend QCALT 2 into a right forwarding, like header node, if you will, where instead of a backing file, which is usually read-only, we may have a read-write backing file, where QCALT 2 will store the metadata for other images for you, and then that way the managing layers can worry about the backing file relationships, and all we need to worry about is that we now have a QCALT 2 that doesn't actually take rights, it just forwards them to a raw file. So we're hoping this is going to let us do persistent bitmap storage for raw disks for people who want that kind of performance, but would still like incremental backups. So I think I heard in the previous talk we were talking about VMware's CBT a bit. So this was a problem that came up for us because everything I've described so far is what we call a push model, where QEMU knows exactly what's dirty and where it's dirty, and you ask QEMU to author the backup. So this is a largely QEMU-driven process, which isn't necessarily a problem for us or for people who are already using our models, but we do have a lot of requests for making something similar to VMware's CBT, where you have the ability to query for which blocks are dirty and then manually yourself copy those blocks out. So at the moment, push model is already in QEMU, we are working on the pull model, and the way the pull model works is it's going to offer a lightweight temporary snapshot, so you will connect to QEMU, you will request a temporary, like a view of the image using the NBD protocol, and then through the NBD protocol you're going to have the ability to ask, you know, give me a list of the dirty sectors, I want to see what's dirty, and then the client can at its own pace copy out the sectors it thinks are dirty, and perhaps if this third party client knows more than we do about exactly what's dirty, it can make its own decisions about what to backup, where, why, for instance. But this is a feature that was requested by Virtuozo, and they've been working very hard on this, and we are hoping that this is going to add a lot of flexibility to existing clients, both for people who use VMware, or for people who already use their own scripts for QEMU, we're hoping that one of the two push or pull models is going to fit nicely into the workflow. So a little bit more about the snapshot, it is a point in time snapshot, we call it image fleecing, you can open up an image and get a still view of the image at that point in time, but because of that it's going to require an on-disk cache, because as the VM continues to run, we need somewhere to flush those writes if we run out of memory. So it has some downsides perhaps compared to the QEMU authored backups, but not that different, but like I said it's going to offer kind of full control, and it's probably the most QEMU agnostic method where you're kind of cutting QEMU out of having to manage these a bit, and it is at the moment the only way to query exactly which blocks are dirty, so it may be useful for antivirus research or things like that, if you want to see what programs are dirty and where and so on. So we do have some to-dos, we still have to work on the interface for some of the more advanced features, the pull model, managing persistence, you know, if you say I want to delete this out of the drive, I want to put a new bitmap in, or I don't want this bitmap to be persistent, so there's a lot of interface questions that we're still working on, we don't have like an integrity checker just yet, but we're working on one, we would like to add the ability to do offline incremental backups, so we would like a reference tool to make sure that if you have shut the VM down, you can still parse this dirty bitmap and complete a backup offline if you would like to. So, and I would also like to give a quick shout out, we do have a GSOC project coming up, if anybody was actually interested, we want to write a reference implementation, just a simple CLI tool, probably using Python to take advantage of our existing test infrastructure. If it sounds like something that you'd be interested in, we're definitely looking for people to help us with this, and so, just at the end here, so the project status, the interface is merged, the incremental mode is merged, transaction support was merged in 2.5, the persistence spec got merged in 2.6, grouped transactions got added just in 2.8, migration support is in review, it's on the list right now, we're hoping to get it in for 2.9, persistence, it's the same story, and pull model is still kind of an early design, and then there were some specs written for NBD to allow us to use NBD for this purpose, and we are hoping to get a lot of that in 2.10. And, any questions? Other questions about the report, so in the end you get the plus and non-volumen incremental, what do you do with those files? You're not going to like this, so that's the management layer's job, I don't care. So, no, we tried to keep that complexity out of QME, because we didn't want to replicate what Libvert was already doing, so Libvert has whatever it has to manage its own storage pools and things like that, so we tried to not care exactly about how many incrementals were already created, we say that it's up to the management layer to remember the last incremental it made, and to properly hook up the chain, so it's a very unix philosophy, where if you ask for the wrong thing, you will get the wrong thing. So, in that way we kept our part simple, and we just kind of... Libvert's problem now. Okay, so the question was if we were collaborating with Overt or not. So, downstream at Red Hat, yes, I've been talking to some project managers to help them explain exactly what we've made, and I think this info is getting back to them, but I haven't been personally working with someone in Overt, but I did write a white paper for it that I distributed to some people, and if there are any requests or anything like that, if you know somebody directly who would definitely want to talk to me, send me an email. Yes? Could you repeat that last part for me? As storage gets cheaper and deduplication technology is so far proliferating, how much life is left and personally incremental or financial backup strategy? Probably more than you would hope for, or I should repeat the question, which was as storage gets cheaper and deduplication proliferates, how much life is left in this kind of a technique? More than you would hope, because we still use floppy drives for some things, so it's always too early to say, oh, this is dead. People do care about this a lot. Other companies besides Red Hat are pouring a lot of time and effort into making sure that this works. So as for how much sense it makes, I'm not able to give you an incredibly great answer for that, but I can tell you that businesses are extremely concerned with it and are spending a lot of time and money to make sure that it works well. So it's probably going to be around for however long you think it should, and then another decade past that. I guess I'm quite a bit early. Oh, yeah? Maybe I missed understanding this part of the handling of the bitmaps, but can you describe when the number of the bitmaps per drive is more than one? Okay, is the key so one is... Oh, sure, sure. Okay, I see what you're asking. So the question is, when would you want more than one bitmap per drive? So if you wanted to implement separate backup paradigms at the same time, so say you want a daily backup, and you could use one bitmap to do every day, you could have a second bitmap to do every week, you could in theory just do the daily backup and then the back end can manage the different chains for you to give you different views, but it was almost already there for us. It would have taken more effort to remove that feature. So the flexibility is there for us. So some further reading, if anybody's interested in this stuff. The Kimi project main page wiki has status and updates for us. There's a project page for this in particular. The bitmaps documentation for the QMP aspect of managing all the push model backups is in the source tree under the docs folder bitmaps.md. For the closest thing we have to a reference implementation right now, we have an IO test which is written in Python and that's going to run through several backup scenarios like failure on one node, failure on two nodes, failure on individual mode, group mode and so on. So that's actually the closest we have to seeing exactly how a program would request these backups. The status white paper that I was talking about, which is written for the benefit of Libvert and other layers above QEMU, is currently at this little Google tiny URL link. So if that's something that you were interested in reading, take a picture or write it down or grab the slides or whatever you will. And then for people who are interested in writing tools to interface with QEMU, the jobs talk from the last KVM forum is up through the Linux foundation. If you search KVM forum 2016, find my name and that'll be up there. So I think I'm actually a little bit early but it'll give you time to scoot out.
 Okay, sure. My name is John Snow. I work for Red Hat. I am a developer on the QEMU project.  I've been working on backup solutions for the past two years alongside my other project,  which is making sure that ATA drives work well, but you guys are here for the backup part.  So before I jump into it, can I ask how many people are familiar with QEMU or even know what it is?  Just raise your hand if you... Okay, good, good. Tons of people. Good.  Sometimes I worry that anybody only knows what KVM means, and then I'm like,  oh no, there's this other part that does all the work, but anyway.  So before I jump into it, I would like to give my acknowledgments.  This is open source, of course, so this is not just my work. This is the work of dozens of individuals.  So the feature proposal for incremental backups in QEMU started with Ja'gan Sundar,  who gave a talk at KVM Forum 2011 on wanting to get this rolling.  And here we are in 2017, so it's been a work in progress for a while.  Femme Geng, a co-worker from Beijing, submitted basically the drafts that became the current version in 2014.  Stefan Hoinazi and Max Reitz are block-layer maintainers, and they've been doing a lot of the review work,  and they've been very patient with everybody.  We've also had a lot of involvement from Virtuoso, which used to be OpenVZ,  or what was their other company named Parallels, and a few others, and recently they've split off,  but Virtuoso has been doing an incredible amount of work for us, so it would be upsetting  if I did not give them a shout out here for all of their incredible work.  So just as an overview, just so we know where we're going, I want to talk about kind of the problems  with QEMU hacking incremental backup, which is probably obvious to everybody in here.  I want to talk about some of the primitives we use to implement incremental backup in QEMU  and some of the interfaces and how that works.  I'm going to go through some examples showing kind of the life cycle and management of incremental backups.  I'll get into some more advanced applications of this, things that we've been working on the last two years,  and then to close out, I'll talk about some of the features that we're currently working on,  things that we're hoping to push upstream soon, some status, and you can grill me with questions if I stand me too much.  So, to start, the problem, of course, I mean the problem with daily backups,  especially if you have dozens and dozens of virtual machines, this is gross.  It's a waste of space, it's way too much. The storage efficiency is disgusting, it is clunky, it is slow,  you're going to tie up your network, but it is simple and convenient.  If you are doing full backups, you know exactly what you have.  It is a full backup, it is stored on another desk, it is easy to restore, it's not too hard to think about.  So, what we've done, of course, as you might expect, is we've added incremental backups to QEMU.  The storage efficiency, of course, is much better.  It's only going to copy the modified data, it's a lot fast, but maybe it's more complicated,  but I would like to convince you it's worth the extra complication, of course.  QEMU added preliminary support for incremental backup in 2.4, so about two years ago now.  Good luck with that.  Development is ongoing as of 2.8, we're working on 2.9 right now.  It's not included as supported in a Red Hat product yet, not that that matters for the open stream version,  but if you were to come to the upstream list and ask us, we'd say,  well, we're still working on it, we don't recommend this for data center usage yet,  but it's mostly for the brave as of yet, but we are nearing feature completion.  So, the approach to incremental backup in QEMU.  Originally, it was Jagan Sundar's patches, which were a whole series of separate command line tools.  It wasn't baked into the main QEMU binary.  It used an entirely new network protocol, so all the existing management protocols we had,  this tapped another one on top of it.  It ran as an independent thread, so it didn't respect the QEMU state machine at all,  or the main loop, it added its own complete separate thing,  which made it a little dangerous for making sure that the snapshots were safe,  because we had race conditions with this.  It utilized temporary snapshots to make sure that the backups were perfectly atomic,  and it implemented the tracking with an in-memory dirty bitmap.  And it was ultimately not merged due to some of the complexity problems  and how it didn't play nicely with QEMU as a whole.  So, Pham Chiang took another shot at this in 2014.  This was also a dirty sector bitmap-based approach,  but it used a lot of our existing primitives that we already had in QEMU.  So, it was not a separate thread.  It behaved in the main loop.  It used all of our AIO scheduling.  It didn't use any external tooling, no new protocols.  It was managed via our existing protocol, which Libvert uses to talk to QEMU  and make sure we're behaving.  We implemented this simply as just a new backup mode.  We have an existing backup feature, and we just added a new argument to it.  We can use this with any image format, so QCOW 2 is my personal favorite.  Everybody else likes raw, that's okay.  And we're hoping that because this is going to use all of the existing tools  that QEMU has had for a while, that we're going to maximize compatibility  with people's existing scripts, and hopefully it won't be too hard  for people to add incremental support to their management solutions.  So, yeah, design goals.  Already touching on this a little bit,  we wanted to make sure that we were using as much as possible.  QEMU is a frighteningly large project at times,  and every time we reimplement the wheel, it grows and grows and grows.  So, for this task, we wanted to make sure that we were using everything we already had.  So, we have this type of, good, glad that made it in.  So, key structure is the block drive dirty bitmap,  and we were already using this to track dirty sectors for our block migration and drive mirroring operations.  So, when you do a live migration in QEMU and you want to migrate from one physical host to another,  QEMU already needs to know which parts of the disk are being dirty  so that it can make sure that it migrates those as it pivots to the new host.  So, we already had something that was kind of doing what we want.  It had a configurable granularity to depend on the workload.  Some smaller or larger granularities may help the migration pivot sooner,  and we could already hook up any number of bitmaps we wanted per drive.  So, it looks like this was going to be pretty much exactly what we wanted.  We wanted to use the drive backup interface.  I call the QMP protocol well-known.  It's well-known to us, hopefully it's well-known to people using QEMU.  We already have this command creating full backups for us.  It was already capable of point-in-time live backups.  We could already export data to arbitrary destinations via the NBD protocol.  So, like I said, we just simply added a new argument and, well, a couple of bitmap management commands as well.  Another goal for us of coherency, we wanted to make sure that if you were backing up multiple drives,  if you have 10, 20 drives attached to your virtual machine, we wanted to make sure  that the incremental backup across all 20 was point-in-time coherent across all of them.  And we wanted to use an existing QMP transaction feature to accomplish this.  We want to make sure that the bitmaps are persistent.  So, even if the virtual machine is shut down, shuffled around, rebooted, what have you,  we want to make sure that this incremental data, which is kind of precious because it allows us to make very small backups in the future,  we want to make sure that stuff survives.  And we didn't want to depend on the drive data format because everybody has their pet favorites on exactly what kind of data format they're going to be using.  And we didn't want to depend on the backup target format.  So, we wanted to be able to go from whatever format to whatever format,  and trying to make this as kind of a data storage agnostic solution as possible.  Again, we want to make sure that this is migration safe, and we wanted to make sure that on error,  which, you know, hopefully never going to happen, but of course we know that it always will.  So, we want to make sure that we have the ability to resume incremental backups without necessarily having to rely on a brand to do full backup chain, if we can help it.  And for the persistence, we have to make sure that we're not relying on outdated persistence data, which can lead to lost backups.  So, those are the design goals of what we're going for.  For people who are familiar with snapshots, this question comes up a lot, why aren't you just using snapshots for this?  Snapshots and backups have kind of different properties.  Backups are, you know, we'd like those to be off-site and in an area if possible, we would like the leading backup to be free, for instance.  The snapshot mechanisms we often use in FEMU or format specific, so we wanted to avoid using that.  Some people do use various kinds of snapshots and then they kind of parse the snapshot data to pull out kind of a point in time backup.  But we wanted to skip all that mess and just go straight to an incremental backup.  So, the incremental backups we have, they're inert, they don't grow, they're not attached to the running system.  You can do whatever you want with them once they can be used in the system.  Unlike snapshots where you're using a copy-on-write kind of solution, the leading copy-on-write snapshot may incur an IO penalty and maybe you don't know how much, depending on how much you've been using or using the feature.  So, this is kind of why we don't use snapshots, but it's going to depend on everybody's use case, because this is another option that we will offer better for today and for off-site, like that, that's in particular.  Okay, so building blocks. I'm going to talk about the bitmaps and their management just a bit.  So, this is the existing structure we have to track writes to our drives.  Again, I already used the drive mirror and my block migration, and it's implemented using a higher-archicled bitmap, which is just a multi-level data structure to track writes to sectors.  And any number of them can be attached to your drives, which is important because you can have multiple independent backup regimes occurring simultaneously if you wish.  So, if you want to do both dailies and weekends and monthlys, it's just three bitmaps. You have one for each backup regime.  So, the hierarchical bitmap, I'm sure you can visualize this in your mind, but just because I like making graphs, it's actually a seven-layer data structure, but two is good enough.  But this structure allows us to iterate over the data very quickly, especially if it's sparse.  We're able to skip large sections of the disk, so incremental backup should proceed quite fast.  And as for multiple per bitmap, you can create an arbitrary number.  I think we do cap it at some very large number, like 65,000.  Some such thing, but you can attach any number of these bitmaps to any of your virtual drives.  As you saw in the previous slide, they have names.  So, the existing usages are anonymous. We call them anonymous bitmaps.  But if you are intending to use these for backups, they're named so that we can interface with them from the external API.  And the name becomes the unique ID.  But each drive can have the same name of a bitmap, so you need the pair of the drive name and the bitmap name to address this backup sequence.  So just to illustrate, you can absolutely have something named bitmap zero on both drives.  I don't necessarily recommend it, but they drive you insane, but you can.  And then, of course, we have internally used anonymous bitmaps for mirroring and migration.  So I mentioned that we have configurable granularity on this.  The smaller the granularity, this is a block-based backup, not files.  So we have no idea where files begin and end necessarily.  Maybe we could look in and check, but we're not going to.  So we have to go by sector at some point.  And so you can configure the granularity.  So one byte being dirty may incur you a backup size of 64 kilobytes,  but you can configure it larger or smaller depending on your requirements or your expected usages.  Or you could even, like people never do, try to profile it and then use the one that actually works best for them.  But I won't tell you guys what to do.  So we default to 64 kilobytes, but it will attempt to match the cluster size of the actual disk you provide.  And 64K of the default for QCaul 2.  I was kind of getting ahead of myself here.  So we're tracking per sector, but the granularity is configured in bytes  because we're trying to get rid of the concept of a sector in a VM.  So sometimes we run into some interesting cases like this.  So the backup engine itself is going to copy out things per cluster.  It's 64K at the moment.  It's non-configurable at the second, but it's almost for sure going to be something that you can tweak in the future.  So given that QCaul 2 defaults to 64K and the backup engine defaults to 64K,  64K is probably the best right now.  So the management for these things.  Everything is done via QMP, which is really good news if you're a computer.  Not so great news if you like doing everything yourself by hand.  So QMP, for people who aren't maybe super familiar with it,  it's a JSON-based protocol.  It's like JSON-ish.  I think we changed maybe the type of quotes you can use or something like that.  It's almost quite nearly JSON, and it is what Lidvert and everything else will use to be talking to Q me.  So it can be a little cumbersome by hand, but we do have little Python tools and things you can use if you want to play with it manually.  But there are four commands here.  So for the management of dirty bitmaps, there's add, remove, clear,  and then there's the more generic query block command, which will show you status in general about all block devices,  but it includes some interesting information about the bitmaps attached to each drive.  So creation. Creation is pretty simple.  Bitmaps can be created at any time on any note, and for any reason, because we're not watching you.  Bitmaps begin recording writes immediately as soon as you add them, and the granularity is optional.  If you omit it, it's going back to 64K, but here you can see this kind of JSON-ish command,  and we execute the block dirty bitmap add command on drive zero.  We name it bitmap zero, and we've decided that we want a granularity of 128K this time around.  Deletion. Similarly simple.  The bitmaps can only be deleted when they're not in use,  and what counts as in use in this case for us is not actually being used to make a backup when you type it.  But otherwise you can delete them whenever you want.  Bitmaps are addressed by the node name here, and if you delete the bitmap, it has absolutely no effect on any backups you've already made,  it has no effect on any backups that you're still hoping to make from other bitmaps.  So this is a fairly independent kind of an action.  If you wish to clear a bitmap for convenience instead of deleting the bitmap and then re-adding it,  we do have a clear command.  Just a simple...  Oh no! I lost my thing. Thank you.  As for querying the bitmaps, this is the query block command,  which I've truncated a lot of the output here just to show what's relevant to us,  but when you query the block you'll see device drive 0, and it will give you a list of dirty bitmaps.  You can see this one is active.  The count will show you how many sectors are dirty,  so you can get some idea of how big the backup's going to be based on how many bits are dirty in the bitmap.  You'll get the name and the granularity, some things as well.  The status can also indicate frozen, and that's...  Ooh!  Sorry about this.  The frozen status indicates that the bitmap is not allowed to be interacted with,  because it is currently in use for a backup operation.  Yes, again, this is the sector count and the granularity,  so it can be hard to convert in your head exactly all these different formats,  between bytes, sectors, clusters, and so on,  but the computer will do it for you, because the computer is good at math.  So the QMP commands are not super useful alone.  They're not atomic necessarily.  They're only safe when the VM is offline, and there's no cross-drive coherence guarantee,  and that's something that I specifically mentioned we didn't want to have happen.  So we do have these transaction commands  that are going to allow us to create full backups alongside a new bitmap,  like immediately, so that every single write is going to be tracked accordingly.  We're going to be able to reset bitmaps simultaneously across any number of drives,  and we're going to be able to issue any number of backups across as many drives as we have.  The problem with the individual commands, for instance,  is if you try to reset or add a bitmap  and then no matter how fast you are and you try to send the other command,  oh, by the way, I'd like to start a new backup now.  You have no idea. You may have gotten a write that happened in the interim,  and we like everything to be live and fast and QMU,  so pausing the VM to do these operations is not really a great solution for us.  So we do have these transaction commands that allow us to batch the commands together  to make sure that the operations are data safe.  So the transaction actions that interface with bitmaps,  we have the version of dirty bitmap add and dirty bitmap clear.  Don't really need a transaction from a move because we're getting rid of it.  This works in conjunction with the type drive backup for the transactions,  and this can be used for everything I said, the multi-drive coherency,  full backups, new incremental chain sync points.  It's kind of the major interface you'll be using to do anything with more than one drive, really.  So that's a lot of talking. Let's show some pictures.  So this is the life cycle for an incremental backup chain in QMU.  Hopefully it's exactly as simple as you would hope it to be.  You're going to, in general, either create a new backup chain  or you're going to take an existing one and synchronize it with a new full backup.  And then from then on, you can make as many incremental backups as often as you want,  according to whatever policy you may have.  So for an example showing the QMP behind this, this is a transaction.  It's going to batch the add command and the drive backup command together,  such that the bitmap will be cleared exactly in time with the drive backup.  Here I've also highlighted, it doesn't come across so well on the projector,  but this is sync full as the argument we're giving.  So this is a full backup.  And the backup will start at this point in time and it will clear the bitmap at this point in time.  But the backup will still take some time to complete and it will finish later.  And this bitmap is going to be kind of in use until that finishes.  So to show you what will happen, we have this ID drive zero  and we're going to run this transaction and then in one shot,  we'll get a full backup and a new bitmap with no dirty sectors in it.  So from this point forward, we can use it for incrementals.  For a different example, if you didn't want to add a bitmap,  if you want to clear it, just kind of like reset,  you can use the clear and the drive backup command at the same time together.  And that's how we can make it clear.  We can now use it to go back to the first backup.  And similarly, the bitmap's dirty sector count has been reset for us.  And then finally, we get to creating an incremental backup.  So unlike the other commands with the full backup, we're going to create an image separately here.  We're going to tell this image that the backing file is the last full backup we made  so that when we create this sparse image of just the sectors that have changed since the last usage,  if you were to mount this incremental backup, you would be able to see a full and complete image of the hard drive.  It's just each is stored in a separate layer.  So we're going to use an external QMU image tool to create an empty QCOW2 for us.  We're going to set the backing file with the dash B command and the format is QCOW2.  And then at any point we wish, we can issue this drive backup command.  And the only difference here, you can see it's highlighted, is sync is incremental,  and we name the bitmap and the device, and the effect is this.  So if we have this many dirty sectors, we issue the backup command,  and now we have a new QCOW2 file using the backup as a backing file.  And then the count is reset.  And then from here on out, as many times as you wish, you can keep creating incremental backups.  Just keep creating new top layers.  So the problem here, perhaps in the management aspect, is of course,  if you decide to have a backup every single hour or something like this,  you're quickly going to have a chain of several thousand of these images.  But hopefully, if you've been copying them offsite, like I'm asking nicely,  you can use tools at any time to condense these images.  So the management layer on the other side of this can take these long backup chains  and periodically condense them or copy them or move them into whatever other dedupe situation that you have.  So hopefully, that is not too complicated for the management side of things.  So brief break, I'm just going to...  No, I'm kidding. No, I'm kidding. I'm going to talk about more stuff.  Anyway, so I'd like to talk just a bit about what are jobs.  So QMP commands are synchronous, and the QMP socket is going to block each time you send one.  So for long running commands, we don't really have the ability to block the socket for that entire time.  So we have this asynchronous task API that allows us to submit a QMP command  and get some kind of a long running job that we can manage, we can cancel, and all sorts of things.  For more information on this, you can see literally any talk from KVM Form 2016  because three of the block maintainers all gave talks on exactly how block management works.  And figuratively, only three. It was Kashyap Chamarthi, Max Wrights, and myself all gave talks on this.  And there's a link in the end if you want to know how to manage these things.  So transactions in detail. We're going to batch some of the QMP commands.  Each individual item is an action. The transaction succeeds only if all of the actions do.  But the problem is that some actions are going to launch jobs synchronously and some don't.  And I hope that doesn't cause any problems. But of course it did.  The problem was that if you have a transaction and half of the items are launching these long running jobs  and half aren't, the concept of a transaction failure became really muddied  because normally transactions fail if any component action fails.  But the problem here is that as long as the job launched, we considered that a success.  So then later on some of the jobs could fail. So say if you start a two terabyte backup job  because you really hate your network infrastructure.  And then it failed at 1.8 terabytes or something.  The transaction still succeeded. So the management client now is really confused about what exactly just happened  because the transaction, oh, so what's good and what is correlated at what point of time to...  So this was a bit of a problem for us. So yes, before 2.5, the action succeeds if the job is started.  Jobs failing later have no effect on other jobs. So some backup succeeded and some would fail.  And this wasn't quite so great.  So we added a new parameter to the transaction command as a whole  where there was no change from the original behavior where the action succeeds if the job is started.  But any jobs launched by a transaction now will wait for completion from all other jobs before they finish.  So for instance, if you try to do a backup across a two terabyte drive and like a one gig drive,  clearly that one gig drive is going to finish way sooner than that two terabyte behemoth.  But it's going to copy all of its data first, but then it's just going to wait in a completion mode,  waiting for the other job to finish. And if the two terabyte job should fail,  the one gigabyte job knows that it's not safe to clear out the bits in the bitmap  and it will roll back to make sure that nothing is lost.  This isn't necessarily what you want, but it's up to the management client.  Some management clients assume a transaction will be fully all or nothing,  and this will give them that behavior so you can avoid keeping partial completion state in memory  or having to worry about that sort of thing. But it's up to you.  There's no problem technically if one job fails and another doesn't.  You don't have to cancel or delete the other backup.  You could just retry the incremental on both to get a point in time across both.  But this was a hotly requested feature for people to avoid having to think about partial completion and failure and so on.  So this is an example of a multi-drive backup doing incremental across two drives.  There's really no difference other than this thing is all wrapped in a transaction command.  And here I have actually omitted the grouped completion mode that I was talking about,  but it would be an argument below action.  So in addition to the actions array, there would be like a completion mode grouped parameter here.  But this is just to illustrate what happens when we want backup across two drives at the same time.  So you may have something like this, and then upon issuing the transaction point in time,  we get two backups at exactly the same time.  Yeah, just pointing out that those two are absolutely correlated and should be safe to use together.  So for partial failures, again, if one failed and the other does not, we're going to have to delete one.  But then the other backup is still good.  And the way to recover from this would be to add another incremental backup.  So you'd have two incremental backups for drive zero and just one incremental backup for drive one at the end of that  to recover from like a partial failure.  But if that is untenable for you, the other solution is if one fails, we're actually going to delete both of them.  So it is a bit of a waste, but it does save you the trouble of having to think about exactly how to manage the recovery situation for man.  Let's see.  So we have a few more features.  These are kind of things that were a few more features that we're trying to work on.  So bitmap migration at the moment, the initial version of this did not actually support migrating across live hosts.  So the initial version of migration for us was to split these data, the dirty bitmap into one kilobyte chunks,  and then serialize it piece by piece.  And then using the bitmaps themselves, we could kind of copy as the drive got dirtied.  And for things that were small enough, we could just kind of skip the live phase and send it wholesale.  But maybe this doesn't scale so well, depending on how many bitmaps or how many drives you have.  This could really add some time to the migration stream.  So the bitmaps weren't transferred alongside the data.  So the data would go in its own loop, and the bitmaps would go in its own loop.  And then we had this dirty, dirty bitmap kind of solution that was kind of really confusing to name all the variables for.  But this captures changes during live migration, and we could resend the pieces if we needed to.  And it didn't use a lot of memory, but it kind of added some complexity to the migration itself.  So we have a second approach now using a post-copy technique.  If you're familiar with post-copy in general, post-copy migrations are the concept that you could migrate a VM from one machine to another  and then immediately start running the VM on the new machine, and then as it needs the memory, it would false and fetch the memory from the previous host.  And the reason this is useful is because it allows you to guarantee when the machine will pivot,  because if you are not worrying about the rate that the memory is being dirty,  it's possible that the VM will be so active, so busy, so loaded that it will dirty the memory faster than we can migrate it across the stream.  And the same kind of problem happens for disk.  But in post-copy, you just migrate the VM first with an initial set, and then as it needs memory, it fetches it from the old host.  So we figured that we could probably do something like that for bitmaps.  So we're just going to send the whole bitmap post-pivot, and then before the data has fully converged,  we're going to record the new writes on the target with a new bitmap.  We're going to disallow backups until all of the bitmaps have come back to us,  and then we're just going to merge the bitmaps on the target.  So what this gets us is the ability to guarantee that the bitmaps aren't going to be stalling the migration,  and then the price we pay for that is you may not be able to issue a new incremental backup for the next minute or something after you migrate as everything catches up.  This does add a new failure point.  So this is the unfortunate part.  We do consider this kind of a non-critical loss because you've just lost metadata,  but the bitmap chains can be restarted using the clear commands and so on,  but we are looking at ways perhaps to mitigate this kind of a data loss event.  I shouldn't say data loss, I should say metadata loss.  There are other options.  If this really spooks you out, you can always use shared storage migration and flush the bitmaps to the disk,  migrate and then load the bitmaps from the disk again, and that's probably a lot safer.  So for the persistence, like I'm hinting at, we were going to do a format agnostic solution,  but we were going to use QCALT 2 as a generic container to store bitmaps for arbitrary other drives,  but the problem we had with that is that QEMU is not in the business of managing file names or hierarchies for where things are stored.  We leave that to Libvert and other management tools.  So when we were trying to add a spec that allowed us to use QCALT 2 to describe the dirty state of other files and formats,  it quickly became obvious that we were trying to replicate too much from the layer above us.  So now we're just targeting the QCALT 2 format, which some people might be sad to hear, but I'll get to that in a bit.  So for right now, it's a QCALT 2 format.  Multiple bitmaps can be stored per file.  The bitmaps have types.  There's a full spec that was written kind of independent of this feature that allowed us to store these dirty bitmaps in a QCALT 2 file.  So this is extensible and we may use it in the future to restart migrations or things, but for now, it's incremental backup related.  The bitmaps can auto load in QEMU, like if there's a persistent bitmap associated with a storage file.  Once you give that to QEMU, the next time it boots, it sees, ah, yes, good, and it will start up for you.  The spec amendment is already merged and there are patches ready on list from Virtuovo,  and hopefully we're going to get these into 2.9.  So for non-QCALT 2 files, there are some options.  Some formats are going to add their own primary support for it.  I know Virtuovo wants to make sure that they have support in their parallels format,  but we may extend QCALT 2 into a right forwarding, like header node, if you will,  where instead of a backing file, which is usually read-only, we may have a read-write backing file,  where QCALT 2 will store the metadata for other images for you,  and then that way the managing layers can worry about the backing file relationships,  and all we need to worry about is that we now have a QCALT 2 that doesn't actually take rights,  it just forwards them to a raw file.  So we're hoping this is going to let us do persistent bitmap storage for raw disks for people who want that kind of performance,  but would still like incremental backups.  So I think I heard in the previous talk we were talking about VMware's CBT a bit.  So this was a problem that came up for us because everything I've described so far is what we call a push model,  where QEMU knows exactly what's dirty and where it's dirty, and you ask QEMU to author the backup.  So this is a largely QEMU-driven process, which isn't necessarily a problem for us or for people who are already using our models,  but we do have a lot of requests for making something similar to VMware's CBT,  where you have the ability to query for which blocks are dirty and then manually yourself copy those blocks out.  So at the moment, push model is already in QEMU, we are working on the pull model,  and the way the pull model works is it's going to offer a lightweight temporary snapshot,  so you will connect to QEMU, you will request a temporary, like a view of the image using the NBD protocol,  and then through the NBD protocol you're going to have the ability to ask,  you know, give me a list of the dirty sectors, I want to see what's dirty,  and then the client can at its own pace copy out the sectors it thinks are dirty,  and perhaps if this third party client knows more than we do about exactly what's dirty,  it can make its own decisions about what to backup, where, why, for instance.  But this is a feature that was requested by Virtuozo, and they've been working very hard on this,  and we are hoping that this is going to add a lot of flexibility to existing clients,  both for people who use VMware, or for people who already use their own scripts for QEMU,  we're hoping that one of the two push or pull models is going to fit nicely into the workflow.  So a little bit more about the snapshot, it is a point in time snapshot, we call it image fleecing,  you can open up an image and get a still view of the image at that point in time,  but because of that it's going to require an on-disk cache, because as the VM continues to run,  we need somewhere to flush those writes if we run out of memory.  So it has some downsides perhaps compared to the QEMU authored backups, but not that different,  but like I said it's going to offer kind of full control, and it's probably the most QEMU agnostic method  where you're kind of cutting QEMU out of having to manage these a bit,  and it is at the moment the only way to query exactly which blocks are dirty,  so it may be useful for antivirus research or things like that,  if you want to see what programs are dirty and where and so on.  So we do have some to-dos, we still have to work on the interface for some of the more advanced features,  the pull model, managing persistence, you know, if you say I want to delete this out of the drive,  I want to put a new bitmap in, or I don't want this bitmap to be persistent,  so there's a lot of interface questions that we're still working on,  we don't have like an integrity checker just yet, but we're working on one,  we would like to add the ability to do offline incremental backups,  so we would like a reference tool to make sure that if you have shut the VM down,  you can still parse this dirty bitmap and complete a backup offline if you would like to.  So, and I would also like to give a quick shout out,  we do have a GSOC project coming up, if anybody was actually interested,  we want to write a reference implementation, just a simple CLI tool,  probably using Python to take advantage of our existing test infrastructure.  If it sounds like something that you'd be interested in,  we're definitely looking for people to help us with this,  and so, just at the end here, so the project status, the interface is merged,  the incremental mode is merged, transaction support was merged in 2.5,  the persistence spec got merged in 2.6,  grouped transactions got added just in 2.8,  migration support is in review, it's on the list right now,  we're hoping to get it in for 2.9, persistence, it's the same story,  and pull model is still kind of an early design,  and then there were some specs written for NBD to allow us to use NBD for this purpose,  and we are hoping to get a lot of that in 2.10.  And, any questions?  Other questions about the report,  so in the end you get the plus and non-volumen incremental,  what do you do with those files?  You're not going to like this, so that's the management layer's job,  I don't care.  So, no, we tried to keep that complexity out of QME,  because we didn't want to replicate what Libvert was already doing,  so Libvert has whatever it has to manage its own storage pools and things like that,  so we tried to not care exactly about how many incrementals were already created,  we say that it's up to the management layer to remember the last incremental it made,  and to properly hook up the chain, so it's a very unix philosophy,  where if you ask for the wrong thing, you will get the wrong thing.  So, in that way we kept our part simple, and we just kind of...  Libvert's problem now.  Okay, so the question was if we were collaborating with Overt or not.  So, downstream at Red Hat, yes, I've been talking to some project managers  to help them explain exactly what we've made,  and I think this info is getting back to them,  but I haven't been personally working with someone in Overt,  but I did write a white paper for it that I distributed to some people,  and if there are any requests or anything like that,  if you know somebody directly who would definitely want to talk to me, send me an email.  Yes?  Could you repeat that last part for me?  As storage gets cheaper and deduplication technology is so far proliferating,  how much life is left and personally incremental or financial backup strategy?  Probably more than you would hope for, or I should repeat the question,  which was as storage gets cheaper and deduplication proliferates,  how much life is left in this kind of a technique?  More than you would hope, because we still use floppy drives for some things,  so it's always too early to say, oh, this is dead.  People do care about this a lot.  Other companies besides Red Hat are pouring a lot of time and effort into making sure that this works.  So as for how much sense it makes,  I'm not able to give you an incredibly great answer for that,  but I can tell you that businesses are extremely concerned with it  and are spending a lot of time and money to make sure that it works well.  So it's probably going to be around for however long you think it should,  and then another decade past that.  I guess I'm quite a bit early.  Oh, yeah?  Maybe I missed understanding this part of the handling of the bitmaps,  but can you describe when the number of the bitmaps per drive is more than one?  Okay, is the key so one is...  Oh, sure, sure. Okay, I see what you're asking.  So the question is, when would you want more than one bitmap per drive?  So if you wanted to implement separate backup paradigms at the same time,  so say you want a daily backup, and you could use one bitmap to do every day,  you could have a second bitmap to do every week,  you could in theory just do the daily backup  and then the back end can manage the different chains for you to give you different views,  but it was almost already there for us.  It would have taken more effort to remove that feature.  So the flexibility is there for us.  So some further reading, if anybody's interested in this stuff.  The Kimi project main page wiki has status and updates for us.  There's a project page for this in particular.  The bitmaps documentation for the QMP aspect of managing all the push model backups  is in the source tree under the docs folder bitmaps.md.  For the closest thing we have to a reference implementation right now,  we have an IO test which is written in Python  and that's going to run through several backup scenarios like failure on one node,  failure on two nodes, failure on individual mode, group mode and so on.  So that's actually the closest we have to seeing exactly how a program would request these backups.  The status white paper that I was talking about,  which is written for the benefit of Libvert and other layers above QEMU,  is currently at this little Google tiny URL link.  So if that's something that you were interested in reading,  take a picture or write it down or grab the slides or whatever you will.  And then for people who are interested in writing tools to interface with QEMU,  the jobs talk from the last KVM forum is up through the Linux foundation.  If you search KVM forum 2016, find my name and that'll be up there.  So I think I'm actually a little bit early but it'll give you time to scoot out.
Incremental_Backups_Good_things_come_in_small_packages_0_59.png Incremental_Backups_Good_things_come_in_small_packages_60_131.png Incremental_Backups_Good_things_come_in_small_packages_132_1319.png Incremental_Backups_Good_things_come_in_small_packages_1320_2375.png Incremental_Backups_Good_things_come_in_small_packages_2376_2927.png Incremental_Backups_Good_things_come_in_small_packages_2928_3407.png Incremental_Backups_Good_things_come_in_small_packages_3408_3755.png Incremental_Backups_Good_things_come_in_small_packages_3756_4055.png Incremental_Backups_Good_things_come_in_small_packages_4056_4115.png Incremental_Backups_Good_things_come_in_small_packages_4116_4175.png Incremental_Backups_Good_things_come_in_small_packages_4176_4259.png Incremental_Backups_Good_things_come_in_small_packages_4260_4367.png Incremental_Backups_Good_things_come_in_small_packages_4368_4871.png Incremental_Backups_Good_things_come_in_small_packages_4872_4967.png Incremental_Backups_Good_things_come_in_small_packages_4968_5375.png Incremental_Backups_Good_things_come_in_small_packages_5376_6275.png Incremental_Backups_Good_things_come_in_small_packages_6276_6359.png Incremental_Backups_Good_things_come_in_small_packages_6360_7823.png Incremental_Backups_Good_things_come_in_small_packages_7824_9275.png Incremental_Backups_Good_things_come_in_small_packages_9276_11063.png Incremental_Backups_Good_things_come_in_small_packages_11064_11843.png Incremental_Backups_Good_things_come_in_small_packages_11844_11903.png Incremental_Backups_Good_things_come_in_small_packages_11904_12167.png Incremental_Backups_Good_things_come_in_small_packages_12168_12287.png Incremental_Backups_Good_things_come_in_small_packages_12288_13187.png Incremental_Backups_Good_things_come_in_small_packages_13188_13427.png Incremental_Backups_Good_things_come_in_small_packages_13428_14411.png Incremental_Backups_Good_things_come_in_small_packages_14412_16283.png Incremental_Backups_Good_things_come_in_small_packages_16284_16403.png Incremental_Backups_Good_things_come_in_small_packages_16404_17315.png Incremental_Backups_Good_things_come_in_small_packages_17316_17903.png Incremental_Backups_Good_things_come_in_small_packages_17904_18323.png Incremental_Backups_Good_things_come_in_small_packages_18324_19079.png Incremental_Backups_Good_things_come_in_small_packages_19080_19571.png Incremental_Backups_Good_things_come_in_small_packages_19572_20963.png Incremental_Backups_Good_things_come_in_small_packages_20964_22055.png Incremental_Backups_Good_things_come_in_small_packages_22056_23507.png Incremental_Backups_Good_things_come_in_small_packages_23508_24275.png Incremental_Backups_Good_things_come_in_small_packages_24276_25727.png Incremental_Backups_Good_things_come_in_small_packages_25728_27347.png Incremental_Backups_Good_things_come_in_small_packages_27348_27815.png Incremental_Backups_Good_things_come_in_small_packages_27816_29183.png Incremental_Backups_Good_things_come_in_small_packages_29184_30239.png Incremental_Backups_Good_things_come_in_small_packages_30240_30359.png Incremental_Backups_Good_things_come_in_small_packages_30360_30995.png Incremental_Backups_Good_things_come_in_small_packages_30996_32051.png Incremental_Backups_Good_things_come_in_small_packages_32052_32483.png Incremental_Backups_Good_things_come_in_small_packages_32484_32843.png Incremental_Backups_Good_things_come_in_small_packages_32844_33419.png Incremental_Backups_Good_things_come_in_small_packages_33420_34871.png Incremental_Backups_Good_things_come_in_small_packages_34872_35195.png Incremental_Backups_Good_things_come_in_small_packages_35196_35483.png Incremental_Backups_Good_things_come_in_small_packages_35484_36491.png Incremental_Backups_Good_things_come_in_small_packages_36492_36635.png Incremental_Backups_Good_things_come_in_small_packages_36636_36743.png Incremental_Backups_Good_things_come_in_small_packages_36744_38111.png Incremental_Backups_Good_things_come_in_small_packages_38112_40043.png Incremental_Backups_Good_things_come_in_small_packages_40044_40427.png Incremental_Backups_Good_things_come_in_small_packages_40428_42719.png Incremental_Backups_Good_things_come_in_small_packages_42720_43427.png Incremental_Backups_Good_things_come_in_small_packages_43428_44999.png Incremental_Backups_Good_things_come_in_small_packages_45000_45251.png Incremental_Backups_Good_things_come_in_small_packages_45252_46439.png Incremental_Backups_Good_things_come_in_small_packages_46440_47303.png Incremental_Backups_Good_things_come_in_small_packages_47304_49883.png Incremental_Backups_Good_things_come_in_small_packages_49884_50855.png Incremental_Backups_Good_things_come_in_small_packages_50856_52187.png Incremental_Backups_Good_things_come_in_small_packages_52188_53363.png Incremental_Backups_Good_things_come_in_small_packages_53364_54767.png Incremental_Backups_Good_things_come_in_small_packages_54768_57635.png Incremental_Backups_Good_things_come_in_small_packages_57636_59087.png Incremental_Backups_Good_things_come_in_small_packages_59088_60191.png Incremental_Backups_Good_things_come_in_small_packages_60192_60875.png Incremental_Backups_Good_things_come_in_small_packages_60876_60935.png Incremental_Backups_Good_things_come_in_small_packages_60936_61079.png Incremental_Backups_Good_things_come_in_small_packages_61080_62015.png Incremental_Backups_Good_things_come_in_small_packages_62016_67115.png Incremental_Backups_Good_things_come_in_small_packages_67116_68591.png Incremental_Backups_Good_things_come_in_small_packages_68592_70859.png Incremental_Backups_Good_things_come_in_small_packages_70860_70991.png Incremental_Backups_Good_things_come_in_small_packages_70992_71039.png Incremental_Backups_Good_things_come_in_small_packages_71040_71114.png