Transactional Updates - deep dive
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Forster, Ignaz
How to update your systems without breaking them With the release of Leap 15 the new system role called "Transactional Server" will be available during the installation, so this is the perfect opportunity to have a look at the concept behind it and how to work with such a system in practice. In this talk we will have a look at transactional-updates from different angles: * The basic concepts behind transactional-update * How to use the Transactional Server or Kubic (Users & Administrators) * Packaging for transactional systems (Packagers) * How transactional-update compares to other solutions from various distributions (Developers) * Recent developments in the transactional-update world
 So, hello, good afternoon, and welcome to this session about transactional updates. Maybe let's start with a question into the audience before we begin. Who has actually read about transactional updates in one of those numerous channels now? I've read several posts on the OpenSUSE blog, and I've used about the transactional updates on news sites. So, yeah, who has actually read about transactional updates already? Okay, so that's a good portion of the audience. Maybe we can just skip some slides later then. What's the actual reason? The reason is probably that transactional updates was actually introduced in the form of the transactional server. It's been there for at least 15 now. We have had it for a time, but it's not the only place where you can get transactional updates. It originated from the Qubic project and the SusieCast platform. That's where it originally came from. There's two systems, both the transactional server and the Qubic system share one thing in common, namely, they have a read-only-root file system. So that's where transactional update is actually used. However, there's no reason to limit it to read-only-root file systems. Transactional update is just a regular package, which is part of Tumbleweed and Leap. If you want, you can also install it on a read-write file system, and I'll have some notes about that later in the talk. So what am I going to talk about? First of all, a small introduction into transactional updates. What's the architecture and the idea behind it? Maybe a short live demo on how to actually use it. Then you may have heard one of the... Is there a large echo here? Just... Okay. Just what I have in previous talks about transactional updates, what has changed since then, and the talk is called transactional update deep dive. So we'll have a deeper look at some of the mechanics behind it. Another question into the audience, who's actually packaging applications? Okay. Quite some people. The question is what do we have to do to actually be compatible with transactional updates? Get into that later. Sorry, I'll have to skip watching the alternatives from other distributions. We are not the only ones in that field. I just won't have any time to do that anymore. But we'll have a short look at what will be done next. So let's start with what is it and how does it actually work? So you may have read that slide yesterday in Richard's talk. Let's start with the definition of transactional updates. Transactional update is an update that is atomic. So that means it has to be either fully applied or not applied at all. The second part is that it doesn't influence the currently running system. So if something breaks, you certainly don't want to break your running system, but the update has to run somewhere in the background. There's a second criteria, which is the update has to be able to be rolled back. So if something fails, which you just noticed later after the update, you have to be able to get back to a snapshot where you know that you had a previous known good working state. So let's have a look at the current system. Currently, we are using snapper together with zipper. And if you have a look at it, we can see that we both, we get two snapshots, one snapshot before actually applying the system, before actually applying the update. And the second snapshot after the update was actually applied. The problem with this approach is that all the updates are done in the currently running system. So if something breaks, for example, if an RPM package cannot be applied, has some error in its post script or whatever, you have just actually broken your currently running system. So what you can do now is you can go back to a previous known work, a good known snapshot, but of course, we are violating our rule to have that atomic part. So what is transactional update actually doing differently here? Transactional update is also using snapper and serpent in the background. So we are just building up an existing technology and it will just put another layer on top of it. We also get two snapshots, one pre-snapshot and one post-snapshot. The pre-snapshot may be a bit strange if you are using it on a read-only file system. I'll get to that later. And the post-snapshot is not really a backup snapshot as it was in the snapper case, but it will actually be the working snapshot. Let's have a look at the graphical representation of that. We'll have the currently active system. Can I just paint in something here? No, I can't. We have the currently active system, which will get the backup snapshot and then we'll get a second snapshot where the actual update is actually applied into. During that whole time, the active system will just continue to see its own current root file system. It won't see anything about that snapshot over there where the actual update is done. So how do you get the system to actually see the new contents when the update was successful? Then the default B3FS root file system will be set to that new snapshot. You'll have to reboot that's the atomic part of the atomic update and then you'll be in the new snapshot. If you want to go back, you can just go back to any of those previous two snapshots and yeah, that's basically the whole magic behind it. Now maybe who has actually used transactional update already? Well, quite some people, but still a lot of you didn't actually see how it's actually working. Let's have a look at a system which is actually running. Just let me switch the display here. Can you see something? Okay. So this is a current cubic release which is one of those read-only systems and I've installed it a few days ago and that's the current updates. So what can I actually do? I can, for example, still type super list repositories and we can see that it's just a conventional tumbleweed system. Nothing special about it here. Sorry. It's gone. Okay. Back again. Yeah, I've adjusted that. We can see that it's just a conventional tumbleweed system without any special fancy things, but if we just one note, you don't have to remember all those commands. I've put together a cheat sheet. You can see what I've been typing in here. But we can see that the root file system is a read-only file system. If you have a look at the other mounts, you can see a lot of read-write file systems. I'll just get to it in a minute. So we had repositories listed. We can even do a separate refresh. And you may be wondering how is it even possible if it's a root file system. The first one, the repositories are stored in ETC. ETC has a special handling. We like the var directory, I'll come back to that later. Both of it is a read-write directory. So if I actually try to install something now, any suggestions? No, sorry. I still have to be able to use the system. I've taken three routes. Then we can see that this will actually fail because we are on a read-only file system. So that's where the transactional update, the command will come in. Transactional update is just a regular command, like you would expect it, with several operations. One of those operations is, for example, if you want to install or update our system, let's hope nothing has changed. Oh, we are getting a new kernel. Okay. Yeah. So now we have to wait. But now we can see what transactional update is actually doing. We just got those two new snapshots. And the kernel update will be installed into that new snapshot. We'll also get, of course, a new In-A-Di, which the kernel will be generating, and maybe even a new crop settings, maybe. All of those are handled in that snapshot. How long is it going to take? Yeah, sure. Mm-hmm. Yeah. Yeah. Just a second. Yeah. The question was, if there are several packages in a package update, or in a transactional update, will I get a new snapshot for each package that was actually updated? No. That's one snapshot, a transactional update call. Or two snapshots. We get that pre and post snapshot. But all of those updates will be applied in that one snapshot and that post snapshot. I have one question based on that. You just ran transactional update, and it didn't ask you if you want to apply it or automatically said yes to the question, if I'm so correct on the terminal. My first question is, so the transactional update is not interactive, or I think sometime I tried also the transactional update and asked me if I want yes or no. And when I press control C, I want to cancel my action. Basically, I think it removed the snapshot, but I was not sure if it removed the future snapshot or my even current one, which was fine. Whenever you type transactional update, it will depend on if it's interactive or not, depending on the command you type. Transactional update was developed as part of the cubic project where automatic updates were the main reason for it to actually work. So if you type up or up, those will be non-interactive commands. I'll do the second part now if I want to install trace out again. I'll type transactional update PKG in and the command. And in this case, I'll have to confirm I want to actually install it. So now we got transactional trace out installed. And now the question is how do we actually use it? You'll notice that line towards the end. Please reboot your machine to activate the changes and avoid data loss. I just said so before. Yeah, that's the question. That's the next point. We got a warning during the beginning. Just a second, I'll highlight it, which says, warning default snapshot differs from current snapshot. Any changes within the previous snapshot will be discarded. If you have a look at our current file system, you'll see that we are in snapshot number 10. If you actually have a look at snapper, you'll see that we are meanwhile at snapshot number 16. So what has happened? We got three snapshots based on snapshot number 10. All of these three snapshots are independent of each other, which means whenever I type transactional update, I'll get a snapshot based on our current system state. So please take care whenever you want to apply your changes, you have to reboot the system before doing anything else. Otherwise, you'll just discard your changes you just did. I mean, it's still available in that snapshot. You can boot that snapshot explicitly from the grab menu, or you can just say transactional update rollback, for example, 14. And then I'd have set that specific snapshot as the one that I actually want to use. However, please take care. Each call will just create another snapshot based on the current system, just ignoring the other things you've done in the meantime. So the question was still, we just installed trace route. If I just type it in here, as expected, it won't work. So I'd have to reboot into that new snapshot now. I just broke it if you took attention because I just resorted the snapshot to snapshot number 14, which didn't contain trace route. So I'd just have to read that system to snapshot number 16. Before I do that, I'll just create another snapshot, and I'll show you another command which might come in quite handy. If you want to see if your command actually did what it was expected to do, you can just type in transactional update shell. What that will do is it will give you a read write shell just before closing the snapshot for read into read only mode. And you can just see what you actually did. So if you just install trace route again, I've just combined some commands now. You can combine almost any number of commands if you want. And you can see we are stuck in a transactional update shell here. This time, you'll be able to actually type the command, and it will work. If you are leaving the shell again, then it's not working again because we aren't booted into that snapshot yet. Just a second. Yeah, I've already shown you with a snapper list, you can actually see the current snapshots. With find mount, you can see the actual snapshot where we actually are. If you want to see this snapshot we'll next boot into, you can type B3FS, subvolume gets default of the root file system so we can see the next snapshot we'll actually boot into is snapshot number 18. That's because our installation was successful, and the last successful transactional update run will always set the snapshot for the next boot. So that's basically it. I want reboot that will take another minute, which we don't have. So just a second, let's switch back to the presentation. As I said, I've prepared a cheat sheet. All the commands I've just been using are listed on that sheet. And yeah, that's the basic operation. So basically all the writing operations into the actual system will be replaced by a corresponding transactional update call. So I just mentioned that, I think repeatedly. Please consider that the age transactional update call will override your existing call if you're your previous run, if you didn't boot into that system. And be aware, if you're using transactional update on a read-write system, you will have a high chance of overwriting actual system things that actually happened. Because on a read-write system, nothing will prevent you from just modifying anything into the root file system. And you may be surprised that you don't get the files or the things you'd expect in the next run. So if you've, yeah, let's just ask, who have you seen any of the previous talks last year during either FOS-DEM or OpenSusicon about transactional updates? Oh, a small number, so I don't have to mention it. You may still be glad to see that var is one single subvolume now. You probably can't read what's on that screenshot, and that has a good reason, because every line which contains some of those red things there is one dedicated subvolume for one var directory that used to be the case up until tumbled, I think somewhere at the beginning of this year. Now it's all combined into one var directory. That has one specific reason, because previously we had not variable state in var, most prominently the RPM database. The RPM database, of course, has to be part of the snapshot, because you have to roll back to that specific state, the applications and the RPM database of one unit in that case. So it was moved to use this image RPM, you may be wondering about that strange path that's in combination with Fedora, we're also with project atomic having a similar problem, but now that's the reason we, as I said, it will be working with a unified var directory. I'll just skip the next slide. So this is called transactional update deep dive, so let's have some deeper look at it. What's actually happening in the background? As a user's perspective, you just don't have to know that, but what's happening is actually that there are several special directories which have to be taken care of. The first one is var, we just mentioned that, RPM packages stored in there, the RPM databases stored in there, but there are several other things, for example, databases will store the data in there, so that has several implications. First of all, we can't just roll back the var directory, because obviously you don't want to lose your customer data, for example. So another point is, of course, you have to actually have right permissions to that directory, because you actually want to store the information, and that means you'll usually have var on a dedicated subvolume or dedicated partition. Unfortunately, for you, var is not part of the root file system, so if some package will actually change the contents of var, that will be a problem, because it's not mounted in there, so this will usually fail. We have some special dedicated mechanism for it, so if you did an update with transaction update on the next boot after the update, a script will just see what should have been done in var and will generate those directories in files, but what won't work is if the RPM boots just modified something, the post scripts, for example, just modified any contents there, so they will have modified just to some bogus data, which has no direct relation to that data which is actually running in the system. There's another special directory, which is ETC, of course, also in a read-only file system, I assume you don't want to use the stock configuration files, but want to actually use them or modify them, maybe you have a configuration framework like Solve, like we just heard about before, so that one has to be another writable directory. For that, we have another solution, which is ETC is just an overlay. So whenever you modify any of the default files which were distributed by the distribution, you will just get another file in the overlay, the overlay itself is stored in var again, so that's another reason why that's an important file system. And just a second, what will now happen if you create a snapshot is you will get the contents of the overlay file system synced onto the snapshot itself. It has one huge advantage, namely you can just roll back to any older previous snapshot and you still get the correct configuration for that specific snapshot. Let's think about fill-up templates, which will just do merges if you install another major version of some application. You may happen to have different configuration file syntaxes, so you don't want to use the new configuration files with the old application and vice versa. So that's basically solving that problem. That also has the advantage the RPM packages can actually modify the real configuration files which are used in production. Unfortunately, that also means that we have to clean up the overlay file system somehow. So on the next boot of the system, if you put into the new snapshot, a script will run that will detect or basically it will just delete the overlay contents. With one exception, of course, the reboot doesn't necessarily have to be directly after your actual snapshot, so it may be that there are some changes to the ATC directory after you've taken your snapshot. So if it's detected that some files have been modified after the snapshot, then those files will still be left in the overlay of S. So they will probably be added to the real root file system on the next snapshot. There are special sub volumes, namely opt, valoc and bootcrap2. Those will be available in the snapshot. Because they are just there for regular third party packages, for example, for internal reasons. If your package is installing its data somewhere else, then you will probably get into trouble. For example, if your package is installing into SIV, that's not supposed to be used by packages basically. And that's not mounted into the snapshot. So whatever your package will be doing there, it won't be visible into the final system because it will always be over mounted. Let's skip that. House checker and reboot manager is something which we'll see tomorrow in Paul's talk. So let me emphasize that again. Basically transaction updates have the huge advantage that you can basically use any package we have in our OpenSUSE repository with the only exception that you have to take some care. If you are following the packaging guidelines and the file hierarchy standards, then everything will just usually just work out of the box. If you really need to do some handling and bar, then please have a look at the migration update section, you'll usually want to use a system descript which will basically do that modification, so that you want to do on next boot, namely when you actually boot that system that you have just updated. And yeah, in that case, you'll get the required changes. You can do the required changes. So let's just have a short summary of what transaction updates will actually do. It has one disadvantage, namely it's only compatible with B3FS because it heavily relies on its features and it's heavily using its features. But especially when comparing it to other distributions, for example Fedora or Ubuntu or CoreS. Then we have that huge advantage that you don't introduce any new techniques. We can just continue our existing packages. And yeah, that's basically it. The only thing you have to remember is to use transactional update the command itself. Transactional update is a general purpose tool, also in comparison to the other distributions. You can use it for almost everything. It's especially useful, of course, if you're using it together with servers where you want to, I don't want to interrupt your services or only want to interrupt your services in specific service intervals. And of course for clusters where you want to make sure that every cluster has the same state. So what's next? The last slide, maybe you want to see that in SLAS. That's the question we actually get quite a lot. There are plans to include it in the next service pack. It's not set yet because we want to resolve all the VAR packages until then. You may be glad to hear that we have a mechanism similar to DM Verity with IMA and EVM. We are just evaluating that. And the last point, I'll just get into that one. The question is why do we have to type transactional update anyway? Because a snapper also made it to integrate it into SIPA. That's something we are looking into, to also just use it out of the box without even knowing you're using a transactional update system. So yeah, visit Paul Gornan's talk tomorrow if you want to see something else. I hope we have 30 seconds left for questions. Yeah, up there? How did you decide to go to reboot this error? Either, usually you can just type transactional update reboot and depending on your configuration several things may appear. By default, it's using an application called reboot manager, which you can configure for reboot intervals. You can just say we have our downtime at three o'clock and it will just reboot the system. Then if it has to reboot it at all, maybe if the update just fails, then of course nothing will be rebooted. You can also configure it to just use the system to reboot. Then the system will be rebooted immediately. Or you can just type reboot manually by hand. Okay, Richard, how much time of your, may I steal of your time? Richard didn't complain. We can answer two more questions. Yeah, up there? There are ways to convert more related to server and the Rekus. The question is, can we convert an existing non-transactional system into a transactional system? No automated way. You can do it by hand, basically, by just setting the sub volume to read only and of course, modifying the FS tab entry to set it to read only. Please install transactional updates before, because otherwise you won't be able to update the system. That's more or less a manual way to do it, but it's not supported. Let's get it that way. You said just the problem is you do multiple changes for reporting that only the last one wins. The question is why do we only display a warning if a user is doing multiple transactional update calls without rebooting? As said, it's originally developed to be used in a cluster setup where you don't do any automatic steps at all. So by default, you'll have a daily system D timer which will call transactional update by itself. If something has to be updated, it will inform reboot manager which will then just reboot the system. Basically for testing, I think it's quite handy to be able to just type in the commands, but yeah, maybe we could get a bigger warning. Yeah, that's something we should consider. Final point, if you want to contact us or if you have any questions, you can find us on cubic.opencensored.org. You'll also find our communications channel there. So that's it. Thanks a lot for listening. Thank you.
 So, hello, good afternoon, and welcome to this session about transactional updates.  Maybe let's start with a question into the audience before we begin.  Who has actually read about transactional updates in one of those numerous channels  now?  I've read several posts on the OpenSUSE blog, and I've used about the transactional updates  on news sites.  So, yeah, who has actually read about transactional updates already?  Okay, so that's a good portion of the audience.  Maybe we can just skip some slides later then.  What's the actual reason?  The reason is probably that transactional updates was actually introduced in the form  of the transactional server.  It's been there for at least 15 now.  We have had it for a time, but it's not the only place where you can get transactional  updates.  It originated from the Qubic project and the SusieCast platform.  That's where it originally came from.  There's two systems, both the transactional server and the Qubic system share one thing  in common, namely, they have a read-only-root file system.  So that's where transactional update is actually used.  However, there's no reason to limit it to read-only-root file systems.  Transactional update is just a regular package, which is part of Tumbleweed and Leap.  If you want, you can also install it on a read-write file system, and I'll have some  notes about that later in the talk.  So what am I going to talk about?  First of all, a small introduction into transactional updates.  What's the architecture and the idea behind it?  Maybe a short live demo on how to actually use it.  Then you may have heard one of the...  Is there a large echo here?  Just...  Okay.  Just what I have in previous talks about transactional updates, what has changed since  then, and the talk is called transactional update deep dive.  So we'll have a deeper look at some of the mechanics behind it.  Another question into the audience, who's actually packaging applications?  Okay.  Quite some people.  The question is what do we have to do to actually be compatible with transactional updates?  Get into that later.  Sorry, I'll have to skip watching the alternatives from other distributions.  We are not the only ones in that field.  I just won't have any time to do that anymore.  But we'll have a short look at what will be done next.  So let's start with what is it and how does it actually work?  So you may have read that slide yesterday in Richard's talk.  Let's start with the definition of transactional updates.  Transactional update is an update that is atomic.  So that means it has to be either fully applied or not applied at all.  The second part is that it doesn't influence the currently running system.  So if something breaks, you certainly don't want to break your running system, but the  update has to run somewhere in the background.  There's a second criteria, which is the update has to be able to be rolled back.  So if something fails, which you just noticed later after the update, you have to be able  to get back to a snapshot where you know that you had a previous known good working state.  So let's have a look at the current system.  Currently, we are using snapper together with zipper.  And if you have a look at it, we can see that we both, we get two snapshots, one snapshot  before actually applying the system, before actually applying the update.  And the second snapshot after the update was actually applied.  The problem with this approach is that all the updates are done in the currently running  system.  So if something breaks, for example, if an RPM package cannot be applied, has some error  in its post script or whatever, you have just actually broken your currently running system.  So what you can do now is you can go back to a previous known work, a good known snapshot,  but of course, we are violating our rule to have that atomic part.  So what is transactional update actually doing differently here?  Transactional update is also using snapper and serpent in the background.  So we are just building up an existing technology and it will just put another layer on top  of it.  We also get two snapshots, one pre-snapshot and one post-snapshot.  The pre-snapshot may be a bit strange if you are using it on a read-only file system.  I'll get to that later.  And the post-snapshot is not really a backup snapshot as it was in the snapper case, but  it will actually be the working snapshot.  Let's have a look at the graphical representation of that.  We'll have the currently active system.  Can I just paint in something here?  No, I can't.  We have the currently active system, which will get the backup snapshot and then we'll  get a second snapshot where the actual update is actually applied into.  During that whole time, the active system will just continue to see its own current  root file system.  It won't see anything about that snapshot over there where the actual update is done.  So how do you get the system to actually see the new contents when the update was successful?  Then the default B3FS root file system will be set to that new snapshot.  You'll have to reboot that's the atomic part of the atomic update and then you'll be in  the new snapshot.  If you want to go back, you can just go back to any of those previous two snapshots and  yeah, that's basically the whole magic behind it.  Now maybe who has actually used transactional update already?  Well, quite some people, but still a lot of you didn't actually see how it's actually  working.  Let's have a look at a system which is actually running.  Just let me switch the display here.  Can you see something?  Okay.  So this is a current cubic release which is one of those read-only systems and I've  installed it a few days ago and that's the current updates.  So what can I actually do?  I can, for example, still type super list repositories and we can see that it's just  a conventional tumbleweed system.  Nothing special about it here.  Sorry.  It's gone.  Okay.  Back again.  Yeah, I've adjusted that.  We can see that it's just a conventional tumbleweed system without any special fancy things, but  if we just one note, you don't have to remember all those commands.  I've put together a cheat sheet.  You can see what I've been typing in here.  But we can see that the root file system is a read-only file system.  If you have a look at the other mounts, you can see a lot of read-write file systems.  I'll just get to it in a minute.  So we had repositories listed.  We can even do a separate refresh.  And you may be wondering how is it even possible if it's a root file system.  The first one, the repositories are stored in ETC.  ETC has a special handling.  We like the var directory, I'll come back to that later.  Both of it is a read-write directory.  So if I actually try to install something now, any suggestions?  No, sorry.  I still have to be able to use the system.  I've taken three routes.  Then we can see that this will actually fail because we are on a read-only file system.  So that's where the transactional update, the command will come in.  Transactional update is just a regular command, like you would expect it, with several operations.  One of those operations is, for example, if you want to install or update our system,  let's hope nothing has changed.  Oh, we are getting a new kernel.  Okay.  Yeah.  So now we have to wait.  But now we can see what transactional update is actually doing.  We just got those two new snapshots.  And the kernel update will be installed into that new snapshot.  We'll also get, of course, a new In-A-Di, which the kernel will be generating, and maybe  even a new crop settings, maybe.  All of those are handled in that snapshot.  How long is it going to take?  Yeah, sure.  Mm-hmm.  Yeah.  Yeah.  Just a second.  Yeah.  The question was, if there are several packages in a package update, or in a transactional  update, will I get a new snapshot for each package that was actually updated?  No.  That's one snapshot, a transactional update call.  Or two snapshots.  We get that pre and post snapshot.  But all of those updates will be applied in that one snapshot and that post snapshot.  I have one question based on that.  You just ran transactional update, and it didn't ask you if you want to apply it or  automatically said yes to the question, if I'm so correct on the terminal.  My first question is, so the transactional update is not interactive, or I think sometime  I tried also the transactional update and asked me if I want yes or no.  And when I press control C, I want to cancel my action.  Basically, I think it removed the snapshot, but I was not sure if it removed the future  snapshot or my even current one, which was fine.  Whenever you type transactional update, it will depend on if it's interactive or not,  depending on the command you type.  Transactional update was developed as part of the cubic project where automatic updates  were the main reason for it to actually work.  So if you type up or up, those will be non-interactive commands.  I'll do the second part now if I want to install trace out again.  I'll type transactional update PKG in and the command.  And in this case, I'll have to confirm I want to actually install it.  So now we got transactional trace out installed.  And now the question is how do we actually use it?  You'll notice that line towards the end.  Please reboot your machine to activate the changes and avoid data loss.  I just said so before.  Yeah, that's the question.  That's the next point.  We got a warning during the beginning.  Just a second, I'll highlight it, which says, warning default snapshot differs from current  snapshot.  Any changes within the previous snapshot will be discarded.  If you have a look at our current file system, you'll see that we are in snapshot number  10.  If you actually have a look at snapper, you'll see that we are meanwhile at snapshot number  16.  So what has happened?  We got three snapshots based on snapshot number 10.  All of these three snapshots are independent of each other, which means whenever I type  transactional update, I'll get a snapshot based on our current system state.  So please take care whenever you want to apply your changes, you have to reboot the system  before doing anything else.  Otherwise, you'll just discard your changes you just did.  I mean, it's still available in that snapshot.  You can boot that snapshot explicitly from the grab menu, or you can just say transactional  update rollback, for example, 14.  And then I'd have set that specific snapshot as the one that I actually want to use.  However, please take care.  Each call will just create another snapshot based on the current system, just ignoring  the other things you've done in the meantime.  So the question was still, we just installed trace route.  If I just type it in here, as expected, it won't work.  So I'd have to reboot into that new snapshot now.  I just broke it if you took attention because I just resorted the snapshot to snapshot number  14, which didn't contain trace route.  So I'd just have to read that system to snapshot number 16.  Before I do that, I'll just create another snapshot, and I'll show you another command  which might come in quite handy.  If you want to see if your command actually did what it was expected to do, you can just  type in transactional update shell.  What that will do is it will give you a read write shell just before closing the snapshot  for read into read only mode.  And you can just see what you actually did.  So if you just install trace route again, I've just combined some commands now.  You can combine almost any number of commands if you want.  And you can see we are stuck in a transactional update shell here.  This time, you'll be able to actually type the command, and it will work.  If you are leaving the shell again, then it's not working again because we aren't booted  into that snapshot yet.  Just a second.  Yeah, I've already shown you with a snapper list, you can actually see the current snapshots.  With find mount, you can see the actual snapshot where we actually are.  If you want to see this snapshot we'll next boot into, you can type B3FS, subvolume gets  default of the root file system so we can see the next snapshot we'll actually boot  into is snapshot number 18.  That's because our installation was successful, and the last successful transactional update  run will always set the snapshot for the next boot.  So that's basically it.  I want reboot that will take another minute, which we don't have.  So just a second, let's switch back to the presentation.  As I said, I've prepared a cheat sheet.  All the commands I've just been using are listed on that sheet.  And yeah, that's the basic operation.  So basically all the writing operations into the actual system will be replaced by a corresponding  transactional update call.  So I just mentioned that, I think repeatedly.  Please consider that the age transactional update call will override your existing call  if you're your previous run, if you didn't boot into that system.  And be aware, if you're using transactional update on a read-write system, you will have  a high chance of overwriting actual system things that actually happened.  Because on a read-write system, nothing will prevent you from just modifying anything into  the root file system.  And you may be surprised that you don't get the files or the things you'd expect in the  next run.  So if you've, yeah, let's just ask, who have you seen any of the previous talks last year  during either FOS-DEM or OpenSusicon about transactional updates?  Oh, a small number, so I don't have to mention it.  You may still be glad to see that var is one single subvolume now.  You probably can't read what's on that screenshot, and that has a good reason, because every  line which contains some of those red things there is one dedicated subvolume for one var  directory that used to be the case up until tumbled, I think somewhere at the beginning  of this year.  Now it's all combined into one var directory.  That has one specific reason, because previously we had not variable state in var, most prominently  the RPM database.  The RPM database, of course, has to be part of the snapshot, because you have to roll  back to that specific state, the applications and the RPM database of one unit in that case.  So it was moved to use this image RPM, you may be wondering about that strange path that's  in combination with Fedora, we're also with project atomic having a similar problem, but  now that's the reason we, as I said, it will be working with a unified var directory.  I'll just skip the next slide.  So this is called transactional update deep dive, so let's have some deeper look at it.  What's actually happening in the background?  As a user's perspective, you just don't have to know that, but what's happening is actually  that there are several special directories which have to be taken care of.  The first one is var, we just mentioned that, RPM packages stored in there, the RPM databases  stored in there, but there are several other things, for example, databases will store the  data in there, so that has several implications.  First of all, we can't just roll back the var directory, because obviously you don't  want to lose your customer data, for example.  So another point is, of course, you have to actually have right permissions to that directory,  because you actually want to store the information, and that means you'll usually have var on  a dedicated subvolume or dedicated partition.  Unfortunately, for you, var is not part of the root file system, so if some package will  actually change the contents of var, that will be a problem, because it's not mounted  in there, so this will usually fail.  We have some special dedicated mechanism for it, so if you did an update with transaction  update on the next boot after the update, a script will just see what should have been  done in var and will generate those directories in files, but what won't work is if the RPM  boots just modified something, the post scripts, for example, just modified any contents there,  so they will have modified just to some bogus data, which has no direct relation to that  data which is actually running in the system.  There's another special directory, which is ETC, of course, also in a read-only file  system, I assume you don't want to use the stock configuration files, but want to actually  use them or modify them, maybe you have a configuration framework like Solve, like we  just heard about before, so that one has to be another writable directory.  For that, we have another solution, which is ETC is just an overlay.  So whenever you modify any of the default files which were distributed by the distribution,  you will just get another file in the overlay, the overlay itself is stored in var again,  so that's another reason why that's an important file system.  And just a second, what will now happen if you create a snapshot is you will get the  contents of the overlay file system synced onto the snapshot itself.  It has one huge advantage, namely you can just roll back to any older previous snapshot  and you still get the correct configuration for that specific snapshot.  Let's think about fill-up templates, which will just do merges if you install another  major version of some application.  You may happen to have different configuration file syntaxes, so you don't want to use the  new configuration files with the old application and vice versa.  So that's basically solving that problem.  That also has the advantage the RPM packages can actually modify the real configuration  files which are used in production.  Unfortunately, that also means that we have to clean up the overlay file system somehow.  So on the next boot of the system, if you put into the new snapshot, a script will run  that will detect or basically it will just delete the overlay contents.  With one exception, of course, the reboot doesn't necessarily have to be directly after  your actual snapshot, so it may be that there are some changes to the ATC directory after  you've taken your snapshot.  So if it's detected that some files have been modified after the snapshot, then those files  will still be left in the overlay of S. So they will probably be added to the real root  file system on the next snapshot.  There are special sub volumes, namely opt, valoc and bootcrap2.  Those will be available in the snapshot.  Because they are just there for regular third party packages, for example, for internal reasons.  If your package is installing its data somewhere else, then you will probably get into trouble.  For example, if your package is installing into SIV, that's not supposed to be used by  packages basically.  And that's not mounted into the snapshot.  So whatever your package will be doing there, it won't be visible into the final system  because it will always be over mounted.  Let's skip that.  House checker and reboot manager is something which we'll see tomorrow in Paul's talk.  So let me emphasize that again.  Basically transaction updates have the huge advantage that you can basically use any package  we have in our OpenSUSE repository with the only exception that you have to take some  care.  If you are following the packaging guidelines and the file hierarchy standards, then everything  will just usually just work out of the box.  If you really need to do some handling and bar, then please have a look at the migration  update section, you'll usually want to use a system descript which will basically do  that modification, so that you want to do on next boot, namely when you actually boot  that system that you have just updated.  And yeah, in that case, you'll get the required changes.  You can do the required changes.  So let's just have a short summary of what transaction updates will actually do.  It has one disadvantage, namely it's only compatible with B3FS because it heavily relies on its  features and it's heavily using its features.  But especially when comparing it to other distributions, for example Fedora or Ubuntu  or CoreS.  Then we have that huge advantage that you don't introduce any new techniques.  We can just continue our existing packages.  And yeah, that's basically it.  The only thing you have to remember is to use transactional update the command itself.  Transactional update is a general purpose tool, also in comparison to the other distributions.  You can use it for almost everything.  It's especially useful, of course, if you're using it together with servers where you want  to, I don't want to interrupt your services or only want to interrupt your services in  specific service intervals.  And of course for clusters where you want to make sure that every cluster has the same  state.  So what's next?  The last slide, maybe you want to see that in SLAS.  That's the question we actually get quite a lot.  There are plans to include it in the next service pack.  It's not set yet because we want to resolve all the VAR packages until then.  You may be glad to hear that we have a mechanism similar to DM Verity with IMA and EVM.  We are just evaluating that.  And the last point, I'll just get into that one.  The question is why do we have to type transactional update anyway?  Because a snapper also made it to integrate it into SIPA.  That's something we are looking into, to also just use it out of the box without even knowing  you're using a transactional update system.  So yeah, visit Paul Gornan's talk tomorrow if you want to see something else.  I hope we have 30 seconds left for questions.  Yeah, up there?  How did you decide to go to reboot this error?  Either, usually you can just type transactional update reboot and depending on your configuration  several things may appear.  By default, it's using an application called reboot manager, which you can configure for  reboot intervals.  You can just say we have our downtime at three o'clock and it will just reboot the system.  Then if it has to reboot it at all, maybe if the update just fails, then of course nothing  will be rebooted.  You can also configure it to just use the system to reboot.  Then the system will be rebooted immediately.  Or you can just type reboot manually by hand.  Okay, Richard, how much time of your, may I steal of your time?  Richard didn't complain.  We can answer two more questions.  Yeah, up there?  There are ways to convert more related to server and the Rekus.  The question is, can we convert an existing non-transactional system into a transactional  system?  No automated way.  You can do it by hand, basically, by just setting the sub volume to read only and of  course, modifying the FS tab entry to set it to read only.  Please install transactional updates before, because otherwise you won't be able to update  the system.  That's more or less a manual way to do it, but it's not supported.  Let's get it that way.  You said just the problem is you do multiple changes for reporting that only the last one  wins.  The question is why do we only display a warning if a user is doing multiple transactional  update calls without rebooting?  As said, it's originally developed to be used in a cluster setup where you don't do  any automatic steps at all.  So by default, you'll have a daily system D timer which will call transactional update  by itself.  If something has to be updated, it will inform reboot manager which will then just reboot  the system.  Basically for testing, I think it's quite handy to be able to just type in the commands,  but yeah, maybe we could get a bigger warning.  Yeah, that's something we should consider.  Final point, if you want to contact us or if you have any questions, you can find us  on cubic.opencensored.org.  You'll also find our communications channel there.  So that's it.  Thanks a lot for listening.  Thank you.
openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_0_2447.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_2448_10739.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_10740_10991.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_10992_11087.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_11088_11831.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_11832_11879.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_11880_12035.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_12036_12299.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_12300_13247.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_13248_13439.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_13440_14531.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_14532_14771.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_14772_15203.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_15204_15359.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_15360_19775.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_19776_19919.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_19920_20147.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_20148_20939.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_20940_21143.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_21144_22607.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_22608_25043.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_25044_25931.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_25932_26075.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_26076_27335.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_27336_27395.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_27396_27515.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_27516_27563.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_27564_28079.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_28080_40127.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_40128_45359.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_45360_48107.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_48108_50279.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_50280_50387.png openSUSE_Conference_2018_-_Transactional_Updates_-_akehodya7vA_50388_50777.png