A Novel Undo Tool for Plone
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Upton, Sean
A user-facing restoration system built around zc.beforestorage. Modifications or deletions of content items can have unintended consequences, or be themselves unintended. Ages ago, Plone exposed ZODB's transactional undo facilities to users, but these have proven too fragile for many real-world use cases. This talk presents a new tool integrated into the Plone user interface that allows (usually administrative) users to undo modifications that need to be reverted to previous state. More than just a "trash bin" for restoring deleted items, this tool provides some degree of simple content rollback for one or more content items, and is built on zc.beforestorage. Carefully done, a better cherry-pick restoration of your undo-history is both possible and safe.
 How many of you have had to programmatically rescue your customers from human error? Okay. How many of you found it painful? Okay. I still have a site that was broken a week or two ago that I just haven't had. Okay. I just hope he finds someone else. The second question might not seem related, but it's related in concept. How many of you have some kind of need for an audit log of changes that are made to your site for regulatory purposes or any other reason? Okay. My goal here is to talk about an approach that I have and pitch this approach and discuss whether or not this is something that might be appropriate at some future point in the Plone Corps. My initial goal was to demo an actual product. What I have here is a half-finished product that is well-tested but has no user interface. What we're going to talk about is what's going on under the hood, how I'm trying to solve this problem, what I've done before, and what I'm proposing we might be able to do. I'd like to spark a discussion. There's been some interest in this approach in discussions previously on Twitter and GitHub. If there's any questions, my goal here is just to really get people thinking about the viability of this idea and whether or not it's something that we want to, as a community, consider. My disclaimers here. First, this is an experimental approach. I'm developing this in an add-on that can be installed and uninstalled in Plone. It wouldn't necessarily be if it was ever integrated into the Plone Corps via a plip and standard release process, it would probably be refederated into a variety of different places, not necessarily into a single add-on to prevent any kind of package proliferation. But it is an unfinished work in progress. What I have working right now in the package with tests is a pretty reasonable back-end for modification logging. It uses a variety of events for four different kinds of facilities. Those facilities are modification, deletion, addition, and moves or renames. We'd have the audit log that would tell you for any of those four facilities, which kind of, in last and first-out order, and basically reverse chronological order, which changes were made, who made the change, what the content was, if it was a move, we want to do things like log the previous path and the current path, those kinds of things. We're logging some very basic small metadata. We do some of this stuff in other places, like for modifications we have the catalog, we have CMF additions, but in some cases it's not going to be sufficient for the kinds of restores that we need to do. For the customer that I had to do these rescues for, they had some data that was stored in the ZODB that was not necessarily content-ish. The content item was a proxy to a lot of underlying ZODB data stored on that object that could be imported by ZEXP, but could not be simply just restored from something like CMF additions. I want to pitch this idea, I say it's about undo, but really I don't mean undo. I mean restoration in the broader sense. Because when we talk about undo, there was a ticket or an issue that was put on GitHub some months ago asking for us to restore undo into Plone. We had this undo tab in Plone 3 or undo action. We can't do that because transactional undo in ZODB just doesn't really work in the real world for a variety of reasons. On any reasonably loaded site after some time, you just can't get the undo to work. I'm here to pitch an idea. I'm here to show you some of the way in which I manually, programmatically rescued my customer and talk about the audit logging system that I've been working on, but also pitch the talk about the need for a UI. I'm working on it. This is work in progress. I was hoping to have a little more finish before the conference, but this is something that I think that if there's interest in this, I'd like to spark discussion about. We'll look at the pieces, so we'll talk about them and hopefully find some kind of practical way forward. Here's my motivations. There was a reasonable degree of interest in discussion previously on GitHub and Twitter. This happened after I had gone through three different rescues for one customer. These three rescues, one of them was a mistake that our customer service folks made in trying to help out our customer. The other was made by the customer themselves. They got in what I'd say deep-due-due because they gave themselves a situation where they made one modification that cascaded to dozens and dozens and dozens of content items. I needed to actually restore more than one item at a time. On top of this, this wasn't just a modification that affected content. It affected things like embedded indexes that we were storing using Repose.catalog on each of these form items that we were trying to restore. The ZODB content that was stored underneath the content that had nothing to do with the content fields. When we're talking about restoration, there was really only one option, and that was go back in time, grab the snapshot from last Tuesday and do a ZEXP import of that. Rather than do that manually, piece by piece from, say, a data backup, like an old data.fs that I've backed up, what we ended up using was zc.beforeStorage because we had this in the undue history. Our default right now is we packed to seven days. We knew that somebody called us on day two saying, we know we screwed up. We get that call, and we know that we have some fixed number of days to programmatically go in and rescue them with a production database without having to resort to going into backups. What I found doing this over and over again was that this was repeatable. What we were doing was very patterned and very formulaic in terms of how I was programmatically solving this problem. That means I think we can have a user interface for this. Again, we can't use undo in the ZMI slash ZODB sense to really undo things these days. The basics of what we need, we need to have some kind of audit logging system. The goal here, and my approach at this point is to store this in ZODB. There have been other approaches that people have had saying, hey, store this out of band, do things like, if you are logging all requests, not just writes, you might do something like there are things out there, collective.firehoses, the example that comes to mind, something that might log to a separate database using an asynchronous queue. In this case, what I am interested in doing is simply at the very end of things, when there is any kind of modification, using the event subscribers to capture that in a very small record inside the ZODB. Right now, logging asynchronous, it might be asynchronous in the future. That is to be determined. One of the concerns I have is that there are increased potential for conflict errors the way I am doing it now. That is usually not a problem because we retry transactions in ZOP three times. However, if your transaction is very long running, you have a very expensive transaction at the end of it. You have a conflict error because you are trying to insert something into a list, it is basically a queue of integers. It is sort of ridiculous that that one little tiny thing is doing your entire transaction. When it could be retry, the insertion could be retried easily. There may be some opportunity for either using conflict resolution or asynchronous behavior to solve all this. We are going to log all these four facilities. Deletion, modification, moves and additions. The different kinds of writes that we really care about. When we think about moves, we are excluding additions and excluding deletions from the I object removed interface because that is notified in a variety of ways. We log all of this independent of the catalog. We do it in annotations on the site. Notice the way that I am handling this right now. The handlers are simple. They call a logging system which adapts the site and uses annotations and built in out of the box content types. There is a risk to that in that using out of the box content, sorry, data types. I am using things like ob trees, iob trees and persistent lists. I am not subclassing any of them. In theory, my goal here is to make this an easily uninstallable product. By using those out of the box data types, I am making this fairly low risk. I just have some annotations that are stored in your site using standard data types. If you try this and then you uninstall it once this thing is ready and you don't like it, it can go away without being a problem. The goal here is to have a testable, installable and uninstallable add-on that can be a proof of concept to verify that this is a reasonable way to restore things from the past. How we are going to use this. We have the core components I have tested. I have integration tests for a variety of things including things like pruning the logs. It still needs a good user interface. Outside of the core use case, the audit logging could be used for other things like regulatory audit requirements. For example, if you have HIPAA requirements, it doesn't solve everything because in some cases you might actually need to log things like views. Right now we just use Z2.log and collective.username.logger to log all the authenticated users access. In theory, from a regulatory requirement, if I was wanting to do something that was HIPAA compliant for healthcare privacy laws in the United States, what I would have to do, I could potentially get away with just using my Z2.log plus some kind of audit log that I am storing in the ZIDB. My UI plan here is enumerating things and views. By enumerating, I mean we need to be able to view all those four facilities. Modification, deletion, addition, and moves or renames. We need to be able to handle the fact that somebody might delete a parent object and you need to restore something that has a deleted parent. We can handle all those cases potentially in restoration, but we also need to have a way to prune changes depending upon how long we want to keep history. You may pack to seven days, but maybe you want to keep six months of audit history. That could be up to you. How this would be used and how that would be automated potentially, like right now, keep in perpetuity unless you use the view to go prune it. It's not very big, so I don't think it's that big of a deal to keep that information because these are small records. These are records with four different fields. They're very, very small metadata records. They're not going to take up a lot of space, they're not going to take up a lot of overhead. But anyway, you're going to have the RAM overhead of having a few extra B3 buckets. I may need to fix the conflict resolution strategy because we have a queue that basically keeps the order of these inserted records for each of those facilities. That queue, it's not really a queue. It's more like we just do insert zero into a persistent list. We're basically doing it in LIFA order, but we need to avoid conflicts. In order to do that, really, I actually have to do this either asynchronously or we have to look at doing under P, under resolve conflict method on a subclass of persistent list. I don't want to do that now because that becomes yet some data type that you have to have in the package that potentially gets persisted. That's not uninstallable. I want to punt that to a future date. Log in every change. We want to log every change. We want to be able to filter changes by things like by path. I want to see every change that happened in the folder foo slash bar. I want to be able to filter by path. I want to be able to filter by user. I want to know everything that somebody, that particular user has changed in the last 60 days. We want to be able to filter by date. Some of this filtering might be indexed and some of this filtering probably won't be indexed. That's okay because this is really an emergency facility. It doesn't necessarily have to be fast. The records are small to iterate through. I'm not trying to over optimize this at this point. Filtering right now goes through every single record and just tries to find matches. It's not an indexed filter. The goal is to have a view that allows you to list in reverse chronological order all changes for that facility, modification, deletion, addition, or move. For regulatory logging, you could have a data retention threshold before you prune anything. You could potentially never prune if you wanted to keep the audit trail forever. The change records are small, so there's very little risk to that. I talked about the facilities that we would try to have. Each item has two items in an annotation that are stored. There's an IOB tree that stores the actual records and there is a persistent list that stores the keys. The keys are automatically generated integers. Each of these four facilities have the ability to keep track of every change in insertion order. Each is going to log the following four things, UID path authenticated user and timestamp. Right now, that's local time. We may choose to do this in UTC. I haven't made a design decision on that yet. Also for things like moves, what we probably need to do is log both the path before and after the change so that we have the ability to restore something to its previous path. What do we do with this? We fix human errors. Use ZCE.beforeStorage multiple times programmatically to fix these errors. If you have something within your kept history, that's great. If you don't, you've got to resort to going into your backups. Hopefully, you have reasonable backups or snapshots or some system that keeps some way for you to go back in time if you're not able to do it from your production database. Each of the cases where I've had to rescue a customer, it's been from the production database because it's been within that seven-day window. Usually, that oh no call comes very, very quickly once somebody realizes the scope of what they've done. I'm creating a package right now called Plone.Wayback as a transitional package and is an experimental proof of concept. This is currently on my own GitHub account. I don't have the permission to create repositories on the Plone GitHub. If anybody wants to create a repository for me, I will put it there and keep working on it. Also soliciting any client collaboration, if anybody wants to work with me on this. The general idea, Wayback is in reference to the Wayback machine. They named it WABAC, sort of like in reference to ENIAC when Mr. Peabody in Sherman was originally created as a cartoon. It's a transitional add-on. It's half finished. It's probably going to get exploded and federated elsewhere if it ever is integrated into the Plone core. That's assuming that everybody actually likes this thing. We are assuming that I would write a clip for this, assuming that there was sufficient interest in folks working with this, playing with this as an add-on. I got positive feedback, I'll write a clip. I want this to be testable by users. I want it to be uninstallable in the meantime with a very little risk so that folks can give me good feedback. This is probably something that will be finished next week, at least in some very basic state. It's not going to handle all the edge cases, but it should be able to handle basic things like restoring the version of content from last Tuesday, or undoing the deletion that you had last Monday, or whatever is within your actual transaction history in your production database. We want to restore previous known good state before accidental deletion. Modification we potentially could do with CMF editions and other ways, but some of the problems are that CMF editions is dealing with only certain stuff. My assumption is that I have things that CMF editions probably is not taken care of. My belief is I don't necessarily know that to be absolutely sure, I would probably trust going back before storage to grab a copy of the content rather than using CMF editions for a rollback of some kinds of content. The goal is that the user simply just views the audit logs and filters into the audit logs, finds the things they want, clicks a couple check boxes, and clicks restore. That's simple. Potentially more than one item at a time. ZCW before storage is a really interesting thing. It basically wraps your existing storage, whatever that is. That could be something that ZO, FileStorage, any BrailleStorage, anything. It requires you to keep history. If you're using a history-free storage, that's a problem for you. It requires you to keep enough history. My eventual goal would be to have the ability to point to some directory where you have a bunch of file storage instances that have some kind of naming convention where you could ostensibly go back through and find stuff from your backups. That's not in scope and that would require some degree of configuration. It would require file naming conventions. There's a lot of things that I haven't thought about yet. I don't want to solve that problem quite yet. Mostly I just want to undo things easily. When that phone call comes in, I don't want it to be three hours of me writing some script to fix it. I want to be able to tell a site administrator how to fix it. Programmatically using this, basically the way that you would use before storage without some kind of tool like this is you'd load the storage wrapper programmatically in a run script. You would use set site on the time-traveled version, last Wednesday's snapshot of the storage. You'd get your content, you'd export it using the ZEXP functionality, probably to a string IO or to an attempt file depending upon the size of the object. You then set site on the live target that's the current version of the database. You'd restore it from ZEXP. You probably would have to do, there's a few other things that you have to do like restore original. If you're using content creation APIs, you probably have to set things like the original UUID and potentially deal with NVID problems and other things that I haven't quite dealt with yet. But then you'd repeat that. What I'd like to show you at the end of this is some example code of how that's been done. Before I do that, a few disclaimers. First, you may want the ability to not undo things. Somebody comes to you and says, our site's been defaced by an employee. That employee has been fired. You need to delete this thing. It needs to be permanent. Nobody can roll this back. There needs to be the ability to purge things from the undo history so they can't be rolled back even if they still are in the ZODB undo history. My vision right now is a site-wide audit logging facility that would only be accessible at the site route as a view to site administrators and manager roles. You'd basically have to have some kind of superpowers to be able to roll things back. That's fine. I'd rather it be a site administrator than me as a programmer. We can make this placeful, meaning you could actually say, hey, I want to roll back things that are in this folder. But that needs some thought about permissions and local roles and other things that are not yet in scope. Next steps, I need to finish the user interface for enumeration filtering and pruning of the audit data. The backend is done, but the front end needs to work. I also need to come up with a reasonable tactic for avoiding the conflict, write conflicts on the key insertion for the audit logging. That in terms of testing is not a necessity, but if we were to do this in a large-scale production site with long-running transactions, I'd say long-running transactions. I really mean just a transaction that takes longer because you have a high level of concurrency and your databases in high use. We need views and adapters for restoration via before storage. I'm working on that piece. That's basically taking what I've done programmatically, repetitively, and rescuing my users and trying to generalize it. We want to log the restorations themselves. Once you've actually restored something, we need to keep an audit trail of that too. If this proves universally useful, then I'll flip it. Before we get to questions, let's show an example of what using before storage looks like. You can also provide feedback with a link from 6VDUP. Let me go to... Let's load this up. Hopefully, I can... In the show, I'll try to make this brief. In the meantime, while I'm loading this, are there any questions? There's the ZMI undo tab. If you were making a change that really... I wouldn't ever recommend a user use it. That's why it was removed after Plum 3. I think it was Plum 3. When was it removed? I think the Plum 3... One of the Plum 3s, they were removed. Once folks realize it didn't work half the time, I think that that became a fairly sufficient motivation for the community to remove that. I've used undo in the ZMI all the time, usually for stuff that I've done or stuff that somebody comes to me immediately and says, I made this mistake 30 seconds ago. I don't always have that luxury. Once time... It's a function of the passage of time and the addition of more rights to the database, which is why time travel becomes a necessity. There is undo on a single object. There's version. You can refer to a previous version, compare a version with content that's not always enabled by default on all content types, but it's available. I had another question. Yeah, shoot. On to your UI dilemma. Yeah. A very common scenario that I get is somebody updates content. Sure. Two weeks later, somebody else says, oh, this is broken. What happened? Nobody knows when it changed, but they know it changed from one state to another. Okay. Yeah. You can go in and certify a specific date for time, but you need to look through the content to figure out when it got messed up. The solution to that is you need to not just be able to enumerate all the changes that are made on the site, especially a busy site. You need to filter to specific path or users. In your case, you're saying you don't know who made the change. You don't know when it was changed, but you know this thing screwed up. With modifications, I think you probably do have some, in most cases, with most content, you have some degree of history. Use it. If your workflow history on your object includes all the saves and changes, as well as things like state changes, then you absolutely have the ability to know, Joe did this two weeks ago. He's to blame, but we still have to fix this. I know where to look by date. In this case, you're just saying, I know the content screwed up. You're searching by path, right? Unless it was moved and then you've got some other things. Yeah. Yeah. There are some edge cases where you're going to have a needle in a haystack. I don't think if you can filter through these things, even if the filtering is somewhat slow, because you've got to go through linearly search through all of these records and load them up for the past six months or whatever, however you're retaining these things. I am envisioning this as a scenario where I say, hey, you've got two week undo window, and beyond that, we're not doing anything. Your scenario might be, I don't know when they modified it, and maybe it's past that window. At which point you'd say, well, something like this really ought to support the ability to go into your stored backups. You can say, this is the directory where my backups are, and be able to pull things out of load a file storage by naming convention, go in and use before storage to wrap that file storage, not your production database. This is a reasonable thing to do. I say, I have this specific script that says, I want to basically go to before this date. I want to use this specific rel storage database, and I want to wrap it with before storage. This configuration becomes something that I then use to load before storage. What I'm doing is getting two databases. I have the current setup, and I also have this before storage, getting the database connection and the site route right here. I'm doing set site on the source data while I retrieve it, and then I'm doing set site on the target data while I restore it, so that I can have access to local components in the respective time travel context. My functions that are calling these things are typically passing in source and target site, and they're doing specific things. The fact that I'm restoring these things is not germane to the larger discussion, which is we can just simply say, I have content items that need to be restored. This is very easily generalized. While this is a very specific run script that says, do this thing, grab these things, add them back into the site, manipulate them so that they look like what they looked like before, things like change the UUID back to its original UUID, that sort of thing, all of this is totally generalizable. There's no reason why we can't do this with a user interface that allows folks to filter through the audit history, click some check boxes, and say, give me my stuff back. That's the goal. I'm wondering, is there any data you pass to the org storage so that you get the data from a specific date that can be used? You have to. I think the question is, how do you know the date for the before storage? If you're clicking some check boxes, you have to go back for each individual item, you may have to go back to a different date depending upon when it was modified, so you might have to load the storage multiple times. That's not unreasonable to do. There's some overhead in doing that. But specifying the date, what you'd end up doing is you'd say, give me that modification date, subtract, add one second to that with time delta, and then use that as your time stamp for things you want before. So you have some reasonable guarantee that before plus minus one second is basically what we're looking at, because the time stamps in the ZODB, if I remember correctly, are not granular enough to do things at the millisecond level. So we're just saying it's a little crude, but it's going to work most of the time. And that is, take the date of modification that was logged, add one second to that. I'm not sure if the time stamp for the actions are granular. I don't know. Are you referring mainly to this? So yeah, I should be clear. There is a risk here. If the transaction takes a long time to commit and your event logger logs something, one second may not be long enough. If it takes five seconds between the time that the event logger gets a date and the transaction puts a time stamp on that specific transaction, you do have a problem with that drift, that specific, I can't predict what that's going to be. So I need to experiment with it. That is a challenge. I think the ZODE time function that gets used for that stuff has a fixed time once the transaction starts. Like you can call ZODE time over and over within a single transaction and you always have the same time stamp. So there's no difference. I believe that's what you use to record the transaction. But I'm not positive. The start of transaction is basically what you get recorded. So if the start of transaction is what's recorded, then we're in great shape. If it's the end of transaction, I can't predict when the transaction time ends. I'll need to find out. One way or another, I think I can have a workaround if it becomes a problem. Yes? The director of the London presented the site they did for the Brazilian government for the Olympics and they also created an add-on, just the auditing part, collective.fingerpoint. You might want to look at that. Say that again? Collective.fingerpointing. Okay, fingerpointing. Okay. Yeah. They probably had another use case. Yeah. And I have looked at that before. I've looked at that before. Other related things, I think Wildcard in their Castle CMS has some facility for undo. But basically what I think they're doing is they are, when you delete an item, they're actually just marking it as an invisible and they're changing the idea of it or something to that degree. So they're not actually truly deleting the item. So it's like half deleted in a way, bi-bicycly hiding it from you. So there's some kind of like pseudo-trash bin functionality there. That's a different approach. So it's not an invalid approach. This is just a little bit more aggressive. Another question. I'm wondering why you need this evaded block because all the information should be already evaded. For example, if you click on the undo top in the ZMI, even if you move an item, I think you have both paths available. So this information should somehow be already evaded, I guess. So if you're undoing something in the ZMI, what's the problem again? You have an event log. In the beginning you were proposing to write an event log where you record all the changes. That's correct. The changes are already recorded. So what's the difference? The difference is that the transaction history isn't capturing some of the things that you need. Now, you might say, hey, you know, the path is being captured in the transaction log. That's not necessarily very useful. And you'll find that what's actually being recorded in the path is a virtual host monster mangled version of the path, which is complicated to deal with. So you can't go into, if you have an admin URL and you have your front end URL and the change happened on one or the other, you can't go into the other URL and figure out the relative path in relationship. It becomes complicated. So my advice is not to trust the path that's necessarily stored in the transaction history. Doing this at the application level does make some sense. Instead of using the database's history, the other thing that the database lacks is the ability to tell us. It might tell us what user and what path. It doesn't necessarily give us the ability to add on any extra metadata that we might find useful for filtering. So for example, I want to be able to track back to what path. This item was deleted because the parent of that item was deleted. And I have one transaction in ZODB that says folder A was deleted, but inside folder A was folder B, folder C, folder, and 82 items. Those are all sort of buried under the obscure one single transaction record. Because when I do an I object removed event, and I notify object removed event, I get a cascade of these things recursively down to all the removed objects. While that makes things slower, because I have to notify things for all those objects, but that's an expected cost of removing a whole folder with sub-volters and content and other things inside it. But you do have the physical path, the physical path of the one thing you removed, the place where you made the modification. But if that place contained 82 other items, the existence of those 82 other items are gone. You don't know that they were there. And searching through transactions is very tricky. Because using transaction iterators to find things is probably going to be 10 times slower than having an application log where we'll potentially find the pain points and say, hey, we really need to index these things by path. So we have some flexibility from an optimization standpoint. If we find anything is slow, doing it, doing it the application level provides very little additional costs. So yeah, we're storing some of the same information twice, once in the transaction record and once in the, in, in, you know, some B tree somewhere on the, on an annotation on the site. But these are small records. And the cost of that from a storage standpoint is low. The cost of it from a performance standpoint, we'll, we'll see. Okay. Okay. One more. So what you're proposing basically is single object restoration instead of transactional output. So how would you deal with the consistency issues that you're going to make this user-facing? That's very true. So here, here's my, here's my thinking about those different facilities. Addition. We probably don't necessarily need to use the undo log to un, to add something. We already have a nice facility for just deleting the content. So it's just useful from an audit logging standpoint. Modification sometimes we have history that gives us what we need. We're potentially logging some additional history. That's somewhat duplicative. But if you're saying, hey, you know, we have this subscriber that does these things when you delete the, when you delete the item, it does something else and content rules, for example, could be running. And I can't know about those things. I can't, I'm hoping this is general enough that it's solving 98% of the problems that people would have with content restoration and that people would be able to manually work around those other, the other 2%. And the thing is, is that I feel like if I can give the side administrator the ability to try to restore this first before coming to me, that I'm going to save myself as a programmer a lot of time. And save myself as a responsible party for digging into the under the hood. If the site or content administrator has the ability to deal with this, they're going to be more savvy to whether or not it worked or not as than the average end user. So I'm going to trust my power users just to save me a whole lot of time. Because if I'm having to invest, you know, scale this up and think human error happens probably at any given site once, twice a month, something happens, somebody wants something undone. Usually that person just says, oh, oops, I'm going to fix my own problem. Sometimes, you know, if that happens six times a year and each time costs me one to three hours, that's, that's time I want back in my life. So if there's some simple control panel that gives the side administrator the ability to simply just check, find and check items to restore, even if it didn't work in 2% of the cases, if it worked most of those cases, it would be a big time saver for me. It would be a big convenience for the users. And it makes us look good, you know, as people who support systems. So, thank you.
 How many of you have had to programmatically rescue your customers from human error?  Okay.  How many of you found it painful?  Okay.  I still have a site that was broken a week or two ago that I just haven't had.  Okay.  I just hope he finds someone else.  The second question might not seem related, but it's related in concept.  How many of you have some kind of need for an audit log of changes that are made to your  site for regulatory purposes or any other reason?  Okay.  My goal here is to talk about an approach that I have and pitch this approach and discuss  whether or not this is something that might be appropriate at some future point in the  Plone Corps.  My initial goal was to demo an actual product.  What I have here is a half-finished product that is well-tested but has no user interface.  What we're going to talk about is what's going on under the hood, how I'm trying to  solve this problem, what I've done before, and what I'm proposing we might be able to  do.  I'd like to spark a discussion.  There's been some interest in this approach in discussions previously on Twitter and GitHub.  If there's any questions, my goal here is just to really get people thinking about the  viability of this idea and whether or not it's something that we want to, as a community,  consider.  My disclaimers here.  First, this is an experimental approach.  I'm developing this in an add-on that can be installed and uninstalled in Plone.  It wouldn't necessarily be if it was ever integrated into the Plone Corps via a plip  and standard release process, it would probably be refederated into a variety of different  places, not necessarily into a single add-on to prevent any kind of package proliferation.  But it is an unfinished work in progress.  What I have working right now in the package with tests is a pretty reasonable back-end  for modification logging.  It uses a variety of events for four different kinds of facilities.  Those facilities are modification, deletion, addition, and moves or renames.  We'd have the audit log that would tell you for any of those four facilities, which kind  of, in last and first-out order, and basically reverse chronological order, which changes  were made, who made the change, what the content was, if it was a move, we want to do things  like log the previous path and the current path, those kinds of things.  We're logging some very basic small metadata.  We do some of this stuff in other places, like for modifications we have the catalog,  we have CMF additions, but in some cases it's not going to be sufficient for the kinds  of restores that we need to do.  For the customer that I had to do these rescues for, they had some data that was stored in  the ZODB that was not necessarily content-ish.  The content item was a proxy to a lot of underlying ZODB data stored on that object that could  be imported by ZEXP, but could not be simply just restored from something like CMF additions.  I want to pitch this idea, I say it's about undo, but really I don't mean undo.  I mean restoration in the broader sense.  Because when we talk about undo, there was a ticket or an issue that was put on GitHub  some months ago asking for us to restore undo into Plone.  We had this undo tab in Plone 3 or undo action.  We can't do that because transactional undo in ZODB just doesn't really work in the real  world for a variety of reasons.  On any reasonably loaded site after some time, you just can't get the undo to work.  I'm here to pitch an idea.  I'm here to show you some of the way in which I manually, programmatically rescued my customer  and talk about the audit logging system that I've been working on, but also pitch the talk  about the need for a UI.  I'm working on it.  This is work in progress.  I was hoping to have a little more finish before the conference, but this is something that  I think that if there's interest in this, I'd like to spark discussion about.  We'll look at the pieces, so we'll talk about them and hopefully find some kind of practical  way forward.  Here's my motivations.  There was a reasonable degree of interest in discussion previously on GitHub and Twitter.  This happened after I had gone through three different rescues for one customer.  These three rescues, one of them was a mistake that our customer service folks made in trying  to help out our customer.  The other was made by the customer themselves.  They got in what I'd say deep-due-due because they gave themselves a situation where they  made one modification that cascaded to dozens and dozens and dozens of content items.  I needed to actually restore more than one item at a time.  On top of this, this wasn't just a modification that affected content.  It affected things like embedded indexes that we were storing using Repose.catalog on each  of these form items that we were trying to restore.  The ZODB content that was stored underneath the content that had nothing to do with the  content fields.  When we're talking about restoration, there was really only one option, and that was go  back in time, grab the snapshot from last Tuesday and do a ZEXP import of that.  Rather than do that manually, piece by piece from, say, a data backup, like an old data.fs  that I've backed up, what we ended up using was zc.beforeStorage because we had this in  the undue history.  Our default right now is we packed to seven days.  We knew that somebody called us on day two saying, we know we screwed up.  We get that call, and we know that we have some fixed number of days to programmatically  go in and rescue them with a production database without having to resort to going into backups.  What I found doing this over and over again was that this was repeatable.  What we were doing was very patterned and very formulaic in terms of how I was programmatically  solving this problem.  That means I think we can have a user interface for this.  Again, we can't use undo in the ZMI slash ZODB sense to really undo things these days.  The basics of what we need, we need to have some kind of audit logging system.  The goal here, and my approach at this point is to store this in ZODB.  There have been other approaches that people have had saying, hey, store this out of band,  do things like, if you are logging all requests, not just writes, you might do something like  there are things out there, collective.firehoses, the example that comes to mind, something that  might log to a separate database using an asynchronous queue.  In this case, what I am interested in doing is simply at the very end of things, when  there is any kind of modification, using the event subscribers to capture that in a very  small record inside the ZODB.  Right now, logging asynchronous, it might be asynchronous in the future.  That is to be determined.  One of the concerns I have is that there are increased potential for conflict errors the  way I am doing it now.  That is usually not a problem because we retry transactions in ZOP three times.  However, if your transaction is very long running, you have a very expensive transaction  at the end of it.  You have a conflict error because you are trying to insert something into a list, it  is basically a queue of integers.  It is sort of ridiculous that that one little tiny thing is doing your entire transaction.  When it could be retry, the insertion could be retried easily.  There may be some opportunity for either using conflict resolution or asynchronous behavior  to solve all this.  We are going to log all these four facilities.  Deletion, modification, moves and additions.  The different kinds of writes that we really care about.  When we think about moves, we are excluding additions and excluding deletions from the  I object removed interface because that is notified in a variety of ways.  We log all of this independent of the catalog.  We do it in annotations on the site.  Notice the way that I am handling this right now.  The handlers are simple.  They call a logging system which adapts the site and uses annotations and built in out  of the box content types.  There is a risk to that in that using out of the box content, sorry, data types.  I am using things like ob trees, iob trees and persistent lists.  I am not subclassing any of them.  In theory, my goal here is to make this an easily uninstallable product.  By using those out of the box data types, I am making this fairly low risk.  I just have some annotations that are stored in your site using standard data types.  If you try this and then you uninstall it once this thing is ready and you don't like  it, it can go away without being a problem.  The goal here is to have a testable, installable and uninstallable add-on that can be a proof  of concept to verify that this is a reasonable way to restore things from the past.  How we are going to use this.  We have the core components I have tested.  I have integration tests for a variety of things including things like pruning the logs.  It still needs a good user interface.  Outside of the core use case, the audit logging could be used for other things like regulatory  audit requirements.  For example, if you have HIPAA requirements, it doesn't solve everything because in some  cases you might actually need to log things like views.  Right now we just use Z2.log and collective.username.logger to log all the authenticated users access.  In theory, from a regulatory requirement, if I was wanting to do something that was  HIPAA compliant for healthcare privacy laws in the United States, what I would have to  do, I could potentially get away with just using my Z2.log plus some kind of audit log  that I am storing in the ZIDB.  My UI plan here is enumerating things and views.  By enumerating, I mean we need to be able to view all those four facilities.  Modification, deletion, addition, and moves or renames.  We need to be able to handle the fact that somebody might delete a parent object and  you need to restore something that has a deleted parent.  We can handle all those cases potentially in restoration, but we also need to have a  way to prune changes depending upon how long we want to keep history.  You may pack to seven days, but maybe you want to keep six months of audit history.  That could be up to you.  How this would be used and how that would be automated potentially, like right now,  keep in perpetuity unless you use the view to go prune it.  It's not very big, so I don't think it's that big of a deal to keep that information because  these are small records.  These are records with four different fields.  They're very, very small metadata records.  They're not going to take up a lot of space, they're not going to take up a lot of overhead.  But anyway, you're going to have the RAM overhead of having a few extra B3 buckets.  I may need to fix the conflict resolution strategy because we have a queue that basically  keeps the order of these inserted records for each of those facilities.  That queue, it's not really a queue.  It's more like we just do insert zero into a persistent list.  We're basically doing it in LIFA order, but we need to avoid conflicts.  In order to do that, really, I actually have to do this either asynchronously or we have  to look at doing under P, under resolve conflict method on a subclass of persistent list.  I don't want to do that now because that becomes yet some data type that you have to have in  the package that potentially gets persisted.  That's not uninstallable.  I want to punt that to a future date.  Log in every change.  We want to log every change.  We want to be able to filter changes by things like by path.  I want to see every change that happened in the folder foo slash bar.  I want to be able to filter by path.  I want to be able to filter by user.  I want to know everything that somebody, that particular user has changed in the last 60  days.  We want to be able to filter by date.  Some of this filtering might be indexed and some of this filtering probably won't be indexed.  That's okay because this is really an emergency facility.  It doesn't necessarily have to be fast.  The records are small to iterate through.  I'm not trying to over optimize this at this point.  Filtering right now goes through every single record and just tries to find matches.  It's not an indexed filter.  The goal is to have a view that allows you to list in reverse chronological order all  changes for that facility, modification, deletion, addition, or move.  For regulatory logging, you could have a data retention threshold before you prune anything.  You could potentially never prune if you wanted to keep the audit trail forever.  The change records are small, so there's very little risk to that.  I talked about the facilities that we would try to have.  Each item has two items in an annotation that are stored.  There's an IOB tree that stores the actual records and there is a persistent list that  stores the keys.  The keys are automatically generated integers.  Each of these four facilities have the ability to keep track of every change in insertion  order.  Each is going to log the following four things, UID path authenticated user and timestamp.  Right now, that's local time.  We may choose to do this in UTC.  I haven't made a design decision on that yet.  Also for things like moves, what we probably need to do is log both the path before and  after the change so that we have the ability to restore something to its previous path.  What do we do with this?  We fix human errors.  Use ZCE.beforeStorage multiple times programmatically to fix these errors.  If you have something within your kept history, that's great.  If you don't, you've got to resort to going into your backups.  Hopefully, you have reasonable backups or snapshots or some system that keeps some way  for you to go back in time if you're not able to do it from your production database.  Each of the cases where I've had to rescue a customer, it's been from the production  database because it's been within that seven-day window.  Usually, that oh no call comes very, very quickly once somebody realizes the scope of  what they've done.  I'm creating a package right now called Plone.Wayback as a transitional package and is an experimental  proof of concept.  This is currently on my own GitHub account.  I don't have the permission to create repositories on the Plone GitHub.  If anybody wants to create a repository for me, I will put it there and keep working on  it.  Also soliciting any client collaboration, if anybody wants to work with me on this.  The general idea, Wayback is in reference to the Wayback machine.  They named it WABAC, sort of like in reference to ENIAC when Mr. Peabody in Sherman was originally  created as a cartoon.  It's a transitional add-on.  It's half finished.  It's probably going to get exploded and federated elsewhere if it ever is integrated into the  Plone core.  That's assuming that everybody actually likes this thing.  We are assuming that I would write a clip for this, assuming that there was sufficient  interest in folks working with this, playing with this as an add-on.  I got positive feedback, I'll write a clip.  I want this to be testable by users.  I want it to be uninstallable in the meantime with a very little risk so that folks can  give me good feedback.  This is probably something that will be finished next week, at least in some very basic state.  It's not going to handle all the edge cases, but it should be able to handle basic things  like restoring the version of content from last Tuesday, or undoing the deletion that  you had last Monday, or whatever is within your actual transaction history in your production  database.  We want to restore previous known good state before accidental deletion.  Modification we potentially could do with CMF editions and other ways, but some of the  problems are that CMF editions is dealing with only certain stuff.  My assumption is that I have things that CMF editions probably is not taken care of.  My belief is I don't necessarily know that to be absolutely sure, I would probably trust  going back before storage to grab a copy of the content rather than using CMF editions  for a rollback of some kinds of content.  The goal is that the user simply just views the audit logs and filters into the audit logs,  finds the things they want, clicks a couple check boxes, and clicks restore.  That's simple.  Potentially more than one item at a time.  ZCW before storage is a really interesting thing.  It basically wraps your existing storage, whatever that is.  That could be something that ZO, FileStorage, any BrailleStorage, anything.  It requires you to keep history.  If you're using a history-free storage, that's a problem for you.  It requires you to keep enough history.  My eventual goal would be to have the ability to point to some directory where you have  a bunch of file storage instances that have some kind of naming convention where you could  ostensibly go back through and find stuff from your backups.  That's not in scope and that would require some degree of configuration.  It would require file naming conventions.  There's a lot of things that I haven't thought about yet.  I don't want to solve that problem quite yet.  Mostly I just want to undo things easily.  When that phone call comes in, I don't want it to be three hours of me writing some script  to fix it.  I want to be able to tell a site administrator how to fix it.  Programmatically using this, basically the way that you would use before storage without  some kind of tool like this is you'd load the storage wrapper programmatically in a run  script.  You would use set site on the time-traveled version, last Wednesday's snapshot of the  storage.  You'd get your content, you'd export it using the ZEXP functionality, probably to a string  IO or to an attempt file depending upon the size of the object.  You then set site on the live target that's the current version of the database.  You'd restore it from ZEXP.  You probably would have to do, there's a few other things that you have to do like restore  original.  If you're using content creation APIs, you probably have to set things like the original  UUID and potentially deal with NVID problems and other things that I haven't quite dealt  with yet.  But then you'd repeat that.  What I'd like to show you at the end of this is some example code of how that's been done.  Before I do that, a few disclaimers.  First, you may want the ability to not undo things.  Somebody comes to you and says, our site's been defaced by an employee.  That employee has been fired.  You need to delete this thing.  It needs to be permanent.  Nobody can roll this back.  There needs to be the ability to purge things from the undo history so they can't be rolled  back even if they still are in the ZODB undo history.  My vision right now is a site-wide audit logging facility that would only be accessible at  the site route as a view to site administrators and manager roles.  You'd basically have to have some kind of superpowers to be able to roll things back.  That's fine.  I'd rather it be a site administrator than me as a programmer.  We can make this placeful, meaning you could actually say, hey, I want to roll back things  that are in this folder.  But that needs some thought about permissions and local roles and other things that are  not yet in scope.  Next steps, I need to finish the user interface for enumeration filtering and pruning of the  audit data.  The backend is done, but the front end needs to work.  I also need to come up with a reasonable tactic for avoiding the conflict, write conflicts  on the key insertion for the audit logging.  That in terms of testing is not a necessity, but if we were to do this in a large-scale  production site with long-running transactions, I'd say long-running transactions.  I really mean just a transaction that takes longer because you have a high level of concurrency  and your databases in high use.  We need views and adapters for restoration via before storage.  I'm working on that piece.  That's basically taking what I've done programmatically, repetitively, and rescuing my users and trying  to generalize it.  We want to log the restorations themselves.  Once you've actually restored something, we need to keep an audit trail of that too.  If this proves universally useful, then I'll flip it.  Before we get to questions, let's show an example of what using before storage looks  like.  You can also provide feedback with a link from 6VDUP.  Let me go to... Let's load this up.  Hopefully, I can... In the show, I'll try to make this brief.  In the meantime, while I'm loading this, are there any questions?  There's the ZMI undo tab.  If you were making a change that really... I wouldn't ever recommend a user use it.  That's why it was removed after Plum 3.  I think it was Plum 3.  When was it removed?  I think the Plum 3... One of the Plum 3s, they were removed.  Once folks realize it didn't work half the time, I think that that became a fairly sufficient  motivation for the community to remove that.  I've used undo in the ZMI all the time, usually for stuff that I've done or stuff that somebody  comes to me immediately and says, I made this mistake 30 seconds ago.  I don't always have that luxury.  Once time... It's a function of the passage of time and the addition of more rights to  the database, which is why time travel becomes a necessity.  There is undo on a single object.  There's version.  You can refer to a previous version, compare a version with content that's not always  enabled by default on all content types, but it's available.  I had another question.  Yeah, shoot.  On to your UI dilemma.  Yeah.  A very common scenario that I get is somebody updates content.  Sure.  Two weeks later, somebody else says, oh, this is broken.  What happened?  Nobody knows when it changed, but they know it changed from one state to another.  Okay.  Yeah.  You can go in and certify a specific date for time, but you need to look through the content  to figure out when it got messed up.  The solution to that is you need to not just be able to enumerate all the changes that  are made on the site, especially a busy site.  You need to filter to specific path or users.  In your case, you're saying you don't know who made the change.  You don't know when it was changed, but you know this thing screwed up.  With modifications, I think you probably do have some, in most cases, with most content,  you have some degree of history.  Use it.  If your workflow history on your object includes all the saves and changes, as well as things  like state changes, then you absolutely have the ability to know, Joe did this two weeks  ago.  He's to blame, but we still have to fix this.  I know where to look by date.  In this case, you're just saying, I know the content screwed up.  You're searching by path, right?  Unless it was moved and then you've got some other things.  Yeah.  Yeah.  There are some edge cases where you're going to have a needle in a haystack.  I don't think if you can filter through these things, even if the filtering is somewhat  slow, because you've got to go through linearly search through all of these records and load  them up for the past six months or whatever, however you're retaining these things.  I am envisioning this as a scenario where I say, hey, you've got two week undo window,  and beyond that, we're not doing anything.  Your scenario might be, I don't know when they modified it, and maybe it's past that  window.  At which point you'd say, well, something like this really ought to support the ability  to go into your stored backups.  You can say, this is the directory where my backups are, and be able to pull things out  of load a file storage by naming convention, go in and use before storage to wrap that  file storage, not your production database.  This is a reasonable thing to do.  I say, I have this specific script that says, I want to basically go to before this date.  I want to use this specific rel storage database, and I want to wrap it with before storage.  This configuration becomes something that I then use to load before storage.  What I'm doing is getting two databases.  I have the current setup, and I also have this before storage, getting the database connection  and the site route right here.  I'm doing set site on the source data while I retrieve it, and then I'm doing set site  on the target data while I restore it, so that I can have access to local components  in the respective time travel context.  My functions that are calling these things are typically passing in source and target  site, and they're doing specific things.  The fact that I'm restoring these things is not germane to the larger discussion, which  is we can just simply say, I have content items that need to be restored.  This is very easily generalized.  While this is a very specific run script that says, do this thing, grab these things, add  them back into the site, manipulate them so that they look like what they looked like  before, things like change the UUID back to its original UUID, that sort of thing, all  of this is totally generalizable.  There's no reason why we can't do this with a user interface that allows folks to filter  through the audit history, click some check boxes, and say, give me my stuff back.  That's the goal.  I'm wondering, is there any data you pass to the org storage so that you get the data  from a specific date that can be used?  You have to.  I think the question is, how do you know the date for the before storage?  If you're clicking some check boxes, you have to go back for each individual item, you may  have to go back to a different date depending upon when it was modified, so you might have  to load the storage multiple times.  That's not unreasonable to do.  There's some overhead in doing that.  But specifying the date, what you'd end up doing is you'd say, give me that modification  date, subtract, add one second to that with time delta, and then use that as your time  stamp for things you want before.  So you have some reasonable guarantee that before plus minus one second is basically  what we're looking at, because the time stamps in the ZODB, if I remember correctly, are  not granular enough to do things at the millisecond level.  So we're just saying it's a little crude, but it's going to work most of the time.  And that is, take the date of modification that was logged, add one second to that.  I'm not sure if the time stamp for the actions are granular.  I don't know.  Are you referring mainly to this?  So yeah, I should be clear.  There is a risk here.  If the transaction takes a long time to commit and your event logger logs something, one  second may not be long enough.  If it takes five seconds between the time that the event logger gets a date and the transaction  puts a time stamp on that specific transaction, you do have a problem with that drift, that  specific, I can't predict what that's going to be.  So I need to experiment with it.  That is a challenge.  I think the ZODE time function that gets used for that stuff has a fixed time once the transaction  starts.  Like you can call ZODE time over and over within a single transaction and you always have the  same time stamp.  So there's no difference.  I believe that's what you use to record the transaction.  But I'm not positive.  The start of transaction is basically what you get recorded.  So if the start of transaction is what's recorded, then we're in great shape.  If it's the end of transaction, I can't predict when the transaction time ends.  I'll need to find out.  One way or another, I think I can have a workaround if it becomes a problem.  Yes?  The director of the London presented the site they did for the Brazilian government for  the Olympics and they also created an add-on, just the auditing part, collective.fingerpoint.  You might want to look at that.  Say that again?  Collective.fingerpointing.  Okay, fingerpointing.  Okay.  Yeah.  They probably had another use case.  Yeah.  And I have looked at that before.  I've looked at that before.  Other related things, I think Wildcard in their Castle CMS has some facility for undo.  But basically what I think they're doing is they are, when you delete an item, they're  actually just marking it as an invisible and they're changing the idea of it or something  to that degree.  So they're not actually truly deleting the item.  So it's like half deleted in a way, bi-bicycly hiding it from you.  So there's some kind of like pseudo-trash bin functionality there.  That's a different approach.  So it's not an invalid approach.  This is just a little bit more aggressive.  Another question.  I'm wondering why you need this evaded block because all the information should be already  evaded.  For example, if you click on the undo top in the ZMI, even if you move an item, I think  you have both paths available.  So this information should somehow be already evaded, I guess.  So if you're undoing something in the ZMI, what's the problem again?  You have an event log.  In the beginning you were proposing to write an event log where you record all the changes.  That's correct.  The changes are already recorded.  So what's the difference?  The difference is that the transaction history isn't capturing some of the things that you  need.  Now, you might say, hey, you know, the path is being captured in the transaction log.  That's not necessarily very useful.  And you'll find that what's actually being recorded in the path is a virtual host monster  mangled version of the path, which is complicated to deal with.  So you can't go into, if you have an admin URL and you have your front end URL and the  change happened on one or the other, you can't go into the other URL and figure out the relative  path in relationship.  It becomes complicated.  So my advice is not to trust the path that's necessarily stored in the transaction history.  Doing this at the application level does make some sense.  Instead of using the database's history, the other thing that the database lacks is the  ability to tell us.  It might tell us what user and what path.  It doesn't necessarily give us the ability to add on any extra metadata that we might  find useful for filtering.  So for example, I want to be able to track back to what path.  This item was deleted because the parent of that item was deleted.  And I have one transaction in ZODB that says folder A was deleted, but inside folder A  was folder B, folder C, folder, and 82 items.  Those are all sort of buried under the obscure one single transaction record.  Because when I do an I object removed event, and I notify object removed event, I get a  cascade of these things recursively down to all the removed objects.  While that makes things slower, because I have to notify things for all those objects,  but that's an expected cost of removing a whole folder with sub-volters and content  and other things inside it.  But you do have the physical path, the physical path of the one thing you removed, the place  where you made the modification.  But if that place contained 82 other items, the existence of those 82 other items are  gone.  You don't know that they were there.  And searching through transactions is very tricky.  Because using transaction iterators to find things is probably going to be 10 times slower  than having an application log where we'll potentially find the pain points and say,  hey, we really need to index these things by path.  So we have some flexibility from an optimization standpoint.  If we find anything is slow, doing it, doing it the application level provides very little  additional costs.  So yeah, we're storing some of the same information twice, once in the transaction record and once  in the, in, in, you know, some B tree somewhere on the, on an annotation on the site.  But these are small records.  And the cost of that from a storage standpoint is low.  The cost of it from a performance standpoint, we'll, we'll see.  Okay.  Okay.  One more.  So what you're proposing basically is single object restoration instead of transactional  output.  So how would you deal with the consistency issues that you're going to make this user-facing?  That's very true.  So here, here's my, here's my thinking about those different facilities.  Addition.  We probably don't necessarily need to use the undo log to un, to add something.  We already have a nice facility for just deleting the content.  So it's just useful from an audit logging standpoint.  Modification sometimes we have history that gives us what we need.  We're potentially logging some additional history.  That's somewhat duplicative.  But if you're saying, hey, you know, we have this subscriber that does these things when  you delete the, when you delete the item, it does something else and content rules,  for example, could be running.  And I can't know about those things.  I can't, I'm hoping this is general enough that it's solving 98% of the problems that  people would have with content restoration and that people would be able to manually  work around those other, the other 2%.  And the thing is, is that I feel like if I can give the side administrator the ability  to try to restore this first before coming to me, that I'm going to save myself as a  programmer a lot of time.  And save myself as a responsible party for digging into the under the hood.  If the site or content administrator has the ability to deal with this, they're going to  be more savvy to whether or not it worked or not as than the average end user.  So I'm going to trust my power users just to save me a whole lot of time.  Because if I'm having to invest, you know, scale this up and think human error happens  probably at any given site once, twice a month, something happens, somebody wants something  undone.  Usually that person just says, oh, oops, I'm going to fix my own problem.  Sometimes, you know, if that happens six times a year and each time costs me one to three  hours, that's, that's time I want back in my life.  So if there's some simple control panel that gives the side administrator the ability to  simply just check, find and check items to restore, even if it didn't work in 2% of the  cases, if it worked most of those cases, it would be a big time saver for me.  It would be a big convenience for the users.  And it makes us look good, you know, as people who support systems.  So, thank you.
Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_0_71.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_72_167.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_168_2327.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_2328_6431.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_6432_6731.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_6732_9659.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_9660_13727.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_13728_17627.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_17628_19631.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_19632_20963.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_20964_22523.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_22524_23795.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_23796_25619.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_25620_27035.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_27036_28547.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_28548_31571.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_31572_32075.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_32076_32363.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_32364_33047.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_33048_33443.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_33444_33575.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_33576_33839.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_33840_33971.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_33972_35939.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_35940_36335.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_36336_38267.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_38268_38699.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_38700_38771.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_38772_38819.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_38820_43031.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_43032_43427.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_43428_45791.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_45792_45923.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_45924_47267.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_47268_47891.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_47892_48155.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_48156_50123.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_50124_50519.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_50520_50651.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_50652_52163.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_52164_52487.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_52488_54455.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_54456_54851.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_54852_56399.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_56400_56483.png Day_2__A_Novel_Undo_Tool_for_Plone__by_Sean_Upton_DwyxL27PqbU_56484_56503.png