Migrations! Migrations! Migrations!
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Bauer, Philip
In recent years we have migrated a good number of sites from Plone 4 to 5, from Archetypes to Dexterity, from LinguaPlone to plone.app.multilingual and more recently from Python 2 to 3. Some migrations even combined all of the above. In this talk I will try to cover all the technical aspects of such large-scale migrations, will walk through many code-examples and discuss best-practices. All upgrade-steps and documentation will be provided for you to reuse.
 So recently we did a lot of migrations and a lot of them now with Python 3 we have an additional reason to a good argument to sell Plon5 to Plon4 users. So we're doing migrations from Plon4 to Plon5 at the moment. A lot of them, a lot of these sites still have linguaplon, they need to be migrated to Plon at multilingual, they still have archetypes, lots of that and also need to be migrated from Python 2 to Python 3. So one of the things I learned in the past years is that every migration is different, only not so much because they have kind of the same problems in maybe not identical but very similar ways. So I found that I can reuse most of not only the lessons I learned but also the code that I use for these migrations and since I got a lot of requests I did a little package called collective migration helpers that contains almost all the code that I'm showing today, I'm going to show code from other packages as well. So if you're interested in reusing any of the methods in there just go there. So the most important lesson I learned from doing migrations is automate everything. So write upgrade steps, put everything in code and put it in code so it can run multiple times, it can be run easily and it can be run in different contexts. Let's look at an example what I mean by that. It's a very simple, small method that disables solar. So it does a couple of things that are interesting here. So one is that you might recognize, see the context equals none. In an upgrade step the first argument is usually the portal setup tool. So you can run that as an upgrade step if you register it but I'm not using the context at all. So that means I can also run it from a PDB for example, from a debug console and instance debug, I can even run it from an instance run script. I can run it from a browser view, whatever. So it's very flexible. This is number one. So the second thing is I keep imports inside the upgrade step or the upgrade function in this case. So in case collective solar doesn't exist, I don't have an import at the top and I also do make the import conditionally so I check if I actually can import solar, if not I'm actually not going to do anything. So it doesn't hurt me if I run this code in a site that's never heard of solar. And also if it's been run before, so the solar was disabled like 10 minutes ago, it doesn't hurt at all. And the last thing is that I log what I'm doing. So log in for solar disabled. If I'm not disabling solar, I'm not logging anything. So that's kind of easy things but if you follow these simple patterns, you will be able to combine your upgrade steps into something that is easily reusable. Another example is this, it can run multiple times again. It can be run in different contexts. It disables a couple of LDAP authentication plugins and tries to delete them and logs it if it's there, doesn't log it if it isn't there. As you can see, a lot of migrations are similar but different. So probably none of your projects have an AD plugin underscore BG48. Just one of mine did. This is why the code in the repository is there for you to copy and paste into your own modules. It's not meant, you can use it to import certain things but that's not the initial intention. So also these two are actually the first two things you should always run. Maybe not exactly these. Maybe you don't have solar but elastic search and you don't have LDAP but have something, I don't know, blown up LDAP. This is whatever. But these are two things you don't want while doing migrations, calling some external search index. The worst would be writing into that, the production search index while you're doing migration on a different machine because the blown instance keeps the URL from the IP from the solar instance for example, that would be terrible. And also you don't want LDAP to be asked every time you do something on your development. But before you do that, you try to get some information. Get the bird's eye view, that's how I called it. Say you want to learn as much as you can about your site, what content types are there, what add-ons are there, how old your items are, how many items there are, how heavy, I mean in megabytes they are, about local roles, about placeful workflows, about all kinds of things. And here's a small example from the statistics module in the package I made public. This is a helper view, it's a browser view in this case that will, you don't have to read all this, will list a lot of useful information that you usually want to know. For example, how many items of which type there are, so there's obviously 884 help center leaf pages in this portal, where that content is, so there's obviously some weird folder that, I don't know, services that contains a lot of stuff. So how old your content is, so something was very weird in 2019 because they had 1,900 events, so pretty busy year probably. And also, where you actually, the biggest objects live, sometimes you have people uploading whole ISO images into clone instances. So we have a huge zip file of 600 megabytes on the top and it sorts them by size. Just a small example, you can adapt the code to your need. Another thing I like to do is find out where the content is that needs to be removed at some point, so when you find out what kind of add-ons you have, you'll see here we have platform gen help center, collage and calendar X folder, these are in that project, these were used quite a lot and it tells me now, okay, where is that, how many items are in there because most of them are container-ish, which one was created, from which year are these items, and with this information you can send emails to people who are in charge of the site and ask them, okay, this is what you have, this is where it is, what do you want to do with this and more often than not the answer is going to be, oh, I had no idea this actually still existed, please remove it. So you don't have to deal with that, but unless you can't make that decision yourself, unless you're the project manager content-wise, which usually are not as a developer. So the second big lesson is to divide and conquer. In an abstract way that means deal with one problem at a time and ignore and replace, remove and replace one add-on at a time, solve one problem at a time and ignore problems that don't block you, put them in a to-do list and they may have just gone away in platform 5.2. So that is important advice. Divide and conquer also means divide your migrations into a multitude of upgrade steps. So have one upgrade step for every single thing you do, so you can run them separately, either in the PDB or with portal set-up, so these are a couple of upgrade steps, migrating images, migrate everything else to plan up content types, uninstall archetypes, and only when this actually works for enough times and you're content and you don't want to click so many buttons anymore, you can combine them into one upgrade steps group as PLONE itself does. So this allows you to run them separately so you get an individual commit after each upgrade step, as long as you do it in the user interface. So if step 2 takes 5 minutes and in step 3 you're going to get an error and you run them sequentially without a commit in between, you get very angry at yourself, you'll hate yourself, so don't do that. Also you might be tempted to use branches for your upgrade steps and migration code, which don't, in PLONE we have things that can help you not to do that, one would be moving imports into the code, have conditional imports that fail when you don't have that, like I don't know, no more lingual PLONE, you don't have to do anything because you're now in PLONE 5.1. Also you can register upgrade steps conditionally with ZCML condition, have PLONE BLAR version as PLONE itself does it. You can decide to register your steps like this. Here's one example, only if you have PLONE 5 you should be able to run this step to uninstall archetypes. So this sounds really easy but it's totally not, but there are a couple of nice tricks that you should use and keep in mind. One thing super obvious, which nobody does, is just remove 98% of your content from the portal that you want to migrate. I often have very huge databases to deal with your migrations and okay my laptop is fast but still it takes a lot of time to move this data around, so write something like this that deletes all files and images that are bigger than one megabyte, just one example. Don't remove any folders, but if you have ATE help centers and 50,000 documents and 900,000 news items, just remove 99% of each of these items and try to deal with everything else, but keep the content structure in place because that don't delete any folders because A, they're not heavy, I mean in megabytes, only their content is, and B, this will save you from running into issues later like local roles and portlets are mostly assigned on folders and these are things that sometimes break your migrations, so if you keep these, something that is container-ish, keep everything there unless it's something like a Plone Help Center which is always kind of the same, but if you have folders or other folder types, keep them around. And you can still fix any corner cases after your migration runs through, so I showed that already. Another tip I got from Alessandro is do not migrate blobs because the blobs, the blob itself will not change, neither from archetype, the to dexterity, nor from Python 2 to Python 3, the PDF that lives on the file system is always going to be the same, and if you use an add-on called experimental graceful blob missing, which actually even works with Rails storage now, since I haven't merged that yet, but it's there, you move the blob storage away, just move it to a backup folder and you use that. This add-on will create an empty file with a marker text in it for everything that it touches, it's really fast because it's a very small file and you can re-add the blobs once the whole migration is done later. Since the UIDs in any same migration will stay identical, the path to the blob file will also stay identical. You need to keep the blob files during cleanup, so if you decide, ah, I need to remove this and this and this and this, and for this you need the blobs because you really want to lose the blobs as well, so don't forget that. So that is also a nice, tiny thing, when you copy a big database from a production server or a common server to another, also copy the data FS.index because re-indexing a really big database on startup can take, I don't know, up to half an hour, depends on how big it is. This is nice, so you got to forget the past and forgive your sins, the easiest, and you need permission to, for all that, but you will, it's a technical requirement anyway, but you need to talk to your clients. We're going to remove all revisions, for example, this is a five line code which changed a data FS from 21 gigabytes to three gigabytes and the blob storage from 91 gigabytes to 50 gigabytes just by removing all revisions. And yes, binary files can be versioned, you should not do that, but some clients do, so you will have, this is how we lost, went from 91 to 50 gigabytes because binary files are versioned. And an alternative to this would be to use Collective Revision Manager, if you want to control this manually, you can decide how many items you want to keep for, which objects and so on, so that's nice. The second part of your past is previous commits, you need to pack the database, it takes a while, the important thing is set it to day zero, that is very important to keep zero days, that's very important when you migrate to Python three later on. Okay, now let's go to the hard part, add-ons. So here's a simple example, removes an add-on that is easy to remove and after that it's actually gone, and it's a good pattern, you could add more logging, but that's fine. Moving on to a slightly more complicated example, this is also a very small add-on, but it changes existing content, so when you remove an add-on sometimes it adds additional views, you have to unretches to these in generic setup for example, so that they're no longer allowed, but you also have to unset them on the objects that use these views for example. This is one small example, the important thing is to drop the layout attribute property from all objects that have this first, that's like a replacement for default views, don't really recommend it, but yeah. Okay, a bit more advanced, here is a portlet, because portlets create content, portlets in this case, and you have to go through the site and find them, in this case I actually knew where it was, it's an NM example intranet, that's where it was, so it's easy to find, but in the packages also code to find portlets anywhere of a certain kind, so you can use that to remove portlets. Here's a, that's tricky, so for those who don't know that, the portal quick installer has a feature, has its own registry that stores information about all add-ons that were imported, that were installed, and depending on the age and the state of the add-on it was in when it was first installed, this will contain different information, including types, some add-ons have a types XML that contains all types where they set additional information on, and not only a types XML that contains the types that it actually registers, which is a simple beginner bug, the result from that is that upon uninstalling this add-on, this type will be gone, if that type is your folder, like in this case, or a collection, like in this case, you're in big trouble because then you have to try to, you have to reinstall the type or re-register the type, taking it from Planner Content Types or wherever it came from, the one you want, so, but when uninstalling an add-on you can pass it a modified cascade that is different from the default cascade in the quick installer and that will save you from that trouble. That's the same thing happens with the registry or used to happen with the registry, it was totally nuked when uninstalling some add-ons and nothing ever worked anymore unless you reinstalled the registry, which was easy but not obvious and the error message was not, oh, you just lost the registry, please reinstall the registry was something very much more obscure. Just quickly about profiles, you can and you should use uninstalled and upgrade profiles, like Plone does, so personally I'd like to, I admit that I like the approach to have all the stuff during a migration in Python, so I only have to look in one place what it actually does, but some things are actually much, much easier to achieve with generic setup than in Python, so I don't know, I'm kind of on the fence on this. Here's an example of an upgrade step in Plone itself, this is how it's registered and it has, oh, I was supposed to show the structure now in my editor, which I don't have, okay, you can check that yourself, it is in, it's in Plone Up upgrades and there is a folder that is called 2RC4, release candidate 4 and it contains a folder XML that removes a view and actions XML and the tool set XML and stuff like that, just look up these examples. Okay, moving on to persistent, this is hard, there are really long blog posts about that and it's, you can uninstall utilities and tools and components with generic setup, but it doesn't always die the way you want it, so you can do that in Python as well. Here's one example from the package I released that removes utilities and adapters that are used by Linguo Plone, it doesn't hurt to run this code 50 times, I have, actually I have a migration where I run this after each individual upgrade step because I had cases when, I don't know, maybe it was my fault, my commit was not, didn't go through properly, so I just put this at the end of every single upgrade step I had there and it worked well. I'm not going to walk through the code, but it's, if you know what you want to remove, removing it is actually really simple because utility subscriber and then you put the name of the thing and remove it, bam, not that hard. But since it is hard to figure out what to remove and it's a tedious process to iterate over that, okay, yeah, something that's not dying, let's write a method to kill it, it's often more easy to actually patch it away and there is a nice tool called alias module, it's Implona butyls and it helps you to get rid of things that are, have a hard time dying. This is an example from webdav, so what it does is it adds IFTP access into the path, so it's importable by the persistent data in your database when it's unpickling the pickle that is referencing that, but it's replacing whatever it was before with the empty BBB interface, BBB means backward compatibility, I don't know that doesn't make sense but someone came up with that at some point. Here's another example from collective easy slideshow, so this is not a core thing but this is an add-on that had a really hard time dying and this was a very easy way to get rid of it, so if you feel dirty now, if you look at this, you shouldn't because this is how ploneup upgrade actually looks, these are only the first 30 lines from it, the file is much, much longer and it runs every single time you start a plone and it patches away a lot of things that you've plone and we got rid of over the years, but some of these things are really hard to uninstall and clear out the database because there might be old pickles that are no longer used but if you go back in your database like you revert commits, then they are resurrected again and this is this really hard technical problem and this just, this solves this in a very nice, nice way. Moving on to linguaplone, okay. So very important, migrate from linguaplone to ploneup multilingual while running in Python 3.4.3, not 3.4 please, and while still using ArchiTimes, there is a build-in migration in ploneup multilingual, sorry, that actually works fine but it mostly works if your content is not a total mess but your content is very likely to be a total mess because editors in linguaplone had the ability to do things that are not useful, for example, have a folder in one language and have mixed languages inside that folder but part of that, these translations are then in a different folder again so it's a complete mess and this is some code that you totally should read right now that helps you clean up this so only, I can't point but somewhere in there are actually two lines of code that run the whole migration from ploneup, linguaplone to ploneup multilingual, they're conveniently called step one and two and the other was called step three, so call them method step one and two, I like that, still, everything else, it's in the package, it tries to make an educated guess if you have two, this works with two languages, only can try to adapt it and it tries to move stuff around in the same way, same way so that your content actually if it doesn't have a language, it guesses the language from the language of the folder and from the language of the majority of the content in the folder and if it's a different language, it creates a translation for the parent folder and for an untranslated folders, it looks into what language has the content and so on and so on so it does a lot of stuff and you can use it for your own good and at your own risk, please, it is, I'm not going to walk through that, that would take too long. There's also, yeah, look at the whole linguaplone module in migration for helpers, if you still have that, so once you did that, all these things like clean up, remove add-ons, slim your database, remove your revisions, then you can safely upgrade to plone five two, either from plone five one or plone zero, plone three or whatever you want. So do that naturally while still in python two, do that keeping archetypes still if you have it, keep, have it, have multilingual enabled if you have it and if you come from plone four as a gesture, you directly upgrade to plone five two or rather five two one, which will be out soon, I saw a pending folder already on this plone org, so expect that within the week. In some rare cases, you might want to upgrade to five zero ten or five one six first, but I haven't had such a case actually. So how you do that? Plone five two with archetypes, that's easy, just add the archetypes extra to your build out and it will pull in all the archetypes dependencies and don't stop there, don't run plone on five two with archetypes, that is just so insulting to everyone who put in work into plone in the last five years. Go further from there. And yeah, there's also documentation about changes in plone five two, they're really well documented, so please read the FM. And then migrate to x30 in five two. I've covered this a couple of times already, so I'm not going to go into too much detail, but I'm going to show you a couple of best practices, especially for sites with a lot of content and complex things. If you have a default site, you can use it for it, if you want, it's fine. Automate it, migrate one type at a time, start with the folders and first again try it with a reduced database. So this is an example to migrate all files from archetypes to dexterity. This is not new. This has been in the documentation of plone up content types since, I don't know, many years now and it works fine. It has been improved, that's why I say use it in five two. For example, the patch searchable text and a couple of other things in this method, the PAC migration, it makes it much faster if you skip indexing and patch searchable text away because your searchable text is going to be the same anyway, so why would you want to reindex them just for the sake of migrating? If everything works fine, do that, do it all at once. Even topics, works out of the box, you can migrate topics to dexterity collections, thank you. Much more interesting, custom content. So for example, plone help center, who uses or who uses plone help center? It's easy to migrate, that's why I picked it for this purpose demonstration because it's mostly default content with constraints and a couple of views that look slightly different and it has one additional feature that folders don't have so you can manage versions but my client doesn't use that feature at all, so I decided I'll just migrate it to default content like folders and pages. So this is an example to migrate the help center how to folders, as I said, it's a couple of folders that contain how tos with a constraint and this is all you need, this does everything, you don't need anything else, this could be run from a PDB thing, it goes through the whole site and it turns every single instance of a plone help center how to folder into a normal folder, bam. Okay there's a couple more complicated items, for example the help center FAQ is a document, so you can specify a field mapping to map the text field to the text field and say that's a rich text and so it's migrating the help center FAQ to a document and it's keeping the rich text. You need to specify the fields because the source type doesn't come, it can't pick up the migration helper, the migrate custom AT thingy here, it doesn't know about help center FAQ, it knows about all the default content types in plone so if you have a migrated document it knows there's a rich text field I need to migrate but for help center FAQ everything that is not like doubling core or one of the default fields needs to be specified here in a very simple way. You can specify something more complicated here, this is a nice pattern you can use if you have a content type that you want to migrate to default content but the content type you had before is mostly for archive purposes only but it had like 50 rich text fields and five text fields on top of that but you don't want that anymore, you just want the information that's in there so what it does it goes, it's a field migrator that concatenates these text fields one after the other adding a small heading with the name of the field before that so you have like 20 fields and now they're in one rich text field because nobody's using the plone help center anymore, actually they're using it but now it's not called plone help center, it's called a document and you use sensible headings to add your content and not a structure. So this would then migrate the help center itself because the help center has an additional field called writes, I actually don't have no idea what it holds so I never, some text so I add that into the text field that it also has so it has a rich text field, the help center I should have specified that text to text as well, I forgot that and also the writes to text field so first it's text to text and then writes to text is appended afterwards so it's a nice pattern you could use. Once it's done you can, this has a lot of content types, the plone help center as you can see, migrate the containers first and the items later and you're done, all methods look exactly the same like this mostly so they're super short but it's one for each item that's how it works. So for content where you have multiple items like in plone form gen it's like you don't want to port the plone form, the form folder to a easy form folder and then the AT whatever field to some other content type that's not how it works because easy form has a different way for that, you need to specify a more low level migrator with this pattern but this pattern will give you all the freedom you can, you have so with the migrate plone form gen using JSON this would be called for every single instance of a plone form gen folder, self-old would be the plone form gen, the form folder, self-new will be a new instance of easy form it will already have the text, the title and the description and like local roles and all that nonsense but what you can do in data self-json if you're 5jpeg, doesn't exist yet and self-easy form from JSON, passing it the data, it can put in whatever is required to move the archetypes field descriptions to JSON and then turn them into the what is it, supermodel XML schema. So but there's alternatives, one is the in place migrator form for teamwork upgrade, it is mostly interesting for folders because migrating folders can be very slow because what the default migration does it moves all the content from one, it creates a new folder and moves the content over there which is a process that can take a long time if you have a large tree, this is why I say you need to reduce your database size, we're patching away in the default migration, we're patching away a lot of indexing and link integrity, check all the things that are really time consuming but if you have a very large tree it can still take quite some time and this would be an interesting alternative to look at it's well documented, I haven't tried it I have to say and also I have to ask why is there no pull request for Plano content types that replaces the default migration of folders like not every single step but the moving of content from one to the other with this, I'm super open to discuss this because the default migration for folders uses the code from Produs content migration. The second alternative is another beautiful thing I learned from Alessandro Pisa, he's a well of beautiful information, he gave a talk in 2017 and he proposes to use this pattern which is actually also there's a helper method in Plano content types that does the same thing, it changes the base class of the object, this is again very useful for folders, you just simply rip out the old base class, push in the new base class and then set the object in there and the content is still there, you still need to re-instantiate the underscore tree thingy but it doesn't have to move the content from one to another, that is useful. Okay, if you're interested in that talk to Alessandro, he did a lot of big migrations maybe using that pattern for now, it would be nice to have some more documentation, blog post or stuff but the talk is really good, there's nice examples in. So then you're in 5.2, you're running dexterity, you need to, oh god I skipped this part, you need to remove archetypes after you migrate to dexterity, I forgot to write this chapter because I thought my talk is at 2 o'clock after lunch. But there is a method in the package I released that removes archetypes because there are a couple of tools in there that are again, won't die easily but using the pattern there and what I pasted on collective clone org, community clone org will deal with that. So then start up clone without archetypes, so remove the extra in Python 2 and you should be good to go. Okay, the migration to Python 3 is also really well documented, I'm not going to repeat it, I'm pointing you at the documentation, it is here, please read it, it's well documented how to port your add-ons and also well documented how to migrate your database, I'm not going to switch back to the browser but there's a lot there. Keep a copy of your database, please, that's like the basics, don't do a migration without a copy of it, this is how it basically looks like, you invoke this one command, this is the ideal of output because it's a very small site, usually it's bigger, before you do that you pack your database to do zero days again because the old transactions will not be ported as far as I know and important advice, I would ignore most of the things that ZODB verifies says because if it doesn't stop the migration, if you have problems after start up, again alias module is your friend, it will help you patch away any problems of stuff that is still there and you just keep it, the site is probably not going to live for 20 years into the future, so for the 18 years that it will live into the future, it can live with a couple of additional alias modules. Another tip, if you use rel storage migrate to data to ZODB first, this is how you do that, then ZODB convert and then you get a ZODB because it's just very much easier to move a ZODB around than a Postgres database, at least for me, I don't know, maybe your mileage may vary and also the migration with the ZODB has been tested, it's not been tested so far while you keep your content in rel storage, for example Postgres in this case. And then you clean up again, there's a couple things that are still broken, they're not taken care of, they need to move into core, I put them in this package so far because I'm super busy, yeah, I know everyone else. This is one example, the searchable text index will be broken, so it will not be automatically converted, this takes care of that, again it's an upgrade step, just copy and paste it or import it and run it, it checks for the first item if it's a byte, then it has to clear the index and then rebuilds it, it takes a while, but you need to do that only once. And then also event indexes are still, in some old cases are still old daytime, not new daytime, and also navigation portlets will be broken as if they have a custom navigation route because that's a computed field which is not properly, where are my post Python 3 fixes, fixed portlets, it's a computed value, and this is the example how you get any portlet and here you fix the portlet, just set it to something else, five minutes, excellent, if I can manage, if I find my mouse. Then you finalize, which basically means do some cleanup, you enable a new theme, or you get a disabled theme. There's examples for all kinds of stuff in that package, including dropping all of your steam, there is code for removing all skin template, skin customizations and portal customizations, please do that before you migrate because otherwise you will have them in the database after you migrate it and you will wonder why you can't access a certain folder tree because there was a template overwritten and that no longer works, so yeah, there's a lot of useful stuff. There's one thing that is important here, there's a couple, when you migrate aside from Plone 4 to Plone 5.2 and don't create it a new, the drop down navigation will not be enabled by default, it's set on zero I think, maybe we changed that, can't remember, things are changing fast. But the other thing is that the registry, sometimes in the whole process your registry might be gone and you will get weird messages, I said that before, that's easy to fix by just loading the Plone app registry settings that you can have there and never say purge old equals true because that will nuke your site. So why do it this way? This is, because there's lots of talks, we have this magic code that does the migration, we have this transmogrifier pipeline for our super company and we're doing it this way and I hear these interesting things and also hear the question like on Wednesday, how can Plone gain more market share and I get a severe case of cognitive dissonance when I hear that because I think Plone needs a very good out of the box experience regarding migrations and it doesn't help Plone if every company has their hidden away migration code and tries to sell migrations. I don't want to sell migrations, I want my clients to be able to do their migration for themselves. If it's a bigger project, sure you need someone professional and they probably want a new theme and they want new features so it's really good as a project but the important thing about that is the new stuff and not the migration, I don't want to have my clients pay lots of money for okay we need to upgrade to Plone 5 and that's going to cost you 20,000 euros. It's not going to be able to be done any other way. So I really hate that and for this reason I've written the migration code from archetypes to dexterity. I've written the, or helped write the migration from Python 2 to Python 3 and all that stuff. I've written documentation and examples and given talks about that so please use it and if there is an issue, speed wise for example or something is broken, file a ticket, don't send an email, file a ticket and ask the community to help you because we all should use that. If you have custom code that is fixing a problem that you had or if you have custom code that is migrating I don't know, Platform Gen or something else into something new. I would love to see something migrate Mosaic to Volta about that probably not possible. Contribute that to the community, that would be so nice. Here's again the package. Here's again the short thing about how you do that in which order and one last thing. I guess everybody thinks this is going to be super hard and it's going to be tough. Let's just try it. So here is a site I have migrated to Python 3. It's an internet from a client. Uses a Mosaic front page now so we're not moving to Volta right now. They're a bit conservative. So this is, oh you can't see anything. This is it. This is the site. This is another thing. This is Volta. So while Volta starts up I'm going to go to the site. Localhost 8080. There it is. So I log in, I install a PlonDrest API. That's what I actually already did. Then I go to localhost and see anything. I go to 3000 for Pully and see what happens. And hey, there's my content. There's my website. It is, again the front page was, is Mosaic so it doesn't show up. But all the content is there. So if I go to, this is a folder that has a custom content, a custom content type called staff member. I guess everyone has a content type called staff member somewhere. Thank you. I'm almost done. And you click on it. It contains all the content. You get all the file fields from the, from the database. Get all the data. And migrating to Plon6 is actually nothing. The only thing you do is you start up Volta and then you open it, you install PlonResta API and that's it. There is, I know, things to do and I'm very excited to hear the talk about the migration from Dexterity form schema to blocks and how, how the guys at KitsConcept approach that. I'm even more excited to see the code released again. Thank you for your time.
 So recently we did a lot of migrations and a lot of them now with Python 3 we have an  additional reason to a good argument to sell Plon5 to Plon4 users.  So we're doing migrations from Plon4 to Plon5 at the moment.  A lot of them, a lot of these sites still have linguaplon, they need to be migrated  to Plon at multilingual, they still have archetypes, lots of that and also need to be migrated  from Python 2 to Python 3.  So one of the things I learned in the past years is that every migration is different,  only not so much because they have kind of the same problems in maybe not identical but  very similar ways.  So I found that I can reuse most of not only the lessons I learned but also the code that  I use for these migrations and since I got a lot of requests I did a little package called  collective migration helpers that contains almost all the code that I'm showing today,  I'm going to show code from other packages as well.  So if you're interested in reusing any of the methods in there just go there.  So the most important lesson I learned from doing migrations is automate everything.  So write upgrade steps, put everything in code and put it in code so it can run multiple  times, it can be run easily and it can be run in different contexts.  Let's look at an example what I mean by that.  It's a very simple, small method that disables solar.  So it does a couple of things that are interesting here.  So one is that you might recognize, see the context equals none.  In an upgrade step the first argument is usually the portal setup tool.  So you can run that as an upgrade step if you register it but I'm not using the context  at all.  So that means I can also run it from a PDB for example, from a debug console and instance  debug, I can even run it from an instance run script.  I can run it from a browser view, whatever.  So it's very flexible.  This is number one.  So the second thing is I keep imports inside the upgrade step or the upgrade function in  this case.  So in case collective solar doesn't exist, I don't have an import at the top and I also  do make the import conditionally so I check if I actually can import solar, if not I'm  actually not going to do anything.  So it doesn't hurt me if I run this code in a site that's never heard of solar.  And also if it's been run before, so the solar was disabled like 10 minutes ago, it doesn't  hurt at all.  And the last thing is that I log what I'm doing.  So log in for solar disabled.  If I'm not disabling solar, I'm not logging anything.  So that's kind of easy things but if you follow these simple patterns, you will be able to  combine your upgrade steps into something that is easily reusable.  Another example is this, it can run multiple times again.  It can be run in different contexts.  It disables a couple of LDAP authentication plugins and tries to delete them and logs  it if it's there, doesn't log it if it isn't there.  As you can see, a lot of migrations are similar but different.  So probably none of your projects have an AD plugin underscore BG48.  Just one of mine did.  This is why the code in the repository is there for you to copy and paste into your  own modules.  It's not meant, you can use it to import certain things but that's not the initial intention.  So also these two are actually the first two things you should always run.  Maybe not exactly these.  Maybe you don't have solar but elastic search and you don't have LDAP but have something,  I don't know, blown up LDAP.  This is whatever.  But these are two things you don't want while doing migrations, calling some external search  index.  The worst would be writing into that, the production search index while you're doing  migration on a different machine because the blown instance keeps the URL from the IP from  the solar instance for example, that would be terrible.  And also you don't want LDAP to be asked every time you do something on your development.  But before you do that, you try to get some information.  Get the bird's eye view, that's how I called it.  Say you want to learn as much as you can about your site, what content types are there, what  add-ons are there, how old your items are, how many items there are, how heavy, I mean  in megabytes they are, about local roles, about placeful workflows, about all kinds  of things.  And here's a small example from the statistics module in the package I made public.  This is a helper view, it's a browser view in this case that will, you don't have to  read all this, will list a lot of useful information that you usually want to know.  For example, how many items of which type there are, so there's obviously 884 help  center leaf pages in this portal, where that content is, so there's obviously some weird  folder that, I don't know, services that contains a lot of stuff.  So how old your content is, so something was very weird in 2019 because they had 1,900  events, so pretty busy year probably.  And also, where you actually, the biggest objects live, sometimes you have people uploading  whole ISO images into clone instances.  So we have a huge zip file of 600 megabytes on the top and it sorts them by size.  Just a small example, you can adapt the code to your need.  Another thing I like to do is find out where the content is that needs to be removed at  some point, so when you find out what kind of add-ons you have, you'll see here we have  platform gen help center, collage and calendar X folder, these are in that project, these  were used quite a lot and it tells me now, okay, where is that, how many items are in  there because most of them are container-ish, which one was created, from which year are  these items, and with this information you can send emails to people who are in charge  of the site and ask them, okay, this is what you have, this is where it is, what do you  want to do with this and more often than not the answer is going to be, oh, I had no idea  this actually still existed, please remove it.  So you don't have to deal with that, but unless you can't make that decision yourself, unless  you're the project manager content-wise, which usually are not as a developer.  So the second big lesson is to divide and conquer.  In an abstract way that means deal with one problem at a time and ignore and replace,  remove and replace one add-on at a time, solve one problem at a time and ignore problems  that don't block you, put them in a to-do list and they may have just gone away in platform  5.2.  So that is important advice.  Divide and conquer also means divide your migrations into a multitude of upgrade steps.  So have one upgrade step for every single thing you do, so you can run them separately,  either in the PDB or with portal set-up, so these are a couple of upgrade steps, migrating  images, migrate everything else to plan up content types, uninstall archetypes, and only  when this actually works for enough times and you're content and you don't want to  click so many buttons anymore, you can combine them into one upgrade steps group as PLONE  itself does.  So this allows you to run them separately so you get an individual commit after each  upgrade step, as long as you do it in the user interface.  So if step 2 takes 5 minutes and in step 3 you're going to get an error and you run  them sequentially without a commit in between, you get very angry at yourself, you'll hate  yourself, so don't do that.  Also you might be tempted to use branches for your upgrade steps and migration code,  which don't, in PLONE we have things that can help you not to do that, one would be  moving imports into the code, have conditional imports that fail when you don't have that,  like I don't know, no more lingual PLONE, you don't have to do anything because you're  now in PLONE 5.1.  Also you can register upgrade steps conditionally with ZCML condition, have PLONE BLAR version  as PLONE itself does it.  You can decide to register your steps like this.  Here's one example, only if you have PLONE 5 you should be able to run this step to uninstall  archetypes.  So this sounds really easy but it's totally not, but there are a couple of nice tricks  that you should use and keep in mind.  One thing super obvious, which nobody does, is just remove 98% of your content from the  portal that you want to migrate.  I often have very huge databases to deal with your migrations and okay my laptop is fast  but still it takes a lot of time to move this data around, so write something like this  that deletes all files and images that are bigger than one megabyte, just one example.  Don't remove any folders, but if you have ATE help centers and 50,000 documents and  900,000 news items, just remove 99% of each of these items and try to deal with everything  else, but keep the content structure in place because that don't delete any folders because  A, they're not heavy, I mean in megabytes, only their content is, and B, this will save  you from running into issues later like local roles and portlets are mostly assigned on  folders and these are things that sometimes break your migrations, so if you keep these,  something that is container-ish, keep everything there unless it's something like a Plone Help  Center which is always kind of the same, but if you have folders or other folder types,  keep them around.  And you can still fix any corner cases after your migration runs through, so I showed that  already.  Another tip I got from Alessandro is do not migrate blobs because the blobs, the blob  itself will not change, neither from archetype, the to dexterity, nor from Python 2 to Python  3, the PDF that lives on the file system is always going to be the same, and if you use  an add-on called experimental graceful blob missing, which actually even works with Rails  storage now, since I haven't merged that yet, but it's there, you move the blob storage  away, just move it to a backup folder and you use that.  This add-on will create an empty file with a marker text in it for everything that it  touches, it's really fast because it's a very small file and you can re-add the blobs  once the whole migration is done later.  Since the UIDs in any same migration will stay identical, the path to the blob file will  also stay identical.  You need to keep the blob files during cleanup, so if you decide, ah, I need to remove this  and this and this and this, and for this you need the blobs because you really want to  lose the blobs as well, so don't forget that.  So that is also a nice, tiny thing, when you copy a big database from a production server  or a common server to another, also copy the data FS.index because re-indexing a really  big database on startup can take, I don't know, up to half an hour, depends on how big  it is.  This is nice, so you got to forget the past and forgive your sins, the easiest, and you  need permission to, for all that, but you will, it's a technical requirement anyway,  but you need to talk to your clients.  We're going to remove all revisions, for example, this is a five line code which changed  a data FS from 21 gigabytes to three gigabytes and the blob storage from 91 gigabytes to 50  gigabytes just by removing all revisions.  And yes, binary files can be versioned, you should not do that, but some clients do, so  you will have, this is how we lost, went from 91 to 50 gigabytes because binary files are  versioned.  And an alternative to this would be to use Collective Revision Manager, if you want to  control this manually, you can decide how many items you want to keep for, which objects  and so on, so that's nice.  The second part of your past is previous commits, you need to pack the database, it takes a  while, the important thing is set it to day zero, that is very important to keep zero  days, that's very important when you migrate to Python three later on.  Okay, now let's go to the hard part, add-ons.  So here's a simple example, removes an add-on that is easy to remove and after that it's  actually gone, and it's a good pattern, you could add more logging, but that's fine.  Moving on to a slightly more complicated example, this is also a very small add-on, but it changes  existing content, so when you remove an add-on sometimes it adds additional views, you have  to unretches to these in generic setup for example, so that they're no longer allowed,  but you also have to unset them on the objects that use these views for example.  This is one small example, the important thing is to drop the layout attribute property from  all objects that have this first, that's like a replacement for default views, don't really  recommend it, but yeah.  Okay, a bit more advanced, here is a portlet, because portlets create content, portlets  in this case, and you have to go through the site and find them, in this case I actually  knew where it was, it's an NM example intranet, that's where it was, so it's easy to find,  but in the packages also code to find portlets anywhere of a certain kind, so you can use  that to remove portlets.  Here's a, that's tricky, so for those who don't know that, the portal quick installer  has a feature, has its own registry that stores information about all add-ons that were imported,  that were installed, and depending on the age and the state of the add-on it was in when  it was first installed, this will contain different information, including types, some  add-ons have a types XML that contains all types where they set additional information  on, and not only a types XML that contains the types that it actually registers, which  is a simple beginner bug, the result from that is that upon uninstalling this add-on,  this type will be gone, if that type is your folder, like in this case, or a collection,  like in this case, you're in big trouble because then you have to try to, you have to reinstall  the type or re-register the type, taking it from Planner Content Types or wherever it  came from, the one you want, so, but when uninstalling an add-on you can pass it a modified cascade  that is different from the default cascade in the quick installer and that will save  you from that trouble. That's the same thing happens with the registry or used to happen  with the registry, it was totally nuked when uninstalling some add-ons and nothing ever  worked anymore unless you reinstalled the registry, which was easy but not obvious and  the error message was not, oh, you just lost the registry, please reinstall the registry  was something very much more obscure. Just quickly about profiles, you can and you should  use uninstalled and upgrade profiles, like Plone does, so personally I'd like to, I admit  that I like the approach to have all the stuff during a migration in Python, so I only have  to look in one place what it actually does, but some things are actually much, much easier  to achieve with generic setup than in Python, so I don't know, I'm kind of on the fence  on this. Here's an example of an upgrade step in Plone itself, this is how it's registered  and it has, oh, I was supposed to show the structure now in my editor, which I don't  have, okay, you can check that yourself, it is in, it's in Plone Up upgrades and there  is a folder that is called 2RC4, release candidate 4 and it contains a folder XML that removes  a view and actions XML and the tool set XML and stuff like that, just look up these examples.  Okay, moving on to persistent, this is hard, there are really long blog posts about that  and it's, you can uninstall utilities and tools and components with generic setup, but it  doesn't always die the way you want it, so you can do that in Python as well. Here's  one example from the package I released that removes utilities and adapters that are used  by Linguo Plone, it doesn't hurt to run this code 50 times, I have, actually I have a migration  where I run this after each individual upgrade step because I had cases when, I don't know,  maybe it was my fault, my commit was not, didn't go through properly, so I just put  this at the end of every single upgrade step I had there and it worked well. I'm not going  to walk through the code, but it's, if you know what you want to remove, removing it  is actually really simple because utility subscriber and then you put the name of the  thing and remove it, bam, not that hard. But since it is hard to figure out what to  remove and it's a tedious process to iterate over that, okay, yeah, something that's not  dying, let's write a method to kill it, it's often more easy to actually patch it away  and there is a nice tool called alias module, it's Implona butyls and it helps you to get  rid of things that are, have a hard time dying. This is an example from webdav, so what it  does is it adds IFTP access into the path, so it's importable by the persistent data  in your database when it's unpickling the pickle that is referencing that, but it's  replacing whatever it was before with the empty BBB interface, BBB means backward compatibility,  I don't know that doesn't make sense but someone came up with that at some point. Here's another  example from collective easy slideshow, so this is not a core thing but this is an add-on  that had a really hard time dying and this was a very easy way to get rid of it, so if  you feel dirty now, if you look at this, you shouldn't because this is how ploneup upgrade  actually looks, these are only the first 30 lines from it, the file is much, much longer  and it runs every single time you start a plone and it patches away a lot of things that  you've plone and we got rid of over the years, but some of these things are really hard to  uninstall and clear out the database because there might be old pickles that are no longer  used but if you go back in your database like you revert commits, then they are resurrected  again and this is this really hard technical problem and this just, this solves this in  a very nice, nice way. Moving on to linguaplone, okay. So very important, migrate from linguaplone  to ploneup multilingual while running in Python 3.4.3, not 3.4 please, and while still using  ArchiTimes, there is a build-in migration in ploneup multilingual, sorry, that actually  works fine but it mostly works if your content is not a total mess but your content is very  likely to be a total mess because editors in linguaplone had the ability to do things  that are not useful, for example, have a folder in one language and have mixed languages inside  that folder but part of that, these translations are then in a different folder again so it's  a complete mess and this is some code that you totally should read right now that helps  you clean up this so only, I can't point but somewhere in there are actually two lines  of code that run the whole migration from ploneup, linguaplone to ploneup multilingual,  they're conveniently called step one and two and the other was called step three, so call  them method step one and two, I like that, still, everything else, it's in the package,  it tries to make an educated guess if you have two, this works with two languages, only  can try to adapt it and it tries to move stuff around in the same way, same way so that your  content actually if it doesn't have a language, it guesses the language from the language of  the folder and from the language of the majority of the content in the folder and if it's a  different language, it creates a translation for the parent folder and for an untranslated  folders, it looks into what language has the content and so on and so on so it does a lot  of stuff and you can use it for your own good and at your own risk, please, it is, I'm not  going to walk through that, that would take too long.  There's also, yeah, look at the whole linguaplone module in migration for helpers, if you still  have that, so once you did that, all these things like clean up, remove add-ons, slim  your database, remove your revisions, then you can safely upgrade to plone five two,  either from plone five one or plone zero, plone three or whatever you want.  So do that naturally while still in python two, do that keeping archetypes still if you  have it, keep, have it, have multilingual enabled if you have it and if you come from  plone four as a gesture, you directly upgrade to plone five two or rather five two one,  which will be out soon, I saw a pending folder already on this plone org, so expect that  within the week.  In some rare cases, you might want to upgrade to five zero ten or five one six first, but  I haven't had such a case actually.  So how you do that?  Plone five two with archetypes, that's easy, just add the archetypes extra to your build  out and it will pull in all the archetypes dependencies and don't stop there, don't run  plone on five two with archetypes, that is just so insulting to everyone who put in work  into plone in the last five years.  Go further from there.  And yeah, there's also documentation about changes in plone five two, they're really  well documented, so please read the FM.  And then migrate to x30 in five two.  I've covered this a couple of times already, so I'm not going to go into too much detail,  but I'm going to show you a couple of best practices, especially for sites with a lot  of content and complex things.  If you have a default site, you can use it for it, if you want, it's fine.  Automate it, migrate one type at a time, start with the folders and first again try it with  a reduced database.  So this is an example to migrate all files from archetypes to dexterity.  This is not new.  This has been in the documentation of plone up content types since, I don't know, many  years now and it works fine.  It has been improved, that's why I say use it in five two.  For example, the patch searchable text and a couple of other things in this method, the  PAC migration, it makes it much faster if you skip indexing and patch searchable text away  because your searchable text is going to be the same anyway, so why would you want to  reindex them just for the sake of migrating?  If everything works fine, do that, do it all at once.  Even topics, works out of the box, you can migrate topics to dexterity collections, thank  you.  Much more interesting, custom content.  So for example, plone help center, who uses or who uses plone help center?  It's easy to migrate, that's why I picked it for this purpose demonstration because  it's mostly default content with constraints and a couple of views that look slightly different  and it has one additional feature that folders don't have so you can manage versions but  my client doesn't use that feature at all, so I decided I'll just migrate it to default  content like folders and pages.  So this is an example to migrate the help center how to folders, as I said, it's a couple  of folders that contain how tos with a constraint and this is all you need, this does everything,  you don't need anything else, this could be run from a PDB thing, it goes through the  whole site and it turns every single instance of a plone help center how to folder into  a normal folder, bam.  Okay there's a couple more complicated items, for example the help center FAQ is a document,  so you can specify a field mapping to map the text field to the text field and say that's  a rich text and so it's migrating the help center FAQ to a document and it's keeping  the rich text.  You need to specify the fields because the source type doesn't come, it can't pick up  the migration helper, the migrate custom AT thingy here, it doesn't know about help center  FAQ, it knows about all the default content types in plone so if you have a migrated document  it knows there's a rich text field I need to migrate but for help center FAQ everything  that is not like doubling core or one of the default fields needs to be specified here in  a very simple way.  You can specify something more complicated here, this is a nice pattern you can use if  you have a content type that you want to migrate to default content but the content type you  had before is mostly for archive purposes only but it had like 50 rich text fields and  five text fields on top of that but you don't want that anymore, you just want the information  that's in there so what it does it goes, it's a field migrator that concatenates these text  fields one after the other adding a small heading with the name of the field before  that so you have like 20 fields and now they're in one rich text field because nobody's using  the plone help center anymore, actually they're using it but now it's not called plone help  center, it's called a document and you use sensible headings to add your content and  not a structure.  So this would then migrate the help center itself because the help center has an additional  field called writes, I actually don't have no idea what it holds so I never, some text  so I add that into the text field that it also has so it has a rich text field, the  help center I should have specified that text to text as well, I forgot that and also the  writes to text field so first it's text to text and then writes to text is appended afterwards  so it's a nice pattern you could use.  Once it's done you can, this has a lot of content types, the plone help center as you  can see, migrate the containers first and the items later and you're done, all methods  look exactly the same like this mostly so they're super short but it's one for each  item that's how it works.  So for content where you have multiple items like in plone form gen it's like you don't  want to port the plone form, the form folder to a easy form folder and then the AT whatever  field to some other content type that's not how it works because easy form has a different  way for that, you need to specify a more low level migrator with this pattern but this  pattern will give you all the freedom you can, you have so with the migrate plone form  gen using JSON this would be called for every single instance of a plone form gen folder,  self-old would be the plone form gen, the form folder, self-new will be a new instance  of easy form it will already have the text, the title and the description and like local  roles and all that nonsense but what you can do in data self-json if you're 5jpeg, doesn't  exist yet and self-easy form from JSON, passing it the data, it can put in whatever is required  to move the archetypes field descriptions to JSON and then turn them into the what is  it, supermodel XML schema.  So but there's alternatives, one is the in place migrator form for teamwork upgrade,  it is mostly interesting for folders because migrating folders can be very slow because  what the default migration does it moves all the content from one, it creates a new folder  and moves the content over there which is a process that can take a long time if you  have a large tree, this is why I say you need to reduce your database size, we're patching  away in the default migration, we're patching away a lot of indexing and link integrity,  check all the things that are really time consuming but if you have a very large tree  it can still take quite some time and this would be an interesting alternative to look  at it's well documented, I haven't tried it I have to say and also I have to ask why is  there no pull request for Plano content types that replaces the default migration of folders  like not every single step but the moving of content from one to the other with this,  I'm super open to discuss this because the default migration for folders uses the code  from Produs content migration.  The second alternative is another beautiful thing I learned from Alessandro Pisa, he's  a well of beautiful information, he gave a talk in 2017 and he proposes to use this pattern  which is actually also there's a helper method in Plano content types that does the same  thing, it changes the base class of the object, this is again very useful for folders, you  just simply rip out the old base class, push in the new base class and then set the object  in there and the content is still there, you still need to re-instantiate the underscore  tree thingy but it doesn't have to move the content from one to another, that is useful.  Okay, if you're interested in that talk to Alessandro, he did a lot of big migrations  maybe using that pattern for now, it would be nice to have some more documentation, blog  post or stuff but the talk is really good, there's nice examples in.  So then you're in 5.2, you're running dexterity, you need to, oh god I skipped this part, you  need to remove archetypes after you migrate to dexterity, I forgot to write this chapter  because I thought my talk is at 2 o'clock after lunch.  But there is a method in the package I released that removes archetypes because there are  a couple of tools in there that are again, won't die easily but using the pattern there  and what I pasted on collective clone org, community clone org will deal with that.  So then start up clone without archetypes, so remove the extra in Python 2 and you should  be good to go.  Okay, the migration to Python 3 is also really well documented, I'm not going to repeat it,  I'm pointing you at the documentation, it is here, please read it, it's well documented  how to port your add-ons and also well documented how to migrate your database, I'm not going  to switch back to the browser but there's a lot there.  Keep a copy of your database, please, that's like the basics, don't do a migration without  a copy of it, this is how it basically looks like, you invoke this one command, this is  the ideal of output because it's a very small site, usually it's bigger, before you do that  you pack your database to do zero days again because the old transactions will not be ported  as far as I know and important advice, I would ignore most of the things that ZODB verifies  says because if it doesn't stop the migration, if you have problems after start up, again  alias module is your friend, it will help you patch away any problems of stuff that is still  there and you just keep it, the site is probably not going to live for 20 years into the future,  so for the 18 years that it will live into the future, it can live with a couple of additional  alias modules.  Another tip, if you use rel storage migrate to data to ZODB first, this is how you do that,  then ZODB convert and then you get a ZODB because it's just very much easier to move  a ZODB around than a Postgres database, at least for me, I don't know, maybe your mileage  may vary and also the migration with the ZODB has been tested, it's not been tested so far  while you keep your content in rel storage, for example Postgres in this case.  And then you clean up again, there's a couple things that are still broken, they're not  taken care of, they need to move into core, I put them in this package so far because  I'm super busy, yeah, I know everyone else.  This is one example, the searchable text index will be broken, so it will not be automatically  converted, this takes care of that, again it's an upgrade step, just copy and paste it or  import it and run it, it checks for the first item if it's a byte, then it has to clear  the index and then rebuilds it, it takes a while, but you need to do that only once.  And then also event indexes are still, in some old cases are still old daytime, not  new daytime, and also navigation portlets will be broken as if they have a custom navigation  route because that's a computed field which is not properly, where are my post Python  3 fixes, fixed portlets, it's a computed value, and this is the example how you get any portlet  and here you fix the portlet, just set it to something else, five minutes, excellent,  if I can manage, if I find my mouse.  Then you finalize, which basically means do some cleanup, you enable a new theme, or you  get a disabled theme.  There's examples for all kinds of stuff in that package, including dropping all of your  steam, there is code for removing all skin template, skin customizations and portal  customizations, please do that before you migrate because otherwise you will have them  in the database after you migrate it and you will wonder why you can't access a certain  folder tree because there was a template overwritten and that no longer works, so yeah, there's  a lot of useful stuff.  There's one thing that is important here, there's a couple, when you migrate aside from  Plone 4 to Plone 5.2 and don't create it a new, the drop down navigation will not be  enabled by default, it's set on zero I think, maybe we changed that, can't remember, things  are changing fast.  But the other thing is that the registry, sometimes in the whole process your registry  might be gone and you will get weird messages, I said that before, that's easy to fix by  just loading the Plone app registry settings that you can have there and never say purge  old equals true because that will nuke your site.  So why do it this way?  This is, because there's lots of talks, we have this magic code that does the migration,  we have this transmogrifier pipeline for our super company and we're doing it this way  and I hear these interesting things and also hear the question like on Wednesday, how can  Plone gain more market share and I get a severe case of cognitive dissonance when I hear  that because I think Plone needs a very good out of the box experience regarding migrations  and it doesn't help Plone if every company has their hidden away migration code and tries  to sell migrations.  I don't want to sell migrations, I want my clients to be able to do their migration for  themselves.  If it's a bigger project, sure you need someone professional and they probably want a new  theme and they want new features so it's really good as a project but the important thing  about that is the new stuff and not the migration, I don't want to have my clients pay lots of  money for okay we need to upgrade to Plone 5 and that's going to cost you 20,000 euros.  It's not going to be able to be done any other way.  So I really hate that and for this reason I've written the migration code from archetypes  to dexterity.  I've written the, or helped write the migration from Python 2 to Python 3 and all that stuff.  I've written documentation and examples and given talks about that so please use it and  if there is an issue, speed wise for example or something is broken, file a ticket, don't  send an email, file a ticket and ask the community to help you because we all should use that.  If you have custom code that is fixing a problem that you had or if you have custom code that  is migrating I don't know, Platform Gen or something else into something new.  I would love to see something migrate Mosaic to Volta about that probably not possible.  Contribute that to the community, that would be so nice.  Here's again the package.  Here's again the short thing about how you do that in which order and one last thing.  I guess everybody thinks this is going to be super hard and it's going to be tough.  Let's just try it.  So here is a site I have migrated to Python 3.  It's an internet from a client.  Uses a Mosaic front page now so we're not moving to Volta right now.  They're a bit conservative.  So this is, oh you can't see anything.  This is it.  This is the site.  This is another thing.  This is Volta.  So while Volta starts up I'm going to go to the site.  Localhost 8080.  There it is.  So I log in, I install a PlonDrest API.  That's what I actually already did.  Then I go to localhost and see anything.  I go to 3000 for Pully and see what happens.  And hey, there's my content.  There's my website.  It is, again the front page was, is Mosaic so it doesn't show up.  But all the content is there.  So if I go to, this is a folder that has a custom content, a custom content type called  staff member.  I guess everyone has a content type called staff member somewhere.  Thank you.  I'm almost done.  And you click on it.  It contains all the content.  You get all the file fields from the, from the database.  Get all the data.  And migrating to Plon6 is actually nothing.  The only thing you do is you start up Volta and then you open it, you install PlonResta  API and that's it.  There is, I know, things to do and I'm very excited to hear the talk about the migration  from Dexterity form schema to blocks and how, how the guys at KitsConcept approach that.  I'm even more excited to see the code released again.  Thank you for your time.
Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_0_47.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_48_95.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_96_155.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_156_215.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_216_275.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_276_323.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_324_671.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_672_719.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_720_767.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_768_875.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_876_923.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_924_1031.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_1032_1091.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_1092_1919.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_1920_9359.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_9360_28907.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_28908_30047.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_30048_30203.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_30204_31307.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_31308_37211.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_37212_41231.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_41232_41891.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_41892_42095.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_42096_45671.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_45672_55223.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_55224_56231.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_56232_56603.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_56604_58127.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_58128_58235.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_58236_58307.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_58308_58463.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_58464_61655.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_61656_62747.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_62748_62831.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_62832_63395.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_63396_65783.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_65784_69371.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_69372_69443.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_69444_69923.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_69924_70991.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_70992_71051.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_71052_71183.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_71184_71423.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_71424_71615.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_71616_72083.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_72084_72479.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_72480_72563.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_72564_73115.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_73116_73271.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_73272_73331.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_73332_73535.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_73536_73583.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_73584_73643.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_73644_73691.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_73692_73763.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_73764_73811.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_73812_73859.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_73860_73907.png Philip_Bauer_-_Migrations__Migrations__Migrations__ZIN1qmhMHJ4_73908_73915.png