Doing the impossible
CC Attribution 3.0 Germany:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Loechel, Alexander
Zope and Plone are fantastic systems that provide a lot of features out of the box. A primary focus of Zope and Plone has always been about security and empowering of users, and one of the things that provides this is RestrictedPython. A lot of code in Zope and Plone has become legacy and remained unmaintained for several years, RestrictedPython was one of those core packages that got less attention and became the major blocker for the porting of Zope and Plone to Python 3. Many thought it would be impossible to port RestrictedPython to Python 3, but in May 2017 we released a version that is compatible with both Python 2.7 and Python 3.4-3.6 and even PyPy. Nothing is impossible within the Plone community! This talk will focus on RestrictedPython: • what it is, • how and who could use it, • what was the problem with porting to Python 3, • why things seem impossible and why that is almost always not true, • thoughts on Zope/Plone and other framework security
 So, I'm going to show you a little bit of the application server. It's one of the largest piece of software in the Python world that has not been Python 3 compatible two years ago from now. That was a huge problem because a lot of our projects depend on Plown and without a path to the future, it could be horrible. So, yeah, what was the problem? We started in Bristol. I think it was 2014 in Bristol. We had the first discussion about the roadmap again, how to move, and some of the quotes was it is impossible to get Plown to Python 3 because there are too many blockers in Zope. No one has touched restricted Python or access control and they are from the beginning. So, it's the really hard stuff. But, hey, when it is time, our third C-Clock, the author of 2001 Odyssey in Space, has made some fantastic quotes about it. When it is distinguished, what elderly scientists states that something is possible is almost certain right. When he states that something is impossible is very probably wrong. And if someone tells you that's impossible, well, try to change it, make it possible, that's about what it is. So, why should be restricted Python be the major blocker for Plown on Python 3? Well, if you look at the dependency graph, all of the underlying frameworks that we depend on, access control, document templates, Python scripts, the Z catalog, Zope 2, everything directly depends on restricted Python. And a lot of the other up-way framework parts of Plown that makes Plown Plown depends on them. So, yeah, we need to touch restricted Python. And, you know, every piece of Zope that was not adopted by Plown is literally dead. Not a lot of the Plown people touched those parts of Zope. Restricted Python has been around till the end of the 1990s. So, in the code I've seen, there was a Python version 0.8 something or so that it has started on. So, really weird shit and stuff, yeah, but hey, let's try to adopt it. And that's the fantastic thing in the Plown community. We have a diverse group. Everybody has a capability of doing something. So, we started it, or I started it. And, well, what was the problem restricted Python has almost had no documentation. The test coverage on its own was extremely low. And, you know, debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, you're not smart enough to debug it. Horrible stuff. Restricted Python for that time was probably one of the most advanced Python packages around. It took a long time to go into it and understand it. But, hey, the problem was created by ourselves where Zope leads Python follows. A lot of the Python 2 standard library are directly influenced by Zope. There are a lot of packages that has been gone now in the Python 3 world that are just left overs because Zope needed them. So, there are deprecated modules like the compiler package, which was 80% of the implementation on restricted Python, depending on, and there was no documentation at all on the insides of the compiler package. Horrible. The only way to discover the limits of the possible is to venture a little way past them into the impossible. And so, let's go there. It's fun getting into the new stuff and, or the old stuff. Archaeology is also, but, hey, what doesn't mean any sufficient advanced technologies in this thing for magic. So, if you're looking at the stuff we have in clone and Zope, there are so many fantastic things. And we have a lot of things that we don't have, or other frameworks don't have. So, if you look through the web, we can write scripts in Zope that could be used by, or written by our users that are not destroying your server completely or something or can hurt that extreme. We have a fantastic workflow machinery. And so, that directly depends on restricted Python to only show you the things you're allowed to see. So, hell, there is a lot of shit in for someone who's outside. That's like magic. How can they accomplish that? And that's the thing that makes clones so special. And that's the thing we want to keep. So, yeah. But the problem is what is restricted Python? And especially what it is not. Because if you're looking, in the early days of Python, the first Python workshop at NIST, the SPAM 1, the first Python conference, in 1994 already has an S, their first topic, requirements for a saved Python interpreter. And Jim Fulton and Paul Everett were at NIST at the conference. That was the origin where the idea for restricted Python is originated from. So, there is the requirement for that. We have it for almost 15 years now. But Zelp and clone are the only ones that use that. So, yeah. Most of the biggest problems in software are problems of misconceptions. So, the people don't understand the real problem or the real needs of the users. So, that's where we need to figure out what to do. And, well, restricted Python is not a sandbox. That's important to know. Sandboxes most often don't work. You cannot escape the sandbox. So, if you're relying on a sandbox to think, to secure your system, there's always someone smarter than you that found a way to escape. Security by obscurity or by putting everything into one place is problematic. Doing security is a long process. There are different approaches to that. And Python has done a way that has been shown to work in another language also. So, what restricted Python is, is a limited, safe subset of the Python programming language. The grammar, the grammar that we love to work on. But it is so limited that you get it to be not anymore Turing complete. So, programming languages depends on being Turing complete to be able to do everything. So, if you limit that, you could do it and decrease the possible things that could be done with it. There are simple things that you would like to allow the people, but not everything. You don't need file access or something else. And there was another project that's have done that, that was the other project in 1995 with the Ravenscot profile file, which is a limited subset of the programming language order that makes it restricted and logical testable so that you can show that the program, a program in it could be run correctly. So, yeah, there is a way. And that's also the way to restrict the Python 4.0, the way to Python 3. So, well, if you want to start it, first solve the problem, then write the code. So, understanding what it is, think. And, yeah, the first read of the complete code says it uses the compiler module and its AST class. Well, go to docs.python.org, compiler package. There is not, there may be 100 lines, a short overview, but no details, nothing. It's not fully documented. There's no described upgrade path or something like that. Just say it's gone in Python 3. Deplicate it. And the other thing is, if you're looking into the code of restricted Python 3. They do manual byte code generation. Manual byte code generation? Holy fuck, what are they doing there? You have interpreter specifics. They just let it run on C Python 2. Oh, yeah. It's hard. But, well, if you think in some scientific way, it's not documented, it's not usable. If it's not tested, it did not work. If it's not checked in into version control, it did not exist. And if it's not repeatable, it's not science. So, someone has already found a way to shield it. So, there must be another way to do it. If someone has already done it, it should be repeatable. That's science. So, yeah, it's hard. And compiler knowledge is necessary to port it. Hell, we are such a diverse community that we have a specialist for everything. I have studied computer science. I have taken compiler lessons or written one of my thesis about compilers. Well, it probably will take time, but it's not impossible. So, let's give it a try. And, well, where did it start? In Plon OpenGarden 2015. Then we discussed it again and start looking into the code. And, yeah, the people already decided, if that's not possible, we need an alternative way. So, Plon Server, which now has become GioTiner, and some other ways to make a turnaround on that, probably get rid of restricted Python if we can't handle it. But, hey, if we can do it, let's do it. And so, reading and understanding the code is the necessity in the beginning. And, hey, any fool can write code that a computer can understand. Good programmers write code that humans can understand. That's all about it. You need to get down to the Zen in Python. Make the code readable. Make it understandable. So, now I have taken over the maintenance, but I'm probably not smart enough to do it from the whole of my life or something like that. So, someone else, if I'm probably gone, could take over the maintenance. So, make the code more easy, more documented, more commanded, and everything so that the community can work with it. So, yeah, what did I do? I started, and the first half year I was working on porting was just writing documentation. What I see in the code, just annotate everything, making the whole set, getting some feedback from some of the Zop developers that has already worked on it, and, hey, if you don't have any requirements or design, programming is just the art of editing back to the empty text file. So, it did not work that way. So, yeah, compiler AST. Hey, there is a new preferred way in Python 3 that already was established in Python 2.6. That's the new AST model. And AST could completely replace the compiler AST model. And the compiler function itself in Python accept AST as an input. So, you do not need to manually generate bytecode anymore. You just drop in the modified AST and get it worked. And you can get out of the compiler function the AST and get bytecode out of it. And that works from Python 3.6 up and Python 3.4 up in a very, very smooth way. The packages have already been there in the beginning of Python 3.0. Python 3.0 implementation from 3.3 to 3.4 changed a bit. So, we have tried to make it comparable with 3.2 and 3.3 in the beginning. It was hard, but they are already out of support. So, we skip that and make it easier. Python 3.4, 3.5, 3.6, 3.7 are going like a gland. So, that is not a problem. We can get to work on that. And, well, the first thing that was necessary was to get access to the foundation or the foundation repositories. And I am glad that Minabu organized the Plones Symposium in Tokyo, 2015. That was the first place where I met Pressiever and get access to the repositories and everything. So, that was necessary. And it is sometimes the simple things that take you to travel 20,000 miles. Yeah. And, hey, another thing is we have done a lot of code in the old Zope way. But we are now tending to do more and more coding conventions to get the code readable. And that is what the Zen of Python is about. It is the lessons learned by the people at Zope Corporation at that time from Zope and translated into the whole Python community. So, beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. And, hey, you have to read your code. And after two weeks, three months, six months, looking at your own code, it could be the code of someone else. You need to understand it again. If you have not written it the way that you can read and understand it, quickly again, you are lost. So, the first thing I tried to make was applying our new coding conventions on it. So, I did not change any of the implementation details. I just make it following our coding conventions so that it is more readable. The second thing is writing tests. And I really have raised the test coverage. I thought in the beginning it was about 18%, 18%, less than 20%. Test coverage over the whole package. We have raised the test coverage now to more than 95% of the complete code. We've done it completely tested. Every syntax statement that is in the Python language is explicitly tested. What happens if it is allowed or disallowed in Python 3? Or in Restricted Python. And we do a functional test of all of that. So, hey, in tests, the first principle is that you must not fool yourself. You are the easiest person to fool. Because you believe that the things you are doing is good and right. So, get another person into the system. Have them have a look at it. That's the thing you need. So, I make a decision together with two others and said we have to move. The test set up, Restricted Python did not depend on any other external package. It just depends on the Python standard library. But writing tests in the Python standard unit test library is not that straightforward. And PyTest and TOX give you a lot of easier stuff to start at. And it's actually at the moment the defective standard in modern Python packages. And TOX gives you the possibility to test multiple Python versions locally on your machine against the same thing. That's fantastic. And PyTest gives you the possibility to parametrize your tests. Not only with input values, but also, that's the advantage of Python, you can pass a function in. So, I do paramize the tests with the old implementation and the new implementation. So, we had the left the old implementation, complete impact, do just another file next to it with the new implementation. So, we can test if everything works like it should be. If it's on the one implementation or on the other. So, at least for Python 2, we know it is exactly the same. And that's fantastic. And, well, PyTest has the advantage you just can use the answer. You don't have all the time look up, is it now as true, is it as raised with something or so? Just easier. I give directly after lunch another talk about modern Python testing. If you're interested in the lessons I have learned on porting restricted Python, what we should do and get to get better best practices on developing stuff compatible. You can come there. That's one of the examples of the tests we have done in restricted Python. So, we take both implementations. So, the new implementation, the second line is the old implementation we pass it in and just look, execute a statement and look if we get errors, if the result is also translatable or a hand load, we export it and see what it gets out. So, you can prove both implementations work the same. Perfect. So, yeah, it takes a long time. And doing it alone is very hard. I'm glad that during the up-and-city sprint, the people from GoSep and some other core developer joined in, stepped up and helped through, we were able to do it. And hey, if someone says to you, it is impossible, well, things that are impossible just take longer. It's the fun thing. You can get it work. That's the nice thing. So, at the beginning, all the people were saying, Zope is that. But we as Plon has not adopted is that. Hey, Zope is not that. The people came up again today. Or if you're looking at the conference, there are several people that originates from the Zope community, are here and doing stuff. And, yeah, we get it working during one of the Zope sprints in May 2017 to cut a first release of restricted Python 4.0. That is Python 3.0. We can already test that. Zope 4, the talk from Hano tomorrow, will tell about runs completely on Python 3. And it depends on restricted Python and access control. And you see on the right, we now have a test coverage of 98.58% test coverage. Python 2, 7, 3, 4, 3, 5, 3, 6, all working, all test passes. Each time we do something. So, yeah, we can get it done. And how has Ice Mac Michael Hovitz from GoSep said it? Welcome to the Python 3.0. Thank you him for his work doing it, helping me. And, well, for those people that are outside, or knowing some of the old stuff from Zope, PJAB has given a fantastic quote on it. Those who do not study Zope are condemned to reinvent it. We do see a lot of fantastic stuff in the Python community today. They tried to do stuff that has Zope done before. But they are still stuck in the path where they are not can give the equivalent of Zope some way. So, hey, on the other hand, they start and say, that couldn't be that hard. But, hey, if you think it's simple, then you probably have misunderstood the problem. You have to really think and look how it works. It is a long path, but it could do. And my wish is that we get the strength of the Python world. We have the Zope and Plon community into other frameworks. Get them to use the stuff we produced. And they are adapted by more people so they can work with it. Project Jupiter, Jupiter Notebook, which was originally iPython Notebook, is a fantastic environment for learning and teaching on Python, doing data science stuff in Python through the web, but you can kill your complete server through the web. So, you have to shield it. You have to do everything. And they have a sandbox. But, hey, everybody of us should be able, with our skills, got from the Plon world, to escape that sandbox and kill the server if we like. So, that's not the thing. So, if you do something, and there is TryJepeter.org, where you can get the server up and running with iPython Notebook, that means you have to secure that more or get things done. Django, Pyramid, and Guillotine have other security features, but they do not allow writing code through the web. Giving the people the possibility to write something. A user in a system. And even if it's just a God method or a short script, to move a file to another folder is something that empowers the users. We should be able to give them and help them. So, I would like to see others using it. But, I'm probably also a person that sees, if we have something complex, like Plon, it is not easy to recreate from scratch. Django has said it correctly. A complex system that works is invariably found to have involved from a simple system that works. A complex system designed from scratch never works and cannot be patched to make it up to work. You have to start over with a working simple system. I like the ideas that Guillotine also takes, but the problem is if you're knowing the Pareto principle, so that the first 80% takes 20% of the time and the last, the harder 20% takes 80% of the time, Guillotine has started with the easy parts of Zope and Plon. They not even have reached the hard part. And that's the problem. At the moment, it's a fantastic framework for doing REST, but all the empowerment that Plon and Zope has done to the community is not there at the moment. And that's the thing that I like. Yeah, legacy is boring. The old new stuff is shiny and makes fun doing something quick, but doing something that lasts, but doing something that enables large-scale things is hard. And getting there is something that our community has shown in the past and can show in the future. We don't should let that down. So yeah, what are the lessons learned I have done? Don't take impossible to port or fix that moment as serious. It is possible. It probably takes longer, but you can do. You should adopt modern tools and framework for your help. So Talks and Titus is a fantastic thing. I said, if possible, we should do that in the Plon community and we should probably update our best practices for Plon. And if we can use Talks, we probably can easier also so for all of our add-ons that they work together with. Different Plon versions with different Python versions, and so that's possible. For all of you who are interested in the way to Python 3, I can recommend three other Talks. So Hanoz talked more about Zope on Python 3. Mike talked about modern Python testing, and Mike has worked a lot on sub-templates and Bob-templates-plon. And that's the package where the best practices for Python and Plon packages are currently defined for the Plon community. So give it a try and hey, every revolutionary idea seems to evoke three stages of reaction. There may be some up by the praises. It's completely impossible. Don't waste my time. Oh, it's possible, but it's not worth doing. Oh, we have done it. I said it, it was a good idea at all. And that's the way the people think. So thank you. We are done. We have it on Python 3 now, so that's the way to go. Thank you. Thank you. Thank you. Thank you. Any questions? Any other questions? Yeah. Okay. Philip Bauer already had a running Plon 5.1 on the 4 on Python 3. It has some errors, so it's just about fixing all the bugs with some of the print statements and some of the code or so. So there's a few things, but it just takes time. I guess if we work on it and get more people testing it, we could have a Python 3-compatible Plon next year. Okay. Hano is giving tomorrow a talk, but Zulp 4 is running smooth already. There is some stuff on generating the new startup templates and so on. So Zulp 4 has got rid of that server as the default handler for presenting the generated code. It's now using whiskey as the default. So you need to get all the stuff on the configuration side. So probably Zulp Recype, Zetzi Recype Zulp Server needs to be adjusted for Plon, but hey, that's possible. So there is some stuff to do, but it's just the normal stuff we do on those prints, isn't it? Any other questions? I guess that's it then. Let's thank our speaker once again. Thank you. Thank you.
 So, I'm going to show you a little bit of the  application server.  It's one of the largest piece of software in the  Python world that has not been Python 3 compatible two years  ago from now.  That was a huge problem because a lot of our projects depend on  Plown and without a path to the future, it could be horrible.  So, yeah, what was the problem?  We started in Bristol.  I think it was 2014 in Bristol.  We had the first discussion about the roadmap again, how to move,  and some of the quotes was it is impossible to get Plown to Python  3 because there are too many blockers in Zope.  No one has touched restricted Python or access control and they are  from the beginning.  So, it's the really hard stuff.  But, hey, when it is time, our third C-Clock, the author of  2001 Odyssey in Space, has made some fantastic quotes about it.  When it is distinguished, what elderly scientists states that  something is possible is almost certain right.  When he states that something is impossible is very probably wrong.  And if someone tells you that's impossible, well, try to change it,  make it possible, that's about what it is.  So, why should be restricted Python be the major blocker for Plown on  Python 3?  Well, if you look at the dependency graph, all of the underlying  frameworks that we depend on, access control, document templates,  Python scripts, the Z catalog, Zope 2, everything directly depends on  restricted Python.  And a lot of the other up-way framework parts of Plown that makes  Plown Plown depends on them.  So, yeah, we need to touch restricted Python.  And, you know, every piece of Zope that was not adopted by Plown  is literally dead.  Not a lot of the Plown people touched those parts of Zope.  Restricted Python has been around till the end of the 1990s.  So, in the code I've seen, there was a Python version 0.8 something or  so that it has started on.  So, really weird shit and stuff, yeah, but hey, let's try to adopt it.  And that's the fantastic thing in the Plown community.  We have a diverse group.  Everybody has a capability of doing something.  So, we started it, or I started it.  And, well, what was the problem restricted Python has almost had  no documentation.  The test coverage on its own was extremely low.  And, you know, debugging is twice as hard as writing the code in the  first place.  Therefore, if you write the code as cleverly as possible, you are,  by definition, you're not smart enough to debug it.  Horrible stuff.  Restricted Python for that time was probably one of the most advanced  Python packages around.  It took a long time to go into it and understand it.  But, hey, the problem was created by ourselves where Zope leads Python  follows.  A lot of the Python 2 standard library are directly influenced by Zope.  There are a lot of packages that has been gone now in the Python 3 world  that are just left overs because Zope needed them.  So, there are deprecated modules like the compiler package,  which was 80% of the implementation on restricted Python,  depending on, and there was no documentation at all on the  insides of the compiler package.  Horrible.  The only way to discover the limits of the possible is to venture a little  way past them into the impossible.  And so, let's go there.  It's fun getting into the new stuff and, or the old stuff.  Archaeology is also, but, hey, what doesn't mean any sufficient advanced  technologies in this thing for magic.  So, if you're looking at the stuff we have in clone and Zope, there are so  many fantastic things.  And we have a lot of things that we don't have, or other frameworks don't  have.  So, if you look through the web, we can write scripts in Zope that could be  used by, or written by our users that are not destroying your server  completely or something or can hurt that extreme.  We have a fantastic workflow machinery.  And so, that directly depends on restricted Python to only show you the  things you're allowed to see.  So, hell, there is a lot of shit in for someone who's outside.  That's like magic.  How can they accomplish that?  And that's the thing that makes clones so special.  And that's the thing we want to keep.  So, yeah.  But the problem is what is restricted Python?  And especially what it is not.  Because if you're looking, in the early days of Python, the first Python  workshop at NIST, the SPAM 1, the first Python conference, in 1994 already  has an S, their first topic, requirements for a saved Python  interpreter. And Jim Fulton and Paul Everett were at NIST at the conference.  That was the origin where the idea for restricted Python is originated from.  So, there is the requirement for that.  We have it for almost 15 years now.  But Zelp and clone are the only ones that use that.  So, yeah.  Most of the biggest problems in software are problems of misconceptions.  So, the people don't understand the real problem or the real needs of the users.  So, that's where we need to figure out what to do.  And, well, restricted Python is not a sandbox.  That's important to know.  Sandboxes most often don't work.  You cannot escape the sandbox.  So, if you're relying on a sandbox to think, to secure your system, there's always  someone smarter than you that found a way to escape.  Security by obscurity or by putting everything into one place is problematic.  Doing security is a long process.  There are different approaches to that.  And Python has done a way that has been shown to work in another language also.  So, what restricted Python is, is a limited, safe subset of the Python programming language.  The grammar, the grammar that we love to work on.  But it is so limited that you get it to be not anymore Turing complete.  So, programming languages depends on being Turing complete to be able to do everything.  So, if you limit that, you could do it and decrease the possible things that could be done with it.  There are simple things that you would like to allow the people, but not everything.  You don't need file access or something else.  And there was another project that's have done that, that was the other project in 1995  with the Ravenscot profile file, which is a limited subset of the programming language order  that makes it restricted and logical testable so that you can show that the program,  a program in it could be run correctly.  So, yeah, there is a way.  And that's also the way to restrict the Python 4.0, the way to Python 3.  So, well, if you want to start it, first solve the problem, then write the code.  So, understanding what it is, think.  And, yeah, the first read of the complete code says it uses the compiler module and its AST class.  Well, go to docs.python.org, compiler package.  There is not, there may be 100 lines, a short overview, but no details, nothing.  It's not fully documented.  There's no described upgrade path or something like that.  Just say it's gone in Python 3.  Deplicate it.  And the other thing is, if you're looking into the code of restricted Python 3.  They do manual byte code generation.  Manual byte code generation?  Holy fuck, what are they doing there?  You have interpreter specifics.  They just let it run on C Python 2.  Oh, yeah.  It's hard.  But, well, if you think in some scientific way,  it's not documented, it's not usable.  If it's not tested, it did not work.  If it's not checked in into version control, it did not exist.  And if it's not repeatable, it's not science.  So, someone has already found a way to shield it.  So, there must be another way to do it.  If someone has already done it, it should be repeatable.  That's science.  So, yeah, it's hard.  And compiler knowledge is necessary to port it.  Hell, we are such a diverse community that we have a specialist for everything.  I have studied computer science.  I have taken compiler lessons or written one of my thesis about compilers.  Well, it probably will take time, but it's not impossible.  So, let's give it a try.  And, well, where did it start?  In Plon OpenGarden 2015.  Then we discussed it again and start looking into the code.  And, yeah, the people already decided, if that's not possible,  we need an alternative way.  So, Plon Server, which now has become GioTiner,  and some other ways to make a turnaround on that,  probably get rid of restricted Python if we can't handle it.  But, hey, if we can do it, let's do it.  And so, reading and understanding the code is the necessity in the beginning.  And, hey, any fool can write code that a computer can understand.  Good programmers write code that humans can understand.  That's all about it.  You need to get down to the Zen in Python.  Make the code readable.  Make it understandable.  So, now I have taken over the maintenance,  but I'm probably not smart enough to do it from the whole of my life  or something like that.  So, someone else, if I'm probably gone, could take over the maintenance.  So, make the code more easy, more documented, more commanded,  and everything so that the community can work with it.  So, yeah, what did I do?  I started, and the first half year I was working on porting  was just writing documentation.  What I see in the code, just annotate everything,  making the whole set, getting some feedback from some of the Zop developers  that has already worked on it, and, hey,  if you don't have any requirements or design,  programming is just the art of editing back to the empty text file.  So, it did not work that way.  So, yeah, compiler AST.  Hey, there is a new preferred way in Python 3  that already was established in Python 2.6.  That's the new AST model.  And AST could completely replace the compiler AST model.  And the compiler function itself in Python  accept AST as an input.  So, you do not need to manually generate bytecode anymore.  You just drop in the modified AST  and get it worked.  And you can get out of the compiler function the AST  and get bytecode out of it.  And that works from Python 3.6 up and Python 3.4 up  in a very, very smooth way.  The packages have already been there in the beginning of Python 3.0.  Python 3.0 implementation from 3.3 to 3.4 changed a bit.  So, we have tried to make it comparable with 3.2 and 3.3 in the beginning.  It was hard, but they are already out of support.  So, we skip that and make it easier.  Python 3.4, 3.5, 3.6, 3.7 are going like a gland.  So, that is not a problem. We can get to work on that.  And, well, the first thing that was necessary was to get access to the foundation  or the foundation repositories.  And I am glad that Minabu organized the Plones Symposium in Tokyo, 2015.  That was the first place where I met Pressiever  and get access to the repositories and everything.  So, that was necessary.  And it is sometimes the simple things that take you to travel 20,000 miles.  Yeah.  And, hey, another thing is we have done a lot of code in the old Zope way.  But we are now tending to do more and more coding conventions  to get the code readable.  And that is what the Zen of Python is about.  It is the lessons learned by the people at Zope Corporation at that time  from Zope and translated into the whole Python community.  So, beautiful is better than ugly.  Explicit is better than implicit.  Simple is better than complex.  Complex is better than complicated.  Flat is better than nested.  Sparse is better than dense.  Readability counts.  And, hey, you have to read your code.  And after two weeks, three months, six months,  looking at your own code, it could be the code of someone else.  You need to understand it again.  If you have not written it the way that you can read and understand it,  quickly again, you are lost.  So, the first thing I tried to make was applying our new coding conventions on it.  So, I did not change any of the implementation details.  I just make it following our coding conventions so that it is more readable.  The second thing is writing tests.  And I really have raised the test coverage.  I thought in the beginning it was about 18%, 18%, less than 20%.  Test coverage over the whole package.  We have raised the test coverage now to more than 95% of the complete code.  We've done it completely tested.  Every syntax statement that is in the Python language is explicitly tested.  What happens if it is allowed or disallowed in Python 3?  Or in Restricted Python.  And we do a functional test of all of that.  So, hey, in tests, the first principle is that you must not fool yourself.  You are the easiest person to fool.  Because you believe that the things you are doing is good and right.  So, get another person into the system.  Have them have a look at it.  That's the thing you need.  So, I make a decision together with two others and said we have to move.  The test set up, Restricted Python did not depend on any other external package.  It just depends on the Python standard library.  But writing tests in the Python standard unit test library is not that straightforward.  And PyTest and TOX give you a lot of easier stuff to start at.  And it's actually at the moment the defective standard in modern Python packages.  And TOX gives you the possibility to test multiple Python versions locally on your machine against the same thing.  That's fantastic.  And PyTest gives you the possibility to parametrize your tests.  Not only with input values, but also, that's the advantage of Python, you can pass a function in.  So, I do paramize the tests with the old implementation and the new implementation.  So, we had the left the old implementation, complete impact, do just another file next to it with the new implementation.  So, we can test if everything works like it should be.  If it's on the one implementation or on the other.  So, at least for Python 2, we know it is exactly the same.  And that's fantastic.  And, well, PyTest has the advantage you just can use the answer.  You don't have all the time look up, is it now as true, is it as raised with something or so?  Just easier.  I give directly after lunch another talk about modern Python testing.  If you're interested in the lessons I have learned on porting restricted Python,  what we should do and get to get better best practices on developing stuff compatible.  You can come there.  That's one of the examples of the tests we have done in restricted Python.  So, we take both implementations.  So, the new implementation, the second line is the old implementation we pass it in and just look,  execute a statement and look if we get errors, if the result is also translatable or a hand load,  we export it and see what it gets out. So, you can prove both implementations work the same.  Perfect.  So, yeah, it takes a long time.  And doing it alone is very hard.  I'm glad that during the up-and-city sprint, the people from GoSep and some other core developer joined in,  stepped up and helped through, we were able to do it.  And hey, if someone says to you, it is impossible, well, things that are impossible just take longer.  It's the fun thing. You can get it work.  That's the nice thing.  So, at the beginning, all the people were saying, Zope is that.  But we as Plon has not adopted is that.  Hey, Zope is not that. The people came up again today.  Or if you're looking at the conference, there are several people that originates from the Zope community,  are here and doing stuff.  And, yeah, we get it working during one of the Zope sprints in May 2017 to cut a first release of restricted Python 4.0.  That is Python 3.0.  We can already test that.  Zope 4, the talk from Hano tomorrow, will tell about runs completely on Python 3.  And it depends on restricted Python and access control.  And you see on the right, we now have a test coverage of 98.58% test coverage.  Python 2, 7, 3, 4, 3, 5, 3, 6, all working, all test passes.  Each time we do something.  So, yeah, we can get it done.  And how has Ice Mac Michael Hovitz from GoSep said it?  Welcome to the Python 3.0.  Thank you him for his work doing it, helping me.  And, well, for those people that are outside, or knowing some of the old stuff from Zope,  PJAB has given a fantastic quote on it.  Those who do not study Zope are condemned to reinvent it.  We do see a lot of fantastic stuff in the Python community today.  They tried to do stuff that has Zope done before.  But they are still stuck in the path where they are not can give the equivalent of Zope some way.  So, hey, on the other hand, they start and say, that couldn't be that hard.  But, hey, if you think it's simple, then you probably have misunderstood the problem.  You have to really think and look how it works.  It is a long path, but it could do.  And my wish is that we get the strength of the Python world.  We have the Zope and Plon community into other frameworks.  Get them to use the stuff we produced.  And they are adapted by more people so they can work with it.  Project Jupiter, Jupiter Notebook, which was originally iPython Notebook,  is a fantastic environment for learning and teaching on Python,  doing data science stuff in Python through the web,  but you can kill your complete server through the web.  So, you have to shield it. You have to do everything.  And they have a sandbox.  But, hey, everybody of us should be able, with our skills,  got from the Plon world, to escape that sandbox and kill the server if we like.  So, that's not the thing.  So, if you do something, and there is TryJepeter.org,  where you can get the server up and running with iPython Notebook,  that means you have to secure that more or get things done.  Django, Pyramid, and Guillotine have other security features,  but they do not allow writing code through the web.  Giving the people the possibility to write something.  A user in a system.  And even if it's just a God method or a short script,  to move a file to another folder is something that empowers the users.  We should be able to give them and help them.  So, I would like to see others using it.  But, I'm probably also a person that sees, if we have something complex,  like Plon, it is not easy to recreate from scratch.  Django has said it correctly.  A complex system that works is invariably found to have involved from a simple system that works.  A complex system designed from scratch never works and cannot be patched to make it up to work.  You have to start over with a working simple system.  I like the ideas that Guillotine also takes,  but the problem is if you're knowing the Pareto principle,  so that the first 80% takes 20% of the time and the last, the harder 20% takes 80% of the time,  Guillotine has started with the easy parts of Zope and Plon.  They not even have reached the hard part.  And that's the problem.  At the moment, it's a fantastic framework for doing REST,  but all the empowerment that Plon and Zope has done to the community is not there at the moment.  And that's the thing that I like.  Yeah, legacy is boring.  The old new stuff is shiny and makes fun doing something quick,  but doing something that lasts, but doing something that enables large-scale things is hard.  And getting there is something that our community has shown in the past and can show in the future.  We don't should let that down.  So yeah, what are the lessons learned I have done?  Don't take impossible to port or fix that moment as serious.  It is possible.  It probably takes longer, but you can do.  You should adopt modern tools and framework for your help.  So Talks and Titus is a fantastic thing.  I said, if possible, we should do that in the Plon community  and we should probably update our best practices for Plon.  And if we can use Talks, we probably can easier also so for all of our add-ons that they work together with.  Different Plon versions with different Python versions, and so that's possible.  For all of you who are interested in the way to Python 3, I can recommend three other Talks.  So Hanoz talked more about Zope on Python 3.  Mike talked about modern Python testing, and Mike has worked a lot on sub-templates and Bob-templates-plon.  And that's the package where the best practices for Python and Plon packages are currently defined for the Plon community.  So give it a try and hey, every revolutionary idea seems to evoke three stages of reaction.  There may be some up by the praises.  It's completely impossible. Don't waste my time.  Oh, it's possible, but it's not worth doing.  Oh, we have done it.  I said it, it was a good idea at all.  And that's the way the people think.  So thank you.  We are done.  We have it on Python 3 now, so that's the way to go.  Thank you.  Thank you.  Thank you.  Thank you.  Any questions?  Any other questions?  Yeah.  Okay.  Philip Bauer already had a running Plon 5.1 on the 4 on Python 3.  It has some errors, so it's just about fixing all the bugs with some of the print statements and some of the code or so.  So there's a few things, but it just takes time.  I guess if we work on it and get more people testing it, we could have a Python 3-compatible Plon next year.  Okay.  Hano is giving tomorrow a talk, but Zulp 4 is running smooth already.  There is some stuff on generating the new startup templates and so on.  So Zulp 4 has got rid of that server as the default handler for presenting the generated code.  It's now using whiskey as the default.  So you need to get all the stuff on the configuration side.  So probably Zulp Recype, Zetzi Recype Zulp Server needs to be adjusted for Plon, but hey, that's possible.  So there is some stuff to do, but it's just the normal stuff we do on those prints, isn't it?  Any other questions?  I guess that's it then.  Let's thank our speaker once again.  Thank you.  Thank you.
Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_0_131.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_132_251.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_252_983.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_984_1931.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_1932_2843.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_2844_3827.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_3828_5171.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_5172_6359.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_6360_7427.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_7428_7739.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_7740_8207.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_8208_9419.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_9420_9467.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_9468_9563.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_9564_9647.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_9648_10871.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_10872_11243.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_11244_12371.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_12372_14771.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_14772_14975.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_14976_15251.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_15252_16967.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_16968_18503.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_18504_19379.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_19380_20651.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_20652_21539.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_21540_23867.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_23868_24683.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_24684_26843.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_26844_28499.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_28500_31955.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_31956_32519.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_32520_32567.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_32568_33107.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_33108_33635.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_33636_34199.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_34200_35339.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_35340_36431.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_36432_37163.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_37164_37859.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_37860_40583.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_40584_43415.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_43416_44699.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_44700_45395.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_45396_46139.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_46140_46199.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_46200_46583.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_46584_46643.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_46644_50339.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_50340_50519.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_50520_50591.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_50592_50639.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_50640_51047.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_51048_51095.png Day_2__Alexander_Loechel_-_Doing_the_impossible_EFFX3MmMFJQ_51096_51241.png