Practical usage of advanced Python constructs
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Syschikov, Andrey
Andrey Syschikov - Practical usage of advanced Python constructs Python is a language of choice for developers with wide range of experience, for some it is a first programming language, others switch to Python after years of experience. Python provides friendly syntax and smooth learning curve. This sometimes leads to developers lacking comprehension of some more advanced constructs. It happens that experienced developers jump into using Python and sometimes miss less known Python language constructs. On the other hands people who purposefully learned Python sometimes lack practical ideas for how to apply those constructs. This talk will be specifically focused on the practical usages of advanced Python constructs like iterators, generators, decorators and context managers. Goal of the talk is to share ideas about how those constructs can be used for practical purposes in real projects. Prior knowledge is not required, there will be a brief introduction to every construct being presented.
 So, I would assume that we hear all of Python or maybe your company offered to pay for the trip and you couldn't say no. We love Python because of its expression power and nice learning curve and its coding style. The fact that many language features look like native languages, the general straightforwardness of Python and the lack of hidden tricks makes it appealing both as the first language for people who just started programming and as a new language for experienced developers. Practice shows that you can quickly start doing useful things in Python if you are already an experienced programmer. I used to program for Microsoft.net and C-Sharp for five years before I got into my first Python project. I didn't really know that much about Python. I didn't have any formal training in Python or read any book in Python. My approach was purely practical. I was only interested in getting my task done and not really getting too deep into theories. That was three years ago. Since that, I learned a lot about Python. But what I noticed is that many tutorials, books, blog posts that feature advanced concepts, they usually use really impractical examples. Like for example, how to generate Fibonacci numbers. Who really generates Fibonacci numbers in production? That's why I decided to make this talk and provide some practical examples on usage of advanced language constructs in Python. So I'm not really here today to teach you but more to provide interesting ideas, examples, to inspire you to practically use those constructs. So quick check. Please raise your hands. Those who use yield keyword in production code within last year. Wow. I'm surprised. A lot of people. Good. I'm going to introduce himself as a decorator in last year in production. Okay. Great. Who wrote context manager within last year? Okay. So it's good. You're more familiar with the theory so I can not really focus too much on it and jump to examples. So for my examples in this talk, I used code from my project that I worked on and I collected the code from different open source libraries. So what is iterable in Python? All the definitions I took from official glossary. So well, it used a pretty vague abstract definitions. So what it practically is a terrible. It's a terrible object that has an iterable method. Where can it be used in Python? It can be used in for statement in the loop. It can be used in list comprehensions or in generator expressions. Or you can pass it to functions that expect iterable like all any of some filter or whatever. And I would like to draw your attention to turtles for those who don't know. It provides a lot of very useful functionality to work with iterables and iterators. So I said that iterable returns iterator. So what's iterator? It's a writer. Basically the whole functionality of iterator is to keep the current state of iteration like for example if you were iterating over list is to remember the current element and that's being returned. And basically only thing it has to do it has to provide a next method which basically produces a new value on every iteration or raises stop iteration exception if the iterator exhausts it. Well, how it can be created? You can create a class implement method next and instantiate it. I would like to point out this completely impractical way and I won't be really talking about it. Then generator expression. Generator expression looks very much like list comprehensions except that it uses around brackets. It's an important difference. Basically in list comprehensions you immediately force the creation of the whole list and production of the whole values. So it takes memory and it's not very actually useful and well. So my strong opinion is that for most of the time you should prefer generator expressions over list comprehensions because it saves memory and sometimes you don't even need the whole list of the values. Then generator functions. So generators. Generators is a function that has a yield keyword in it. Well, many of you raise hands so I won't go too deep. So this is the most practical way to create a complex iterators in Python. It was introduced in the same version of Python as iterators because yeah, author of Python immediately understood that creating classes for iteration is not very convenient. You can read the discussion about it in PEP 255. It grew into something way larger into coroutines. This is outside of the scope. So quick glance how it works. So if function has a yield keyword in it, it's marked as a generator. When you call it, it doesn't actually execute a function. It just creates iterator object. When you iterate over it, the control flow will actually step into function and go to the first yield, then return it to the loop outside, and then stop until next iteration. So let's quickly run through it. So this how it will be executed. It will print yield. It will return. But on next iteration, it will start from the next line and go until next yield or end the function. So this will print the zero to. So what's the difference between the regular function is that it remembers its states between the calls. So what can it be practically used for? Because many people, when they study the subject, they don't really know, well, okay, this seems cool, but what can I do? One of the very common usage of generators is basically to create a view over a collection, do some filtering and mapping. So this is the code from a project I worked on. We had some list of fields, some of them were considered dynamics, started with some prefix, and basically we needed to get a list of the dynamic field names in different places. So we wrote this simple generator. What's good about it? First, you don't need to copy paste this for and if everywhere you need the list. Second, it looks clean. It's easy to read. Well, it's multi-line list comprehension, so generator expressions are, well, ugly. So this is the, well, this one's some filtering and then we want some mapping. So another very common way to write generators is to flatten lists. So this code comes from Django framework. So flattening lists, if you have a list of lists and you want to iterate over all sub-elements as if it was a single stream of values. So it's basically usually done with nested force, and so they created this iterator so they don't need to copy paste this to force everywhere. It's actually a good and bad example because iterator module provides a much easier way to flatten lists with iterables. That's how it can be done in one line. So it's a very popular question on Stack or Flow, so I would assume that, well, yeah, people have difficulties with it. But there can be more complicated flattening list logic. This code comes from Django, or from JINJA. It's a templating engine often used with Plask framework. So they have a list of items, and item can't be a node or a list. So if it's not, we return it, we yield it. If it's a list, we go into sub-loop. So this is, here you can't use really iter tools. So this is an example of more complex flattening. So what it can be also good for generators is to save memory. So this code comes from request library, is so-called HTTP library for humans, whatever it means. But it's very convenient library to do HTTP requests. So somewhere in the depths of it, they have a socket object or I know some sort of wrapper, self.roll, that can, that, well, it's a network stream. It's a sequence of bytes. But responses from servers can be fairly large. We don't really want to load them in memory completely. So they wrote the iterator that breaks it into chunks, and you can iterate over those chunks without actually loading the whole content of the response into memory. That's very convenient because it saves memory. And it's convenient basically because users of this function may not really need a whole response in memory. It can be used to directly write those chunks into file. So you can save it locally, and it's good for memory and performance. So it's a little bit more complicated. So this is their internal usage of this chunks iterator. So what's they doing here? So it's iterator that takes iterator of chunks and produces iterator of strings. So chunks have fixed lengths, but strings can be arbitrary lengths, and they're divided by delimiter. So what they do here, they break a chunk into strings. But it can happen that the remainder of chunk is incomplete line, and we need to get another chunk to complete it. So here they introduce a state, a local variable, pending, which carries over, a leftover of chunk that is uncompleted string. They keep it between yields, and they add it to chunk so that they can, they remember it between calls so that they can connect it with the rest of the string and return it. So if there are no more chunks, then they assume that pending is complete string and return it. It's a convenient example how also you can save memory and create an iterator over iterator with the generators. So generators are also convenient to traverse complex data structures. So this is example of first code of standard OS module, the function walk. I simplified it significantly. Well, it's like 200 lines of code, but this is the core of it. So it uses recursive generators. So it's very convenient in this case because file system is tree data structure. So walk returns a tuple of current path, the list of deers in this path and the list of files. Then it goes over the list of directories in current path and calls itself basically and yields the results. So in the end, if we iterate over walk generator, we will get a flat list of hierarchical data. That's very convenient. Also iterators can be infinite. So some people may wondering why would you need an infinite iterator? You can't even iterate over it. You can't do four over it because it will eat all your memory and burn CPU. This is example. This is Django templating language for those who are unfamiliar. It just basically copies the four tag, copies its contents. It's here, it generates a rows in a table. So we would like to often, you know, in web, we have tables. The rows come in different colors, one after another. So Django has this cycle tag which actually will produce on every call row one, row two, row one, row two. Internally it's implemented with the iter tools cycle iterator, which is infinite iterator in the lower part. In the lower part you can see, well, it basically takes it terrible and then repeats it one by one, items one by one forever. So every time this tag is called, it calls render method and it just calls next item on the iterator. So it's not a problem that the iterator here is infinite. It will never try to iterate over it completely. It will call it exactly as many times as four iterates. So this is very nice and clean code. You don't need to maintain any state like which value was returned previously. You just call next on cycle. So again, I advertise you iter tools module. So those are the most practical examples for using. But I recommend you to research deeper because iterators, generators is very complex topic in Python. It has way more advanced usage. So I recommend you to look more into iter tools, iter tools module and master it because it can improve your productivity. Then actually yield is not the statement. It is expression. So it can be used as a mechanism to pass a values from color to generator. This can later led Python to even more advanced stuff. So then I recommend you to read about yield from its so-called generation delegations. There those two features are used to implement coroutines in Python, which is also very interesting. And based on coroutines, there is a new, relatively new module, the whole concept in Python called Async or X project. It's for asynchronous input output. They are heavily based on coroutines, generators and iterators. Actually don't need to photo the slides. We'll put them online. So yeah, you can, don't worry if you miss something, you can check it out later. So another advanced feature that's often not used in practice by, well, maybe starter developers, it's decorators. So again, many of you raise hands. So I would assume that many of you know how decorator works. Basically decorator wraps your function with some inner function. What it basically all it does, it assigns to your function the decorated, the wrapped version of your function. What is it good for practically? With a decorator, you can modify input arguments. You can modify return value. Well you can do things before functions called, after functions called. You can actually not call the function at all. So your decorator may decide not to call function for some reason. You can modify some global states, some outer variable, some thread locals and stuff like this for this function and then set them back to the values that were before that. It can use, decorators can be useful to assign metadata to functions of all sorts. So this is example from Flask web framework. They heavily use decorators. So they demonstrate two things. Here it is, hello is a web view. It's basically a function that is called when you call your web server slash hello or slash hello name. So they do here, they use decorators to parse URL into function arguments. And another usage of decorator here is that it basically makes it possible to discover the function. So the framework knows which functions are web views. So in this sense it provides a metadata for your application. This is the small snippet from my current project. So we have this. We want to make extendable list of, well, some filters, whatever that means. And we would like to show them on UI so users can select from a list. So we need some sort of label for those fields. So I created this decorator that provides a human readable description that is used on UI. Also, I can query all the functions in the modules and check if they have this decorator. So it's also good for discovery and for providing metadata that can be used, for example, in UI later. So color or not. So with a decorator you can make a decorator that decides maybe actually we shouldn't call the underlying function at all. This is again simplified example from Django. They have a decorator called permission required. So you can apply this decorator on a web view function and check if the currently logged user has certain permission to call this web view. So what we can see here, they check if the current user has this permission. If user has this permission, they call the actual function. If user doesn't have permission, they don't even call the function. So that's how you can check whether you should actually call function or not or raise error or whatever. This is decorator rate limit. We created it in one of my projects. Basically it's also for web views. It counts in a cache how often the page is called by a certain IP address. And for example, if it's called more than 10 times in a minute, we will not generate the response. We'll show an error. It's very simple mechanism from prevent of abusing of service. So if the user calls it too often, we can prevent call completely. Caching is very big feature that can be done with decorators. So this is in our project, we created really simple and I have to confess a little bit stupid caching decorator that we apply on properties. Exclusively, that's important point. So we have this numpersons and it counts, it's a Django ORM. So basically just counts related objects. So this function is called, this property is called a lot in reports. So we have reports that call this function 20 times. And we can just drop this decorator cache result here and it can very conveniently prevent too many of the calls. So how we implemented it? It's also very straightforward. So you can see here we get the func name. That will be numpersons in this case. So if the object has this attribute, we do get attribute. If it's not in the object, we actually call the property, calculate it and then set corresponding attribute and the subsequent calls to this property will return the cached value. So what's bad about it? It's basically that if save underlying object or data changes, well, there is no way to enforce recalculation. But this is good for simple cases. Like for example, if you're sure that your object won't change. So the proper way to cache with decorators. There is this awesome, I really like this, library called doc.pl. It actually creates proper and very sophisticated way to cache functions and method everything. So the key decorator is cache arguments. You can provide different backends, memcache, local memory, anything. And they will actually cache the functions for corresponding arguments as well. So the cache key will include the function arguments. It's very smart. You can refresh values and everything. I suggest everyone to look into this because that's probably the most sophisticated caching library with the decorators I've seen so far. My personal award for most creative user of decorators goes to Ansible. Ansible is a very powerful tool for IT automation. You can use it for deploying your code and servers, manage your infrastructure. I also recommend you to look into it. It's very powerful. But inside of it, they have a very interesting decorator called timeout. So you can just apply to function. So it does, if the function takes longer than, well, by default, 10 seconds to execute, they stop the function execution and raise an error. How they implemented it? So they implemented it using unique signals. So before the function is called, they set alarm with a callback. Then they set the alarm to seconds. So what happens? How does it work? This kernel feature of Linux, well, any other POSIX compatible OS, it will set this alarm internally and when this alarm goes off, it will call the handler method. And handler method will rise timeout error and this timeout error will appear within a function and it will exit the function immediately. So if it takes longer, it just stops the function and exits. So this is a pretty nice way to do a function by timeout without using threads. Obviously, this will not work on Windows. Sorry. This is mostly for unique operating systems. Digging deeper. So decorators can be way more complex than examples that I showed. There was a previous talk today, unfortunately I missed it, but they covered, I hope maybe someone of you were there, in the program, they wrote that they will cover advanced cases of decorators. So in decorators, you can decorate classes and decorators itself can be class that maintains the state. So you can create way more complex things. But I would say in for more practical cases, the simple decorators are enough. But I recommend you to research more into this. So context managers. Context manager, this is how we call with statement. It is just a very simple thing. It's basically it calls enter on a context manager object and returns a value and signs to variable. Then it calls the actual code that is within the statement and finally it calls exit. So what it can be good for, it's good for if you want deterministic release of unmanaged resources, for example, files, everyone must use with files in Python. Because it will close the file immediately after you stop using it. It's also good to modify some global states. For example, you can select a piece of code that will be executed within transactions or you can set some setting just for this piece of code and then restore the settings. And it can be used a lot of interesting tricks for logging, for more advanced logging and debugging. So a couple of examples, Django. So they have session transaction. So on exit, on exit, basically they check if the current scope was within a transaction. If yes, then we need to commit the transaction or, well, on error, roll back. So that will put it in a transaction. The important here is that finally it's called always. So whatever there is error in transactions or not, it will call the exit method always. So for example, in request library, they have a session, context manager, when you exit it, it closes the session. It's also good because you should close it as fast as you stop using it and not when garbage collector picks it and calls it. This is a small context manager that I wrote for debugging. Basically when it enters, it stores internally the time when you entered into a function, that function of the piece of code and when you exit, it just prints the amount of milliseconds it took to execute your piece of code. This is very simple and very easy to do micro-profiling small chunks of codes without doing too much calling profiler and all this stuff. So digging deeper. Again, I of course didn't cover all the uses of context managers. So Django does a lot of stuff with context managers, especially when it comes to transactions. They also have, it's both decorator and context manager because it wraps some piece of code. So also you can implement with decorators and context managers, you can implement database locality. So you can select, for example, let's say that this function or this piece of code, depending if it's decorator or context manager, all the queries will be executed against slave database. It's, for example, useful for reporting. You have a function generating report. You can create a context manager saying, okay, use slave database for all queries in this. Also more sophisticated log and debug, not just stupid milliseconds, which is actually very, has very low precision. And yeah, you can do some more sophisticated profiling. So that's it. Thank you very much. This is my email. You can send me suggestions, questions, whatever. I registered that stupid domain for my personal blog. I'm going to put the slides and there, and I started a series of posts, which basically I cover the same stuff, but in more details with better explanation. So if you miss something, you can come there and yeah, all the content from the slides will be there within the blog. So thank you very much. Do we have time for questions? Okay. One question, did you already see this LRU cache module in the standard library? That's kind of doing the same thing as one of your examples. It just caches the result code, the result value for specific parameters sets. So maybe you can also use that. One question, did you already discover a good pattern to handle open files? Sorry, it's covered what? To handle open files, if the context is not as local as in sometimes with, so you basically need to give around open file handle and still you need to make sure that it is closed at some time. Is there a good pattern for that? Well, yeah, well, with the context, well, answering the first question, yes, there is a lot of this caching, there are a lot of caching done with the decays in very different libraries, but I recommend to check out DocPile because as I said, it's the most advanced one. It has the richest functionality. Coming to the next one, well, if you want to use file in different place, well, then it's a little bit more complicated than, yeah, you can use it. Well, context managers are good for small chunks of code. So if you have few lines, but yeah, you can write your own class that will store a file handler and once this, and you can call close on your own class when you finish doing with this. So as far as I know, there is no straightforward thing to do with the context managers. Hi, hi, is the order of the decorator important? If I have, for instance, three decorators, sorry, is it going to grab the following decorator? Yeah, this is, you're right, this is a tricky part. Yes, the order of decorators affects it. So some decorators, for some decorators, it's important. The order is important in which you do it, but for some it's not. So yeah, order matters in general case and more specific. So you can write the decorators that, for example, I will give you a very small example specific where was my wonderful cache. So here it's important a property should come before a cache result. Otherwise if you do it the other way, it won't work, it just won't work properly. But because the property is a special, it's a building decorator. But for simple decorator, you can write the logic in a way that it wouldn't matter. So it really depends on, yeah, particularly the decorators. But evaluate the full amount, right? I think it actually, no, it actually, first it will call cache result decorator and only then it will call property. So that's why it's important that property will be like outermost. Yeah, yeah. This is a tricky part here. Any other questions? Comments? We still have a few minutes. Hello, thank you for the talk. Just one observation about the wrap decorator that the standard library offers. So I saw that in one example that's present and in the others when you are writing a wrapper function you are not using that. So I think it's worth mentioning that it is better to use because it copies the doc string and functions name and those things. Yeah, you're absolutely right. You're absolutely right. And practically you should use wraps from Funcutils module. That's absolutely right because it will keep the function name and doc string and all this stuff. But it was just like too much for the talk. But you're right. Thanks for the note. Can we push for one more? Questions, comments? Or if not, let's thank the speakers again. Thank you. Thank you.
 So, I would assume that we hear all of Python or maybe your company offered to pay for the  trip and you couldn't say no.  We love Python because of its expression power and nice learning curve and its coding style.  The fact that many language features look like native languages, the general straightforwardness  of Python and the lack of hidden tricks makes it appealing both as the first language for  people who just started programming and as a new language for experienced developers.  Practice shows that you can quickly start doing useful things in Python if you are already  an experienced programmer.  I used to program for Microsoft.net and C-Sharp for five years before I got into my first  Python project.  I didn't really know that much about Python.  I didn't have any formal training in Python or read any book in Python.  My approach was purely practical.  I was only interested in getting my task done and not really getting too deep into theories.  That was three years ago.  Since that, I learned a lot about Python.  But what I noticed is that many tutorials, books, blog posts that feature advanced concepts,  they usually use really impractical examples.  Like for example, how to generate Fibonacci numbers.  Who really generates Fibonacci numbers in production?  That's why I decided to make this talk and provide some practical examples on usage of  advanced language constructs in Python.  So I'm not really here today to teach you but more to provide interesting ideas, examples,  to inspire you to practically use those constructs.  So quick check.  Please raise your hands.  Those who use yield keyword in production code within last year.  Wow.  I'm surprised.  A lot of people.  Good.  I'm going to introduce himself as a decorator in last year in production.  Okay.  Great.  Who wrote context manager within last year?  Okay.  So it's good.  You're more familiar with the theory so I can not really focus too much on it and jump  to examples.  So for my examples in this talk, I used code from my project that I worked on and I collected  the code from different open source libraries.  So what is iterable in Python?  All the definitions I took from official glossary.  So well, it used a pretty vague abstract definitions.  So what it practically is a terrible.  It's a terrible object that has an iterable method.  Where can it be used in Python?  It can be used in for statement in the loop.  It can be used in list comprehensions or in generator expressions.  Or you can pass it to functions that expect iterable like all any of some filter or whatever.  And I would like to draw your attention to turtles for those who don't know.  It provides a lot of very useful functionality to work with iterables and iterators.  So I said that iterable returns iterator.  So what's iterator?  It's a writer.  Basically the whole functionality of iterator is to keep the current state of iteration  like for example if you were iterating over list is to remember the current element and  that's being returned.  And basically only thing it has to do it has to provide a next method which basically produces  a new value on every iteration or raises stop iteration exception if the iterator exhausts  it.  Well, how it can be created?  You can create a class implement method next and instantiate it.  I would like to point out this completely impractical way and I won't be really talking  about it.  Then generator expression.  Generator expression looks very much like list comprehensions except that it uses around  brackets.  It's an important difference.  Basically in list comprehensions you immediately force the creation of the whole list and production  of the whole values.  So it takes memory and it's not very actually useful and well.  So my strong opinion is that for most of the time you should prefer generator expressions  over list comprehensions because it saves memory and sometimes you don't even need the  whole list of the values.  Then generator functions.  So generators.  Generators is a function that has a yield keyword in it.  Well, many of you raise hands so I won't go too deep.  So this is the most practical way to create a complex iterators in Python.  It was introduced in the same version of Python as iterators because yeah, author of Python  immediately understood that creating classes for iteration is not very convenient.  You can read the discussion about it in PEP 255.  It grew into something way larger into coroutines.  This is outside of the scope.  So quick glance how it works.  So if function has a yield keyword in it, it's marked as a generator.  When you call it, it doesn't actually execute a function.  It just creates iterator object.  When you iterate over it, the control flow will actually step into function and go to  the first yield, then return it to the loop outside, and then stop until next iteration.  So let's quickly run through it.  So this how it will be executed.  It will print yield.  It will return.  But on next iteration, it will start from the next line and go until next yield or end  the function.  So this will print the zero to.  So what's the difference between the regular function is that it remembers its states between  the calls.  So what can it be practically used for?  Because many people, when they study the subject, they don't really know, well, okay, this seems  cool, but what can I do?  One of the very common usage of generators is basically to create a view over a collection,  do some filtering and mapping.  So this is the code from a project I worked on.  We had some list of fields, some of them were considered dynamics, started with some  prefix, and basically we needed to get a list of the dynamic field names in different places.  So we wrote this simple generator.  What's good about it?  First, you don't need to copy paste this for and if everywhere you need the list.  Second, it looks clean.  It's easy to read.  Well, it's multi-line list comprehension, so generator expressions are, well, ugly.  So this is the, well, this one's some filtering and then we want some mapping.  So another very common way to write generators is to flatten lists.  So this code comes from Django framework.  So flattening lists, if you have a list of lists and you want to iterate over all sub-elements  as if it was a single stream of values.  So it's basically usually done with nested force, and so they created this iterator so  they don't need to copy paste this to force everywhere.  It's actually a good and bad example because iterator module provides a much easier way  to flatten lists with iterables.  That's how it can be done in one line.  So it's a very popular question on Stack or Flow, so I would assume that, well, yeah,  people have difficulties with it.  But there can be more complicated flattening list logic.  This code comes from Django, or from JINJA.  It's a templating engine often used with Plask framework.  So they have a list of items, and item can't be a node or a list.  So if it's not, we return it, we yield it.  If it's a list, we go into sub-loop.  So this is, here you can't use really iter tools.  So this is an example of more complex flattening.  So what it can be also good for generators is to save memory.  So this code comes from request library, is so-called HTTP library for humans, whatever  it means.  But it's very convenient library to do HTTP requests.  So somewhere in the depths of it, they have a socket object or I know some sort of wrapper,  self.roll, that can, that, well, it's a network stream.  It's a sequence of bytes.  But responses from servers can be fairly large.  We don't really want to load them in memory completely.  So they wrote the iterator that breaks it into chunks, and you can iterate over those  chunks without actually loading the whole content of the response into memory.  That's very convenient because it saves memory.  And it's convenient basically because users of this function may not really need a whole  response in memory.  It can be used to directly write those chunks into file.  So you can save it locally, and it's good for memory and performance.  So it's a little bit more complicated.  So this is their internal usage of this chunks iterator.  So what's they doing here?  So it's iterator that takes iterator of chunks and produces iterator of strings.  So chunks have fixed lengths, but strings can be arbitrary lengths, and they're divided  by delimiter.  So what they do here, they break a chunk into strings.  But it can happen that the remainder of chunk is incomplete line, and we need to get another  chunk to complete it.  So here they introduce a state, a local variable, pending, which carries over, a leftover of  chunk that is uncompleted string.  They keep it between yields, and they add it to chunk so that they can, they remember  it between calls so that they can connect it with the rest of the string and return  it.  So if there are no more chunks, then they assume that pending is complete string and  return it.  It's a convenient example how also you can save memory and create an iterator over iterator  with the generators.  So generators are also convenient to traverse complex data structures.  So this is example of first code of standard OS module, the function walk.  I simplified it significantly.  Well, it's like 200 lines of code, but this is the core of it.  So it uses recursive generators.  So it's very convenient in this case because file system is tree data structure.  So walk returns a tuple of current path, the list of deers in this path and the list of  files.  Then it goes over the list of directories in current path and calls itself basically  and yields the results.  So in the end, if we iterate over walk generator, we will get a flat list of hierarchical data.  That's very convenient.  Also iterators can be infinite.  So some people may wondering why would you need an infinite iterator?  You can't even iterate over it.  You can't do four over it because it will eat all your memory and burn CPU.  This is example.  This is Django templating language for those who are unfamiliar.  It just basically copies the four tag, copies its contents.  It's here, it generates a rows in a table.  So we would like to often, you know, in web, we have tables.  The rows come in different colors, one after another.  So Django has this cycle tag which actually will produce on every call row one, row two,  row one, row two.  Internally it's implemented with the iter tools cycle iterator, which is infinite iterator  in the lower part.  In the lower part you can see, well, it basically takes it terrible and then repeats it one  by one, items one by one forever.  So every time this tag is called, it calls render method and it just calls next item  on the iterator.  So it's not a problem that the iterator here is infinite.  It will never try to iterate over it completely.  It will call it exactly as many times as four iterates.  So this is very nice and clean code.  You don't need to maintain any state like which value was returned previously.  You just call next on cycle.  So again, I advertise you iter tools module.  So those are the most practical examples for using.  But I recommend you to research deeper because iterators, generators is very complex topic  in Python.  It has way more advanced usage.  So I recommend you to look more into iter tools, iter tools module and master it because  it can improve your productivity.  Then actually yield is not the statement.  It is expression.  So it can be used as a mechanism to pass a values from color to generator.  This can later led Python to even more advanced stuff.  So then I recommend you to read about yield from its so-called generation delegations.  There those two features are used to implement coroutines in Python, which is also very interesting.  And based on coroutines, there is a new, relatively new module, the whole concept in Python called  Async or X project.  It's for asynchronous input output.  They are heavily based on coroutines, generators and iterators.  Actually don't need to photo the slides.  We'll put them online.  So yeah, you can, don't worry if you miss something, you can check it out later.  So another advanced feature that's often not used in practice by, well, maybe starter  developers, it's decorators.  So again, many of you raise hands.  So I would assume that many of you know how decorator works.  Basically decorator wraps your function with some inner function.  What it basically all it does, it assigns to your function the decorated, the wrapped  version of your function.  What is it good for practically?  With a decorator, you can modify input arguments.  You can modify return value.  Well you can do things before functions called, after functions called.  You can actually not call the function at all.  So your decorator may decide not to call function for some reason.  You can modify some global states, some outer variable, some thread locals and stuff like  this for this function and then set them back to the values that were before that.  It can use, decorators can be useful to assign metadata to functions of all sorts.  So this is example from Flask web framework.  They heavily use decorators.  So they demonstrate two things.  Here it is, hello is a web view.  It's basically a function that is called when you call your web server slash hello or slash  hello name.  So they do here, they use decorators to parse URL into function arguments.  And another usage of decorator here is that it basically makes it possible to discover  the function.  So the framework knows which functions are web views.  So in this sense it provides a metadata for your application.  This is the small snippet from my current project.  So we have this.  We want to make extendable list of, well, some filters, whatever that means.  And we would like to show them on UI so users can select from a list.  So we need some sort of label for those fields.  So I created this decorator that provides a human readable description that is used on  UI.  Also, I can query all the functions in the modules and check if they have this decorator.  So it's also good for discovery and for providing metadata that can be used, for example, in  UI later.  So color or not.  So with a decorator you can make a decorator that decides maybe actually we shouldn't call  the underlying function at all.  This is again simplified example from Django.  They have a decorator called permission required.  So you can apply this decorator on a web view function and check if the currently logged  user has certain permission to call this web view.  So what we can see here, they check if the current user has this permission.  If user has this permission, they call the actual function.  If user doesn't have permission, they don't even call the function.  So that's how you can check whether you should actually call function or not or raise error  or whatever.  This is decorator rate limit.  We created it in one of my projects.  Basically it's also for web views.  It counts in a cache how often the page is called by a certain IP address.  And for example, if it's called more than 10 times in a minute, we will not generate  the response.  We'll show an error.  It's very simple mechanism from prevent of abusing of service.  So if the user calls it too often, we can prevent call completely.  Caching is very big feature that can be done with decorators.  So this is in our project, we created really simple and I have to confess a little bit  stupid caching decorator that we apply on properties.  Exclusively, that's important point.  So we have this numpersons and it counts, it's a Django ORM.  So basically just counts related objects.  So this function is called, this property is called a lot in reports.  So we have reports that call this function 20 times.  And we can just drop this decorator cache result here and it can very conveniently prevent  too many of the calls.  So how we implemented it?  It's also very straightforward.  So you can see here we get the func name.  That will be numpersons in this case.  So if the object has this attribute, we do get attribute.  If it's not in the object, we actually call the property, calculate it and then set corresponding  attribute and the subsequent calls to this property will return the cached value.  So what's bad about it?  It's basically that if save underlying object or data changes, well, there is no way to  enforce recalculation.  But this is good for simple cases.  Like for example, if you're sure that your object won't change.  So the proper way to cache with decorators.  There is this awesome, I really like this, library called doc.pl.  It actually creates proper and very sophisticated way to cache functions and method everything.  So the key decorator is cache arguments.  You can provide different backends, memcache, local memory, anything.  And they will actually cache the functions for corresponding arguments as well.  So the cache key will include the function arguments.  It's very smart.  You can refresh values and everything.  I suggest everyone to look into this because that's probably the most sophisticated caching  library with the decorators I've seen so far.  My personal award for most creative user of decorators goes to Ansible.  Ansible is a very powerful tool for IT automation.  You can use it for deploying your code and servers, manage your infrastructure.  I also recommend you to look into it.  It's very powerful.  But inside of it, they have a very interesting decorator called timeout.  So you can just apply to function.  So it does, if the function takes longer than, well, by default, 10 seconds to execute, they  stop the function execution and raise an error.  How they implemented it?  So they implemented it using unique signals.  So before the function is called, they set alarm with a callback.  Then they set the alarm to seconds.  So what happens?  How does it work?  This kernel feature of Linux, well, any other POSIX compatible OS, it will set this alarm  internally and when this alarm goes off, it will call the handler method.  And handler method will rise timeout error and this timeout error will appear within a function  and it will exit the function immediately.  So if it takes longer, it just stops the function and exits.  So this is a pretty nice way to do a function by timeout without using threads.  Obviously, this will not work on Windows.  Sorry.  This is mostly for unique operating systems.  Digging deeper.  So decorators can be way more complex than examples that I showed.  There was a previous talk today, unfortunately I missed it, but they covered, I hope maybe  someone of you were there, in the program, they wrote that they will cover advanced cases  of decorators.  So in decorators, you can decorate classes and decorators itself can be class that maintains  the state.  So you can create way more complex things.  But I would say in for more practical cases, the simple decorators are enough.  But I recommend you to research more into this.  So context managers.  Context manager, this is how we call with statement.  It is just a very simple thing.  It's basically it calls enter on a context manager object and returns a value and signs  to variable.  Then it calls the actual code that is within the statement and finally it calls exit.  So what it can be good for, it's good for if you want deterministic release of unmanaged  resources, for example, files, everyone must use with files in Python.  Because it will close the file immediately after you stop using it.  It's also good to modify some global states.  For example, you can select a piece of code that will be executed within transactions  or you can set some setting just for this piece of code and then restore the settings.  And it can be used a lot of interesting tricks for logging, for more advanced logging and  debugging.  So a couple of examples, Django.  So they have session transaction.  So on exit, on exit, basically they check if the current scope was within a transaction.  If yes, then we need to commit the transaction or, well, on error, roll back.  So that will put it in a transaction.  The important here is that finally it's called always.  So whatever there is error in transactions or not, it will call the exit method always.  So for example, in request library, they have a session, context manager, when you exit  it, it closes the session.  It's also good because you should close it as fast as you stop using it and not when  garbage collector picks it and calls it.  This is a small context manager that I wrote for debugging.  Basically when it enters, it stores internally the time when you entered into a function,  that function of the piece of code and when you exit, it just prints the amount of milliseconds  it took to execute your piece of code.  This is very simple and very easy to do micro-profiling small chunks of codes without doing too much  calling profiler and all this stuff.  So digging deeper.  Again, I of course didn't cover all the uses of context managers.  So Django does a lot of stuff with context managers, especially when it comes to transactions.  They also have, it's both decorator and context manager because it wraps some piece of code.  So also you can implement with decorators and context managers, you can implement database  locality.  So you can select, for example, let's say that this function or this piece of code, depending  if it's decorator or context manager, all the queries will be executed against slave  database.  It's, for example, useful for reporting.  You have a function generating report.  You can create a context manager saying, okay, use slave database for all queries in this.  Also more sophisticated log and debug, not just stupid milliseconds, which is actually  very, has very low precision.  And yeah, you can do some more sophisticated profiling.  So that's it.  Thank you very much.  This is my email.  You can send me suggestions, questions, whatever.  I registered that stupid domain for my personal blog.  I'm going to put the slides and there, and I started a series of posts, which basically  I cover the same stuff, but in more details with better explanation.  So if you miss something, you can come there and yeah, all the content from the slides  will be there within the blog.  So thank you very much.  Do we have time for questions?  Okay.  One question, did you already see this LRU cache module in the standard library?  That's kind of doing the same thing as one of your examples.  It just caches the result code, the result value for specific parameters sets.  So maybe you can also use that.  One question, did you already discover a good pattern to handle open files?  Sorry, it's covered what?  To handle open files, if the context is not as local as in sometimes with, so you basically  need to give around open file handle and still you need to make sure that it is closed at  some time.  Is there a good pattern for that?  Well, yeah, well, with the context, well, answering the first question, yes, there is  a lot of this caching, there are a lot of caching done with the decays in very different  libraries, but I recommend to check out DocPile because as I said, it's the most advanced  one.  It has the richest functionality.  Coming to the next one, well, if you want to use file in different place, well, then  it's a little bit more complicated than, yeah, you can use it.  Well, context managers are good for small chunks of code.  So if you have few lines, but yeah, you can write your own class that will store a file  handler and once this, and you can call close on your own class when you finish doing with  this.  So as far as I know, there is no straightforward thing to do with the context managers.  Hi, hi, is the order of the decorator important?  If I have, for instance, three decorators, sorry, is it going to grab the following decorator?  Yeah, this is, you're right, this is a tricky part.  Yes, the order of decorators affects it.  So some decorators, for some decorators, it's important.  The order is important in which you do it, but for some it's not.  So yeah, order matters in general case and more specific.  So you can write the decorators that, for example, I will give you a very small example  specific where was my wonderful cache.  So here it's important a property should come before a cache result.  Otherwise if you do it the other way, it won't work, it just won't work properly.  But because the property is a special, it's a building decorator.  But for simple decorator, you can write the logic in a way that it wouldn't matter.  So it really depends on, yeah, particularly the decorators.  But evaluate the full amount, right?  I think it actually, no, it actually, first it will call cache result decorator and only  then it will call property.  So that's why it's important that property will be like outermost.  Yeah, yeah.  This is a tricky part here.  Any other questions?  Comments?  We still have a few minutes.  Hello, thank you for the talk.  Just one observation about the wrap decorator that the standard library offers.  So I saw that in one example that's present and in the others when you are writing a wrapper  function you are not using that.  So I think it's worth mentioning that it is better to use because it copies the doc string  and functions name and those things.  Yeah, you're absolutely right.  You're absolutely right.  And practically you should use wraps from Funcutils module.  That's absolutely right because it will keep the function name and doc string and all  this stuff.  But it was just like too much for the talk.  But you're right.  Thanks for the note.  Can we push for one more?  Questions, comments?  Or if not, let's thank the speakers again.  Thank you.  Thank you.
Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_0_7619.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_7620_7739.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_7740_8315.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_8316_8555.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_8556_16967.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_16968_17267.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_17268_17639.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_17640_28259.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_28260_28415.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_28416_28559.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_28560_29087.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_29088_29591.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_29592_35903.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_35904_36107.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_36108_36479.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_36480_36767.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_36768_37199.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_37200_37511.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_37512_37583.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_37584_37643.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_37644_38111.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_38112_38351.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_38352_38423.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_38424_38471.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_38472_38567.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_38568_40751.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_40752_45263.png Andrey_Syschikov_-_Practical_usage_of_advanced_Python_constructs_45264_47939.png