How to write pylint plugins
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Todorov, Alexander
Pylint is the most popular Python source code analyzer which looks for programming errors, helps enforce a coding standard and sniffs for some code smells. It is possible to write plugins to add your own checks, for example enforce all documentation strings to be enclosed with 3 double quotes! This presentation will focus on how the pylint plugin system works, how to examine AST tree of a piece of code and create a minimalistic plugin from scratch. It will also examine couple of plugins that are used in the Kiwi TCMS project.
 Hello, good morning. Okay, now that you're awake, my name is Alex and I'm going to talk about pilot plugins. I'm going to talk a little bit about tokenization, a little bit about abstract syntax trees. But the question is, why do we need to have customized plugins? Why do we need to have even more linters on top of whatever we are already using? And why do we need to have two links which will ultimately tell you your source code is wrong, you have to fix this? And my answer is that the existing tools are not always enough, and static analysis tools like plugin, like pilot, are very easy to extend, they can help you make your software better. And I will show you some examples. So one use case is you want to enforce particular coding style. This can be something which is not valid for the broader Python community. This can be only valid for your own company, or maybe only in the team that you work in, or more commonly valid only in the current project. And that can be different from the rest of the projects in the company. In KivitCMS, that's an open source project I work on, we like to use documentation strings with three double quotes. And we don't like to have the rest of the styles which are perfectly valid Python styles for documentation strings, we just like three double quotes. And we have a customized pilot plugin which will discover this for us, we can fix it, and we can keep all of our source code in the same way. So you can do things like, let's say you have client and server relationships in your application, you may want to name them the same way. So both the client, both the server start with the same name, and then you have client at the end, server at the end, easy to find, easy to grab in the source code if you like. You can do all sorts of stuff. Another example is you're using a lot of frameworks, a lot of libraries, they have best practices, they have some recommendations that you need to follow. For example, if you're using Django, this tells you don't hard code authentication user as a foreign key relationship, because this can be changed, you should be using a setting for that. Also Django tells you don't issue queries directly against the user model, because that can be changed. We have a helper function which will give you the actual model at runtime and you can query that. And all of this is designed to facilitate downstream applications who may wish to change the stock user model and provide something else instead of the default. So this is one way to make your application aware of these things, make it follow the practices. Another example is we are using Django simple history to keep track of changes to some objects, and Django simple history works with model safe. So if you're using objects.update or bulk create, this doesn't use the safe method, so we skip history, we don't like to use these methods. Again, customize pilot plugins, so we are aware not doing this. Another example is this is especially true in big projects, in legacy projects, old source code. You can use static analysis tools to help you find possible sources of problems of bugs. And we have had the problem of missing permissions, so we have views which process request from the browser and they are missing the permission required decorator. So that's really bad. And we've seen this a few times, figured, okay, let's create a plugin for piling and find all the places in the source code that may be missing these permissions. So we have a list of them. We can go check them out, figure out what's going on if we add a new view later and forget to add the permissions to that, the plugin will tell us. So it's a nice mechanism to very easily, very quickly find some problems. Before we continue, two things that are important. You just need to know that they happen. It's not necessary to know how they work in details under the hood. This is parsing, first thing is parsing or lexical analysis or tokenization and the other thing is building abstract syntax histories from the source code. So first, you have input which is the files of your program. This is all character input. It doesn't mean anything to the tooling. It goes through this box which is parsing tokenization, lexical analysis and we get another data structure which has a little bit more meaning. So in my example you see we have a keyword, we have identifier, we have operators, we have numeric constants. This is something that static analysis tooling can work with a lot more easily and you can use this information to make decisions about your source code. Tokenization is Python is very easy. We have the tokenized module which provides the tokenized function. This function receives a single argument which must behave like the read line method. So if you're working with file objects, then file object dot read line should work. If you're working with strings, you have to wrap them in a byte say object and use the read line method. The result of tokenize is a generator which we return token info objects. Token info is a name typo type. It has five elements. So token type is an integer constant and also these names in the brackets, these are constants defined in the tokenized module so you can use them as well. You have the token value as a string. You have start and end position of this token in the input character stream as tuples. So these are starting rows, starting column, end row and column of this token. And then you have the entire line which is currently being inspected by the tokenizer. So this is all the output the tokenize gives you. This is how the hello world example looks like. And you can experiment with it. So it's very easy. So experiment with that, see how different pieces of source code look like to the tokenizer. This is the first step that all static analysis tools do. This is also done internally by Python as well. So next thing is abstract syntax trees. Sounds very complicated. Again, used internally by a lot of tooling used internally by Python. But if you want to work with them, you don't really need to know how they are constructed or all the details behind that. You just need to know that it's a tree based structure. It's very similar to DOM trees in the browser or to XML trees. You have child nodes, parent nodes, you have siblings, you have different types of nodes. They have different types of attributes and you can work with them pretty much very easily. All these different colors, they are objects from different types in Python. And this is how you can recognize them internally when you're writing plugins. Creating abstract syntax trees, again, very easy. We have the AST module which is built in Python that is used by Python internally. Also used by some other tooling like Cosmicray. However, Pylint does not use AST. Pylint uses AstroID which is an external dependency. Very similar to the built-in module. Almost everything is named in the same way but you have to be aware that it's a different module. So we have the parse function provided by AstroID. This will receive a string and we will return AstroID node. The root node is the module. So everything that AstroID parses is represented as a module which contains something else inside of it. So we have, in this example, the module doesn't have very interesting attributes because it doesn't come from the file system. It doesn't have a name but it has a body which is a list of expressions, so the expressions in the module. And we have a single expression in this list which is a code to a function with the name print. This function receives arguments which is again a list and we have only one argument which is a string constant in this example and we don't have any keyword arguments. So again, you can experiment in the interactive interpreter or you can create a small script and experiment, see how different pieces of code look like to AST. This is a relatively well documented library. What you're going to need most of the time is the list of classes. You need to know their names and you will see why in a second and you need to know their attributes. There are also some helper methods and helper functions that you may want to use. They are usually defined in the base classes. So again, experiment with that and figure out how it works. Next is Pylint Checker Interfaces. This is the internal machinery that Pylint provides for you, the developer, to hook into the analysis process and be able to create the plugins. This is also the machinery that Pylint itself uses internally. So all the checks that you have, all the errors that you see when you work with Pylint, they are implemented with these four interfaces and they are also implemented under, as if they were plugins. So the same thing. The names are pretty much self-explanatory. So we have open and close, which are executed at the beginning and at the end. Then you have the raw checker interface. This is not very often used, only in a few places used. Process modules receives the result of AstroID parts. So you can scan the entire module as a whole, if you wish. Then you have the token checker interface which provides process tokens method. This receives the result of tokenize. And the most commonly used, more than 90% of the time is the AstroID checker interface. This will respond to visit and leave methods and the exact name of these methods is depending on the class name of the object of the node that you want to inspect. So for example, if you want to inspect a function, the AstroID node is function def. So you can define methods visit underscore function def or leave underscore function def. Or if you want to inspect a class definition, this is visit underscore class def, leave underscore class def. So that's why you need to know the names. And the order of execution is this. So from top to bottom, you can implement more than one interface in your plugin and the order is important. Another important thing, the order of execution of visit methods, leave method, this is depth first. And this is important because you can use it and build some sort of state machine in your plugin, collect some information in the children. And when you are leaving the parent node, you note that all the children have been visited and you have all the information for them you can make decision. Next, let's create a pilot plugin skeleton. So this is the hello world of pilot plugins. Every module which is valid, valid Python module that provides a register function with one argument will be considered a pilot plugin. Pilot will import the module, try to execute this function, you can put anything you like in this function. Really what goes inside is something like that. Linter.register checker and you create an object from something which pilot calls a checker class. This is how the checker class looks like. This is all boilerplate code. This is the better minimum that you need to have for pilot to be able to execute this class. This is where all the logic about discovering coding patterns and deciding if something is an error or not an error is done. So you need this attribute. This is double underscore, implements double underscore and you give it a list of what interfaces you are going to implement. Usually it's only one but it can be more. You need the name attribute. Most of the time this is not used but it's mandatory. You need the messages dictionary. Notice the name. That's how it needs to be written. The key in this dictionary is an alphanumeric ID. This must be unique across the entire pilot installation and all the plugins that you want to enable. The good thing is that if it's not unique, pilot will crash and give you a nice trace back and you will figure out that this is duplicate. And then you have the value in this dictionary is two of three elements. So this defines your error message. The first one is the short error message which you are going to see on the terminal if you use pilot. It's only one line. The second one is the human readable message ID. This is what you're going to use to enable or disable particular checkers on the command line. So for example, disable missing doc. This is something that we do almost all the time. And the last one is a longer help message. This can be several lines long. You can also see this on the terminal with additional options. This is also compiled as HTML documentation. That is the place to explain to the developer that sees the message why that is a problem, how to fix it maybe. And you need to implement some method from these interfaces. So you scan some source code and decide, okay, that's an error. Save.add message. Give it the human readable message ID and the rest of the arguments are used to annotate where this message, this error appears in the source code. So this module, that particular line, that particular column and pilot will print this information nicely for you. Evoking the plugins with minus, minus load plugins. The only thing that you need to be aware of is pilot is looking in the standard Python path for these plugins. So if they are not there, either you have to move them there or modify the Python path settings. And that does it. Nothing else. Now I'm going to show you a few examples from our open source project. All of them are on GitHub. We do have a lot more available. These are things that we use to make our project better. So documentation string checker. This is how it looks like, the essence of it. It implements two interfaces. So in process tokens, we basically scan through all the tokens in the module, find all the string constants and keep reference to them in the dictionary. Key in the dictionary is the string without the quotes and the value in the dictionary is the string with the quotes. Then we implement these Asteroid based methods. So when visiting modules, class definitions and function definitions, we want to inspect the documentual string. And what we basically do is a dictionary lookup. We find this thing in the dictionary and if it starts with three double quotes, that's fine. Otherwise, we consider it an error. Trigger a message for the developer. The checker for Django. So again, visit cost, look for hard-coded strings. We don't really care if this is inside of a foreign key definition or someplace else. If that is a hard-coded string, we raise an error message for the developer. That easy. And also we inspect the imports. If we see something like from Django contrib, old models import user or do a wildcard import, again, error for the developer to inspect this, figure out what's going on. Missing permissions checker. That's probably the biggest one that we have, which fits onto slides, unfortunately. So first, visit module. We try to figure out if this is a views module. And in our project, we have application slash views.py, another application slash views.py. That's the structure and we just inspect the module name. Keep this in a Boolean flag. Next, visit function def. We try to figure out if this function that is in a Django view file, is this a helper function or is this function base view something that responds to HTTP requests? And the way we check for that is if the first argument is named request, then this must be a function base view from Django. And we continue with further inspection. For classes, we do a similar thing. We want to make sure that the class we are inspecting, this is a class base view in Django. This is not some helper class which is defined in the same module. And the way we do this is we inspect the list of base classes. Because when you use class base views in Django, they always inherit from something else. So we use this to make a simple check. It's not very robust sometimes, but it works for us. And the most important thing, the inspection part is we basically scan through the list of decorators for the method or for the class and search for some well-known names. So if you don't have any decorators, that's a problem for us. If you do have some, search for the well-known names and some combinations between them. If we find them, fine. If we don't find them again, error, the developer must figure it out. We do have other checkers in the project. So for example, we're looking for empty modules. We're looking for nested function definitions or nested class definitions because that's a legacy code base. It's been written in not very good way. And we don't like to have these things. When we see these things, they usually mean there are more problems inside. So that's why we have these checkers. Searching for raw SQL. So Django is ORM based. Unfortunately, we did have a lot of hard-coded SQL statements in the source code which were not compatible with different types of databases. Again, we have a checker. We have checkers for the libraries that we use. For example, this thing, tags.py. This is something internal that we have internal behavior in the application and we don't want to use objects.getor create. We want to use an internal method which will enforce some permission and some other logic. So that's why we have this. And also, we have had some checkers which started life inside of our project and then we were later able to contribute to Pilington, to Piling Django because they were valid for other people as well. So, and the last link is we do have ideas for other items, like other plugins, other checkers to create which are important for us. So if you want to experiment, if you want to get your hands dirty and start writing Piling plugins, this is a good place for you to start. We can give you exact examples of pieces of source code which we don't like and why we think is problematic and you can try to create a plugin for that and contribute back to our project if you want to. So the last thing I have to tell you is that we are also having a project stand here at FOSTEMP. So if you want to come visit us and say hi, talk a little bit more about, you know, why or how we are using these plugins, I will be there after this presentation. And now we have five minutes for questions. Thank you. Okay, first question. Okay, do you use Piling? Yes? One person, two person, five people. Oh, everybody, okay. Okay, Flake8, okay. What I didn't hear? Ah, black, okay. Okay. But the thing about black is it's nice too. However, it's more like for formatting. So especially in the latest versions of Piling and Piling and Django, they have checkers to show you things which are just considered bad practice. Yeah? Okay, so the question is how many things we put in a check and because developers don't always agree with something. If you're going to, so first of all, I am a big fan of satisfying all possible checks that come from Piling. I think they are well designed and they are created for a purpose to make your life easier. But then if you're going to create your own customized plugins for your team and you have people that don't agree, then maybe it's a good time to sit down and make some policies about, you know, coding style within the team, why you consider something to be a problem and why not. And when you have this agreement, then you can create a plugins and people will be happy. How many false positives do you have in your plugins? Yeah, how many false positives do we have in our plugins? The answer is quite a few. I haven't counted them and this is for a reason. It is relatively easy to create a plugin that will detect the most common cases and it is relatively hard to create a plugin that will take into account all the edge cases. So we prefer to have very simple plugins and have more false positives just disable them with a comment and ignore them instead of spending a lot of time fine tuning the plugin. Okay, yes? Can you elaborate a little bit on how you would now fix the code actually, also with pilot plugins? How can I fix the code when I see a problem? Okay, so the question is basically can we change the abstract syntax trees with pilot and the answer is sort of yes and no. There are two links which use AST to do dynamic replacement of nodes. So for example Cosmicray is a tool for mutation testing which is based on automatically changing the source code and running your test suite. And you can do this. You can also save this into a file. When you built an abstract syntax tree it is relatively easy to export this into Python source code and it is almost the same as what was the input. Pilot doesn't have the machinery to change abstract syntax trees nodes and then save them to the file system. This can be added of course. I mean it will be relatively easy to add but it's not existing at the moment. The tool is not designed to do these things but it is possible. Okay, thank you. Thank you.
 Hello, good morning. Okay, now that you're awake, my name is Alex and I'm going to talk  about pilot plugins. I'm going to talk a little bit about tokenization, a little bit  about abstract syntax trees. But the question is, why do we need to have customized plugins?  Why do we need to have even more linters on top of whatever we are already using? And  why do we need to have two links which will ultimately tell you your source code is wrong,  you have to fix this? And my answer is that the existing tools are not always enough,  and static analysis tools like plugin, like pilot, are very easy to extend, they can help  you make your software better. And I will show you some examples. So one use case is  you want to enforce particular coding style. This can be something which is not valid for  the broader Python community. This can be only valid for your own company, or maybe  only in the team that you work in, or more commonly valid only in the current project.  And that can be different from the rest of the projects in the company. In KivitCMS,  that's an open source project I work on, we like to use documentation strings with three  double quotes. And we don't like to have the rest of the styles which are perfectly valid  Python styles for documentation strings, we just like three double quotes. And we have  a customized pilot plugin which will discover this for us, we can fix it, and we can keep  all of our source code in the same way. So you can do things like, let's say you have  client and server relationships in your application, you may want to name them the same way. So  both the client, both the server start with the same name, and then you have client at  the end, server at the end, easy to find, easy to grab in the source code if you like.  You can do all sorts of stuff. Another example is you're using a lot of frameworks, a lot  of libraries, they have best practices, they have some recommendations that you need to  follow. For example, if you're using Django, this tells you don't hard code authentication  user as a foreign key relationship, because this can be changed, you should be using a  setting for that. Also Django tells you don't issue queries directly against the user model,  because that can be changed. We have a helper function which will give you the actual model  at runtime and you can query that. And all of this is designed to facilitate downstream  applications who may wish to change the stock user model and provide something else instead  of the default. So this is one way to make your application aware of these things, make  it follow the practices. Another example is we are using Django simple history to keep  track of changes to some objects, and Django simple history works with model safe. So if  you're using objects.update or bulk create, this doesn't use the safe method, so we skip  history, we don't like to use these methods. Again, customize pilot plugins, so we are  aware not doing this. Another example is this is especially true in big projects, in legacy  projects, old source code. You can use static analysis tools to help you find possible sources  of problems of bugs. And we have had the problem of missing permissions, so we have views which  process request from the browser and they are missing the permission required decorator.  So that's really bad. And we've seen this a few times, figured, okay, let's create a  plugin for piling and find all the places in the source code that may be missing these  permissions. So we have a list of them. We can go check them out, figure out what's going  on if we add a new view later and forget to add the permissions to that, the plugin will  tell us. So it's a nice mechanism to very easily, very quickly find some problems. Before  we continue, two things that are important. You just need to know that they happen. It's  not necessary to know how they work in details under the hood. This is parsing, first thing  is parsing or lexical analysis or tokenization and the other thing is building abstract syntax  histories from the source code. So first, you have input which is the files of your  program. This is all character input. It doesn't mean anything to the tooling. It goes through  this box which is parsing tokenization, lexical analysis and we get another data structure  which has a little bit more meaning. So in my example you see we have a keyword, we have  identifier, we have operators, we have numeric constants. This is something that static analysis  tooling can work with a lot more easily and you can use this information to make decisions  about your source code. Tokenization is Python is very easy. We have the tokenized module  which provides the tokenized function. This function receives a single argument which  must behave like the read line method. So if you're working with file objects, then file  object dot read line should work. If you're working with strings, you have to wrap them  in a byte say object and use the read line method. The result of tokenize is a generator  which we return token info objects. Token info is a name typo type. It has five elements.  So token type is an integer constant and also these names in the brackets, these are constants  defined in the tokenized module so you can use them as well. You have the token value  as a string. You have start and end position of this token in the input character stream  as tuples. So these are starting rows, starting column, end row and column of this token.  And then you have the entire line which is currently being inspected by the tokenizer.  So this is all the output the tokenize gives you. This is how the hello world example looks  like. And you can experiment with it. So it's very easy. So experiment with that, see how  different pieces of source code look like to the tokenizer. This is the first step that  all static analysis tools do. This is also done internally by Python as well. So next  thing is abstract syntax trees. Sounds very complicated. Again, used internally by a lot  of tooling used internally by Python. But if you want to work with them, you don't really  need to know how they are constructed or all the details behind that. You just need to  know that it's a tree based structure. It's very similar to DOM trees in the browser or  to XML trees. You have child nodes, parent nodes, you have siblings, you have different  types of nodes. They have different types of attributes and you can work with them pretty  much very easily. All these different colors, they are objects from different types in Python.  And this is how you can recognize them internally when you're writing plugins. Creating abstract  syntax trees, again, very easy. We have the AST module which is built in Python that is  used by Python internally. Also used by some other tooling like Cosmicray. However, Pylint  does not use AST. Pylint uses AstroID which is an external dependency. Very similar to  the built-in module. Almost everything is named in the same way but you have to be aware  that it's a different module. So we have the parse function provided by AstroID. This will  receive a string and we will return AstroID node. The root node is the module. So everything  that AstroID parses is represented as a module which contains something else inside of it.  So we have, in this example, the module doesn't have very interesting attributes because it  doesn't come from the file system. It doesn't have a name but it has a body which is a list  of expressions, so the expressions in the module. And we have a single expression in  this list which is a code to a function with the name print. This function receives arguments  which is again a list and we have only one argument which is a string constant in this  example and we don't have any keyword arguments. So again, you can experiment in the interactive  interpreter or you can create a small script and experiment, see how different pieces of  code look like to AST. This is a relatively well documented library. What you're going  to need most of the time is the list of classes. You need to know their names and you will  see why in a second and you need to know their attributes. There are also some helper methods  and helper functions that you may want to use. They are usually defined in the base  classes. So again, experiment with that and figure out how it works. Next is Pylint Checker  Interfaces. This is the internal machinery that Pylint provides for you, the developer,  to hook into the analysis process and be able to create the plugins. This is also the machinery  that Pylint itself uses internally. So all the checks that you have, all the errors that  you see when you work with Pylint, they are implemented with these four interfaces and  they are also implemented under, as if they were plugins. So the same thing. The names  are pretty much self-explanatory. So we have open and close, which are executed at the  beginning and at the end. Then you have the raw checker interface. This is not very often  used, only in a few places used. Process modules receives the result of AstroID parts. So you  can scan the entire module as a whole, if you wish. Then you have the token checker interface  which provides process tokens method. This receives the result of tokenize. And the most  commonly used, more than 90% of the time is the AstroID checker interface. This will respond  to visit and leave methods and the exact name of these methods is depending on the class  name of the object of the node that you want to inspect. So for example, if you want to  inspect a function, the AstroID node is function def. So you can define methods visit underscore  function def or leave underscore function def. Or if you want to inspect a class definition,  this is visit underscore class def, leave underscore class def. So that's why you need  to know the names. And the order of execution is this. So from top to bottom, you can implement  more than one interface in your plugin and the order is important. Another important  thing, the order of execution of visit methods, leave method, this is depth first. And this  is important because you can use it and build some sort of state machine in your plugin,  collect some information in the children. And when you are leaving the parent node,  you note that all the children have been visited and you have all the information for them  you can make decision. Next, let's create a pilot plugin skeleton. So this is the hello  world of pilot plugins. Every module which is valid, valid Python module that provides  a register function with one argument will be considered a pilot plugin. Pilot will import  the module, try to execute this function, you can put anything you like in this function.  Really what goes inside is something like that. Linter.register checker and you create  an object from something which pilot calls a checker class. This is how the checker class  looks like. This is all boilerplate code. This is the better minimum that you need to  have for pilot to be able to execute this class. This is where all the logic about discovering  coding patterns and deciding if something is an error or not an error is done. So you  need this attribute. This is double underscore, implements double underscore and you give  it a list of what interfaces you are going to implement. Usually it's only one but it  can be more. You need the name attribute. Most of the time this is not used but it's  mandatory. You need the messages dictionary. Notice the name. That's how it needs to be  written. The key in this dictionary is an alphanumeric ID. This must be unique across  the entire pilot installation and all the plugins that you want to enable. The good  thing is that if it's not unique, pilot will crash and give you a nice trace back and you  will figure out that this is duplicate. And then you have the value in this dictionary  is two of three elements. So this defines your error message. The first one is the short  error message which you are going to see on the terminal if you use pilot. It's only one  line. The second one is the human readable message ID. This is what you're going to use  to enable or disable particular checkers on the command line. So for example, disable  missing doc. This is something that we do almost all the time. And the last one is a  longer help message. This can be several lines long. You can also see this on the terminal  with additional options. This is also compiled as HTML documentation. That is the place to  explain to the developer that sees the message why that is a problem, how to fix it maybe.  And you need to implement some method from these interfaces. So you scan some source  code and decide, okay, that's an error. Save.add message. Give it the human readable message  ID and the rest of the arguments are used to annotate where this message, this error  appears in the source code. So this module, that particular line, that particular column  and pilot will print this information nicely for you. Evoking the plugins with minus,  minus load plugins. The only thing that you need to be aware of is pilot is looking in  the standard Python path for these plugins. So if they are not there, either you have  to move them there or modify the Python path settings. And that does it. Nothing else.  Now I'm going to show you a few examples from our open source project. All of them are on  GitHub. We do have a lot more available. These are things that we use to make our project  better. So documentation string checker. This is how it looks like, the essence of it. It  implements two interfaces. So in process tokens, we basically scan through all the  tokens in the module, find all the string constants and keep reference to them in the  dictionary. Key in the dictionary is the string without the quotes and the value in the dictionary  is the string with the quotes. Then we implement these Asteroid based methods. So when visiting  modules, class definitions and function definitions, we want to inspect the documentual string.  And what we basically do is a dictionary lookup. We find this thing in the dictionary and if  it starts with three double quotes, that's fine. Otherwise, we consider it an error.  Trigger a message for the developer. The checker for Django. So again, visit cost, look for  hard-coded strings. We don't really care if this is inside of a foreign key definition  or someplace else. If that is a hard-coded string, we raise an error message for the  developer. That easy. And also we inspect the imports. If we see something like from  Django contrib, old models import user or do a wildcard import, again, error for the developer  to inspect this, figure out what's going on. Missing permissions checker. That's probably  the biggest one that we have, which fits onto slides, unfortunately. So first, visit module.  We try to figure out if this is a views module. And in our project, we have application slash  views.py, another application slash views.py. That's the structure and we just inspect the  module name. Keep this in a Boolean flag. Next, visit function def. We try to figure out if  this function that is in a Django view file, is this a helper function or is this function  base view something that responds to HTTP requests? And the way we check for that is  if the first argument is named request, then this must be a function base view from Django.  And we continue with further inspection. For classes, we do a similar thing. We want to  make sure that the class we are inspecting, this is a class base view in Django. This  is not some helper class which is defined in the same module. And the way we do this  is we inspect the list of base classes. Because when you use class base views in Django, they  always inherit from something else. So we use this to make a simple check. It's not very  robust sometimes, but it works for us. And the most important thing, the inspection part  is we basically scan through the list of decorators for the method or for the class and search  for some well-known names. So if you don't have any decorators, that's a problem for  us. If you do have some, search for the well-known names and some combinations between them.  If we find them, fine. If we don't find them again, error, the developer must figure it  out. We do have other checkers in the project. So for example, we're looking for empty modules.  We're looking for nested function definitions or nested class definitions because that's  a legacy code base. It's been written in not very good way. And we don't like to have these  things. When we see these things, they usually mean there are more problems inside. So that's  why we have these checkers. Searching for raw SQL. So Django is ORM based. Unfortunately,  we did have a lot of hard-coded SQL statements in the source code which were not compatible  with different types of databases. Again, we have a checker. We have checkers for the  libraries that we use. For example, this thing, tags.py. This is something internal that we  have internal behavior in the application and we don't want to use objects.getor create.  We want to use an internal method which will enforce some permission and some other logic.  So that's why we have this. And also, we have had some checkers which started life inside  of our project and then we were later able to contribute to Pilington, to Piling Django  because they were valid for other people as well. So, and the last link is we do have  ideas for other items, like other plugins, other checkers to create which are important  for us. So if you want to experiment, if you want to get your hands dirty and start writing  Piling plugins, this is a good place for you to start. We can give you exact examples  of pieces of source code which we don't like and why we think is problematic and you can  try to create a plugin for that and contribute back to our project if you want to. So the  last thing I have to tell you is that we are also having a project stand here at FOSTEMP.  So if you want to come visit us and say hi, talk a little bit more about, you know, why  or how we are using these plugins, I will be there after this presentation. And now we  have five minutes for questions. Thank you. Okay, first question. Okay, do you use Piling?  Yes? One person, two person, five people. Oh, everybody, okay. Okay, Flake8, okay. What  I didn't hear? Ah, black, okay. Okay. But the thing about black is it's nice too. However,  it's more like for formatting. So especially in the latest versions of Piling and Piling  and Django, they have checkers to show you things which are just considered bad practice.  Yeah? Okay, so the question is how many things we put in a check and because developers don't  always agree with something. If you're going to, so first of all, I am a big fan of satisfying  all possible checks that come from Piling. I think they are well designed and they are  created for a purpose to make your life easier. But then if you're going to create your own  customized plugins for your team and you have people that don't agree, then maybe it's a  good time to sit down and make some policies about, you know, coding style within the team,  why you consider something to be a problem and why not. And when you have this agreement,  then you can create a plugins and people will be happy.  How many false positives do you have in your plugins?  Yeah, how many false positives do we have in our plugins? The answer is quite a few.  I haven't counted them and this is for a reason. It is relatively easy to create a plugin that  will detect the most common cases and it is relatively hard to create a plugin that will  take into account all the edge cases. So we prefer to have very simple plugins and have  more false positives just disable them with a comment and ignore them instead of spending  a lot of time fine tuning the plugin. Okay, yes?  Can you elaborate a little bit on how you would now fix the code actually, also with  pilot plugins? How can I fix the code when I see a problem?  Okay, so the question is basically can we change the abstract syntax trees with pilot  and the answer is sort of yes and no. There are two links which use AST to do dynamic  replacement of nodes. So for example Cosmicray is a tool for mutation testing which is based  on automatically changing the source code and running your test suite. And you can do  this. You can also save this into a file. When you built an abstract syntax tree it is  relatively easy to export this into Python source code and it is almost the same as what  was the input. Pilot doesn't have the machinery to change abstract syntax trees nodes and  then save them to the file system. This can be added of course. I mean it will be relatively  easy to add but it's not existing at the moment. The tool is not designed to do these  things but it is possible. Okay, thank you.  Thank you.
How_to_write_pylint_plugins_mT0SeMD9rpY_0_155.png How_to_write_pylint_plugins_mT0SeMD9rpY_156_491.png How_to_write_pylint_plugins_mT0SeMD9rpY_492_599.png How_to_write_pylint_plugins_mT0SeMD9rpY_600_671.png How_to_write_pylint_plugins_mT0SeMD9rpY_672_1055.png How_to_write_pylint_plugins_mT0SeMD9rpY_1056_1247.png How_to_write_pylint_plugins_mT0SeMD9rpY_1248_6359.png How_to_write_pylint_plugins_mT0SeMD9rpY_6360_6935.png How_to_write_pylint_plugins_mT0SeMD9rpY_6936_7847.png How_to_write_pylint_plugins_mT0SeMD9rpY_7848_10079.png How_to_write_pylint_plugins_mT0SeMD9rpY_10080_11243.png How_to_write_pylint_plugins_mT0SeMD9rpY_11244_13319.png How_to_write_pylint_plugins_mT0SeMD9rpY_13320_13991.png How_to_write_pylint_plugins_mT0SeMD9rpY_13992_16811.png How_to_write_pylint_plugins_mT0SeMD9rpY_16812_17423.png How_to_write_pylint_plugins_mT0SeMD9rpY_17424_21695.png How_to_write_pylint_plugins_mT0SeMD9rpY_21696_22151.png How_to_write_pylint_plugins_mT0SeMD9rpY_22152_22451.png How_to_write_pylint_plugins_mT0SeMD9rpY_22452_23687.png How_to_write_pylint_plugins_mT0SeMD9rpY_23688_24539.png How_to_write_pylint_plugins_mT0SeMD9rpY_24540_26531.png How_to_write_pylint_plugins_mT0SeMD9rpY_26532_29819.png How_to_write_pylint_plugins_mT0SeMD9rpY_29820_29951.png How_to_write_pylint_plugins_mT0SeMD9rpY_29952_30635.png How_to_write_pylint_plugins_mT0SeMD9rpY_30636_30983.png How_to_write_pylint_plugins_mT0SeMD9rpY_30984_31127.png How_to_write_pylint_plugins_mT0SeMD9rpY_31128_31499.png How_to_write_pylint_plugins_mT0SeMD9rpY_31500_31655.png How_to_write_pylint_plugins_mT0SeMD9rpY_31656_31703.png How_to_write_pylint_plugins_mT0SeMD9rpY_31704_33035.png How_to_write_pylint_plugins_mT0SeMD9rpY_33036_33179.png How_to_write_pylint_plugins_mT0SeMD9rpY_33180_33431.png How_to_write_pylint_plugins_mT0SeMD9rpY_33432_33683.png How_to_write_pylint_plugins_mT0SeMD9rpY_33684_34679.png How_to_write_pylint_plugins_mT0SeMD9rpY_34680_34955.png How_to_write_pylint_plugins_mT0SeMD9rpY_34956_37019.png How_to_write_pylint_plugins_mT0SeMD9rpY_37020_37141.png