<commit_before>Linter Bears
============

Welcome. This tutorial aims to show you how to use the ``@linter`` decorator in
order to integrate linters in your bears.

.. note::

  If you are planning to create a bear that does static code analysis without
  wrapping a tool, please refer to :doc:`this link instead<Writing_Bears>`.

Why is This Useful?
-------------------

A lot of programming languages already have linters implemented, so if your
project uses a language that does not already have a linter Bear you might
need to implement it on your own. Don't worry, it's easy!

What do we Need?
----------------

First of all, we need the linter executable that we are going to use.
In this tutorial we will build the PylintTutorialBear so we need Pylint, a
common linter for Python. It can be found `here <https://www.pylint.org/>`__.
Since it is a python package we can go ahead and install it with

::

    $ pip3 install pylint

Writing the Bear
----------------

To write a linter bear, we need to create a class that interfaces with our
linter-bear infrastructure, which is provided via the ``@linter`` decorator.

::

    from coalib.bearlib.abstractions.Linter import linter

    @linter(executable='pylint')
    class PylintTutorialBear:
        pass

As you can see ``pylint`` is already provided as an executable name which gets
invoked on the files you are going to lint. This is a mandatory argument for
the decorator.

The linter class is only capable of processing one file at a time, for this
purpose ``pylint`` or the external tool needs to be invoked every time with the
appropriate parameters. This is done inside ``create_arguments``,

::

    @linter(executable='pylint')
    class PylintTutorialBear:
        @staticmethod
        def create_arguments(filename, file, config_file):
            pass

``create_arguments`` accepts three parameters:

- ``filename``: The absolute path to the file that gets processed.
- ``file``: The contents of the file to process, given as a list of lines
  (including the return character).
- ``config_file``: The absolute path to a config file to use. If no config file
  is used, this parameter is ``None``. More on that later.

You can use these parameters to construct the command line arguments. The
linter expects from you to return an argument sequence here. A tuple is
preferred. We will do this soon for ``PylintTutorialBear``.

.. note::

    ``create_arguments`` doesn't have to be a static method. In this case you
    also need to prepend ``self`` to the parameters in the signature. Some
    functionality of ``@linter`` is only available inside an instance, like
    logging.

    ::

        def create_arguments(self, filename, file, config_file):
            self.log("Hello world")

So which are the exact command line arguments we need to provide? It depends on
the output format of the linter. The ``@linter`` decorator is capable of
handling different output formats:

- ``regex``: This parses issue messages yielded by the underlying executable.
- ``corrected``: Auto-generates results from a fixed/corrected file provided by
  the tool.

In this tutorial we are going to use the ``regex`` output format. But before we
continue with modifying our bear, we need to figure out how exactly output from
Pylint looks like so we can parse it accordingly.

We get some promising output when invoking Pylint with

::

    $ pylint --msg-template="L{line}C{column}: {msg_id} - {msg}" --reports=n

Sample output looks like this:

::

    No config file found, using default configuration
    ************* Module coalib.bearlib.abstractions.Linter
    L1C0: C0111 - Missing module docstring
    L42C48: E1101 - Class 'Enum' has no 'reverse' member
    L77C32: E1101 - Class 'Enum' has no 'reverse' member
    L21C0: R0912 - Too many branches (16/12)
    L121C28: W0613 - Unused argument 'filename'

This is something we can parse easily with a regex. So let's implement
everything we've found out so far:

::

    @linter(executable='pylint',
            output_format='regex',
            output_regex=r'L(?P<line>\d+)C(?P<column>\d+): (?P<message>.*)')
    class PylintTutorialBear:
        @staticmethod
        def create_arguments(filename, file, config_file):
            return ('--msg-template="L{line}C{column}: {msg_id} - {msg}"',
                    '--reports=n', filename)

As you can see, the ``output_regex`` parameter consists of named groups. These
are important to construct a meaningful result that contains the information
that is printed out.

For the exact list of named groups ``@linter`` recognizes, see the API
documentation.

For more info generally on regexes, see `Python re module
<https://docs.python.org/3/library/re.html>`_.

Let's brush up our ``output_regex`` a bit to use even more information:

::

    @linter(...
            output_regex=r'L(?P<line>\d+)C(?P<column>\d+): '
                         r'(?P<message>(?P<origin>.\d+) - .*)'),
            ...)

Now we use the issue identification as the origin so we are able to deactivate
single rules via ignore statements inside code.

This class is already fully functional and allows to parse issues yielded by
Pylint!

Using Severities
----------------

coala uses three types of severities that categorize the importance of a
result:

-  INFO<commit_msg>Writing_Linter_Bears: Link to API documentation<commit_after>Linter Bears
============

Welcome. This tutorial aims to show you how to use the ``@linter`` decorator in
order to integrate linters in your bears.

.. note::

  If you are planning to create a bear that does static code analysis without
  wrapping a tool, please refer to :doc:`this link instead<Writing_Bears>`.

Why is This Useful?
-------------------

A lot of programming languages already have linters implemented, so if your
project uses a language that does not already have a linter Bear you might
need to implement it on your own. Don't worry, it's easy!

What do we Need?
----------------

First of all, we need the linter executable that we are going to use.
In this tutorial we will build the PylintTutorialBear so we need Pylint, a
common linter for Python. It can be found `here <https://www.pylint.org/>`__.
Since it is a python package we can go ahead and install it with

::

    $ pip3 install pylint

Writing the Bear
----------------

To write a linter bear, we need to create a class that interfaces with our
linter-bear infrastructure, which is provided via the ``@linter`` decorator.

::

    from coalib.bearlib.abstractions.Linter import linter

    @linter(executable='pylint')
    class PylintTutorialBear:
        pass

As you can see ``pylint`` is already provided as an executable name which gets
invoked on the files you are going to lint. This is a mandatory argument for
the decorator.

The linter class is only capable of processing one file at a time, for this
purpose ``pylint`` or the external tool needs to be invoked every time with the
appropriate parameters. This is done inside ``create_arguments``,

::

    @linter(executable='pylint')
    class PylintTutorialBear:
        @staticmethod
        def create_arguments(filename, file, config_file):
            pass

``create_arguments`` accepts three parameters:

- ``filename``: The absolute path to the file that gets processed.
- ``file``: The contents of the file to process, given as a list of lines
  (including the return character).
- ``config_file``: The absolute path to a config file to use. If no config file
  is used, this parameter is ``None``. More on that later.

You can use these parameters to construct the command line arguments. The
linter expects from you to return an argument sequence here. A tuple is
preferred. We will do this soon for ``PylintTutorialBear``.

.. note::

    ``create_arguments`` doesn't have to be a static method. In this case you
    also need to prepend ``self`` to the parameters in the signature. Some
    functionality of ``@linter`` is only available inside an instance, like
    logging.

    ::

        def create_arguments(self, filename, file, config_file):
            self.log("Hello world")

So which are the exact command line arguments we need to provide? It depends on
the output format of the linter. The ``@linter`` decorator is capable of
handling different output formats:

- ``regex``: This parses issue messages yielded by the underlying executable.
- ``corrected``: Auto-generates results from a fixed/corrected file provided by
  the tool.

In this tutorial we are going to use the ``regex`` output format. But before we
continue with modifying our bear, we need to figure out how exactly output from
Pylint looks like so we can parse it accordingly.

We get some promising output when invoking Pylint with

::

    $ pylint --msg-template="L{line}C{column}: {msg_id} - {msg}" --reports=n

Sample output looks like this:

::

    No config file found, using default configuration
    ************* Module coalib.bearlib.abstractions.Linter
    L1C0: C0111 - Missing module docstring
    L42C48: E1101 - Class 'Enum' has no 'reverse' member
    L77C32: E1101 - Class 'Enum' has no 'reverse' member
    L21C0: R0912 - Too many branches (16/12)
    L121C28: W0613 - Unused argument 'filename'

This is something we can parse easily with a regex. So let's implement
everything we've found out so far:

::

    @linter(executable='pylint',
            output_format='regex',
            output_regex=r'L(?P<line>\d+)C(?P<column>\d+): (?P<message>.*)')
    class PylintTutorialBear:
        @staticmethod
        def create_arguments(filename, file, config_file):
            return ('--msg-template="L{line}C{column}: {msg_id} - {msg}"',
                    '--reports=n', filename)

As you can see, the ``output_regex`` parameter consists of named groups. These
are important to construct a meaningful result that contains the information
that is printed out.

For the exact list of named groups ``@linter`` recognizes, see the `API
documentation <https://api.coala.io/en/latest/>`_.

For more info generally on regexes, see `Python re module
<https://docs.python.org/3/library/re.html>`_.

Let's brush up our ``output_regex`` a bit to use even more information:

::

    @linter(...
            output_regex=r'L(?P<line>\d+)C(?P<column>\d+): '
                         r'(?P<message>(?P<origin>.\d+) - .*)'),
            ...)

Now we use the issue identification as the origin so we are able to deactivate
single rules via ignore statements inside code.

This class is already fully functional and allows to parse issues yielded by
Pylint!

Using Severities
----------------

coala uses three types of severities that categorize the importance of a
result:

-  INFO
{'fraction_non_alphanumeric': 0.09816008218922201, 'fraction_numerical': 0.008218922200429626, 'mean_word_length': 3.3723968966925275, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5512924', 'n_tokens_mistral': 3075, 'n_tokens_neox': 2921, 'n_words': 1390}