<commit_before>============================
Performance and optimization
============================

This document provides an overview of techniques and tools that can help get
your Django code running more efficiently - faster, and using fewer system
resources.

Introduction
============

Generally one's first concern is to write code that *works*, whose logic
functions as required to produce the expected output. Sometimes, however, this
will not be enough to make the code work as *efficiently* as one would like.

In this case, what's needed is something - and in practice, often a collection
of things - to improve the code's performance without, or only minimally,
affecting its behavior.

General approaches
==================

What are you optimizing *for*?
------------------------------

It's important to have a clear idea what you mean by 'performance'. There is
not just one metric of it.

Improved speed might be the most obvious aim for a program, but sometimes other
performance improvements might be sought, such as lower memory consumption or
fewer demands on the database or network.

Improvements in one area will often bring about improved performance in
another, but not always; sometimes one can even be at the expense of another.
For example, an improvement in a program's speed might cause it to use more
memory. Even worse, it can be self-defeating - if the speed improvement is so
memory-hungry that the system starts to run out of memory, you'll have done
more harm than good.

There are other trade-offs to bear in mind. Your own time is a valuable
resource, more precious than CPU time. Some improvements might be too difficult
to be worth implementing, or might affect the portability or maintainability of
the code. Not all performance improvements are worth the effort.

So, you need to know what performance improvements you are aiming for, and you
also need to know that you have a good reason for aiming in that direction -
and for that you need:

Performance benchmarking
------------------------

It's no good just guessing or assuming where the inefficiencies lie in your
code.

Django tools
^^^^^^^^^^^^

`django-debug-toolbar
<https://github.com/django-debug-toolbar/django-debug-toolbar/>`_ is a very
handy tool that provides insights into what your code is doing and how much
time it spends doing it. In particular it can show you all the SQL queries your
page is generating, and how long each one has taken.

Third-party panels are also available for the toolbar, that can (for example)
report on cache performance and template rendering times.

Third-party services
^^^^^^^^^^^^^^^^^^^^

There are a number of free services that will analyze and report on the
performance of your site's pages from the perspective of a remote HTTP client,
in effect simulating the experience of an actual user.

These can't report on the internals of your code, but can provide a useful
insight into your site's overall performance, including aspects that can't be
adequately measured from within Django environment. Examples include:

* `Yahoo's Yslow <http://yslow.org/>`_
* `Google PageSpeed <https://developers.google.com/speed/pagespeed/>`_

There are also several paid-for services that perform a similar analysis,
including some that are Django-aware and can integrate with your codebase to
profile its performance far more comprehensively.

Get things right from the start
-------------------------------

Some work in optimization involves tackling performance shortcomings, but some
of the work can simply be built in to what you'd do anyway, as part of the good
practices you should adopt even before you start thinking about improving
performance.

In this respect Python is an excellent language to work with, because solutions
that look elegant and feel right usually are the best performing ones. As with
most skills, learning what "looks right" takes practice, but one of the most
useful guidelines is:

Work at the appropriate level
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Django offers many different ways of approaching things, but just because it's
possible to do something in a certain way doesn't mean that it's the most
appropriate way to do it. For example, you might find that you could calculate
the same thing - the number of items in a collection, perhaps - in a
``QuerySet``, in Python, or in a template.

However, it will almost always be faster to do this work at lower rather than
higher levels. At higher levels the system has to deal with objects through
multiple levels of abstraction and layers of machinery.

That is, the database can typically do things faster than Python can, which can
do them faster than the template language can::

    # QuerySet operation on the database
    # fast, because that's what databases are good at
    my_bicycles.count()

    # counting Python objects
    # slower, because it requires a database query anyway, and processing
    # of the Python objects
    len(my_bicycles)

    # Django template filter
    # slower still, because it will have to count them in Python anyway,
    # and because of template language overheads
    {{ my_bicycles|length }}

Generally speaking, the most appropriate level for the job is the lowest-level
one that it is comfortable to code for.

.. note::

    The example above is merely illustrative.

    Firstly, in a real-life case you need to consider what is happening before
    and after your count to work out what's an optimal way of doing it *in that
    particular context*. The database optimization documents describes :ref:`a
    case where counting in the template would be better
    <overuse_of_count_and_exists>`.

    Secondly, there are other options to consider: in a real-life case, ``{{
    my_bicycles.count }}``, which invokes the ``QuerySet`` ``count()`` method
    directly from the template, might be the most appropriate choice.

Caching
=======

Often it is expensive (that is, resource-hungry and slow) to compute a value,
so there can be huge benefit in saving the value to a quickly accessible cache,
ready for the next time it's required.

It's a sufficiently significant and powerful technique that Django includes a
comprehensive caching framework, as well as other smaller pieces of caching
functionality.

:doc:`The caching framework </topics/cache>`
--------------------------------------------

Django's :doc:`caching framework </topics/cache>` offers very significant
opportunities for performance gains, by saving dynamic content so that it
doesn't need to be calculated for each request.

For convenience, Django offers different levels of cache granularity: you can
cache the output of specific views, or only the pieces that are difficult to
produce, or even an entire site.

Implementing caching should not be regarded as an alternative to improving code
that's performing poorly because it has been written badly. It's one of the
final steps towards producing well-performing code, not a shortcut.

:class:`~django.utils.functional.cached_property`
-------------------------------------------------

It's common to have to call a class instances's method more than once. If
that function is expensive, then doing so can be wasteful.

Using the ``@cached_property`` decorator saves the value returned by a
property; the next time the function is called on that instance, it will return
the saved value rather than re-computing it. Note that this only works on
methods that take ``self`` as their only argument and that it changes the
method to a property.

Certain Django components also have their own caching functionality; these are
discussed below in the sections related to those components.

Understanding laziness
======================

*Laziness* is a strategy complementary to caching. Caching avoids
recomputation by saving results; laziness delays computation until it's<commit_msg>Fixed #25727 -- Added a doc link to cached_property.<commit_after>============================
Performance and optimization
============================

This document provides an overview of techniques and tools that can help get
your Django code running more efficiently - faster, and using fewer system
resources.

Introduction
============

Generally one's first concern is to write code that *works*, whose logic
functions as required to produce the expected output. Sometimes, however, this
will not be enough to make the code work as *efficiently* as one would like.

In this case, what's needed is something - and in practice, often a collection
of things - to improve the code's performance without, or only minimally,
affecting its behavior.

General approaches
==================

What are you optimizing *for*?
------------------------------

It's important to have a clear idea what you mean by 'performance'. There is
not just one metric of it.

Improved speed might be the most obvious aim for a program, but sometimes other
performance improvements might be sought, such as lower memory consumption or
fewer demands on the database or network.

Improvements in one area will often bring about improved performance in
another, but not always; sometimes one can even be at the expense of another.
For example, an improvement in a program's speed might cause it to use more
memory. Even worse, it can be self-defeating - if the speed improvement is so
memory-hungry that the system starts to run out of memory, you'll have done
more harm than good.

There are other trade-offs to bear in mind. Your own time is a valuable
resource, more precious than CPU time. Some improvements might be too difficult
to be worth implementing, or might affect the portability or maintainability of
the code. Not all performance improvements are worth the effort.

So, you need to know what performance improvements you are aiming for, and you
also need to know that you have a good reason for aiming in that direction -
and for that you need:

Performance benchmarking
------------------------

It's no good just guessing or assuming where the inefficiencies lie in your
code.

Django tools
^^^^^^^^^^^^

`django-debug-toolbar
<https://github.com/django-debug-toolbar/django-debug-toolbar/>`_ is a very
handy tool that provides insights into what your code is doing and how much
time it spends doing it. In particular it can show you all the SQL queries your
page is generating, and how long each one has taken.

Third-party panels are also available for the toolbar, that can (for example)
report on cache performance and template rendering times.

Third-party services
^^^^^^^^^^^^^^^^^^^^

There are a number of free services that will analyze and report on the
performance of your site's pages from the perspective of a remote HTTP client,
in effect simulating the experience of an actual user.

These can't report on the internals of your code, but can provide a useful
insight into your site's overall performance, including aspects that can't be
adequately measured from within Django environment. Examples include:

* `Yahoo's Yslow <http://yslow.org/>`_
* `Google PageSpeed <https://developers.google.com/speed/pagespeed/>`_

There are also several paid-for services that perform a similar analysis,
including some that are Django-aware and can integrate with your codebase to
profile its performance far more comprehensively.

Get things right from the start
-------------------------------

Some work in optimization involves tackling performance shortcomings, but some
of the work can simply be built in to what you'd do anyway, as part of the good
practices you should adopt even before you start thinking about improving
performance.

In this respect Python is an excellent language to work with, because solutions
that look elegant and feel right usually are the best performing ones. As with
most skills, learning what "looks right" takes practice, but one of the most
useful guidelines is:

Work at the appropriate level
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Django offers many different ways of approaching things, but just because it's
possible to do something in a certain way doesn't mean that it's the most
appropriate way to do it. For example, you might find that you could calculate
the same thing - the number of items in a collection, perhaps - in a
``QuerySet``, in Python, or in a template.

However, it will almost always be faster to do this work at lower rather than
higher levels. At higher levels the system has to deal with objects through
multiple levels of abstraction and layers of machinery.

That is, the database can typically do things faster than Python can, which can
do them faster than the template language can::

    # QuerySet operation on the database
    # fast, because that's what databases are good at
    my_bicycles.count()

    # counting Python objects
    # slower, because it requires a database query anyway, and processing
    # of the Python objects
    len(my_bicycles)

    # Django template filter
    # slower still, because it will have to count them in Python anyway,
    # and because of template language overheads
    {{ my_bicycles|length }}

Generally speaking, the most appropriate level for the job is the lowest-level
one that it is comfortable to code for.

.. note::

    The example above is merely illustrative.

    Firstly, in a real-life case you need to consider what is happening before
    and after your count to work out what's an optimal way of doing it *in that
    particular context*. The database optimization documents describes :ref:`a
    case where counting in the template would be better
    <overuse_of_count_and_exists>`.

    Secondly, there are other options to consider: in a real-life case, ``{{
    my_bicycles.count }}``, which invokes the ``QuerySet`` ``count()`` method
    directly from the template, might be the most appropriate choice.

Caching
=======

Often it is expensive (that is, resource-hungry and slow) to compute a value,
so there can be huge benefit in saving the value to a quickly accessible cache,
ready for the next time it's required.

It's a sufficiently significant and powerful technique that Django includes a
comprehensive caching framework, as well as other smaller pieces of caching
functionality.

:doc:`The caching framework </topics/cache>`
--------------------------------------------

Django's :doc:`caching framework </topics/cache>` offers very significant
opportunities for performance gains, by saving dynamic content so that it
doesn't need to be calculated for each request.

For convenience, Django offers different levels of cache granularity: you can
cache the output of specific views, or only the pieces that are difficult to
produce, or even an entire site.

Implementing caching should not be regarded as an alternative to improving code
that's performing poorly because it has been written badly. It's one of the
final steps towards producing well-performing code, not a shortcut.

:class:`~django.utils.functional.cached_property`
-------------------------------------------------

It's common to have to call a class instances's method more than once. If
that function is expensive, then doing so can be wasteful.

Using the :class:`~django.utils.functional.cached_property` decorator saves the
value returned by a property; the next time the function is called on that
instance, it will return the saved value rather than re-computing it. Note that
this only works on methods that take ``self`` as their only argument and that
it changes the method to a property.

Certain Django components also have their own caching functionality; these are
discussed below in the sections related to those components.

Understanding laziness
======================

*Laziness* is a strategy complementary to caching. Caching avoids
recomputation by saving results; laziness delays computation until it's
{'fraction_non_alphanumeric': 0.0840607210626186, 'fraction_numerical': 0.00031625553447185326, 'mean_word_length': 3.9987353778058803, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7191497', 'n_tokens_mistral': 3784, 'n_tokens_neox': 3594, 'n_words': 2276}