Fixed #20877 -- added a performance optimization guide

docs/index.txt

@@ -233,6 +233,14 @@ regions:
 * :doc:`"Local flavor" <topics/localflavor>`
 * :doc:`Time zones </topics/i18n/timezones>`
 
+Performance and optimization
+============================
+
+There are a variety of techniques and tools that can help get your code running
+more efficiently - faster, and using fewer system resources.
+
+* :doc:`Performance and optimization overview <topics/performance>`
+
 Python compatibility
 ====================
 

docs/ref/utils.txt

@@ -495,8 +495,8 @@ Atom1Feed
 
     For cases like this, use the ``django.utils.functional.allow_lazy()``
     decorator. It modifies the function so that *if* it's called with a lazy
-    translation as the first argument, the function evaluation is delayed until it
-    needs to be converted to a string.
+    translation as one of its arguments, the function evaluation is delayed
+    until it needs to be converted to a string.
 
     For example::
 

docs/topics/cache.txt

@@ -1162,22 +1162,6 @@ Example::
 
 .. _`Cache-Control spec`: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9
 
-Other optimizations
-===================
-
-Django comes with a few other pieces of middleware that can help optimize your
-site's performance:
-
-* ``django.middleware.http.ConditionalGetMiddleware`` adds support for
-  modern browsers to conditionally GET responses based on the ``ETag``
-  and ``Last-Modified`` headers.
-
-* :class:`django.middleware.gzip.GZipMiddleware` compresses responses for all
-  modern browsers, saving bandwidth and transfer time. Be warned, however,
-  that compression techniques like ``GZipMiddleware`` are subject to attacks.
-  See the warning in :class:`~django.middleware.gzip.GZipMiddleware` for
-  details.
-
 Order of MIDDLEWARE_CLASSES
 ===========================
 

docs/topics/db/optimization.txt

@@ -88,7 +88,8 @@ of parentheses, but will call callables automatically, hiding the above
 distinction.
 
 Be careful with your own custom properties - it is up to you to implement
-caching.
+caching when required, for example using the
+:class:`~django.utils.functional.cached_property` decorator.
 
 Use the ``with`` template tag
 -----------------------------
@@ -111,10 +112,11 @@ For instance:
 * At the most basic level, use :ref:`filter and exclude <queryset-api>` to do
   filtering in the database.
 
-* Use :class:`F expressions <django.db.models.F>` to do filtering
-  against other fields within the same model.
+* Use :class:`F expressions <django.db.models.F>` to filter
+  based on other fields within the same model.
 
-* Use :doc:`annotate to do aggregation in the database </topics/db/aggregation>`.
+* Use :doc:`annotate to do aggregation in the database
+  </topics/db/aggregation>`.
 
 If these aren't enough to generate the SQL you need:
 
@@ -233,6 +235,8 @@ queryset``.
 
 But:
 
+.. _overuse_of_count_and_exists:
+
 Don't overuse ``count()`` and ``exists()``
 ------------------------------------------
 

docs/topics/index.txt

@@ -26,6 +26,7 @@ Introductions to all the key parts of Django you'll need to know:
    pagination
    python3
    security
+   performance
    serialization
    settings
    signals

docs/topics/performance.txt

@@ -0,0 +1,384 @@
+============================
+Performance and optimization
+============================
+
+This document provides an overview of techniques and tools that can help get
+your Django code running more efficiently - faster, and using fewer system
+resources.
+
+Introduction
+============
+
+Generally one's first concern is to write code that *works*, whose logic
+functions as required to produce the expected output. Sometimes, however, this
+will not be enough to make the code work as *efficiently* as one would like.
+
+In this case, what's needed is something - and in practice, often a collection
+of things - to improve the code's performance without, or only minimally,
+affecting its behavior.
+
+General approaches
+==================
+
+What are you optimizing *for*?
+------------------------------
+
+It's important to have a clear idea what you mean by 'performance'. There is
+not just one metric of it.
+
+Improved speed might be the most obvious aim for a program, but sometimes other
+performance improvements might be sought, such as lower memory consumption or
+fewer demands on the database or network.
+
+Improvements in one area will often bring about improved performance in
+another, but not always; sometimes one can even be at the expense of another.
+For example, an improvement in a program's speed might cause it to use more
+memory. Even worse, it can be self-defeating - if the speed improvement is so
+memory-hungry that the system starts to run out of memory, you'll have done
+more harm than good.
+
+There are other trade-offs to bear in mind. Your own time is a valuable
+resource, more precious than CPU time. Some improvements might be too difficult
+to be worth implementing, or might affect the portability or maintainability of
+the code. Not all performance improvements are worth the effort.
+
+So, you need to know what performance improvements you are aiming for, and you
+also need to know that you have a good reason for aiming in that direction -
+and for that you need:
+
+Performance benchmarking
+------------------------
+
+It's no good just guessing or assuming where the inefficiencies lie in your
+code.
+
+Django tools
+^^^^^^^^^^^^
+
+`django-debug-toolbar
+<https://github.com/django-debug-toolbar/django-debug-toolbar/>`_ is a very
+handy tool that provides insights into what your code is doing and how much
+time it spends doing it. In particular it can show you all the SQL queries your
+page is generating, and how long each one has taken.
+
+Third-party panels are also available for the toolbar, that can (for example)
+report on cache performance and template rendering times.
+
+Third-party services
+^^^^^^^^^^^^^^^^^^^^
+
+There are a number of free services that will analyse and report on the
+performance of your site's pages from the perspective of a remote HTTP client,
+in effect simulating the experience of an actual user.
+
+These can't report on the internals of your code, but can provide a useful
+insight into your site's overall performance, including aspects that can't be
+adequately measured from within Django environment. Examples include:
+
+* `Yahoo's Yslow <http://developer.yahoo.com/yslow/>`_
+* `Google PageSpeed <https://developers.google.com/speed/pagespeed/>`_
+
+There are also several paid-for services that perform a similar analysis,
+including some that are Django-aware and can integrate with your codebase to
+profile its performance far more comprehensively.
+
+Get things right from the start
+-------------------------------
+
+Some work in optimization involves tackling performance shortcomings, but some
+of the work can simply be built in to what you'd do anyway, as part of the good
+practices you should adopt even before you start thinking about improving
+performance.
+
+In this respect Python is an excellent language to work with, because solutions
+that look elegant and feel right usually are the best performing ones. As with
+most skills, learning what "looks right" takes practice, but one of the most
+useful guidelines is:
+
+Work at the appropriate level
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Django offers many different ways of approaching things, but just because it's
+possible to do something in a certain way doesn't mean that it's the most
+appropriate way to do it. For example, you might find that you could calculate
+the same thing - the number of items in a collection, perhaps - in a
+``QuerySet``, in Python, or in a template.
+
+However, it will almost always be faster to do this work at lower rather than
+higher levels. At higher levels the system has to deal with objects through
+multiple levels of abstraction and layers of machinery.
+
+That is, the database can typically do things faster than Python can, which can
+do them faster than the template language can::
+
+    # QuerySet operation on the database
+    # fast, because that's what databases are good at
+    my_bicycles.count()
+
+    # counting Python objects
+    # slower, because it requires a database query anyway, and processing
+    # of the Python objects
+    len(my_bicycles)
+
+    # Django template filter
+    # slower still, because it will have to count them in Python anyway,
+    # and because of template language overheads
+    {{ my_bicycles|length }}
+
+Generally speaking, the most appropriate level for the job is the lowest-level
+one that it is comfortable to code for.
+
+.. note::
+
+    The example above is merely illustrative.
+
+    Firstly, in a real-life case you need to consider what is happening before
+    and after your count to work out what's an optimal way of doing it *in that
+    particular context*. The database optimization documents describes :ref:`a
+    case where counting in the template would be better
+    <overuse_of_count_and_exists>`.
+
+    Secondly, there are other options to consider: in a real-life case, ``{{
+    my_bicycles.count }}``, which invokes the ``QuerySet`` ``count()`` method
+    directly from the template, might be the most appropriate choice.
+
+Caching
+=======
+
+Often it is expensive (that is, resource-hungry and slow) to compute a value,
+so there can be huge benefit in saving the value to a quickly accessible cache,
+ready for the next time it's required.
+
+It's a sufficiently significant and powerful technique that Django includes a
+comprehensive caching framework, as well as numerous other opportunities to
+make use of caching.
+
+:doc:`The caching framework </topics/cache>`
+--------------------------------------------
+
+Django's :doc:`caching framework </topics/cache>` offers very significant
+opportunities for performance gains, by saving dynamic content so that it
+doesn't need to be calculated for each request.
+
+For convenience, Django offers different levels of cache granularity: you can
+cache the output of specific views, or only the pieces that are difficult to
+produce, or even an entire site.
+
+Implementing caching should not be regarded as an alternative to improving code
+that's performing poorly because it has been written badly. It's one of the
+final steps towards producing well-performing code, not a shortcut.
+
+Other opportunities for caching
+-------------------------------
+
+Beyond the caching framework, Django offers other smaller pieces of caching
+functionality.
+
+:class:`~django.utils.functional.cached_property`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+It's common to have to call a class instances's method more than once. If
+that function is expensive, then doing so can be wasteful.
+
+Using the ``@cached_property`` decorator saves the value returned by a
+property; the next time the function is called on that instance, it will return
+the saved value rather than re-computing it. Note that this only works on
+methods that take ``self`` as their only argument and that it changes the
+method to a property.
+
+:class:`~django.contrib.staticfiles.storage.CachedStaticFilesStorage`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+:class:`~django.contrib.staticfiles.storage.CachedStaticFilesStorage` appends a
+content-dependent tag to the filenames of :doc:`static files
+</ref/contrib/staticfiles>` to make it safe for browsers to cache them
+long-term without missing future changes - when a file changes, so will the
+tag, so browsers will reload the asset automatically.
+
+Understanding laziness
+======================
+
+*Laziness* is a strategy complementary to caching. Caching avoids
+recomputation by saving results; laziness delays computation until it's
+actually required.
+
+Laziness allows us to refer to things before they are instantiated, or even
+before it's possible to instantiate them. This has numerous uses.
+
+For example, :ref:`lazy translation <lazy-translations>` can be used before the
+target language is even known, because it doesn't take place until the
+translated string is actually required, such as in a rendered template.
+
+Laziness is also a way to save effort by trying to avoid work in the first
+place. That is, one aspect of laziness is not doing anything until it has to be
+done, because it may not turn out to be necessary after all. Laziness can
+therefore have performance implications, and the more expensive the work
+concerned, the more there is to gain through laziness.
+
+Python provides a number of tools for lazy evaluation, particularly through the
+:py:term:`generator` and :py:term:`generator expression` constructs. It's worth
+reading up on laziness in Python to discover opportunities for making use of
+lazy patterns in your code.
+
+Laziness in Django
+------------------
+
+Django is itself quite lazy. A good example of this can be found in the
+evaluation of ``QuerySets``. :ref:`QuerySets are lazy <querysets-are-lazy>`.
+Thus a ``QuerySet`` can be created, passed around and combined with other
+``QuerySets``, without actually incurring any trips to the database to fetch
+the items it describes. What gets passed around is the ``QuerySet`` object, not
+the collection of items that - eventually - will be required from the database.
+
+On the other hand, :ref:`certain operations will force the evaluation of a
+QuerySet <when-querysets-are-evaluated>`. Avoiding the premature evaluation of
+a ``QuerySet`` can save making an expensive and unnecessary trip to the
+database.
+
+Django also offers an :meth:`~django.utils.functional.allow_lazy` decorator.
+This allows a function that has been called with a lazy argument to behave
+lazily itself, only being evaluated when it needs to be. Thus the lazy argument
+- which could be an expensive one - will not be called upon for evaluation
+until it's strictly required.
+
+Databases
+=========
+
+:doc:`Database optimization </topics/db/optimization>`
+------------------------------------------------------
+
+Django’s database layer provides various ways to help developers get the best
+performance from their databases. The :doc:`database optimization documentation
+</topics/db/optimization>` gathers together links to the relevant
+documentation and adds various tips that outline the steps to take when
+attempting to optimize your database usage.
+
+Other database-related tips
+---------------------------
+
+Enabling :ref:`persistent-database-connections` can speed up connections to the
+database accounts for a significant part of the request processing time.
+
+This helps a lot on virtualized hosts with limited network performance, for example.
+
+HTTP performance
+================
+
+Middleware
+----------
+
+Django comes with a few helpful pieces of :doc:`middleware </ref/middleware>`
+that can help optimize your site's performance. They include:
+
+:class:`~django.middleware.http.ConditionalGetMiddleware`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Adds support for modern browsers to conditionally GET responses based on the
+``ETag`` and ``Last-Modified`` headers.
+
+:class:`~django.middleware.gzip.GZipMiddleware`
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Compresses responses for all modern browsers, saving bandwidth and transfer
+time. Note that GZipMiddleware is currently considered a security risk, and is
+vulnerable to attacks that nullify the protection provided by TLS/SSL. See the
+warning in :class:`~django.middleware.gzip.GZipMiddleware` for more information.
+
+Third-party HTTP tools
+----------------------
+
+There are numerous third-party Django tools and packages available, notably
+ones that are able to "minify" and compress HTML, CSS and JavaScript.
+
+Template performance
+====================
+
+Note that:
+
+* using ``{% block %}`` is faster than using ``{% include %}``
+* heavily-fragmented templates, assembled from many small pieces, can affect
+  performance
+
+The cached template loader
+--------------------------
+
+Enabling the :class:`cached template loader
+<django.template.loaders.cached.Loader>` often improves performance
+drastically, as it avoids compiling each template every time it needs to be
+rendered.
+
+Using different versions of available software
+==============================================
+
+It can sometimes be worth checking whether different and better-performing
+versions of the software that you're using are available.
+
+This may be helpful, but is unlikely to solve a serious performance problem.
+You won't usually gain performance advantages that are better than marginal.
+
+.. note::
+
+    It's worth repeating: **reaching for alternatives to software you're
+    already using is very rarely the answer to performance problems**.
+
+Newer is often - but not always - better
+----------------------------------------
+
+It's fairly rare for a new release of well-maintained software to be less
+efficient, but the maintainers can't anticipate every possible use-case - so
+while being aware that newer versions are likely to perform better, don't
+simply assume that they always will.
+
+This is true of Django itself. Successive releases have offered a number of
+improvements across the system, but you should still check the real-world
+performance of your application, because in some cases you may find that
+changes mean it performs worse rather than better.
+
+Newer versions of Python, and also of Python packages, will often perform
+better too - but measure, rather than assume.
+
+.. note::
+
+    Unless you've encountered an unusual performance problem in a particular
+    version, you'll generally find better features, reliability, and security
+    in a new release and that these benefits are far more significant than any
+    performance you might win or lose.
+
+Alternatives to Django's template language
+------------------------------------------
+
+For nearly all cases, Django's built-in template language is perfectly
+adequate. However, if the bottlenecks in your Django project seem to lie in the
+template system and you have exhausted other opportunities to remedy this, a
+third-party alternative may be the answer.
+
+`Jinja2 <http://jinja.pocoo.org/docs/>`_ can offer performance improvements,
+particularly when it comes to speed.
+
+Alternative template systems vary in the extent to which they share Django's
+templating language.
+
+.. note::
+
+    *If* you experience performance issues in templates, the first thing to do
+    is to understand exactly why. Using an alternative template system may
+    prove faster, but the same gains may also be available without going to
+    that trouble - for example, expensive processing and logic in your
+    templates could be done more efficiently in your views.
+
+Alternative software implementations
+------------------------------------
+
+It *may* be worth checking whether Python software you're using has been
+provided in a different implementation that can execute the same code faster.
+
+However, most Django performance problems in well-written code are typically
+not to be found at the Python execution level, but rather in inefficient
+database querying, caching, and templates (and if you're relying on
+poorly-written Python code, your performance problems are very unlikely to be
+solved by having it execute faster).
+
+Avoid using C implementations of Python libraries or non-standard Python
+implementations like `PyPy <http://pypy.org/>`_ in search of performance gains,
+unless you are sure they are appropriate for your application. Any gains are
+likely to be small, and compatibility issues are common.