django/docs/testing.txt

===========================
Testing Django applications
===========================

Automated testing is an extremely useful bug-killing tool for the modern
Web developer. You can use a collection of tests -- a **test suite** -- to
solve, or avoid, a number of problems:

    * When you're writing new code, you can use tests to validate your code
      works as expected.

    * When you're refactoring or modifying old code, you can use tests to
      ensure your changes haven't affected your application's behavior
      unexpectedly.

Testing a Web application is a complex task, because a Web application is made
of several layers of logic -- from HTTP-level request handling, to form
validation and processing, to template rendering. With Django's test-execution
framework and assorted utilities, you can simulate requests, insert test data,
inspect your application's output and generally verify your code is doing what
it should be doing.

The best part is, it's really easy.

.. admonition:: Note

    This testing framework is currently under development. It may change
    slightly before the next official Django release.

    (That's *no* excuse not to write tests, though!)

Writing tests
=============

Tests in Django come in two forms: doctests and unit tests.

Writing doctests
----------------

Doctests use Python's standard doctest_ module, which searches for tests in
your docstrings. Django's test runner looks for doctests in your ``models.py``
file, and executes any that it finds. Django will also search for a file
called ``tests.py`` in the application directory (i.e., the directory that
holds ``models.py``). If a ``tests.py`` is found, it will also be searched
for doctests.

.. admonition:: What's a **docstring**?

    A good explanation of docstrings (and some guidlines for using them
    effectively) can be found in :PEP:`257`:

        A docstring is a string literal that occurs as the first statement in
        a module, function, class, or method definition.  Such a docstring
        becomes the ``__doc__`` special attribute of that object.

    Since tests often make great documentation, doctest lets you put your
    tests directly in your docstrings.

You can put doctest strings on any object in your ``models.py``, but it's
common practice to put application-level doctests in the module docstring, and
model-level doctests in the docstring for each model.

For example::

    from django.db import models

    class Animal(models.Model):
        """
        An animal that knows how to make noise

        # Create some animals
        >>> lion = Animal.objects.create(name="lion", sound="roar")
        >>> cat = Animal.objects.create(name="cat", sound="meow")

        # Make 'em speak
        >>> lion.speak()
        'The lion says "roar"'
        >>> cat.speak()
        'The cat says "meow"'
        """

        name = models.CharField(maxlength=20)
        sound = models.CharField(maxlength=20)

        def speak(self):
            return 'The %s says "%s"' % (self.name, self.sound)

When you `run your tests`_, the test utility will find this docstring, notice
that portions of it look like an interactive Python session, and execute those
lines while checking that the results match.

For more details about how doctest works, see the `standard library
documentation for doctest`_

.. _doctest: http://docs.python.org/lib/module-doctest.html
.. _standard library documentation for doctest: doctest_

Writing unittests
-----------------

Like doctests, Django's unit tests use a standard library module: unittest_.
As with doctests, Django's test runner looks for any unit test cases defined
in ``models.py``, or in a ``tests.py`` file stored in the application
directory.

An equivalent unittest test case for the above example would look like::

    import unittest
    from myapp.models import Animal

    class AnimalTestCase(unittest.TestCase):

        def setUp(self):
            self.lion = Animal.objects.create(name="lion", sound="roar")
            self.cat = Animal.objects.create(name="cat", sound="meow")

        def testSpeaking(self):
            self.assertEquals(self.lion.speak(), 'The lion says "roar"')
            self.assertEquals(self.cat.speak(), 'The cat says "meow"')

When you `run your tests`_, the default behavior of the test utility is
to find all the test cases (that is, subclasses of ``unittest.TestCase``)
in ``models.py`` and ``tests.py``, automatically build a test suite out of
those test cases, and run that suite.

**New in Django development version**

However, if you define a method called ``suite()`` in either ``models.py`` or
``tests.py``, that method will be used to construct the test suite for that
module. This follows the `suggested organization`_ for unit tests. See the
Python documentation for more details on how to construct a complex test
suite.

For more details about ``unittest``, see the `standard library unittest
documentation`_.

.. _unittest: http://docs.python.org/lib/module-unittest.html
.. _standard library unittest documentation: unittest_
.. _run your tests: `Running tests`_
.. _suggested organization: http://docs.python.org/lib/organizing-tests.html

Which should I use?
-------------------

Choosing a test framework is often contentious, so Django simply supports
both of the standard Python test frameworks. Choosing one is up to each
developer's personal tastes; each is supported equally. Since each test
system has different benefits, the best approach is probably to use both
together, picking the test system to match the type of tests you need to
write.

For developers new to testing, however, this choice can seem
confusing, so here are a few key differences to help you decide whether
doctests or unit tests are right for you.

If you've been using Python for a while, ``doctest`` will probably feel more
"pythonic". It's designed to make writing tests as easy as possible, so
there's no overhead of writing classes or methods; you simply put tests in
docstrings. This gives the added advantage of giving your modules automatic
documentation -- well-written doctests can kill both the documentation and the
testing bird with a single stone.

For developers just getting started with testing, using doctests will probably
get you started faster.

The ``unittest`` framework will probably feel very familiar to developers
coming from Java.  Since ``unittest`` is inspired by Java's JUnit, if
you've used testing frameworks in other languages that similarly were
inspired by JUnit, ``unittest`` should also feel pretty familiar.

Since ``unittest`` is organized around classes and methods, if you need
to write a bunch of tests that all share similar code, you can easily use
subclass to abstract common tasks; this makes test code shorter and cleaner.
There's also support for explicit setup and/or cleanup routines, which give
you a high level of control over the environment your test cases run in.

Again, remember that you can use both systems side-by-side (even in the same
app). In the end, most projects will eventually end up using both; each shines
in different circumstances.

Testing Tools
=============

To assist in testing various features of your application, Django provides
tools that can be used to establish tests and test conditions.

* `Test Client`_
* `TestCase`_
* `E-mail services`_

Test Client
-----------

The Test Client is a simple dummy browser. It allows you to simulate
GET and POST requests on a URL, and observe the response that is received.
This allows you to test that the correct view is executed for a given URL,
and that the view constructs the correct response.

As the response is generated, the Test Client gathers details on the
Template and Context objects that were used to generate the response. These
Templates and Contexts are then provided as part of the response, and can be
used as test conditions.

.. admonition:: Test Client vs Browser Automation?

    The Test Client is not intended as a replacement for Twill_, Selenium_,
    or other browser automation frameworks - it is intended to allow
    testing of the contexts and templates produced by a view,
    rather than the HTML rendered to the end-user.

    A comprehensive test suite should use a combination of both: Test Client
    tests to establish that the correct view is being called and that
    the view is collecting the correct context data, and Browser Automation
    tests to check that user interface behaves as expected.

.. _Twill: http://twill.idyll.org/
.. _Selenium: http://www.openqa.org/selenium/

Making requests
~~~~~~~~~~~~~~~

Creating an instance of ``Client`` (``django.test.client.Client``) requires
no arguments at time of construction. Once constructed, the following methods
can be invoked on the ``Client`` instance.

``get(path, data={})``
    Make a GET request on the provided ``path``. The key-value pairs in the
    data dictionary will be used to create a GET data payload. For example::

        c = Client()
        c.get('/customers/details/', {'name':'fred', 'age':7})

    will result in the evaluation of a GET request equivalent to::

        http://yoursite.com/customers/details/?name=fred&age=7

``post(path, data={}, content_type=MULTIPART_CONTENT)``
    Make a POST request on the provided ``path``. If you provide a content type
    (e.g., ``text/xml`` for an XML payload), the contents of ``data`` will be
    sent as-is in the POST request, using the content type in the HTTP
    ``Content-Type`` header.

    If you do not provide a value for ``content_type``, the values in
    ``data`` will be transmitted with a content type of ``multipart/form-data``.
    The key-value pairs in the data dictionary will be encoded as a multipart
    message and used to create the POST data payload.

    To submit multiple values for a given key (for example, to specify
    the selections for a multiple selection list), provide the values as a
    list or tuple for the required key. For example, a data dictionary of
    ``{'choices': ('a','b','d')}`` would submit three selected rows for the
    field named ``choices``.

    Submitting files is a special case. To POST a file, you need only
    provide the file field name as a key, and a file handle to the file you wish to
    upload as a value. The Test Client will populate the two POST fields (i.e.,
    ``field`` and ``field_file``) required by Django's FileField. For example::

        c = Client()
        f = open('wishlist.doc')
        c.post('/customers/wishes/', {'name':'fred', 'attachment':f})
        f.close()

    will result in the evaluation of a POST request on ``/customers/wishes/``,
    with a POST dictionary that contains ``name``, ``attachment`` (containing the
    file name), and ``attachment_file`` (containing the file data). Note that you
    need to manually close the file after it has been provided to the POST.

``login(**credentials)``
    **New in Django development version**

    On a production site, it is likely that some views will be protected from
    anonymous access through the use of the @login_required decorator, or some
    other login checking mechanism. The ``login()`` method can be used to
    simulate the effect of a user logging into the site. As a result of calling
    this method, the Client will have all the cookies and session data required
    to pass any login-based tests that may form part of a view.

    In most cases, the ``credentials`` required by this method are the username
    and password of the user that wants to log in, provided as keyword
    arguments::

        c = Client()
        c.login(username='fred', password='secret')
        # Now you can access a login protected view

    If you are using a different authentication backend, this method may
    require different credentials.

    ``login()`` returns ``True`` if it the credentials were accepted and login
    was successful.

    Note that since the test suite will be executed using the test database,
    which contains no users by default. As a result, logins that are valid
    on your production site will not work under test conditions. You will
    need to create users as part of the test suite (either manually, or
    using a test fixture).

Testing Responses
~~~~~~~~~~~~~~~~~

The ``get()`` and ``post()`` methods both return a Response object. This
Response object has the following properties that can be used for testing
purposes:

    ===============  ==========================================================
    Property         Description
    ===============  ==========================================================
    ``status_code``  The HTTP status of the response. See RFC2616_ for a
                     full list of HTTP status codes.

    ``content``      The body of the response. This is the final page
                     content as rendered by the view, or any error message
                     (such as the URL for a 302 redirect).

    ``template``     The Template instance that was used to render the final
                     content. Testing ``template.name`` can be particularly
                     useful; if the template was loaded from a file,
                     ``template.name`` will be the file name that was loaded.

                     If multiple templates were rendered, (e.g., if one
                     template includes another template),``template`` will
                     be a list of Template objects, in the order in which
                     they were rendered.

    ``context``      The Context that was used to render the template that
                     produced the response content.

                     As with ``template``, if multiple templates were rendered
                     ``context`` will be a list of Context objects, stored in
                     the order in which they were rendered.
    ===============  ==========================================================

.. _RFC2616: http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

Exceptions
~~~~~~~~~~

If you point the Test Client at a view that raises an exception, that exception
will be visible in the test case. You can then use a standard ``try...catch``
block, or ``unittest.TestCase.assertRaises()`` to test for exceptions.

The only exceptions that are not visible in a Test Case are ``Http404``,
``PermissionDenied`` and ``SystemExit``. Django catches these exceptions
internally and converts them into the appropriate HTTP responses codes.

Persistent state
~~~~~~~~~~~~~~~~

The Test Client is stateful; if a cookie is returned as part of a response,
that cookie is provided as part of the next request issued by that Client
instance. Expiry policies for these cookies are not followed; if you want
a cookie to expire, either delete it manually or create a new Client
instance (which will effectively delete all cookies).

There are two properties of the Test Client which are used to store persistent
state information. If necessary, these properties can be interrogated as
part of a test condition.

    ===============  ==========================================================
    Property         Description
    ===============  ==========================================================
    ``cookies``      A Python ``SimpleCookie`` object, containing the current
                     values of all the client cookies.

    ``session``      A dictionary-like object containing session information.
                     See the `session documentation`_ for full details.
    ===============  ==========================================================

.. _`session documentation`: ../sessions/

Example
~~~~~~~

The following is a simple unit test using the Test Client::

    import unittest
    from django.test.client import Client

    class SimpleTest(unittest.TestCase):
        def setUp(self):
            # Every test needs a client
            self.client = Client()
        def test_details(self):
            # Issue a GET request
            response = self.client.get('/customer/details/')

            # Check that the respose is 200 OK
            self.failUnlessEqual(response.status_code, 200)
            # Check that the rendered context contains 5 customers
            self.failUnlessEqual(len(response.context['customers']), 5)

TestCase
--------

Normal python unit tests extend a base class of ``unittest.testCase``.
Django provides an extension of this base class - ``django.test.TestCase``
- that provides some additional capabilities that can be useful for
testing web sites.

Moving from a normal unittest TestCase to a Django TestCase is easy - just
change the base class of your test from ``unittest.TestCase`` to
``django.test.TestCase``. All of the standard Python unit test facilities
will continue to be available, but they will be augmented with some useful
extra facilities.

Default Test Client
~~~~~~~~~~~~~~~~~~~
**New in Django development version**

Every test case in a ``django.test.TestCase`` instance has access to an
instance of a Django `Test Client`_. This Client can be accessed as
``self.client``. This client is recreated for each test.

Fixture loading
~~~~~~~~~~~~~~~

A test case for a database-backed website isn't much use if there isn't any
data in the database. To make it easy to put test data into the database,
Django provides a fixtures framework.

A *Fixture* is a collection of files that contain the serialized contents of
the database. Each fixture has a unique name; however, the files that
comprise the fixture can be distributed over multiple directories, in
multiple applications.

.. note::
    If you have synchronized a Django project, you have already experienced
    the use of one fixture -- the ``initial_data`` fixture. Every time you
    synchronize the database, Django installs the ``initial_data`` fixture.
    This provides a mechanism to populate a new database with any initial
    data (such as a default set of categories). Fixtures with other names
    can be installed manually using ``django-admin.py loaddata``.

However, for the purposes of unit testing, each test must be able to
guarantee the contents of the database at the start of each and every
test.

To define a fixture for a test, all you need to do is add a class
attribute to your test describing the fixtures you want the test to use.
For example, the test case from `Writing unittests`_ would
look like::

    from django.test import TestCase
    from myapp.models import Animal

    class AnimalTestCase(TestCase):
        fixtures = ['mammals.json', 'birds']

        def setUp(self):
            # test definitions as before

        def testFluffyAnimals(self):
            # A test that uses the fixtures

At the start of each test case, before ``setUp()`` is run, Django will
flush the database, returning the database the state it was in directly
after ``syncdb`` was called. Then, all the named fixtures are installed.
In this example, any JSON fixture called ``mammals``, and any fixture
named ``birds`` will be installed. See the documentation on
`loading fixtures`_ for more details on defining and installing fixtures.

.. _`loading fixtures`: ../django-admin/#loaddata-fixture-fixture

This flush/load procedure is repeated for each test in the test case, so you
can be certain that the outcome of a test will not be affected by
another test, or the order of test execution.

Emptying the test outbox
~~~~~~~~~~~~~~~~~~~~~~~~
**New in Django development version**

At the start of each test case, in addition to installing fixtures,
Django clears the contents of the test e-mail outbox.

For more detail on e-mail services during tests, see `E-mail services`_.

Assertions
~~~~~~~~~~
**New in Django development version**

Normal Python unit tests have a wide range of assertions, such as
``assertTrue`` and ``assertEquals`` that can be used to validate behavior.
``django.TestCase`` adds to these, providing some assertions
that can be useful in testing the behavior of web sites.

``assertContains(response, text, count=None, status_code=200)``
    Assert that a response indicates that a page could be retrieved and
    produced the nominated status code, and that ``text`` in the content
    of the response. If ``count`` is provided, ``text`` must occur exactly
    ``count`` times in the response.

``assertFormError(response, form, field, errors)``
    Assert that a field on a form raised the provided list of errors when
    rendered on the form.

    ``form`` is the name the form object was given in the template context.

    ``field`` is the name of the field on the form to check. If ``field``
    has a value of ``None``, non-field errors will be checked.

    ``errors`` is an error string, or a list of error strings, that are
    expected as a result of form validation.

``assertTemplateNotUsed(response, template_name)``
    Assert that the template with the given name was *not* used in rendering
    the response.

``assertRedirects(response, expected_path, status_code=302, target_status_code=200)``
    Assert that the response received produced the nominated status code,
    redirects the browser to the provided path, and that retrieving the provided
    path yields a response with the target status code.

``assertTemplateUsed(response, template_name)``
    Assert that the template with the given name was used in rendering the
    response.

E-mail services
---------------

**New in Django development version**

If your view makes use of the `Django e-mail services`_, you don't really
want e-mail to be sent every time you run a test using that view.

When the Django test framework is initialized, it transparently replaces the
normal `SMTPConnection`_ class with a dummy implementation that redirects all
e-mail to a dummy outbox. This outbox, stored as ``django.core.mail.outbox``,
is a simple list of all `EmailMessage`_ instances that have been sent.
For example, during test conditions, it would be possible to run the following
code::

    from django.core import mail

    # Send message
    mail.send_mail('Subject here', 'Here is the message.', 'from@example.com',
        ['to@example.com'], fail_silently=False)

    # One message has been sent
    self.assertEqual(len(mail.outbox), 1)
    # Subject of first message is correct
    self.assertEqual(mail.outbox[0].subject, 'Subject here')

The ``mail.outbox`` object does not exist under normal execution conditions.
The outbox is created during test setup, along with the dummy `SMTPConnection`_.
When the test framework is torn down, the standard `SMTPConnection`_ class
is restored, and the test outbox is destroyed.

As noted `previously`_, the test outbox is emptied at the start of every
test in a Django TestCase. To empty the outbox manually, assign the empty list
to mail.outbox::

    from django.core import mail

    # Empty the test outbox
    mail.outbox = []

.. _`Django e-mail services`: ../email/
.. _`SMTPConnection`: ../email/#the-emailmessage-and-smtpconnection-classes
.. _`EmailMessage`: ../email/#the-emailmessage-and-smtpconnection-classes
.. _`previously`: #emptying-the-test-outbox

Running tests
=============

Run your tests using your project's ``manage.py`` utility::

    $ ./manage.py test

If you only want to run tests for a particular application, add the
application name to the command line. For example, if your
``INSTALLED_APPS`` contains ``myproject.polls`` and ``myproject.animals``,
but you only want to run the animals unit tests, run::

    $ ./manage.py test animals

**New in Django development version:** If you use unit tests, you can be more
specific in the tests that are executed. To run a single test case in an
application (for example, the AnimalTestCase described previously), add the
name of the test case to the label on the command line::

    $ ./manage.py test animals.AnimalTestCase

**New in Django development version:** To run a single test method inside a
test case, add the name of the test method to the label::

    $ ./manage.py test animals.AnimalTestCase.testFluffyAnimals

When you run your tests, you'll see a bunch of text flow by as the test
database is created and models are initialized. This test database is
created from scratch every time you run your tests.

By default, the test database gets its name by prepending ``test_`` to
the database name specified by the ``DATABASE_NAME`` setting; all other
database settings will the same as they would be for the project normally.
If you wish to use a name other than the default for the test database,
you can use the ``TEST_DATABASE_NAME`` setting to provide a name.

**New in Django development version:** For fine-grained control over the
character encoding of your database, use the ``TEST_DATABASE_CHARSET`` setting.
If you're using MySQL, you can also use the ``TEST_DATABASE_COLLATION`` setting
to control the particular collation used by the test database. See the
settings_ documentation for details of these advanced settings.

.. _settings: ../settings/

The test database is created by the user in the ``DATABASE_USER`` setting.
This user needs to have sufficient privileges to create a new database on the
system.

Once the test database has been established, Django will run your tests.
If everything goes well, at the end you'll see::

    ----------------------------------------------------------------------
    Ran 22 tests in 0.221s

    OK

If there are test failures, however, you'll see full details about what tests
failed::

    ======================================================================
    FAIL: Doctest: ellington.core.throttle.models
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/dev/django/test/doctest.py", line 2153, in runTest
        raise self.failureException(self.format_failure(new.getvalue()))
    AssertionError: Failed doctest test for myapp.models
      File "/dev/myapp/models.py", line 0, in models

    ----------------------------------------------------------------------
    File "/dev/myapp/models.py", line 14, in myapp.models
    Failed example:
        throttle.check("actor A", "action one", limit=2, hours=1)
    Expected:
        True
    Got:
        False

    ----------------------------------------------------------------------
    Ran 2 tests in 0.048s

    FAILED (failures=1)

The return code for the script is the total number of failed and erroneous
tests. If all the tests pass, the return code is 0.

Regardless of whether the tests pass or fail, the test database is destroyed when
all the tests have been executed.

Using a different testing framework
===================================

Doctest and Unittest are not the only Python testing frameworks. While
Django doesn't provide explicit support these alternative frameworks,
it does provide a mechanism to allow you to invoke tests constructed for
an alternative framework as if they were normal Django tests.

When you run ``./manage.py test``, Django looks at the ``TEST_RUNNER``
setting to determine what to do. By default, ``TEST_RUNNER`` points to
``django.test.simple.run_tests``. This method defines the default Django
testing behavior. This behavior involves:

#. Performing global pre-test setup
#. Creating the test database
#. Running ``syncdb`` to install models and initial data into the test database
#. Looking for Unit Tests and Doctests in ``models.py`` and ``tests.py`` file for each installed application
#. Running the Unit Tests and Doctests that are found
#. Destroying the test database
#. Performing global post-test teardown

If you define your own test runner method and point ``TEST_RUNNER``
at that method, Django will execute your test runner whenever you run
``./manage.py test``. In this way, it is possible to use any test
framework that can be executed from Python code.

Defining a test runner
----------------------
By convention, a test runner should be called ``run_tests``; however, you
can call it anything you want. The only requirement is that it has the
same arguments as the Django test runner:

``run_tests(test_labels, verbosity=1, interactive=True, extra_tests=[])``
    
    **New in Django development version:** ``test_labels`` is a list of
    strings describing the tests to be run. A test label can take one of
    three forms:

        * ``app.TestCase.test_method`` - Run a single test method in a test case
        * ``app.TestCase`` - Run all the test methods in a test case
        * ``app`` - Search for and run all tests in the named application.

    If ``test_labels`` has a value of ``None``, the test runner should run
    search for tests in all the applications in ``INSTALLED_APPS``.

    Verbosity determines the amount of notification and debug information that
    will be printed to the console; ``0`` is no output, ``1`` is normal output,
    and ``2`` is verbose output.

    **New in Django development version:** If ``interactive`` is ``True``, the
    test suite may ask the user for instructions when the test suite is
    executed. An example of this behavior would be asking for permission to
    delete an existing test database. If ``interactive`` is ``False``, the
    test suite must be able to run without any manual intervention.

    ``extra_tests`` is a list of extra ``TestCase`` instances to add to the
    suite that is executed by the test runner. These extra tests are run
    in addition to those discovered in the modules listed in ``module_list``.

    This method should return the number of tests that failed.

Testing utilities
-----------------

To assist in the creation of your own test runner, Django provides
a number of utility methods in the ``django.test.utils`` module.

``setup_test_environment()``
    Performs any global pre-test setup, such as the installing the
    instrumentation of the template rendering system and setting up
    the dummy SMTPConnection.

``teardown_test_environment()``
    Performs any global post-test teardown, such as removing the instrumentation
    of the template rendering system and restoring normal e-mail services.

``create_test_db(verbosity=1, autoclobber=False)``
    Creates a new test database, and run ``syncdb`` against it.

    ``verbosity`` has the same behavior as in the test runner.

    ``Autoclobber`` describes the behavior that will occur if a database with
    the same name as the test database is discovered. If ``autoclobber`` is False,
    the user will be asked to approve destroying the existing database. ``sys.exit``
    is called if the user does not approve. If autoclobber is ``True``, the database
    will be destroyed without consulting the user.

    ``create_test_db()`` has the side effect of modifying
    ``settings.DATABASE_NAME`` to match the name of the test database.

``destroy_test_db(old_database_name, verbosity=1)``
    Destroys the database with the name ``settings.DATABASE_NAME`` matching,
    and restores the value of ``settings.DATABASE_NAME`` to the provided name.

    ``verbosity`` has the same behavior as in the test runner.