From a9cf476e013f45dc0cbe671c36c6ae121bc5f411 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Thu, 17 Oct 2024 13:06:48 +0100 Subject: [PATCH] Add the starts of some documentation Still lots more to write --- docs/ref/index.txt | 1 + docs/ref/settings.txt | 62 +++++++++ docs/ref/signals.txt | 31 +++++ docs/ref/tasks.txt | 296 ++++++++++++++++++++++++++++++++++++++++++ docs/topics/index.txt | 1 + docs/topics/tasks.txt | 135 +++++++++++++++++++ 6 files changed, 526 insertions(+) create mode 100644 docs/ref/tasks.txt create mode 100644 docs/topics/tasks.txt diff --git a/docs/ref/index.txt b/docs/ref/index.txt index 8fc99ada81..6fb9f72121 100644 --- a/docs/ref/index.txt +++ b/docs/ref/index.txt @@ -25,6 +25,7 @@ API Reference schema-editor settings signals + tasks templates/index template-response unicode diff --git a/docs/ref/settings.txt b/docs/ref/settings.txt index e3a0f6d32a..0b237049c7 100644 --- a/docs/ref/settings.txt +++ b/docs/ref/settings.txt @@ -2660,6 +2660,68 @@ backend definition in :setting:`STORAGES`. Defining this setting overrides the default value and is *not* merged with it. +.. setting:: TASKS + +``TASKS`` +---------- + +Default:: + + { + "default": { + "BACKEND": "django.tasks.backends.immediate.ImmediateBackend", + } + } + +A dictionary containing the settings for all task backends to be used with +Django. It is a nested dictionary whose contents maps backend aliases to a +dictionary containing the options for each backend. + +The :setting:`TASKS` setting must configure a ``default`` backend; any number +of additional backends may also be specified. Depending on which backend is used, +other options may be required. The following options are available as standard: + +.. setting:: TASKS-BACKEND + +``BACKEND`` +~~~~~~~~~~~ + +Default: ``''`` (Empty string) + +The task backend to use. The built-in backends are: + +* ``'django.tasks.backends.dummy.DummyBackend'`` +* ``'django.tasks.backends.immediate.ImmediateBackend'`` + +You can use a backend that doesn't ship with Django by setting +:setting:`BACKEND ` to a fully-qualified path of a backend +class (i.e. ``mypackage.backends.whatever.WhateverBackend``). + +.. setting:: TASKS-ENQUEUE_ON_COMMIT + +``ENQUEUE_ON_COMMIT`` +~~~~~~~~~~~~~~~~~~~~~ + +Default: ``True`` + +Whether a task should be enqueued at the end of the current transaction (if there +is one) commits successfully, rather than enqueueing immediately. + +This can also be configured on a per-task basis. + +.. setting:: TASKS-QUEUES + +``QUEUES`` +~~~~~~~~~~~~~~~~~~~~~ + +Default: ``["default"]`` + +Specify the queue names supported by the backend. This can be used to ensure +tasks aren't enqueued to queues which don't exist. + +To disable queue name validation, set :setting:`QUEUES ` to an +empty list (``[]``). + .. setting:: TEMPLATES ``TEMPLATES`` diff --git a/docs/ref/signals.txt b/docs/ref/signals.txt index 953b18c1f6..ec4df1a71b 100644 --- a/docs/ref/signals.txt +++ b/docs/ref/signals.txt @@ -703,3 +703,34 @@ Arguments sent with this signal: The database connection that was opened. This can be used in a multiple-database configuration to differentiate connection signals from different databases. + +Tasks signals +============= + +Signals sent by the :doc:`tasks ` framework. + +.. data:: django.tasks.signals.task_enqueued + +Sent once a task has been enqueued. If +:attr:`django.tasks.task.Task.enqueue_on_commit` is set, the signal is only sent +once the transaction commits successfully. + +Arguments that are sent with this signal: + +``sender`` + The backend class which the task was enqueued on to. + +``task_result`` + The enqueued :class:`TaskResult `. + +.. data:: django.tasks.signals.task_finished + +Sent once a task has finished executing, successfully or otherwise. + +Arguments that are sent with this signal: + +``sender`` + The backend class which the task was enqueued on to. + +``task_result`` + The finished :class:`TaskResult `. diff --git a/docs/ref/tasks.txt b/docs/ref/tasks.txt new file mode 100644 index 0000000000..6ca890b15c --- /dev/null +++ b/docs/ref/tasks.txt @@ -0,0 +1,296 @@ +===== +Tasks +===== + +.. module:: django.tasks + :synopsis: Django's built-in background task system. + +Backends +======== + +Base backend +------------ + +.. module:: django.tasks.backends.base + +.. class:: BaseTaskBackend + +``BaseTaskBackend`` is the parent class for all task backends. + +.. method:: BaseTaskBackend.get_result(result_id) + + Retrieve a result by its id. If the result does not exist, + :exc:`ResultDoesNotExist ` + is raised. + + If the backend does not support ``get_result``, :exc:`NotImplementedError` + is raised. + +.. method:: BaseTaskBackend.aget_result(result_id) + + The ``async`` variant of :meth:`BaseTaskBackend.get_result`. + +Introspection +~~~~~~~~~~~~~ + +Introspection allows for identifying the supported functionality of a +backend, and potentially changing behavior accordingly. + +.. attribute:: BaseTaskBackend.supports_defer + + Whether the backend supports enqueueing tasks to be + executed after a specific time using the ``run_after`` attribute. + +.. attribute:: BaseTaskBackend.supports_async_task + + Whether the backend supports enqueueing async functions (coroutines). + +.. attribute:: BaseTaskBackend.supports_get_result + + Whether the backend supports retrieving task results from another thread + after they have been enqueued. + +Available backends +------------------ + +Immediate backend +~~~~~~~~~~~~~~~~~ + +.. module:: django.tasks.backends.immediate + +.. class:: ImmediateBackend + +The immediate backend executes tasks immediately, rather than in the background. + +Dummy backend +~~~~~~~~~~~~~ + +.. module:: django.tasks.backends.dummy + +.. class:: DummyBackend + +The dummy backend doesn't execute enqueued tasks at all, instead storing results +for later use. + +.. attribute:: DummyBackend.results + + A list of results for the enqueued tasks, in the order they were enqueued. + +.. method:: DummyBackend.clear + + Clears the list of stored results. + +Task definition +=============== + +.. module:: django.tasks.task + +.. function:: task + +A decorator defining a :class:`Task`:: + + from django.tasks import task + + + @task + def calculate_meaning_of_life() -> int: + return 42 + +In the above example, ``calculate_meaning_of_life`` is a :class:`Task` instance. + +``task`` accepts a number of arguments, which are used as-is for the ``Task``: + +* :attr:`priority ` +* :attr:`backend ` +* :attr:`queue_name ` +* :attr:`enqueue_on_commit ` + +If the task is not valid, according to the backend, +:exc:`django.tasks.exceptions.InvalidTaskError` is raised. + +.. class:: Task + +An class representing a task to be run in the background. Tasks should be +defined using the :func:`task` decorator. + +Attributes of ``Task`` cannot be modified. + +.. attribute:: Task.priority + + The priority of the task. Priorities must be between -100 and 100. + + By default, tasks are enqueued with a priority of 0. + +.. attribute:: Task.backend + + The alias of the backend the task should be enqueued to. + +.. attribute:: Task.queue_name + + The name of the queue the task will be enqueued on to. Defaults to + ``"default"``. + + This must match a queue defined in :setting:`QUEUES `. + +.. attribute:: Task.run_after + + The earliest time the task will be executed, or ``None`` to have no time set. + +.. attribute:: Task.enqueue_on_commit + + Whether the task should be enqueued when the transaction commits successfully, + or immediately. + + By default, the behavior of the backend's + :setting:`ENQUEUE_ON_COMMIT ` is used. + +.. attribute:: Task.name + + An identifier for a task. Potentially useful for debugging. + + A task's name is not necessarily unique. + +.. method:: Task.using() + + Create a new task with modified defaults. The existing task is left unchanged. + + ``using`` allows modifying the following attributes: + + * :attr:`priority ` + * :attr:`backend ` + * :attr:`queue_name ` + * :attr:`run_after ` + + ``run_after`` may also be provided as a :class:`timedelta `, which + is used relative to the current time (when ``using`` is called). + +.. method:: Task.enqueue(*args, **kwargs) + + Enqueue the task for later execution. + + Arguments and keyword arguments are passed to the task's function as-is. + + If the task is not valid, according to the backend, + :exc:`django.tasks.exceptions.InvalidTaskError` is raised. + +.. method:: Task.aenqueue(*args, **kwargs) + + The ``async`` variant of :meth:`enqueue `. + +.. method:: Task.get_result(result_id) + + Retrieve a result by its id. If the result does not exist, or is not the same + type as the current task, + :exc:`ResultDoesNotExist ` + is raised. + + If the backend does not support ``get_result``, :exc:`NotImplementedError` + is raised. + +.. method:: Task.aget_result(*args, **kwargs) + + The ``async`` variant of :meth:`get_result `. + +Task results +============ + +.. class:: ResultStatus + + An Enum representing the status of a :class:`TaskResult`. + +.. attribute:: ResultStatus.NEW + + The :class:`Task` has not been run. + +.. attribute:: ResultStatus.RUNNING + + The :class:`Task` is currently being executed. + +.. attribute:: ResultStatus.FAILED + + The :class:`Task` raised an exception during execution, or was unable + to start. + +.. attribute:: ResultStatus.COMPLETE + + The :class:`Task` has finished executing successfully. + +.. class:: TaskResult + +The ``TaskResult`` stores the information about a specific execution of a +:class:`Task`. + +Attributes of ``TaskResult`` cannot be modified. + +.. attribute:: TaskResult.task + + The :class:`Task` the result was enqueued for. + +.. attribute:: TaskResult.id + + A unique identifier for the result, which can be passed to + :meth:`Task.get_result`. + + The id must be a UUID4, represented as a dashed string. + +.. attribute:: TaskResult.status + + The :class:`status ` of the result. + +.. attribute:: TaskResult.enqueued_at + + The time when the task was enqueued. + + If :attr:`Task.enqueue_on_commit` was set, this is the time the transaction + completed. + +.. attribute:: TaskResult.started_at + + The time when the task began execution. + +.. attribute:: TaskResult.finished_at + + The time when the task finished execution, whether it failed or completed. + +.. attribute:: TaskResult.backend + + The backend the result is from. + +.. attribute:: TaskResult.exception + + The exception raised when executing the task, or ``None`` if no exception + was raised. + +.. attribute:: TaskResult.traceback + + The exception traceback from the raised exception when the task failed. + +.. attribute:: TaskResult.return_value + + The return value from the task function. + + If the task has not finished yet, or failed, :exc:`ValueError` is raised. + +.. method:: TaskResult.refresh + + Refresh the result's attributes from the queue store. + +.. method:: TaskResult.arefresh + + The ``async`` variant of :meth:`TaskResult.refresh`. + + +Exceptions +========== + +.. module:: django.tasks.exceptions + +.. exception:: ResultDoesNotExist + + Raised by :meth:`get_result ` + when the provided ``result_id`` does not exist. + +.. exception:: InvalidTaskError + + Raised when the :class:`Task ` attempting to be + enqueued is invalid. diff --git a/docs/topics/index.txt b/docs/topics/index.txt index ffb9fa9d92..408946f9e0 100644 --- a/docs/topics/index.txt +++ b/docs/topics/index.txt @@ -32,3 +32,4 @@ Introductions to all the key parts of Django you'll need to know: checks external-packages async + tasks diff --git a/docs/topics/tasks.txt b/docs/topics/tasks.txt new file mode 100644 index 0000000000..184eb1772c --- /dev/null +++ b/docs/topics/tasks.txt @@ -0,0 +1,135 @@ +======================== +Django's tasks framework +======================== + +For a web application, there's often more than just turning HTTP requests into +HTTP responses. For some functionality, it may be beneficial to run code outside +of the request-response cycle. + +That's where background tasks come in. + +Background tasks can offload complexity outside of the request-response cycle, +to be run somewhere else, potentially at a later date. This keeps requests fast, +latency down, and improves the user's experience. + +Background task fundamentals +============================ + +When work needs to be done in the background, Django creates a Task, which is +stored in the Queue Store. This task contains all of the metadata needed to +execute the task, as well as a unique identifier for Django to retrieve the +result later. + +Outside of Django, a Worker looks at the Queue Store for new Tasks to run. When +a new Task is added, the Worker claims the task, executes it, and saves the +status and result back to the Queue Store. + +.. _configuring-a-task-backend: + +Configuring a Task backend +========================== + +Background tasks require some work to set up. + +Different task backends have different characteristics and configuration options, +which may impact the performance and reliability of your application. + +Task backends are configured using the :setting:`TASKS` setting in your settings +file. + +.. _immediate-task-backend: + +Immediate execution +------------------- + +This is the default backend if another is not specified in your settings file. +The immediate backend runs enqueued tasks immediately, rather than in the +background. This allows background task functionality to be slowly added +to an application, before the required infrastructure is available. + +.. warning:: + + When :setting:`ENQUEUE_ON_COMMIT ` is ``False``, + the task will be executed within the same transaction it was enqueued in. + + This may lead to unexpected behavior changes when changing backend in future. + +To use it, set :setting:`BACKEND ` to +``"django.tasks.backends.immediate.ImmediateBackend"``:: + + TASKS = {"default": {"BACKEND": "django.tasks.backends.immediate.ImmediateBackend"}} + +.. _dummy-task-backend: + +Dummy backend +------------- + +The dummy backend doesn't execute enqueued tasks at all, instead storing results +for later use. + +This backend is not intended for use in production - it is provided as a +convenience that can be used during development and testing. + +To use it, set :setting:`BACKEND ` to +``"django.tasks.backends.dummy.DummyBackend"``:: + + TASKS = {"default": {"BACKEND": "django.tasks.backends.dummy.DummyBackend"}} + +The results for enqueued tasks can be retrieved from the backend's +:attr:`results ` attribute:: + + from django.tasks import default_task_backend + + my_task.enqueue() + + assert len(default_task_backend.results) == 1 + +Stored results can be cleared using the +:meth:`clear ` method:: + + default_task_backend.clear() + + assert len(default_task_backend.results) == 0 + +Using a custom backend +---------------------- + +While Django includes support for a number of task backends out-of-the-box, +sometimes you might want to customize the the task backend. To use an external +task backend with Django, use the Python import path as the +:setting:`BACKEND ` of the :setting:`TASKS` setting, like so:: + + TASKS = { + "default": { + "BACKEND": "path.to.backend", + } + } + +If you're building your own backend, you can use the standard task backends +as reference implementations. You'll find the code in the +:source:`django/tasks/backends/` directory of the Django source. + +Backend arguments +----------------- + +Each backend can be given additional arguments to control its behavior. +These arguments are provided as additional keys in the :setting:`TASKS` setting. +Valid arguments are as follows: + +* :setting:`ENQUEUE_ON_COMMIT `: Whether a task should + be enqueued at the end of the current transaction (if there + is one) commits successfully, rather than enqueueing immediately. This argument + defaults to ``True``. + +* :setting:`QUEUES `: Restrict the queue names a task may be + enqueued to. By default, only the ``"default"`` queue is allowed. Queue name + validation can be disabled by setting this to an empty list. + +Asynchronous support +-------------------- + +Django has developing support for asynchronous task backends. + +``django.tasks.backends.base.BaseTaskBackend`` has async variants of all base +methods. By convention, the asynchronous versions of all methods are prefixed +with ``a``. The arguments for both variants are the same.