From 725355c8b1257906d669a2bccc826501f0f7e436 Mon Sep 17 00:00:00 2001 From: Jake Howard Date: Fri, 13 Dec 2024 17:25:22 +0000 Subject: [PATCH] Various docs improvements --- docs/ref/tasks.txt | 79 ++++++++++++++++------------ docs/topics/tasks.txt | 117 ++++++++++++++++++++++++++---------------- 2 files changed, 120 insertions(+), 76 deletions(-) diff --git a/docs/ref/tasks.txt b/docs/ref/tasks.txt index 4dd0ae9dc9..7f6db7df2e 100644 --- a/docs/ref/tasks.txt +++ b/docs/ref/tasks.txt @@ -28,41 +28,41 @@ In the above example, ``calculate_meaning_of_life`` is a :class:`Task` instance. * :attr:`queue_name ` * :attr:`enqueue_on_commit ` -If the task is not valid, according to the backend, +If the Task is not valid, according to the backend, :exc:`django.tasks.exceptions.InvalidTaskError` is raised. .. class:: Task -An class representing a task to be run in the background. Tasks should be +An class representing a Task to be run in the background. Tasks should be defined using the :func:`task` decorator. Attributes of ``Task`` cannot be modified. .. attribute:: Task.priority - The priority of the task. Priorities must be between -100 and 100, where + The priority of the Task. Priorities must be between -100 and 100, where larger numbers are higher priority, and will be run sooner. - By default, tasks are enqueued with a priority of 0. + By default, Tasks are enqueued with a priority of 0. .. attribute:: Task.backend - The alias of the backend the task should be enqueued to. + The alias of the backend the Task should be enqueued to. .. attribute:: Task.queue_name - The name of the queue the task will be enqueued on to. Defaults to + The name of the queue the Task will be enqueued on to. Defaults to ``"default"``. This must match a queue defined in :setting:`QUEUES `. .. attribute:: Task.run_after - The earliest time the task will be executed, or ``None`` to have no time set. + The earliest time the Task will be executed, or ``None`` to have no time set. .. attribute:: Task.enqueue_on_commit - Whether the task should be enqueued when the transaction commits successfully, + Whether the Task should be enqueued when the transaction commits successfully, or immediately. By default, the behavior of the backend's @@ -70,13 +70,13 @@ Attributes of ``Task`` cannot be modified. .. attribute:: Task.name - An identifier for a task. Potentially useful for debugging. + An identifier for a Task. Potentially useful for debugging. - A task's name is not necessarily unique. + A Task's name is not necessarily unique. .. method:: Task.using() - Create a new task with modified defaults. The existing task is left unchanged. + Create a new Task with modified defaults. The existing Task is left unchanged. ``using`` allows modifying the following attributes: @@ -87,11 +87,11 @@ Attributes of ``Task`` cannot be modified. .. method:: Task.enqueue(*args, **kwargs) - Enqueue the task for later execution. + Enqueue the Task for later execution. - Arguments and keyword arguments are passed to the task's function as-is. + Arguments and keyword arguments are passed to the Task's function as-is. - If the task is not valid, according to the backend, + If the Task is not valid, according to the backend, :exc:`django.tasks.exceptions.InvalidTaskError` is raised. .. method:: Task.aenqueue(*args, **kwargs) @@ -101,7 +101,7 @@ Attributes of ``Task`` cannot be modified. .. method:: Task.get_result(result_id) Retrieve a result by its id. If the result does not exist, or is not the same - type as the current task, + type as the current Task, :exc:`ResultDoesNotExist ` is raised. @@ -152,7 +152,7 @@ Attributes of ``TaskResult`` cannot be modified. A unique identifier for the result, which can be passed to :meth:`Task.get_result`. - The format of a task result id will depend on the backend being used. + The format of a Task result id will depend on the backend being used. Some may use numbers, others UUIDs. Task ids are always strings less than 64 characters. @@ -162,18 +162,18 @@ Attributes of ``TaskResult`` cannot be modified. .. attribute:: TaskResult.enqueued_at - The time when the task was enqueued. + The time when the Task was enqueued. If :attr:`Task.enqueue_on_commit` was set, this is the time the transaction completed. .. attribute:: TaskResult.started_at - The time when the task began execution. + The time when the Task began execution. .. attribute:: TaskResult.finished_at - The time when the task finished execution, whether it failed or succeeded. + The time when the Task finished execution, whether it failed or succeeded. .. attribute:: TaskResult.backend @@ -181,23 +181,23 @@ Attributes of ``TaskResult`` cannot be modified. .. attribute:: TaskResult.exception_class - The exception class raised when executing the task. + The exception class raised when executing the Task. - If the task has not finished, ``ValueError`` is raised. If the task finished + If the Task has not finished, ``ValueError`` is raised. If the Task finished successfully, the exception class is ``None``. .. attribute:: TaskResult.traceback - The exception traceback from the raised exception when the task failed. + The exception traceback from the raised exception when the Task failed. - If the task has not finished, ``ValueError`` is raised. If the task finished + If the Task has not finished, ``ValueError`` is raised. If the Task finished successfully, the traceback is ``None``. .. attribute:: TaskResult.return_value - The return value from the task function. + The return value from the Task function. - If the task has not finished yet, or failed, :exc:`ValueError` is raised. + If the Task did not finish successfully, :exc:`ValueError` is raised. .. method:: TaskResult.refresh @@ -209,7 +209,7 @@ Attributes of ``TaskResult`` cannot be modified. .. attribute:: TaskResult.is_finished - Whether the task has finished (successfully or not). + Whether the Task has finished (successfully or not). Backends ======== @@ -221,7 +221,7 @@ Base backend .. class:: BaseTaskBackend -``BaseTaskBackend`` is the parent class for all task backends. +``BaseTaskBackend`` is the parent class for all Task backends. .. method:: BaseTaskBackend.get_result(result_id) @@ -245,7 +245,7 @@ changing behavior accordingly. .. attribute:: BaseTaskBackend.supports_defer - Whether the backend supports enqueueing tasks to be + Whether the backend supports enqueueing Tasks to be executed after a specific time using the ``run_after`` attribute. .. attribute:: BaseTaskBackend.supports_async_task @@ -254,9 +254,19 @@ changing behavior accordingly. .. attribute:: BaseTaskBackend.supports_get_result - Whether the backend supports retrieving task results from another thread + Whether the backend supports retrieving Task results from another thread after they have been enqueued. +The below table notes which backends support which features: + +======================= ================ ==================== +Feature ``DummyBackend`` ``ImmediateBackend`` +======================= ================ ==================== +``supports_defer`` Yes No +``supports_async_task`` Yes Yes +``supports_get_result`` No No [#fimmediateresult]_ +======================= ================ ==================== + Available backends ------------------ @@ -267,7 +277,7 @@ Immediate backend .. class:: ImmediateBackend -The immediate backend executes tasks immediately, rather than in the background. +The immediate backend executes Tasks immediately, rather than in the background. Dummy backend ~~~~~~~~~~~~~ @@ -276,12 +286,12 @@ Dummy backend .. class:: DummyBackend -The dummy backend doesn't execute enqueued tasks at all, instead storing results +The dummy backend doesn't execute enqueued Tasks at all, instead storing results for later use. .. attribute:: DummyBackend.results - A list of results for the enqueued tasks, in the order they were enqueued. + A list of results for the enqueued Tasks, in the order they were enqueued. .. method:: DummyBackend.clear @@ -301,3 +311,8 @@ Exceptions Raised when the :class:`Task ` attempting to be enqueued is invalid. + +.. rubric:: Footnotes +.. [#fimmediateresult] The immediate backend doesn't officially support ``get_result``, + despite implementing the API, since the result cannot be retrieved from a + different thread. diff --git a/docs/topics/tasks.txt b/docs/topics/tasks.txt index 58854516c9..813ae48b90 100644 --- a/docs/topics/tasks.txt +++ b/docs/topics/tasks.txt @@ -21,7 +21,7 @@ execute the task, as well as a unique identifier for Django to retrieve the result later. Outside of Django, a Worker looks at the Queue Store for new Tasks to run. When -a new Task is added, the Worker claims the task, executes it, and saves the +a new Task is added, the Worker claims the Task, executes it, and saves the status and result back to the Queue Store. .. _configuring-a-task-backend: @@ -29,13 +29,14 @@ status and result back to the Queue Store. Configuring a Task backend ========================== -Background tasks require some work to set up. - -Different task backends have different characteristics and configuration options, -which may impact the performance and reliability of your application. +The Task backend determines how and where tasks are stored for execution and how +they are executed. Different task backends have different characteristics and +configuration options, which may impact the performance and reliability of your +application. Task backends are configured using the :setting:`TASKS` setting in your settings -file. +file. Whilst most applications will only need a single backend, multiple are +supported. .. _immediate-task-backend: @@ -47,25 +48,28 @@ The immediate backend runs enqueued tasks immediately, rather than in the background. This allows background task functionality to be slowly added to an application, before the required infrastructure is available. +To use it, set :setting:`BACKEND ` to +``"django.tasks.backends.immediate.ImmediateBackend"``:: + + TASKS = {"default": {"BACKEND": "django.tasks.backends.immediate.ImmediateBackend"}} + +The ``ImmediateBackend`` may also be useful in tests, to bypass the need to run a +real background worker in your tests. + .. warning:: When :setting:`ENQUEUE_ON_COMMIT ` is ``False``, the task will be executed within the same transaction it was enqueued in. - This may lead to unexpected behavior changes when changing backend in future. - -To use it, set :setting:`BACKEND ` to -``"django.tasks.backends.immediate.ImmediateBackend"``:: - - TASKS = {"default": {"BACKEND": "django.tasks.backends.immediate.ImmediateBackend"}} + See :ref:`task_transactions` for more. .. _dummy-task-backend: Dummy backend ------------- -The dummy backend doesn't execute enqueued tasks at all, instead storing results -for later use. +The dummy backend doesn't execute enqueued Tasks at all, instead storing results +for later use. Task results will forever remain in the ``NEW`` state. This backend is not intended for use in production - it is provided as a convenience that can be used during development and testing. @@ -75,7 +79,7 @@ To use it, set :setting:`BACKEND ` to TASKS = {"default": {"BACKEND": "django.tasks.backends.dummy.DummyBackend"}} -The results for enqueued tasks can be retrieved from the backend's +The results for enqueued Tasks can be retrieved from the backend's :attr:`results ` attribute:: from django.tasks import default_task_backend @@ -94,9 +98,9 @@ Stored results can be cleared using the Using a custom backend ---------------------- -While Django includes support for a number of task backends out-of-the-box, -sometimes you might want to customize the the task backend. To use an external -task backend with Django, use the Python import path as the +While Django includes support for a number of Task backends out-of-the-box, +sometimes you might want to customize the the Task backend. To use an external +Task backend with Django, use the Python import path as the :setting:`BACKEND ` of the :setting:`TASKS` setting, like so:: TASKS = { @@ -105,7 +109,7 @@ task backend with Django, use the Python import path as the } } -If you're building your own backend, you can use the standard task backends +If you're building your own backend, you can use the built-in Task backends as reference implementations. You'll find the code in the :source:`django/tasks/backends/` directory of the Django source. @@ -116,19 +120,19 @@ Each backend can be given additional arguments to control its behavior. These arguments are provided as additional keys in the :setting:`TASKS` setting. Valid arguments are as follows: -* :setting:`ENQUEUE_ON_COMMIT `: Whether a task should +* :setting:`ENQUEUE_ON_COMMIT `: Whether a Task should be enqueued at the end of the current transaction (if there is one) commits successfully, rather than enqueueing immediately. This argument defaults to ``True``. -* :setting:`QUEUES `: Restrict the queue names a task may be +* :setting:`QUEUES `: Restrict the queue names a Task may be enqueued to. By default, only the ``"default"`` queue is allowed. Queue name validation can be disabled by setting this to an empty list. Asynchronous support -------------------- -Django has developing support for asynchronous task backends. +Django has developing support for asynchronous Task backends. ``django.tasks.backends.base.BaseTaskBackend`` has async variants of all base methods. By convention, the asynchronous versions of all methods are prefixed @@ -161,13 +165,13 @@ functionality, or use a system check to prevent the application from starting at To facilitate this, certain features can be checked on a backend: * :attr:`supports_defer `: - Can tasks be executed after a specific time using ``run_after``? + Can Tasks be executed after a specific time using ``run_after``? * :attr:`supports_async_task `: Can ``async`` functions (coroutines) be used as task functions? * :attr:`supports_get_result `: - Can a task's results be retrieved from another thread or process? + Can a Task's results be retrieved from another thread or process? Defining tasks ============== @@ -182,7 +186,7 @@ module-level function:: def calculate_meaning_of_life(answer=42): return answer -Returned in a :class:`django.tasks.Task` instance. +The return value is a :class:`django.tasks.Task` instance. .. note:: @@ -190,31 +194,31 @@ Returned in a :class:`django.tasks.Task` instance. as the same types. This means complex types like model instances, as well as many built-in types - like ``datetime`` and ``tuple`` cannot be used in tasks without additional + like ``datetime`` and ``tuple`` cannot be used in Tasks without additional conversion. -The ``task`` decorator accepts a few keyword arguments to customize the task: +The ``task`` decorator accepts a few keyword arguments to customize the Task: -* ``priority``: The priority of the task. Higher numbers will be executed sooner. +* ``priority``: The priority of the Task. Higher numbers will be executed sooner. -* ``queue_name``: The name of the queue the task will be executed on +* ``queue_name``: The name of the queue the Task will be executed on -* ``backend``: The name of the backend this task must use (as defined in +* ``backend``: The name of the backend this Task must use (as defined in :setting:`TASKS`). -* ``enqueue_on_commit``: Whether the task is enqueued when the current transaction +* ``enqueue_on_commit``: Whether the Task is enqueued when the current transaction commits successfully, or enqueued immediately. These arguments correspond to attributes on the created :class:`Task `. -By convention, tasks should be defined in a ``tasks.py`` file, however this is +By convention, Tasks should be defined in a ``tasks.py`` file, however this is not enforced. Modifying tasks --------------- -Before enqueueing tasks, it may be necessary to modify certain parameters of the task. +Before enqueueing Tasks, it may be necessary to modify certain parameters of the Task. For example, to give it a higher priority than it would normally. A ``Task`` instance cannot be modified directly. Instead, a modified instance @@ -236,10 +240,10 @@ the original as-is:: which is used relative to the current time (when ``using`` is called), or a timezone-aware :class:`datetime `. -Enqueueing tasks +Enqueueing Tasks ================ -To add the task to the queue store, so it will be executed, call the ``enqueue`` +To add the Task to the queue store, so it will be executed, call the ``enqueue`` method on it:: result = calculate_meaning_of_life.enqueue() @@ -254,17 +258,38 @@ If the task takes arguments, these can be passed as-is to ``enqueue``:: To enqueue tasks in an ``async`` context, :meth:`aenqueue ` is available as an ``async`` variant of ``enqueue``. +.. _task_transactions: + Transactions ------------ -By default, tasks are enqueued after the current transaction (if there is one) +By default, tasks are enqueued after the current database transaction (if there is one) commits successfully (using :meth:`transaction.on_commit `), -rather than enqueueing immediately. +rather than enqueueing immediately. For most backends, tasks are run in a +separate process, using a different database connection. Without waiting for the +transaction to commit, workers could start process a Task which uses objects which it +can't access yet. This behavior can be changed by changing the :setting:`TASKS-ENQUEUE_ON_COMMIT` setting for the backend, or for a specific task using the ``enqueue_on_commit`` parameter. +For example, consider this simplified example:: + + @task + def my_task(): + Thing.objects.get() + + + with transaction.atomic(): + Thing.objects.create() + my_task.enqueue() + + +If ``ENQUEUE_ON_COMMIT = False``, then it is possible for ``my_task`` to run before +the ``Thing`` is committed to the database, and the task won't be able to see +the created object within your transaction. + Task results ============ @@ -287,7 +312,11 @@ result for ``calculate_meaning_of_life``, :exc:`django.tasks.exceptions.ResultDo is raised. To retrieve a ``TaskResult``, regardless of which kind of ``Task`` it was from, -use the ``get_result`` method on the API +use the ``get_result`` method on the backend:: + + from django.tasks import default_task_backend + + result = default_task_backend.get_result(result_id) To retrieve results in an ``async`` context, :meth:`aget_result ` is available as an ``async`` variant of ``get_result`` on both the backend and ``Task``. @@ -298,8 +327,8 @@ Calling ``get_result`` on these backends will raise :exc:`NotImplementedError`. Updating results ---------------- -A ``TaskResult`` contains the status of a task's execution at the point it was -retrieved. If the task finishes after ``get_result`` is called, it will not update. +A ``TaskResult`` contains the status of a Task's execution at the point it was +retrieved. If the Task finishes after ``get_result`` is called, it will not update. To refresh the values, call the :meth:`django.tasks.TaskResult.refresh` method:: @@ -312,19 +341,19 @@ To refresh the values, call the :meth:`django.tasks.TaskResult.refresh` method:: Return values ------------- -If your task function returns something, it can be retrieved from the +If your Task function returns something, it can be retrieved from the :attr:`django.tasks.TaskResult.return_value` attribute:: if result.status == ResultStatus.SUCCEEDED: result.return_value # 42 -If the task has not finished executing, or has failed, :exc:`ValueError` is raised. +If the Task has not finished executing, or has failed, :exc:`ValueError` is raised. Exceptions ---------- -If the task doesn't succeed, and instead raises an exception, either -as part of the task or as part of running it, the exception class is saved +If the Task doesn't succeed, and instead raises an exception, either +as part of the Task or as part of running it, the exception class is saved to the :attr:`django.tasks.TaskResult.exception_class` attribute:: assert result.exception_class == ValueError