Add reference documentation for operations and stubs for schemaeditor.

docs/index.txt

@@ -73,7 +73,9 @@ manipulating the data of your Web application. Learn more about it below:
   :doc:`Accessing related objects <ref/models/relations>`
 
 * **Migrations:**
-  :doc:`Introduction to Migrations<topics/migrations>`
+  :doc:`Introduction to Migrations<topics/migrations>` |
+  :doc:`Operations reference <ref/migration-operations>` |
+  :doc:`SchemaEditor <ref/schema-editor>`
 
 * **Advanced:**
   :doc:`Managers <topics/db/managers>` |

docs/ref/migration-operations.txt

@@ -0,0 +1,320 @@
+====================
+Migration Operations
+====================
+
+Migration files are composed of one or more Operations, objects that
+declaratively record what the migration should do to your database.
+
+Django also uses these Operation objects to work out what your models
+looked like historically, and to calculate what changes you've made to
+your models since the last migration so it can automatically write
+your migrations; that's why they're declarative, as it means Django can
+easily load them all into memory and run through them without touching
+the database to work out what your project should look like.
+
+There are also more specialised Operation objects which are for things like
+:ref:`data migrations <data-migrations>` and for advanced manual database
+manipulation. You can also write your own Operation classes if you want
+to encapsulate a custom change you commonly make.
+
+If you need an empty migration file to write your own Operation objects
+into, just use ``python manage.py makemigrations --empty yourappname``,
+but be aware that manually adding schema-altering operations can confuse the
+migration autodetector and make resulting runs of ``makemigrations`` output
+incorrect code.
+
+All of the core Django operations are available from the
+``django.db.migrations.operations`` module.
+
+
+Schema Operations
+=================
+
+CreateModel
+-----------
+::
+
+    CreateModel(name, fields, options=None, bases=None)
+
+Creates a new model in the project history and a corresponding table in the
+database to match it.
+
+``name`` is the model name, as would be written in the ``models.py`` file.
+
+``fields`` is a list of 2-tuples of ``(field_name, field_instance)``.
+The field instance should be an unbound field (so just ``models.CharField()``,
+rather than a field takes from another model).
+
+``options`` is an optional dictionary of values from the model's ``Meta`` class.
+
+``bases`` is an optional list of other classes to have this model inheirit from;
+it can contain both class objects as well as strings in the format
+``"appname.ModelName"`` if you want to depend on another model (so you inherit
+from the historical version). If it's not supplied, it defaults to just
+inheriting from the standard ``models.Model``.
+
+
+DeleteModel
+-----------
+::
+
+    DeleteModel(name)
+
+Deletes the model from the project history and its table from the database.
+
+
+RenameModel
+-----------
+::
+
+    RenameModel(old_name, new_name)
+
+Renames the model from an old name to a new one.
+
+You may have to manually add
+this if you change the model's name and quite a few of its fields at once; to
+the autodetector, this will look like you deleted a model with the old name
+and added a new one with a different name, and the migration it creates will
+lose any data in the old table.
+
+
+AlterModelTable
+---------------
+::
+
+    AlterModelTable(name, table)
+
+Changes the model's table name (the ``db_table`` option on the ``Meta`` subclass)
+
+
+AlterUniqueTogether
+-------------------
+::
+
+    AlterUniqueTogether(name, unique_together)
+
+Changes the model's set of unique constraints
+(the ``unique_together`` option on the ``Meta`` subclass)
+
+
+AlterIndexTogether
+------------------
+::
+
+    AlterIndexTogether(name, index_together)
+
+Changes the model's set of custom indexes
+(the ``index_together`` option on the ``Meta`` subclass)
+
+
+AddField
+--------
+::
+
+    AddField(model_name, name, field, preserve_default=True)
+
+Adds a field to a model. ``model_name`` is the model's name, ``name`` is
+the field's name, and ``field`` is an unbound Field instance (the thing
+you would put in the field declaration in ``models.py`` - for example,
+``models.IntegerField(null=True)``.
+
+The ``preserve_default`` argument indicates whether the field's default
+value is permanent and should be baked into the project state (``True``),
+or if it is temporary and just for this migration (``False``) - usually
+because the migration is adding a non-nullable field to a table and needs
+a default value to put into existing rows. It does not effect the behaviour
+of setting defaults in the database directly - Django never sets database
+defaults, and always applies them in the Django ORM code.
+
+
+RemoveField
+-----------
+::
+
+    RemoveField(model_name, name)
+
+Removes a field from a model.
+
+Bear in mind that when reversed this is actually adding a field to a model;
+if the field is not nullable this may make this operation irreversible (apart
+from any data loss, which of course is irreversible).
+
+
+AlterField
+----------
+::
+
+    AlterField(model_name, name, field)
+
+Alters a field's definition, including changes to its type, ``null``, ``unique``,
+``db_column`` and other field attributes.
+
+Note that not all changes are possible on all databases - for example, you
+cannot change a text-type field like ``models.TextField()`` into a number-type
+field like ``models.IntegerField()`` on most databases.
+
+
+RenameField
+-----------
+::
+
+    RenameField(model_name, old_name, new_name)
+
+Changes a field's name (and, unless ``db_column`` is set, its column name).
+
+
+
+Special Operations
+==================
+
+RunSQL
+------
+
+::
+
+    RunSQL(sql, reverse_sql=None, state_operations=None, multiple=False)
+
+Allows runnning of arbitrary SQL on the database - useful for more advanced
+features of database backends that Django doesn't support directly, like
+partial indexes.
+
+``sql``, and ``reverse_sql`` if provided, should be strings of SQL to run on the
+database. They will be passed to the database as a single SQL statement unless
+``multiple`` is set to ``True``, in which case they will be split into separate
+statements manually by the operation before being passed through.
+
+In some extreme cases, the built-in statement splitter may not be able to split
+correctly, in which case you should manually split the SQL into multiple calls
+to ``RunSQL``.
+
+The ``state_operations`` argument is so you can supply operations that are
+equivalent to the SQL in terms of project state; for example, if you are 
+manually creating a column, you should pass in a list containing an ``AddField``
+operation here so that the autodetector still has an up-to-date state of the
+model (otherwise, when you next run ``makemigrations``, it won't see any
+operation that adds that field and so will try to run it again).
+
+
+.. _operation-run-python:
+
+RunPython
+---------
+
+::
+
+    RunPython(code, reverse_code=None)
+
+Runs custom Python code in a historical context. ``code`` (and ``reverse_code``
+if supplied) should be callable objects that accept two arguments; the first is
+an instance of ``django.apps.registry.Apps`` containing historical models that
+match the operation's place in the project history, and the second is an
+instance of SchemaEditor.
+
+You are advised to write the code as a separate function above the ``Migration``
+class in the migration file, and just pass it to ``RunPython``.
+
+
+SeparateDatabaseAndState
+------------------------
+
+::
+
+    SeparateDatabaseAndState(database_operations=None, state_operations=None)
+
+A highly specalist operation that let you mix and match the database
+(schema-changing) and state (autodetector-powering) aspects of operations.
+
+It accepts two list of operations, and when asked to apply state will use the
+state list, and when asked to apply changes to the database will use the database
+list. Do not use this operation unless you're very sure you know what you're doing.
+
+
+Writing your own
+================
+
+Operations have a relatively simple API, and they're designed so that you can
+easily write your own to supplement the built-in Django ones. The basic structure
+of an Operation looks like this::
+
+    from django.db.migrations.operations.base import Operation
+
+    class MyCustomOperation(Operation):
+
+        # If this is False, it means that this operation will be ignored by
+        # sqlmigrate; if true, it will be run and the SQL collected for its output.
+        reduces_to_sql = False
+
+        # If this is False, Django will refuse to reverse past this operation.
+        reversible = False
+
+        def __init__(self, arg1, arg2):
+            # Operations are usually instantiated with arguments in migration
+            # files. Store the values of them on self for later use.
+            pass
+
+        def state_forwards(self, app_label, state):
+            # The Operation should take the 'state' parameter (an instance of
+            # django.db.migrations.state.ProjectState) and mutate it to match
+            # any schema changes that have occurred.
+            pass
+
+        def database_forwards(self, app_label, schema_editor, from_state, to_state):
+            # The Operation should use schema_editor to apply any changes it
+            # wants to make to the database.
+            pass
+
+        def database_backwards(self, app_label, schema_editor, from_state, to_state):
+            # If reversible is True, this is called when the operation is reversed.
+            pass
+
+        def describe(self):
+            # This is used to describe what the operation does in console output.
+            return "Custom Operation"
+
+You can take this template and work from it, though we suggest looking at the
+built-in Django operations in ``django.db.migrations.operations`` - they're
+easy to read and cover a lot of the example usage of semi-internal aspects
+of the migration framework like ``ProjectState`` and the patterns used to get
+historical models.
+
+Some things to note:
+
+* You don't need to learn too much about ProjectState to just write simple
+  migrations; just know that it has a ``.render()`` method that turns it into
+  an app registry (which you can then call ``get_model`` on).
+
+* ``database_forwards`` and ``database_backwards`` both get two states passed
+  to them; these just represent the difference the ``state_forwards`` method
+  would have applied, but are given to you for convenience and speed reasons.
+
+* ``to_state`` in the database_backwards method is the *older* state; that is,
+  the one that will be the current state once the migration has finished reversing.
+
+* You might see implementations of ``references_model`` on the built-in
+  operations; this is part of the autodetection code and does not matter for
+  custom operations.
+
+As a simple example, let's make an operation that loads PostgreSQL extensions
+(which contain some of PostgreSQL's more exciting features). It's simple enough;
+there's no model state changes, and all it does is run one command::
+
+    from django.db.migrations.operations.base import Operation
+
+    class LoadExtension(Operation):
+
+        reversible = True
+
+        def __init__(self, name):
+            self.name = name
+
+        def state_forwards(self, app_label, state):
+            pass
+
+        def database_forwards(self, app_label, schema_editor, from_state, to_state):
+            schema_editor.execute("CREATE EXTENSION IF NOT EXISTS %s" % self.name)
+
+        def database_backwards(self, app_label, schema_editor, from_state, to_state):
+            schema_editor.execute("DROP EXTENSION %s" % self.name)
+
+        def describe(self):
+            return "Creates extension %s" % self.name

docs/ref/schema-editor.txt

@@ -0,0 +1,112 @@
+============
+SchemaEditor
+============
+
+Django's migration system is split into two parts; the logic for calculating
+and storing what operations should be run (``django.db.migrations``), and the
+database abstraction layer that turns things like "create a model" or
+"delete a field" into SQL - which is the job of the ``SchemaEditor``.
+
+It's unlikely that you will want to interact directly with ``SchemaEditor`` as
+a normal developer using Django, but if you want to write your own migration
+system, or have more advanced needs, it's a lot nicer than writing SQL.
+
+Each database backend in Django supplies its own version of ``SchemaEditor``,
+and it's always accessible via the ``connection.schema_editor()`` context
+manager::
+
+    with connection.schema_editor() as schema_editor:
+        schema_editor.delete_model(MyModel)
+
+It must be used via the context manager as this allows it to manage things
+like transactions and deferred SQL (like creating ``ForeignKey`` constraints).
+
+It exposes all possible operations as methods, that should be called in
+the order you wish changes to be applied. Some possible operations or types
+of change are not possible on all databases - for example, MyISAM does not
+support foreign key constraints.
+
+Methods
+=======
+
+execute
+-------
+
+::
+
+    execute(sql, params=[])
+
+Executes the SQL statement passed in, with parameters if supplied. This
+is a simple wrapper around the normal database cursors that allows
+capture of the SQL to a ``.sql`` file if the user wishes.
+
+create_model
+------------
+
+::
+
+    create_model(model)
+
+
+delete_model
+------------
+
+::
+
+    delete_model(model)
+
+
+alter_unique_together
+---------------------
+
+::
+
+    alter_unique_together(model, old_unique_together, new_unique_together)
+
+
+alter_index_together
+--------------------
+
+::
+
+    alter_index_together(model, old_index_together, new_index_together)
+
+
+alter_db_table
+--------------
+
+::
+
+    alter_db_table(model, old_db_table, new_db_table)
+
+
+alter_db_tablespace
+-------------------
+
+::
+
+    alter_db_tablespace(model, old_db_tablespace, new_db_tablespace)
+
+
+add_field
+---------
+
+::
+
+    add_field(model, field)
+
+
+remove_field
+------------
+
+::
+
+    remove_field(model, field)
+
+
+alter_field
+------------
+
+::
+
+    alter_field(model, old_field, new_field, strict=False)

docs/topics/migrations.txt

@@ -60,7 +60,7 @@ Backend Support
 
 Migrations are supported on all backends that Django ships with, as well
 as any third-party backends if they have programmed in support for schema
-alteration (done via the ``SchemaEditor`` class).
+alteration (done via the :doc:`SchemaEditor </ref/schema-editor>` class).
 
 However, some databases are more capable than others when it comes to
 schema migrations; some of the caveats are covered below.
@@ -311,6 +311,84 @@ from these base classes inherit normally, so if you absolutely need access
 to these you can opt to move them into a superclass.
 
 
+.. _data-migrations:
+
+Data Migrations
+---------------
+
+As well as changing the database schema, you can also use migrations to change
+the data in the database itself, in conjunction with the schema if you want.
+
+Migrations that alter data are usually called "data migrations"; they're best
+written as separate migrations, sitting alongside your schema migrations.
+
+Django can't automatically generate data migrations for you, as it does with
+schema migrations, but it's not very hard to write them. Migration files in
+Django are made up of :doc:`Operations </ref/migration-operations>`, and
+the main operation you use for data migrations is
+:ref:`RunPython <operation-run-python>`.
+
+To start, make an empty migration file you can work from (Django will put
+the file in the right place, suggest a name, and add dependencies for you)::
+
+    python manage.py makemigrations --empty yourappname
+
+Then, open up the file; it should look something like this::
+
+    # encoding: utf8
+    from django.db import models, migrations
+
+    class Migration(migrations.Migration):
+
+        dependencies = [
+            ('yourappname', '0001_initial'),
+        ]
+
+        operations = [
+        ]
+
+Now, all you need to do is create a new function and have RunPython use it.
+RunPython expects a callable as its argument which takes two arguments - the
+first is an :doc:`app registry </ref/applications/>` that has the historical
+versions of all your models loaded into it to match where in your history the
+migration sits, and the second is a :doc:`SchemaEditor </ref/schema-editor>`,
+which you can use to manually effect database schema changes (but beware,
+doing this can confuse the migration autodetector!)
+
+Let's write a simple migration that populates our new ``name`` field with the
+combined values of ``first_name`` and ``last_name`` (we've come to our senses
+and realised that not everyone has first and last names). All we
+need to do is use the historical model and iterate over the rows::
+
+    # encoding: utf8
+    from django.db import models, migrations
+
+    def combine_names(apps, schema_editor):
+        # We can't import the Person model directly as it may be a newer
+        # version than this migration expects. We use the historical version.
+        Person = apps.get_model("yourappname", "Person")
+        for person in Person.objects.all():
+            person.name = "%s %s" % (person.first_name, person.last_name)
+            person.save()
+
+    class Migration(migrations.Migration):
+
+        dependencies = [
+            ('yourappname', '0001_initial'),
+        ]
+
+        operations = [
+            migrations.RunPython(combine_names),
+        ]
+
+Once that's done, we can just run ``python manage.py migrate`` as normal and
+the data migration will run in place alongside other migrations.
+
+If you're interested in the more advanced migration operations, or want
+to be able to write your own, see our
+:doc:`migration operations reference </ref/migration-operations>`.
+
+
 .. _migration-serializing:
 
 Serializing values