django_transaction_barrier - transaction barriers for Django and Celery

django_transaction_barrier provides a barrier-like abstraction for transactions. A Django application developer can use a "transaction barrier" to spawn a task within a transaction and guarantee that the task blocks until it's able to access the updates made in the transaction. django_transaction_barrier is designed with Celery in mind and provides a Celery task base class. Using the base class it's easy to write code that atomically modifies the database and spawns a Celery task that executes after the transaction commits.

Installation

Install from source or use pip:

pip install django_transaction_barrier

and add "django_transaction_barrier" to INSTALLED_APPS in settings.py:

INSTALLED_APPS = (
    'django_transaction_barrier',
    ...
)

Usage

from celery import task
from django.db import transaction
from django_transaction_barrier.celery import TransactionBarrierTask

@task(base=TransactionBarrierTask)
def do_something_task(model_id):
  value = Model.objects.get(id=model_id).value
  ...

@transaction.atomic
def kick_off_task(model, value):
    model.value = value
    do_something_task.apply_async_with_barrier(args=(model.id,))
    model.save()

Details

If an application spawns an asynchronous TransactionBarrierTask the task is guaranteed to execute eventually (assuming a durable task queue) after the transaction commits. If the transaction aborts, the task raises a TransactionAborted exception and does not execute. In autocommit mode (i.e., "outside of a transaction") TransactionBarrierTasks behave like normal Celery tasks.

If an application synchrnously executes a TransactionBarrierTask (e.g., with Celery eager mode) within a transaction, the task executes immediately without waiting for the transaction to commit.

Implementation

django_transaction_barrier implements transaction barriers using row insertion to signify a committed transaction and some DB-specifc logic to detect an abort.

Related

• https://pypi.python.org/pypi/django-transaction-signals-do-not-use/1.0
• https://django-transaction-hooks.readthedocs.org/en/latest/
• https://code.djangoproject.com/ticket/14051
• https://github.com/nickbruun/django_atomic_celery

django_transaction_barrier provides diffferent semantics than related projects. It guarantees TransactionBarrierTask execution if the transaction commits. Most related projects rely on monkey patching Django's database backends to provide post commit hooks, which results in a race: they (non-atomically) commit the transaction and then execute the post commit hook, so they do not guarantee task execution.

As noted above, if an application synchrnously executes a TransactionBarrierTask (e.g., with Celery eager mode) within a transaction, the task executes immediately without waiting for the transaction to commit.

Tests

docker build -t tests . && docker run tests

TODO

• Add support for a mysql backend.