django-icetea

django-icetea is a package built on top of Django and provides abstractions for creating REST APIs.

It has been influenced by the architecture of django-piston and piston-perfect.

I decided to build django-icetea, in order to have an API framework with tight foundations, consistent and intuitive behaviour, readable code, and of course, easy to use.

Installation

django-icetea is registered in PyPI, so installing it is as easy as running:

pip install django-icetea

However, I would suggest using the latest version from github. The master branch is always stable.

Settings parameters

In your application's settings.py file, you can specify the following parameters related to django-icetea:

• ICETEA_ERRORS: With True, enables the sending of emails to the applications's admins, in the case of Server Errors. Default is True.

• ICETEA_DISPLAY_ERRORS: With True, returns well-formed error messages in the case of Server Errors. It requires that DEBUG=True. Default is True.

Documentation

The code is thoroughly documented. Use epydoc to parse it and generate a document out of it. For example, in order to create an html page with the documentation, cd into the django-icetea folder, and issue:

epydoc --html icetea -o docs

This will create folder docs. Open the file index.html for the whole code documentation.

Philosophy

django-icetea aims to provide the abstractions for providing out-of-the-box functionality for creating APIs. It strives to keep things clear and explicit, without any unnecessary magic behind the scenes.

It is very extensible, and the default behaviour can be overridden, extended and modified at will.

As in any project though, some assumptions have to be made, and some conventions need to be predefined.

For example, although HTTP is an application protocol, it is mostly used for interaction with Web browsers. When applied in more generic request/response schemes, there are scenarios that the protocol itself does not indicate the correct behavior. For this reason, I mostly view HTTP as the means of transmission for requests and responses. It is unaware of business logic, and therefore it lacks the means of mapping application specific semantics or errors to HTTP Responses.

A specific case in which the HTTP protocol doesn't really specify the behaviour, is the following:

Say we need to create a model instance; We issue a POST request to the API, and we expect a response which will indicate if the resource has been created, and if yes, return the resource. The server first needs to validate the data it has received. If the data are invalid, the API should return a 400 Bad Request response. If the data are valid, but upon creation the database fails, what do we do? Do we return 400 Bad Request? No way. This will confuse the user and indicate that the data provided were invalid. The request was validated successfully, so this is not the case, Do we return a Successful response 200 OK, and empty data? I choose for the latter. This indicates that the data were indeed valid, but the resource failed to be created.

In anycase developing an API is all about consistent and unambiguous communication between the client and the server. This has been one of my main goals with this project. If different applications require different semantics, django-icetea's code can easily be modified to support them.

Moreover, following the Principle of the least astonishment which is what Python in general, and Django in particular encourage, I have tried to follow the general behavior that Django users are familiar with. An example of this is the validation method of the ModelHandler class (in the handlers.py module). It cleans the data, creates model instances (without committing them to the database) and validates them using the model's full-clean method. Once this is done, we are certain that we are dealing with perfectly valid model instances, which we can safely write to the database. This means that we don't have the need to do all these steps manually, since they are offered by django-icetea out of the box. It is exactly how validation for Django Modelform works, and how most Django developers are used to doing things.. The means have changed (REST API instead of Forms), but the procedure is still the same.

Short Introduction

django-icetea offers 2 types of handlers:

• ModelHandler: Used to expose Django models to the API. Offers CRUD functionality out of the box.

• BaseHandler: Used to expose data that don't map on a model. Most of the functionality will need to be written manually.

Glossary

Singular Request: A request that refers to a single resource. The resource is usually identified by the url. For example a GET/PUT/DELETE request on /resource/<id>/ is a singular request.

Plural Request: A request that affects(retrieves or modifies) a group of resources (usually a subset or all the resources that the client has the right to view). It could be plural GET, plural PUT, or plural DELETE.

Bulk Request: Request with an array of data in its request body. It only makes sense for POST requests, and aims to create multiple instances in one request. For Bulk POST requests there is no recommended behavior or semantics, so we defined our own semantics, in order to make sure that the functionality is predictable and makes the most sense. More details can be seen in section Bulk POST requests.

Assumptions

The only assumption that django-icetea makes is that singular requests are denoted by the keyword argument id. So for example a GET request of the following form /resource/<id>/ requests the resource with id=<id>.

This is essential mostly for security related checks, which mainly control whether the request is plural, and if such a request has been explicitly allowed.

Incoming Requests

The Content-type header for incoming requests should be application/json. This is currently the only request body format that django-icetea recognizes.

Outgoing responses

The outgoing responses of django-icetea can be of one of the following formats:

• application/json
• text/xml
• text/html
• application/vnd.ms-excel

The default is application/json. Please not that in the case of outputting json, or xml, it is easy to serialize data structures in the response. However in the case of html and especially xls format, there should (probably) be application specific semantics applied to the output emitters.

Status codes

The Status codes have the following meanings:

• 200 OK: Request was served successfully
• 400 Bad Request: Validation error on request body
• 403 Forbidden: The client is not authenticated
• 405 Method Not Allowed: The request method was performed on a resource that does not support that method
• 410 Gone: The resource is not available
• 422 UnprocessableEntity: The request was valid but could not be processed due to the semantics of the resource (for example, a DELETE request on a resource that belongs to the authenticated client. We might choose not to allow deletion of the specific resource if its field x has a specific value. In that case we respond with a 422 UnprocessableEntity response).

For a very detailed description of the expected responses for any kind of request, please check section Request and response protocol.

Tests

django-icetea comes with its own test suite, found in the tests.py module. This module defines Base Test Classes, which are used to test django-icetea itself, and can also be used to test any API implementation.

CSRF tokens

Django uses CSRF tokens, in order to deal with web browsers' CSRF vulnerability. Django's CsrfViewMiddleware inserts a CSRF token as a hidden field in every form using the POST method, before sending the form to the web browser. For every subsequent POST request from the web browser, the same middleware checks the token, to ensure that it contains the expected value. If not a 403 Forbidden response is returned.

However, since django-icetea is an API and does not make use of forms, the CSRF token doesn't make a lot of sense. So by default django-icetea views are CSRF exempted, meaning they don't require the CSRF token.

Usage

Say we have an API built on top of django-icetea. Let's assume we have a Django app called foo, with a model FooModel.

We want to define 2 API handlers; One that exposes the model FooModel, and another one that exposes some other non-model data.

Other than defining the business logic, handlers also act as means of representation. For example, ModelHandler classes, define how the corresponding model will be represented within that handler(which fields should be exposed), but also in cases that it is nested in the responses of other handlers.

foo/handlers.py

TODO: Example with a BaseHandler

Here we define our API handler, which is the implementer of the business logic

from foo.models import foomodel
from icetea.handlers import ModelHandler

class FooHandler(ModelHandler):
    authentication = True
    model = FooModel

    read = True
    create = True

    allowed_out_fields = (
        'id',
        'field1', 
        'field2',
    )

    allowed_in_fields = (
        'field1',
        'field2',
    )

foo/urls.py

We need to create resources(equivalent to Django views), which will initiate the serving of API requests

from djanco.conf.urls.defaults import *
from foo.handlers import FooHandler
from icetea.resource import Resource

foo_handler = Resource(FooHandler)

urlpatterns = patterns('',
    url(r'^foo/$ ', foo_handler),
    url(r'^foo/(?P<id>\d+)/$ ', foo_handler),
)

Handler level attributes

Relevant for all handlers

read, create, update, delete

If any of these parameters is True, then the handler allows GET, POST, PUT and DELETE requests respectively.

If instead they are defined as methods, eg:

def read(self, request, *args, **kwargs):
    pass

Then the corresponding action is enabled, and the default functionality is overridden by the method we defined.

bulk_create

If True enables bulk-POST requests. Default is False. See section Notes for more information.

Requires that create = True.

When enabled, you should anticipate on 400 Bad Request responses, with a list in their body.

plural_update

If True, enables plural PUT requests, which means updating multiple resources in one request. It is a potentially catastrophic operation, and for this reason is should be explicitly allowed. Default is False.

Requires that update = True.

When enabled, you should anticipate on 400 Bad Request responses, with a list in their body.

plural_delete

If True enables plural DELETE requests, which means deleting multiple resources in one request. It is a potentially catastrophic operation, and for this reason it should be explicitly allowed. Default is False.

Requires that delete=True.

When enabled, you should anticipate on 422 Unprocessable Entity responses, with a list in their body.

request_fields

Indicates which querystring parameter will act as a a request-level field selector. If True, then the selector is field. If False, there will be no field selection. Default is True.

order

Indicates which querystring parameter will act as the order-type selector on the result set of the requested operation. If True, then the parameter is order. If False, no order-type selection can be performed. Default is False. The order logic, should be implemented in the handler's order_data method.

slice

Indicates which querystring parameter will be used to request slicing of the result set of the requested operation. If True, then the parameter is slice. If False, no slicing will be possible. Default is False. The slicing notation follows Python's slice notation, of start:stop:step.

authentication

If True, only authenticated users can access the handler. The Django authentication is used. Default value is False.

allowed_out_fields

Tuple of fields, which indicates the fields that the handler is allowed to output.

In the case of a ModelHandler, it indicates model fields.

In the case of a BaseHandler, it only has sense if the handler returns dictionaries, or lists of dictionaries, and it indicates the dictionary keys that the handler is allowed to output.

The actual fields that a request will eventually output, is a function of this parameter, as well as the request-level field selection, indicated by the field.

allowed_in_fields

Tuple of fields, which indicates the fields that the handler is allowed to take from the incoming request body. In the case of ModelHandler classes, no primary keys or related keys are allowed.

Relevant only for handlers extending ModelHandler

model

The database model which the Handler exposes.

filters

A dictionary of filter name: filter_operation couples. filter name defines the querystring parameter used to apply the filtering on the current request. filter_operation corresponds to a Django lookup filter, which will be applied on the request's resuls data.

exclude_nested

Fields which should be excluded when the model is nested in another handler's response.

excel_filename

The filename to be given to the attachment, if the the request needs to output to excel format. It can either be a string or a handler method that returns a string. Default value is file.xls

Notes

Adding extra (fake) fields on the response

Sometimes we want to add fake fields on the output of a ModelHandler. Fake fields are fields which are not actual physical model fields, but simply extra information that we wish to include on the handler's response for every model instance.

Note that fake fields have no sense in BaseHandler handlers, since in that case there are no fields, but simply information that the handler emits.

Fake fields can be divided in 2 categories, static and dynamic:

• Fake static fields are fields that are not aware of requests, and for a given model instance X are generated based on other fields of X, or even some external factor like some timestamp. These are declared in the Model class.

• Fake dynamic fields are fields that are aware of requests, and based on the request, generate their values. These are declared in the ModelHandler class.

Fake static fields

Assume that a model has defined fields value1, value2, value3, of type IntegerField. We want to define the fake fields average and median.

In the model class:

_fake_static_fields = ('average', 'median')

def _compute_fake_static_field(self, field):
    if field == 'average':
        return float((self.value1 + self.value2 + self.value3)) / 3

    elif field == 'median':
        numbers = [self.value1, self.value2, self.value3]
        numbers.sort()
        return numbers[1]

The method _compute_fake_static_field is invoked by the Emitter class, which constructs the output of the handler. The field parameter is the field name that is evaluated. So the _compute_fake_static_field method should be able to compute all the field names in the _fake_static_fields tuple.

Fake dynamic fields

Assume that for some requests on a ModelHandler, we want to return, along with the regular model fields, a field that indicates whether the authenticated user who has invoked the request, has a permission to perform some action. So, for every model instance that the response contains we add the permission field, which has a boolean value True of False.

Since this requires checking the request session, it's part of the handler class, instead of the model class.

In the handler class:

def inject_fake_dynamic_fields(self, request, data, fields):
    if 'permission' in fields:
        if isinstance(data, self.model):
            if request.user...:
                data.permission = True
            else:
                data.permission = False
        else:
            for instance in data:
                if request.user...:
                    instance.permission = True
                else:
                    instance.permission = False
    return data

Note that this method is called inject... because it injects the fields in the already generated data response, and returns the enriched response. In the case of the static fake fields, it's called compute..., since it's called by the Emitter class for every static fake field, and returns the value of the field.

From this point on, the API handlers can treat these fields as normal model fields. Meaning, they can be included in the tuples allowed_out_fields, exclude_nested, etc, depending on how you want to treat them.

Bulk POST requests

Bulk POST request refers to a single POST request which attempts to create multiple resources. The specifications of REST or HTTP don't specify any standard behaviour for such requests, and instead discourage its use, due to poor semantics.

For example, how would the API signal an error on one of the data objects in the request body? Or, how would it signal a database error, when all the data objects in the request body were valid?

I chose the following behavior:

• Any error in the request body, will return a Bad Request response. For example when the data in the request body refer to Django models, if even one of the models fails to validate, the response will be 400 Bad Request. The response body will include a list, with all the objects that could not be validated. Every object should have an index parameter, that specifies a zero-bazed index of the request body parameter that could not be validated.

  (Similarly a POST request for a single instance, returns Bad request if the request body does not contain valid data)

• If the request body is valid, the response is OK, and its body contains a list of all the successfully added model instances. If one model instance failed to be created (due for example to a database error), although it contained valid data, it will not be part of the response data.

  (Similarly a POST request for a single instance, returns an OK response, and the model instance in the request body. If the model instance failed to be created, although it was valid, we return an OK response, with null in the response body)

This is in my opinion the most intuitive behavior. However I think that it all depends on the requirements of each application, and the clients using the API. So feel free to modify the existing behavior.

By default bulk POST requests are disabled. They can be enabled by setting bulk_create = True in the handler class.

Building inheritable handlers... Metaclass magic

In this subsection, the term operation means one of read, create, update, delete.

When a handler class sets read = True, basically it says to the system:

I want to inherit the standard read functionality. Please provide me with it.

This works with some metaclass magic. Clearly some magic needs to be in please in order to convert the boolean attribute read, to a method.

The way metaclasses work, is that when a class is initialized, the Python interpreter scans its own member attributes, and then runs the code of the metaclass. In this case, what the metaclass does, is remove all those operations that have been defined with True, like `read = True``, in order to make space for them to be inherited. The metaclass runs on class initialization, whereas the Python MRO runs on runtime.

For this reason, when a handler class is defined, in order to provide inheritable behaviour for other handlers, unless it defines the operations as methods, it needs to provide them as read = True. This way its metaclass will remove these attributes and make "space", for classes that inherit from it, to inherit the behaviour that these operations define. Of course the handlers that inherit from a base handler, will need to first explicitly allow an operation, in case they want to inherit its functionality.

So the way to see it when building handlers:

Setting read = True, means that the handler itself and handlers that inherit from it, will inherit the read functionality, given that they allow so.

Setting read = False, or not setting the read attribute at all, will block the read functionality for the handler and handlers that inherit from it.

Request and response protocol

Here we describe the responses (status code and response body) for all types of HTTP requests, successful or not.

• GET

  • Singular

    • Successful:
      • Status code: 200
        • Response body: Dictionary
    • Errors:
      • Status code: 403
        • Response body: Empty
      • Status code: 405
        • Response body: Empty
      • Status code: 410
        • Response body: Empty

  • Plural

    • Successful:
      • Status code: 200
        • Response body: List
    • Errors:
      • Status code: 403
        • Response body: Empty
      • Status code: 405
        • Response body: Empty

• POST

  • Refers to single resource (Dictionary in request body)
    • Successful:
      • Status code: 200
        • Response body: Dictionary
        • Response body: null (Happens in the case when a database failure prevents the data object from being written to the database)
    • Errors:
      • Status code: 400
        • Response body: Dictionary
      • Status code: 403
        • Response body: Empty
      • Status code: 405
        • Response body: Empty
      • Status code: 422
        • Response body: Dictionary
  • Bulk (List in request body)
    • Successful
      • Status code: 200
        • Response body: List
    • Errors
      • Status code: 400
        • Response body: List (list items are dictionaries. Every dictionary should have anindex parameter which defines a zero-based index of the request body instance that was invalid)
      • Status code: 403
        • Response body: Empty
      • Status code: 405
        • Response body: Empty
      • Status code: 422
        • Response body: List (list items are dictionaries. Every dictionary should have an index parameter which defines a zero-based index of the request body instance that caused the error)

• PUT

  • Singular
    • Successful
      • Status code: 200
        • Response body: Dictionary
    • Errors
      • Status code: 400
        • Response body: Dictionary
      • Status code: 403
        • Response body: Empty
      • Status code: 405
        • Response body: Empty
      • Status code: 410
        • Response body: Empty
      • Status code: 422
        • Response body: Dictionary
  • Plural
    • Successful
      • Status code: 200
        • Response body: List
    • Errors
      • Status code: 400
        • Response body: List (list items are dictionaries. Every dictionary should provide an id parameter which defines the id of the (model) instance that was invalid)
      • Status code: 403
        • Response body: Empty
      • Status code: 405
        • Response body: Empty
      • Status code: 422
        • Response body: List (list items are dictionaries. Every dictionary should provide an id parameter which defines the ``id` of the (model) instance that caused the error)

• DELETE

  • Singular
    • Successful
      • Status code: 200
        • Response body: Dictionary
    • Errors
      • Status code: 403
        • Response body: Empty
      • Status code: 405
        • Response body: Empty
      • Status code: 410
        • Response body: Empty
      • Status code: 422
        • Response body: Dictionary
  • Plural
    • Successful
      • Status code: 200
        • Response body: List
    • Errors
      • Status code: 403
        • Response body: Empty
      • Status code: 405
        • Response body: Empty
      • Status code: 422
        • Response body: List (list items are dictionaries. Every dictionary should provide an id parameter which defined the id of the (model) instance that caused the error)

Note

In the presence of errors, if the response body is a dictionary or a list, every error instance(one in the case of dictionary, multiple in the case of a list) will contain the following keys:

• errors: It will be a dictionary of {field: [error]} pairs where possible, or a list of strings describing the errors.
• type : Error type

In the case of a list of errors, every item will contain the key index, or id, which will specify which request body item, or which data model instance caused the corresponding error.

Examples

# Error response of a POST request for a single resource
{
    "errors": {
        "text": [
            "This field cannot be blank"
        ]
    },
    "type": "Validation Error"
}

# Error response of a Bulk POST request
[
    {
        "index": 0,
        "errors": {
            "gender": [
                "Value u'bi' is not a valid choice."
            ], 
            "email": [
                "Invalid Email"
            ]
        },
        "type": "Validation Error"
    },
    {
        "index": 3,
        "errors":   {
            "postcode": [
                "Invalid Postcode"
            ]
        "type": "Validation Error"
        }
    }
]    

# Error response of a plural DELETE request
[
    {
        "id": 2,
        "errors": [
            "Instance cannot be deleted"
        ],
        "type": "Unprocessable Entity Error"
    },
    {
        "id": 4,
        "errors": [
            "Instance cannot be deleted"
        ],
        "type": "Unprocessable Entity Error"
    }
]