apitools

Tools to play with json-schema and rest apis


License
MIT
Install
pip install apitools==0.1.4

Documentation

Tools to play with json-schemas defined APIs.

These tools are based on json-schema draft 3 from http://tools.ietf.org/html/draft-zyp-json-schema-03 Not all features of the schema are supported and probably won't be. Handling of not supported feature varies between the different tools.

All these tools are proofs of concept and work in progress, they need more extensive testing and documentation.

datagenerator

Class to generate random values given a json-schema.
Doesn't support all json-schema monstruousities, only a subset I find useful.
See TODO.md for what is likely to be implemented next.

Examples

from datagenerator import DataGenerator

generator = DataGenerator()

Basic

Generate random values of each basic type using

>>> generator.random_value("string")
'Olzq3LV'
>>> generator.random_value("number")
-6.675904074356879
>>> generator.random_value("integer")
30
>>> generator.random_value("boolean")
True

Basic with constraints

number

>>> generator.random_value({"type":"number", "minimum":30})
32.34295327292445
>>> generator.random_value({"type":"number", "maximum":30})
-35.80704939879546
>>> generator.random_value({"type":"number", "maximum":30, "minimum":12})
16.45747265846327

integer supports minimum and maximum like number and more

>>> generator.random_value({"type":"integer", "maximum":30, "divisibleBy":4, "minimum":12})
24
>>> generator.random_value({"type":"integer", "maximum":30, "exclusiveMaximum":True, "minimum":28})
29

(same for exclusiveMinimum)

string supports minLength, maxLength, pattern (ignores minLength and maxLength if pattern is used)

>>> generator.random_value({"type":"string", "maxLength":20, "minLength":15})
'VytPCEdAImX11188HU'
>>> generator.random_value({"type":"string", "pattern":"[0-9]{3}[a-zA-Z]{2,5}"})
u'806FoNP'

boolean doesn't have any constraints.

Arrays

Without constraints the array size will be picked the same way as a random integer.
Each item in the array is generated using the default generator for the type given in items.

>>> generator.random_value({"type":"array", "items": {"type":"string"}})
[
    '39yxcpvS5tfPf6O', 
    'sNDk7SlGNQstxxx', 
    'nPcRSD9yIP7j ', 
    'PWP7KQfjc1', 
    'tt6F6Z2YEp'
]

minItems, maxItems and uniqueItems are supported

The type of object in items can be anything that the generator knows about, either one of the basic types or a user defined one available from the generator's schemas store.

from schemasstore import SchemasStore

...
>>> from schemasstore import SchemasStore
>>> store = SchemasStore()
>>> generator.schemas_store = store
>>> store.add_schema({"type":"integer", "name":"small_integer", "minimum":0,"maximum":9})
True
>>> generator.random_value({"type":"array", "uniqueItems":True, "minItems":10, "items":{"type":"small_integer"}})
[0, 7, 2, 5, 3, 6, 1, 4, 8, 9]

See datagenerator for other examples.

Objects

Objects can be generated the same way as the other types.

Example generating search_result.json

>>> store.load_folder("data/schemas/")
>>> generator.random_value("search_result")
{u'price': 21.980325774975253, u'name': 'wdvfXYrrt', u'reference': 26}

Generating arrays of objects is fine as well

>>> generator.random_value({"type":"array", "maxItems":3, "minItems":2, "items":{"type":"search_result"}})
[
    {u'price': 20.304440535786522, u'name': 'VUIgjaPbs', u'reference': 40}, 
    {u'price': 28.45387747055259, u'name': 'JTycBU1V78X1S', u'reference': 27}
]

Or generating objects with arrays of other objects in them, see search_resuts with an array of search_result

>>> generator.random_value("search_results")
{
    u'total_results': 41, 
    u'total_pages': 26, 
    u'current_page': 33, 
    u'items_per_page': 27, 
    u'results': [
        {u'price': 26.218704680177446, u'name': 'B4p1Z1pOFQO', u'reference': 38}, 
        {u'price': 21.205089550441276, u'name': 'FQPHdLds', u'reference': 7}, 
        {u'price': 20.610536930894398, u'name': '8D862p1XVupP', u'reference': 38}, 
        {u'price': 9.543934434058526, u'name': 'PmqBA0e DIWisf', u'reference': 32}
    ]
}

Schemas

Why not generate random schemas?

>>> r_schema = generator.random_schema()
>>> r_schema
{
    'type': 'object', 
    'properties': {
        u'viYXjhu': {'required': False, 'type': 'boolean'}, 
        u'TO': {'required': False, 'type': 'string'}, 
        u'NTSd': {'required': False, 'type': 'string'}, 
        u'WjaL': {'required': False, 'type': 'string'}, 
        u'PtvhZ': {'required': False, 'type': 'boolean'}
    }, 
    'name': u'zJllGkKosmocOVO'
}

And then generate an array of random values of it

>>> store.add_schema(r_schema)
True
>>> generator.random_value({"type":"array", "minItems":1, "maxItems":3, "items":{"type":"zJllGkKosmocOVO"}})
[
    {u'TO': 'jamKFpdwY'}, 
    {u'WjaL': '8LnibWUdsSI', u'PtvhZ': True}, 
    {}
]

Notes on the generation

All the values are generated using the random module, so please don't use the generate values for anything requiring reliable randomness == don't use it to generate passwords.

To generate the data, the generator has to limit the range of possible values, so the values generated don't vary too wildly. The ranges are controlled by variables in DataGenerator. Feel free to tweak them, especially if you need values that don't fall into those ranges without having to set both minimum and maximum on your properties.


urlsgenerator

Class to generate links defined in the links section of a json-schema.

Example

Generate links from book.json

Input

...
    "isbn" : {
        "type":"string",
        "required":true,
        "pattern":"^\\d{12}(\\d|X)$"
    }

    },
    "links" : [
    {
        "rel":"self",
        "href":"books/{isbn}"
    },
    {
        "rel":"instances",
        "href": "books"
    }
    ]
...

Output

{
    u'instances': [u'books'], 
    u'self'     : [u'books/525259838909X']
}

{isbn} got replaced by a random value 525259838909X satisfying the constraints on isbn (matches the regex).


invaliddatagenerator

Class to generate invalid data for a given schema

Basically does the opposite of datagenerator. WIP, needs documentation and examples.


modelgenerator

Base class to generate models from a schema, nothing too visible on its own, check resourceserver.


flasksqlalchemymodelgenerator

Generate SQLAlchemy models to be used with flask-sqlalchemy from a schema. Uses modelgenerator. Used in resourceserver to store and query items.


backbonemodelgenerator

Generate models and collections for Backbone.js from a schema.
The models generated use the primary key defined in the rel=self link or id by default.
To be able to use collections, make sure your schema has a rel=instances link or fetch won't work.

Usage

$ python backbonemodelgenerator.py -h
Usage: backbonemodelgenerator.py jsonfile1 [jsonfile2]...

Options:
  -h, --help            show this help message and exit
  -t OUTPUT_TYPE, --type=OUTPUT_TYPE
                        Output type (js|wrapped|html)

Output types

js

Outputs only the js code for the models/collections

$ python backbonemodelgenerator.py -t js data/schemas/message.json

App.Models.Message = Backbone.Model.extend({
    urlRoot: '/messages',
    idAttribute: 'id'
});

App.Collections.Messages = Backbone.Collection.extend({
    model : App.Models.Message,
    url : "/messages"
});

wrapped

Wraps the js code into $(document).ready()

$ python backbonemodelgenerator.py -t wrapped data/schemas/message.json

$(document).ready(function() {

    window.App = { Models : {}, Collections : {} };

    App.Models.Message = Backbone.Model.extend({
        urlRoot: '/messages',
        idAttribute: 'id'
    });

    App.Collections.Messages = Backbone.Collection.extend({
        model : App.Models.Message,
        url : "/messages"
    });

});

html

Same as wrapped but generate a whole html page including jQuery, Backbone and Underscore to easily test.

Example usage

Setup

You can use it with resource server for example

$ mkdir static
$ python backbonemodelgenerator.py -t html data/schemas/message.json > static/index.html
$ python resourceserver.py data/schemas/message.json
Added message
 * Running on http://0.0.0.0:5000/

Now open your browser at http://0.0.0.0:5000/static/index.html Open your js console to start playing

Create a collection and fetch them

var col = new App.Collections.Messages()
col.fetch()

You should see backbone talking to the resource server in the server shell

127.0.0.1 - - [20/Nov/2012 01:17:15] "GET /messages HTTP/1.1" 200 -

You can inspect the results using

col.models

Using fetch() only works if your schema includes a link with rel=instances

Create a new message

var msg = new App.Models.Message({recipient:"01234567890", text:"test message"})
msg.attributes

At that point the message is not saved yet, you can verify by using

msg.isNew()

You can save it on the server using

msg.save()

You can verify that the message was sent to the server in the server shell

127.0.0.1 - - [20/Nov/2012 01:23:24] "POST /messages HTTP/1.1" 201 -

Now you should have an id for the message and it shouldn't be marked as new anymore.

msg.id
msg.isNew()

Fetch an existing message

Create a message with the id of the message to fetch

var msg = new App.Models.Message({id: 3})

The message is not marked as new as it has an id.
We can then fetch the actual message from the server using

msg.fetch()
msg.attributes()

You can see the query in the server shell again

127.0.0.1 - - [20/Nov/2012 01:25:41] "PUT /messages/3 HTTP/1.1" 200 -

Update a message

Once you have a message object, you can update it using save.

> msg.attributes.recipient
"01234567890"
> msg.save({recipient:"00123456789"})
> msg.attributes.recipient
"00123456789"

This is done by doing a PUT on the server

127.0.0.1 - - [20/Nov/2012 01:33:35] "PUT /messages/3 HTTP/1.1" 200 -

Delete a message

Simply use destroy on the object

msg.destroy()

And see the DELETE happening on the server

127.0.0.1 - - [20/Nov/2012 01:34:48] "DELETE /messages/3 HTTP/1.1" 204 -

resourceserver

Class to implement the REST api of resources defined in a schema.
Supports creation, update, retrieval, deletion, listing of instances and schema.

Usage

Run the server using

$ python resourceserver.py [jsonfile1, jsonfile2, ...]

Example using data/schemas/message.json

$ python resourceserver.py data/schemas/message.json
Added message
 * Running on http://0.0.0.0:5000/

Create a new message

$ curl -i -X POST    http://0.0.0.0:5000/messages -d "recipient=07771818335&text=nice message"
$ curl -i -X POST    http://0.0.0.0:5000/messages -d '{"recipient":"01234567890", "text":"test"}' \
       -H "Content-Type: application/json"
HTTP/1.0 201 CREATED
Content-Type: application/json
Content-Length: 13
Location: http://0.0.0.0:5000/messages/2
Server: Werkzeug/0.8.3 Python/2.7.3
Date: Sun, 18 Nov 2012 19:28:56 GMT

{
  "id": 2
}

List messages

$ curl -i -X GET     http://0.0.0.0:5000/messages
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 126
Server: Werkzeug/0.8.3 Python/2.7.3
Date: Sun, 18 Nov 2012 19:32:09 GMT

[
  {"text": "I </3 ninjas", "recipient": "07771818337", "id": 1},
  {"text": "nice message", "recipient": "07771818335", "id": 2}
]

Retrieve a message

$ curl -i -X GET     http://0.0.0.0:5000/messages/2
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 71
Server: Werkzeug/0.8.3 Python/2.7.3
Date: Sun, 18 Nov 2012 19:35:42 GMT

{
  "text": "nice message",
  "recipient": "07771818335",
  "id": 2
}

Get the json-schema of a message

$ curl -i -X OPTIONS http://0.0.0.0:5000/messages/2
HTTP/1.0 200 OK
Content-Type: application/json
Content-Length: 590
Server: Werkzeug/0.8.3 Python/2.7.3
Date: Sun, 18 Nov 2012 19:37:06 GMT

{
  "description": "Simple message structure",
  "type": "object",
  "properties": {
    "text": {
      "required": true, 
      "type": "string", 
      "maxLength": 140
    }, 
    "recipient": {
      "pattern": "0[0-9]{10}", 
      "required": true,
      "type": "string"
    },
    "id": {
      "minimum": 0,
      "type": "integer"
    }
  },
  "links": [
    {
      "href": "/messages",
      "rel": "root"
    },
    {
      "href": "{id}",
      "rel": "self"
    },
    {
      "rel": "instances"
    },
    {
      "rel": "create"
    }
  ],
  "name": "message"
}

Update a message

Supports partial updates

$ curl -i -X PUT     http://0.0.0.0:5000/messages/2 -d 'recipient=07771818336'
$ curl -i -X PUT     http://0.0.0.0:5000/messages/1 -d '{"text":"foo"}' \
          -H "Content-Type: application/json"
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 0
Server: Werkzeug/0.8.3 Python/2.7.3
Date: Sun, 18 Nov 2012 19:38:02 GMT

Delete a message

$ curl -i -X DELETE  http://0.0.0.0:5000/messages/2
HTTP/1.0 204 NO CONTENT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Server: Werkzeug/0.8.3 Python/2.7.3
Date: Sun, 18 Nov 2012 19:38:38 GMT

Errors examples

Trying to set an implicit key

The message.json doesn't define an explicit primary key, but defines id as the key in the rel=self link.
Each message then gets an additional id key managed by the server.
Trying to set or update the id results in errors

$ curl -i -X POST    http://0.0.0.0:5000/messages   -d "recipient=07771818335&text=nice message&id=7"
$ curl -i -X PUT     http://0.0.0.0:5000/messages/1 -d "recipient=07771818335&text=nice message&id=3"
HTTP/1.0 400 BAD REQUEST
Content-Type: application/json
Content-Length: 43
Server: Werkzeug/0.8.3 Python/2.7.3
Date: Sun, 18 Nov 2012 19:43:48 GMT

{
  "error": "id is read only in message"
}

Trying to create or update unknown properties

$ curl -i -X POST    http://0.0.0.0:5000/messages   -d "recipient=07771818335&tet=test&haxxy=foo"
$ curl -i -X PUT     http://0.0.0.0:5000/messages/1 -d "haxxy=foo"
HTTP/1.0 400 BAD REQUEST
Content-Type: application/json
Content-Length: 57
Server: Werkzeug/0.8.3 Python/2.7.3
Date: Sun, 18 Nov 2012 19:56:19 GMT

{
  "error": "message does not have a 'haxxy' property"
}

Trying to create or update properties with values not respecting constraints

$ curl -i -X PUT     http://0.0.0.0:5000/messages/1 -d "recipient=0notanumber&text=nice message"
$ curl -i -X POST    http://0.0.0.0:5000/messages   -d "recipient=0notanumber"
HTTP/1.0 400 BAD REQUEST
Content-Type: application/json
Content-Length: 86
Server: Werkzeug/0.8.3 Python/2.7.3
Date: Sun, 18 Nov 2012 20:03:34 GMT

{
  "error": "'0notanumber' is an invalid recipient value: must match u'0[0-9]{10}'"
}

Trying to create a message without all the required properties

$ curl -i -X POST    http://0.0.0.0:5000/messages -d "recipient=012345678901"HTTP/1.0 400 BAD REQUEST
Content-Type: application/json
Content-Length: 44
Server: Werkzeug/0.8.3 Python/2.7.3
Date: Sun, 18 Nov 2012 20:06:00 GMT

{
  "error": "text is required in message"
}

Trying to create a message in json with invalid data

$ curl -i -X POST    http://0.0.0.0:5000/messages  -d '{"recipient":"01234567890", "text":"test}' -H "Content-Type: application/json"
HTTP/1.0 400 BAD REQUEST
Content-Type: application/json
Content-Length: 90
Server: Werkzeug/0.8.3 Python/2.7.3
Date: Tue, 20 Nov 2012 00:23:05 GMT

{
  "error": "Invalid data: Unterminated string starting at: line 1 column 35 (char 35)"
}

Primary keys

Each model needs a primary key. There are 3 ways to define the primary key of the model:

If there is no rel=self link, an additional id (or appended with as many _ as necessary to make the name unique) attribute is created. This type of key is called implicit and can only be set by the server (read only).

If there is a rel=self link and it contains a {variable} part, the variable name is used as the primary key.

  • If variable is the name of an existing property, this property is used as the primary key, and can be updated ( explicit key )
  • Otherwise an implicit key is created using the variable name (stil read-only).

Example of an explicit key

This schema uses isbn as the explicit key. Instances can be created using a specific isbn, and its value can be updated.

...
    "isbn" : {
        "type":"string",
        "required":true,
        "pattern":"^\\d{12}[\\d|X]$"
    }

    },
    "links" : [
    {
        "rel":"self",
        "href":"books/{isbn}"
    },
...

Example of implicit key

This schema defines an implicit key order_id (assuming no property is called order_id).

...
    "links" : [
        {
            "rel":"self",
            "href":"{order_id}"
        },
...

Dependencies

Optional

datagenerator, invaliddatagenerator and urlgenerator

Use rstr hosted in a mercurial repo on bitbucket. Run init.sh in dependencies to fetch it. If you don't have mercurial, apt-get install mercurial should help.

flasksqlalchemymodelgenerator and resourceserver

flask-sqlalchemy is required, use flasksqlalchemy-requirements.txt with virtualenv

backbonemodelgenerator

jinja2 is required, comes with flask if you use the flasksqlalchemy-requirements.txt