eve-utils

Templates and scripts to rapidly spin up a production-ready Eve-based API.

Please note: although I currently use these tools to create production-ready APIs, the tools themselves are still under development. Use at your own risk. This doc is under a heavy rewrite. Information here is correct, but there are gaps and it's messy.

Introduction

Eve is amazing. The full power of Flask/Python, optimized for an API over mongodb. Nice.

It does take a bit of work to go from the simple example in the docs...

settings = {'DOMAIN': {'people': {}}}

app = Eve(settings=settings)
app.run()

...to a production-ready API, with robust exception handling, logging, control endpoints, configurability, (and so much more).

eve-utils helps make some of that work easier.

Install eve-utils with pip.

pip install eve-utils

Getting Started

All utilities are accessed through the eve-utils command. You can also use eu for short.

Get help

To see what you can do these tools:

eve-utils --help

or

eu --help

As the length of your commands grow, at each step you can always add --help at the end to see your options. It is always safe to tack on a --help as it only shows the help text - the command itself is not executed.

Quick start

NOTE: this 1-2-3 quick start assumes you have a mongodb instance running at localhost. If you have docker installed, you can do this quickly with

docker run --rm -d -p 27017:27017 --name my-mongo mongo

when you are done, clean up with

docker stop my-mongo

Get started with three easy steps

1. Create your API (I recommend creating a virtual environment first). In an empty folder named my-api

   eve-utils api create .

2. Add a resource to the domain

   eve-utils resource create people

3. Launch the API

   eve-utils run

Now, kick the tires of your freshly minted service. Try it out with the following curl command (or use Postman if you prefer)

curl http://localhost:2112
curl http://localhost:2112/_settings?pretty
curl http://localhost:2112/people  (notice _items is an empty array)
curl -X POST http://localhost:2112/people -H "Content-type: application/json" -d "{\"name\":\"Michael\"}"
curl http://localhost:2112/people  (now there is one person in the _items array)

Stop the service (Ctrl+C where the service is running)

Command Cheat Sheet

Use eve-utils to create and manage several objects that make up your API. Below are some definitions and tips to help you develop your API service to its fullest.

Note: you can see these commands listed when you type eve-utils --help.

Commands

Sub-commands

Most of the commands above require a sub-command. For example, to use eve-utils api... you need to say what you want to do with the api.

To see what sub-commands and options are available:

eve-utils command --help

e.g.,

eve-utils api --help

Some commands have unique sub-commands, but many share the following:

• create - create the thing you're talking about, e.g. eve-utils api create... or eve-utils resource create...
• list - show the things you previously created
• remove - removes the thing you previously created. Note - most commands have not yet implemented remove

Command Details

api

There are two commands that run against api

create

Creates the API service. This command is best run in an empty folder.

Service name

The first choice you must make is the name of the service. This is the only required parameter.

e.g.,

eve-utils create . (the service name will be the name of the folder you are in)

eve-utils create whizbang (the service name will be whizbang)

Add-ins

After the name, you can select from several add-ins which enhance your service. All are optional, and you can choose as many as you wish. You can add them at create time, or any time later.

To see the add-ins available. they are listed as Options when you type:

eve utils api create --help

The add-ins are:

NOTE: You will find more details on each add-in in the next section.

You can mix and match these add-ins, e.g.,

eve-utils api create foobar --add_docker --add_git

This is the same as

eve-utils api create foobar -dg

If you want all add-ins, the easiest way is:

eve-utils api create foobar -davwsg

NOTE: when you select --add_git, it will always be added last as it performs the initial commit for you. This way all the add-ins that are installed first will be part of the commit.

addin

If you didn't select an add-in when you created the API, you can always select it later with the addin command.

In other words...

eve-utils api create foobar --add_validation

...is the same as...

eve-utils api create foobar
...
...
eve-utils api addin --add_validation

...no matter how much time passes between those two statements.

All of the add-ins were introduced in the section above. This section provides more details:

--add_git

details

--add_docker

• Adds the following files: Dockerfile`` ``docker-compose.yml (note: by default this file does not use a volume for mongodb, so killing the container also kills your data) .docker-ignore`` ``image-build`` ``image-build.bat

  ...

--add_auth

• Adds a folder named auth with modules to add authorization to your API (docs to come)
  • NOTE: the only supported IdP is Auth0 at the moment, but it will be fairly easy to manually tweak to use any OAuth2 JWT issuer. (I have used a forked Glewlwyd with very minimal changes)

--add_validation

• adds a folder named validation with a module that adds custom validator to EveService. Use this to extend custom validations. It comes with two validations:

  • unique_ignorecase - works exactly like the built-in unique validator except case is ignored

  • unique_to_parent - set this to a string of a resource's parent (singular!). Uniqueness will only be applied to sibling resources, i.e. the same name can be used if the resource has a different parent.

    • e.g.

      eve-utils resource create region
      eve-utils resource create store
      eve-utils link create region store

      Now in domain.store, change the name field definition from this:

      'name": {
        'type': 'string',
              ...
        'unique': True
      }

      to this:

      'name": {
        'type': 'string',
              ...
        'unique_to_parent': 'region'
      }

--add_web_socket

• Define other events/listeners, emitters/senders in web_socket/__init__.py - feel free to remove the default stuff you see there
• There is a test client at {{BASE_API_URL}}/_ws (which you can remove in web_socket/__init__.py by removing the /_ws/chat route)
  • This is useful to see how to configure the Javascript socket.io client to connect to the web socket now running in the API
  • It is also useful to test messages - the chat app merely re-emits what it receives

--add_serverless

• Adds the following files:

  serverless.py - instantiates, but doesn't run, the Eve app object. This object is made available to the serverless framework and is referenced in the .yml files

  serverless-aws.yml

  serverless-azure.yml

  serverless-google.yml

  logging_no-files.yml - copy this over the original logging.yml to eliminate logging to the file system (which is not available with serverless)

• Also installs serverless globally with npm, does an npm init in the root api folder, and locally installs some serverless plugins (node modules).

resource

intro

create

• adds resource-name to the domain

• default fields are name, description

• add fields by modifying domain/resource-name.py - as you would any Eve resource

• NOTE: resources in Eve are collections, so eve-utils names resources as plural by convention,

  • i.e. if you enter mkresource dog it will create an endpoint named /dogs

  • eve-utils rely on the inflect library for pluralization, which is very accurate but can make mistakes

list

details

remove

details

link

Use link to manage parent/child relationships amongst resources.

create

• For example:

  eve-utils resource create person
  eve-utils resource create cars
  eve-utils link create person car

  • you could also have typed eve-utils link create people cars or eve-utils link create person cars - they all are equivalent

• If you followed the example above, you have already POSTed a person named Michael:

  curl -X POST http://localhost:2112/people -H "Content-type: application/json" -d "{\"name\":\"Michael\"}"

• Normally GET a person by _id. eve-utils wires up the name field as an additonal_lookup, so you can also GET by name.

  curl http://localhost:2112/people/Michael?pretty

  {
    "_id": "606f5453b43a8f480a1b8fc6",
    "name": "Michael",
    "_updated": "2021-04-08T19:06:59",
    "_created": "2021-04-08T19:06:59",
    "_etag": "6e91d500cbb0a2f6645d9b4dced422d429a69820",
    "_links": {
      "self": { "href": "/people/606f5453b43a8f480a1b8fc6", "title": "person" },
      "parent": { "title": "home", "href": "/" },
      "collection": { "title": "people", "href": "people" },
      "cars": { "href": "/people/606f5453b43a8f480a1b8fc6/cars", "title": "cars" }
    }
  }

• Notice the _links field includes a rel named cars. You can POST a car to that href (I'll demonstrate with Javascript):

  const axios = require('axios')
  axios.defaults.baseURL = 'http://localhost:2112'
  
  axios.get('/people/Michael').then((response) => {
      const person = response.data
      const car = {
          name: 'Mustang'
      }
      axios.post(person._links.cars.href, car)
  })

• -p --as_parent_ref: field name defaults to _ parent-resource _ref, e.g. if the parent name was dogs the field would be _dog_ref. Using this parameter, the field name become literally _parent_ref. Useful to implement generic parent traversals.

list

details

remove

details

affordance

intro

create

details

list

details

remove

details

endpoint

intro

create

details

list

details

remove

details

integration

intro

create

details

list

details

remove

details

setting

intro

create

details

list

details

remove

details

run

details

Features

This section is under heavy construction. In no particular order...

Settings/configuration system

The EveService ships with a sophisticated settings/configuration management system based on these principles:

• Easy to run: An API should run right out of the box without requiring configuration
• Container compatible: All settings can be set with environment variables.
• Developer convenience: Developers can easily modify settings without accidentally commiting code with experimental values.
  • use _env.conf to set these values - included when your API was created
  • This file is in .gitignore and .dockerignore so it will never accidentally be shipped anywhere but your dev environment
• Visible: all setting values are logged at startup and can be viewed with GET /_settings - no more wondering what value was set when debugging a problem at 3:00 AM

(TODO: cancellable, optional, prefix stuff?, utils.is_enabled() [0] == 'YyTtEe')

Logging

• out of the box, the python logging module is configured with
  • TRACE level
    • use as you wish (LOG.trace('very verbose message here'))
    • Use @trace decorator to automatically log a function's enter and exit (even if exception is thrown)
    • Disable with ES_TRACE_LOGGING to avoid stepping into the detailed logger when debugging (disabled by default in your _env.conf)
  • Enable ES_LOG_TO_FOLDER and logs are created in /var/logs/service-name (which the dev docker-compose.yml maps to local volume so you can view the logs even if the container is stopped)
  • You can be notified by email if the server sends a 5xx response
    • Enable ES_SEND_ERROR_EMAILS
    • then configure ES_SMTP_PORT, ES_SMTP_SERVER, ES_ERROR_EMAIL_RECIPIENTS, and ES_ERROR_EMAIL_FROM
  • View and change the logging verbosity at runtime with a GET and/or PUT to /_logging (provide details)
• Easily extend the logging capabilities using the standard Python logging modules
  • Publish to Slack on 5xx, or whatever circumstance you wish (the 500th GET to a particular resource? - only limit is your imagination)
  • See and modify utils/log_setup.py

Echo

Sometimes you need to test a client's ability to respond to various message or error/status codes. It may be difficult to purposefully generate a 500. In these cases you can enable ES_ADD_ECHO. Then you can PUT to /_echo a JSON as follows:

{
  "status_code": ###,
  "message": {...}
}

This produce a response with status code as specified, with the message value as the body. It also goes through the service logging system, so you can test receiving emails on 5xx, etc. (or however you choose to extend the logging)

Exception Handling

• improves on Eve's out-of-box behaviour by standardizing the error response - even in the case of 5xx's {_status: "ERR", _error: {code:422, message: ""}, _issues: [] }
• your code can call utils.make_error_response() to emit custom error messages that follow this standard.

HAL media type

• Eve's out-of-box JSON structure is very close to HAL. With eve-utils, you get even closer
  • Content-type is application/hal+json
  • _links are tidied up, some superfluous metadata is removed
  • hierarchical navigation follows IANA conventions
  • (this is constantly improving - coming soon curies, _embedded, as well as other standard hypermedia types like Siren, Atom, Collection+JSON, etc.)

API Gateway Registration

(details)

Miscellaneous

• enforced pluralization

  • collections are pluralized
  • items are singularized

• CORS permissively set by default

• utils.get_db() to quickly access the mongodb collections

• utils.get_api() to make http requests in your code to the API itself

• eve-utils errorlevels / exit codes: 1 - not run in API folder structure

  10x - git 101 - git already added

  20x - auth 201 - auth already added

  30x - validation 301 - validation already installed

  40x - docker 401 - docker already installed

  50x - web_socket 501 - web_socket already installed

  60x - serverless 601 - serverless already added 602 - node not installed 603 - serverless not installed 604 - node not initialized 605 - serverless plugin not installed

  70x - resource 701 - resource already exists

  80x - link 801 - link already exists

  90x - integration 901 - integration already exists 902 - name required when choosing empty integration

• organized folder structure

  • designed with more than simple api in mind (e.g. scripts)

  • src, doc, etc

  • FAQ why my-api/src/my-api ?

  • domain decomposition

  • hooks

  • integrations

  • affordances

  • (addins: validation, authorization, etc...)

• common.py

  _x name description