Eryx Deploy

Object-oriented devops. Period.

BETA status.

Features

• Fully object-oriented: Intuitive and almost self-documented usage.
• Easy to extend.
• Safe: Ask for confirmation on EVERY step (unless configured otherwise, see global settings below).
• Idempotent tasks: All tasks can run multiple times without colateral effects.
• Tested manually on common escenarios (see below).

Usage

First time setup

1. Add eryxdeploy to your dev.txt / local.txt requirements and update them. (Create a new file and a virtualenv if your project is not a python project.)

2. Create a python package in your project root called devops with 2 empty files: secrets.json and stacks.py.

3. Add devops/secrets.json to .gitignore. This file will be used to store info that is insecure to share in your repo.

4. Create an empty fabfile.py file on your project's root dir, importing this lib and the stacks package:

   from eryx_deploy import *
   from devops import stacks
   

5. Configure your app stack(s) in devops/stacks.py.

   Example:

   import json
   
   from fabric.state import env
   
   from eryx_deploy import DjangoBasicStack, UbuntuMachine, AmazonRDSMachine, SassCompileWithGrunt, NPM
   
   secrets = json.load(open('devops/secrets.json'))
   
   env.local_stack = DjangoBasicStack(host=UbuntuMachine.new_local(project_path=secrets['local_project_path']),
                                  project_name='project_xxx',
                                  db_name='project_xxx_dev',
                                  db_user=secrets['local_db_user'],
                                  db_password=secrets['local_db_password'],
                                  remote_url='git@gitlab.com:eryx/project_xxx.git',
                                  requirements_file_path='requirements/local.txt',
                                  virtual_env_path=secrets['local_virtual_env_path'],
                                  web_assets_pipeline=SassCompileWithGrunt,
                                  frontend_package_manager=NPM)
   
   env.stacks['staging'] = DjangoBasicStack(host=UbuntuMachine.new_remote(hostname='project_xxx.eryx.co',
                                                                          ssh_connection='project-xxx',
                                                                          project_path=secrets['staging_project_path']),
                                            project_name='project_xxx',
                                            db_name='project_xxx_staging',
                                            db_user=secrets['staging_db_user'],
                                            db_password=secrets['staging_db_password'],
                                            db_port=secrets['staging_db_port'],
                                            db_server=AmazonRDSMachine(
                                                hostname=secrets['staging_db_server']),
                                            remote_url='git@gitlab.com:eryx/project_xxx.git',
                                            requirements_file_path='requirements/master.txt',
                                            virtual_env_path='%s/bootstrap' % secrets['staging_project_path'],
                                            web_assets_pipeline=SassCompileWithGrunt,
                                            frontend_package_manager=NPM)
   

   The only required thing here is to set the env.stacks dictionary with an app stack tied to an environment configuration. In the example above, we're setting the staging environment with a DjangoBasicStack instance.

   Configuring a local_stack with your workstation configuration is also recommended as some devops tasks depends on it.

   Please check eryx_deploy/app_stacks directory for configuring this and other app stacks.

6. Add your secrets in JSON format on devops/secrets.json.

7. You are now ready to run tasks on your app stacks! Before finishing, please take a moment to document deployment instructions on your README.md to help you and other developers in the future.

Running tasks

First Deploy

To perform an initial deploy of your remote stack on a specific environment, just run from the command line:

$ fab --set env=ENV first_deploy

Replace ENV for staging, production or other environments.

Deploy

To upgrade your remote stack from your local machine, just run:

$ fab --set env=ENV deploy

Replace ENV for staging, production or other environments.

Sync Local DB

To sync your local db with the db from a remote environment, just run:

$ fab --set env=YOUR_ENV sync_local_db

Replace YOUR_ENV for staging, production or other environments.

Global settings

You can configure additional global settings on your fabfile.py.

• env.confirm_before_run: Ask for confirmation before running EVERY command. Disable after you are confident with the tool. Default: True.
• env.allow_skipping_cmd: Allows skipping commands: Saying NO when asking for confirmation to run a command, will ask again if you want to abort or continue. Useful for debugging. Default: False.

Check default_config.py for all options.

Assets

App stacks are configured using many different technology assets. Please check assets dir for all the available classes, segmented by technology type.

Migrating from previous versions

From 0.3.x -> 0.4.x

1. Rename your secrets.py to secrets.json and format the values as a JSON. Change .gitignore accordingly.

2. Change stacks.py to use secrets from the new file. Check example above to see how to import the JSON file.

Tested scenarios

• Django project with PostgreSQL 9.5, Nginx with Gunicorn, NPM, SASS with grunt, on:

  • Digital Ocean (Ubuntu 16.04.3 x64 image).
  • Amazon (EC2 Ubuntu 16.04 x64 AMI) using RDS.

• Django project with PostgreSQL 9.5, Nginx with Gunicorn, on:

  • Amazon (EC2 Ubuntu 16.04 x64 AMI) using RDS (and without it).

• Ruby on Rails project with PostgreSQL 9.5, Nginx with Gunicorn, on:

  • Amazon (EC2 Ubuntu 16.04 x64 AMI) using RDS.

Please add your scenario to this list after testing!

Contributing

Setup

1. Clone this repo on your machine.

2. Make sure to install dev_requirements.txt on a new virtual environment.

3. On the project you are using the tool, uninstall the stable version with pip uninstall eryxdeploy and install a local copy with:

   $ pip install -e /path/to/eryxdeploy
   

4. Any changes you make on the cloned repo will be available instantly from your project.

To debug from a Jetbrain IDE, add a python runner with a full path to fab binary installed on your project's virtualenv (NOT from erxydeploy's project, but from the project were eryxdeploy is installed!). Open eryxdeploy source directly from your project and add a breakpoint there.

Extending the tool

Assets are divided by technology. Any new asset of an existing technology type, like a new frontend package manager should subclass from the corresponding abstract class. Any new type of technology should define a new package dir and an abstract class. Asset constructors should provide default collaborators when possible. The criteria here is to use the most COMMON/DEFAULT implementation.

Machines are an important/special kind of asset. Those assets are responsible for performing the actions ultimately. Most other assets collaborate with machines.

When creating a new app stack, try to define all configuration parameters directly from the constructor. Add most RECOMMENDED default assets (considering nowadays standards) if the stack can use multiple variants.

As this is a library, make sure not to break backwards-compatibility unless is really necessary. Some guidelines:

• Changing the __init__ signature of any asset will probably break compatibility. If you add a new parameter, set a default value.
• Take special care NOT to change App Stack subclasses __init__ signature.
• If you make a breaking change, add a migration section on this README file.
• Import new classes on __init__.py files so the user of the lib can always import doing eryx_deploy.AssetClass, no matter internal package organization.
• Always use or extend from an abstract class when dealing with an asset that may have alternative implementations. Do not override __init__ on subclasses unless you are sure what you are doing.

Design considerations

• When adding a new tool/asset consider that it may coexist with other apps that may also use the same tools. This implies that configuration for a project should not collide with other configuration. For example, if you implement a webserver that uses supervisor to run it, consider that other services may also use supervisor.
• Don't assume your app will always be the only app running on a host machine. Take this into account when building config files.
• All side effects (commands that perform tasks) must be done through a Machine/ExecutableEnvironment instance. Add a machine as an internal collaborator if your assets is supposed to run commands.
• When possible, try not to couple an Asset to a particular environment (production, staging, etc).

Running test suite

• Requires Docker and docker-compose to be configured.

$ python test.py

No tests for now!

Uploading a new release to PyPi

You must have an account on https://pypi.python.org to upload a new version.

1. Be sure to move version number up first.

2. Create package and upload with:

   $ python setup.py bdist_wheel
   $ twine upload dist/*
   

Changelog

0.4.7 (01/08/2019)

• Rails: Fix "TypeError: 'unicode' object is not callable" when deploying Puma

0.4.6 (08/10/2018)

• Rails: Use rails environment to install package group with Bundle

0.4.5 (01/10/2018)

• Add option to select custom Django project root.

0.4.4 (20/03/2018)

• Fix: Also run precompile assets for Rails on staging environment.

0.4.3 (05/03/2018)

• Support for Python 3 projects for the Django stack.

0.4.2 (16/01/2018)

• Bugfix: Fix broken Django environment ('virtual_environment' argument error)

0.4.1 (20/12/2017)

• Fix: Support other RAILS_ENV values than 'production'

0.4.0 (19/12/2017)

• Support for Ruby on Rails stack.
• Move secrets to JSON file format instead of Python to be more language-agnostic.
• More refactoring
• Improved README.md

0.3.0 (02/11/2017)

• New task SyncLocalDB to sync a remote db with your local environment!
• Nginx config improvement: Now supports multiple webservers on the same machine without collision.
• Minor supervisor fix: Reload config before restarting daemon.
• Some refactoring

0.2.0 (27/10/2017)

• Object-oriented deep refactor.
• Added support for initial deploys.
• Idempotent tasks!
• Ask confirmation before every step (can be turned off).
• Use Fabric3 fork for Python 3 support.