hot

WARNING: Use at your own risk! This comes with no warranties, guarantees, or promises of any kind.

Command line helper utility for authoring and performing extended validation testing on Heat templates. The name comes from the acronym for Heat Orchestration Templates. This is a very Rackspace-y tool. If people are willing to help contribute, we can open up some of the more provider specific things like the linting functions.

Requirements

• For Ubuntu 12.04/14.04, install the necessary packages:
  • sudo apt-get install -y python-dev python-pip gcc libxslt1-dev python-lxml git
• Python 2.7 - hot will NOT work with Python 3 since it leverages fabric.
• All systems: pip 1.5.6+ must be installed.
• virtualenv is recommended.
• The following environmental variables must be properly set before running hot:
  • OS_PASSWORD
  • OS_USERNAME
  • OS_TENANT_ID
  • OS_AUTH_URL
  • HEAT_URL
• Optionally, you can set OS_AUTH_TOKEN as opposed to OS_PASSWORD.

Installation

Install with pip (recommended):

pip install hot

Clone it down, install it:

git clone git@github.com:heat-tools/hot.git
cd hot
virtualenv .hot
source .hot/bin/activate
pip install -r requirements.txt
python setup.py install

Usage

Ensure you have activated your virtual environment before every use.

The hot command takes a single verb, ie: 'test'. Issuing the command without a verb will result in the usage output:

(.hot)~/src/hot $ hot
usage: hot [-h] {test,docs,init,lint} ...
hot: error: too few arguments

hot must be run inside of a template repository that contains a tests.yaml file. tests.yaml dictates how hot will spin up deployments and test them. Use the public memcached template repo as a starting point for using hot.

Creating Template Tests

Tests are specified in the tests.yaml file in the root of the template repo. If your test scripts require additional dependencies or libraries, ensure those are installed prior to hot running. hot will not resolve dependencies for you.

get_output

There is a useful function called get_output that can be called to plug in outputs from your heat stack. It will replace itself with the output value for a given output-list key. The output you fetch, must be in the outputs section of your Heat template. This function will only work in the resource_tests section of your tests file since the stack must be up before the values can be retrieved. An example of use would be:

hosts: { get_output: server_ip }

This example will look for the key server_ip in the output-list of the stack once it comes up. This get_output function will then replace itself with the value of server_ip. Assuming the IP comes out to be 1.2.3.4, the dictionary will be updated and interpreted like this when the tests are run:

hosts: 1.2.3.4

get_env

The get_env function is very similar to get_output. If included in a tests file, it will replace the value with the corresponding environment variable. If that environment variable is not set, it will throw a KeyError. Example:

server_hostname: { get_env: TESTING_HOSTNAME }

This function will work inside of any section within your tests file.

Creating/Launching a stack

The test-cases are a list of tests to run. They are run sequentially, and there is no limit to the number you can run. There are only a couple options:

• name: Required value that must be a string
• create: Hash of
  • parameters: Optional hash. If no parameters are provided, a heat stack will be launched with the default values specified in the template.
    
  • timeout: Optional integr, in minutes. Set how long to wait for a stack to complete building. If it takes longer than the timeout value, mark it as a failure, and delete the stack. If something is taking hours to build, something's wrong.

Here's an example:

test-cases:
- name: Default Build Test # Deploy using all default options
  create:
    parameters:
      flavor: 2GB Standard Instance
      memcached_port: 11212  
    timeout: 30 # Deployment should complete in under 30 minutes

The test will be considered successful if the template builds successfully within the user defined timeout window.

Test Options

The test options are documented if you run hot test --help:

$ hot test --help
usage: hot test [-h] [--template TEMPLATE] [--tests-file TESTS_FILE] [-k]
                [-s SLEEP] [--test-cases TEST_CASES [TEST_CASES ...]]
                [-P <KEY1=VALUE1;KEY2=VALUE2...>] [--insecure]

 Test a template by going through the test scenarios in 'tests.yaml' or
    the tests file specified by the user


optional arguments:
  -h, --help            show this help message and exit
  --template TEMPLATE   Heat template to launch. (default: '.catalog')
  --tests-file TESTS_FILE
                        Test file to use. (default: 'tests.yaml')
  -k, --keep-failed     Do not delete a failed test deployment. (default:
                        False)
  -s SLEEP, --sleep SLEEP
                        Frequency for checking test stack status. (default:
                        15)
  --test-cases TEST_CASES [TEST_CASES ...]
                        Space delimited list of tests to run. If none are
                        specified, all will be run. (default: -)
  -P <KEY1=VALUE1;KEY2=VALUE2...>, --parameters <KEY1=VALUE1;KEY2=VALUE2...>
                        Parameter values used to create the stack. This can be
                        specified multiple times, or once with parameters
                        separated by a semicolon. The parameters specified
                        here will override anything defined in the tests.
                        (default: -)
  --insecure            Same as to -k flag with curl, do not strictly validate
                        SSL certificates. (default: False)

As a note, if you have spaces, commas, or other special characters, put the test name in double quotes, and each string will be interpreted individually. Example:

hot test --test-cases "First Test" "Second Test, with special options"

Resource Tests

These are a list of tests with arbitrary names to run. The keys under the test name define what test suite to run. As of now, your options are fabric or script.

There are two values that can be put directly under resource tests to be laid down for all tests that are run:

• ssh_private_key: Actual value of the SSH private key for logging into Linux resources.
• ssh_key_file: Path where we should lay down the ssh_private_key. This is in place since the key file has specific permission requirements.

Fabric

hot supports template yaml syntax checking, stack build success verification, and highly-flexible fabric-based integration testing to test the resulting servers using envassert and any other methods possible in a python script. You really can do anything.

The ideal use case for fabric tests are when you want to tests from within a resource. This is great for checking firewall rules, running processes, and anything else you would like to check on a system.

Under the fabric section, the only thing you need to define is your fabric environment. All of the settings under env are fabric environmental variables. At a minimum, the fabfile, which defines where your test script is located, the tasks, which are the @task sections in your fab file, and the hosts, which define the IP(s) to test are required.

Here's an example tests.yaml showing how to setup fabric tests for our Memcached templates:

test-cases:
- name: Default Build Test # Deploy using all default options
  create:
    timeout: 30 # Deployment should complete in under 30 minutes
  resource_tests: # Tests to run on the resources themselves
    ssh_private_key: { get_output: private_key } # Fetch from output-list of stack
    ssh_key_file: tmp/private_key # File to write with ssh_private_key
    tests:
    - memcached_server:
        fabric:
          # Fabric environment settings to use while running envassert script
          # http://docs.fabfile.org/en/latest/usage/env.html
          env:
            user: root
            key_filename: tmp/private_key
            hosts: { get_output: server_ip } # Fetch from output-list of stack
            tasks:
              - check
            abort_on_prompts: True
            connection_attempts: 3
            disable_known_hosts: True
            use_ssh_config: True
            fabfile: test/fabric/memcached_server.py # Path to envassert test

The yaml syntax checks and build success verification happen by default for each test-case. The test memcached-server invokes fabric against the ip addresses contained in the stack output server_ip using the fabfile at the path test/fabric/memcached_server.py, running the fabric task check.

This task uses envassert to ensure the server is in the desired state:

@task
def check():
    assert package.installed("memcached")
    assert file.exists("/etc/memcached.conf")
    assert port.is_listening(11211)
    assert process.is_up("memcached")
    assert service.is_enabled("memcached")

If an exception is encountered, the test is considered failed.

While envassert is useful for basic server state checks, any python code may be used in the fabfile. The code runs in the context of a fabric session, so fabric methods such as run() and get() are available to execute remote commands and inspect their output. See the fabric API docs for more options.

Here is an example function to check the state of an application and the associated fabric task to invoke it:

import re
from fabric.api import env, run, hide, task


def vestacp_is_responding():
    with hide('running', 'stdout'):
        homepage = run('curl --insecure https://localhost:8083/login/')
        if re.search('Vesta Control Panel', homepage):
            return True
        else:
            return False

@task
def check():
    assert vestacp_is_responding(), 'vestacp did not respond as expected.'

If the /login endpoint does not return a document containing the string Vesta Control Panel, the test will fail, and the string vestacp did not respond as expected. will appear in the stack trace.

Script

The ideal use case for script tests are when you want to tests from outside the environment. Examples could include running selenium tests from your CI server, verifying service availability from outside the environment, or calling third party services to run tests such as vulnerability assessment or performance testing. You can really do anything in here.

The setup is very similar to the fabric tests. Tests will be run in the order they are provided, and it's possible to sequentially run a fabric test and then a script, or vice versa. The parts of the script test are optional, no default values will be assumed or set. Here are the major sections:

• hosts
• files
• environment
• commands

Hosts

A list of hashes that include the IP and all hostnames to associate to that IP. The user running the script either needs write access to the hosts file, or sudo permissions to run sudo mv /tmp/etc_hosts.tmp /etc/hosts. hot will try to overwrite /etc/hosts, if that fails, it'll write /tmp/etc_hosts.tmp and then try to move it with the previously mentioned sudo command. The function will append any values in the hosts section, so all values that were in the hosts file before the test was run should be there after the fact. This does not currently back up or restore the original hosts file after the run. This is designed with containers in mind where a fresh container will be launched for every test.

Within the list of hosts, there should be two keys within each hash:

• ip: string of the IPv4 or IPv6 address to set in the hosts file.
• hostnames: list of hostnames to associate with the ip value.

Files

A list of files to write. Each element will have a key and value. The key will be the file name to write, the value will be what's written into the file. Files will be written as the user that hot was launched as.

Environment

A list of environmental variables to set prior to running the command.

Commands

The commands are run after the optional hosts, files, and environment settings are applied. This is a list of commands and optional command_args to pass in. The breakdown for each command is as follows:

• command: The command to run
• command_args: Optional list of additional arguments to append to the end of the command.

Here's an example config:

test-cases:
- name: Default Build Test # Deploy using all default options
  create:
    timeout: 30 # Deployment should complete in under 30 minutes
  resource_tests:
    tests:
    - site_test:
        script:
          hosts: # Requires user write access to /etc/hosts or passwordless
                 # sudo ability to overwrite the hosts file. See Hosts section
                 # in documentation.
            - ip: 1.2.3.4
              hostnames:
              - example.com
              - stats.example.com
              - { get_output: host }
            - ip: { get_output: server_ip }
              hostnames:
              - example2.com
              - admin.example.com
          files: # Write the following files as user. Syntax: File: "value to put in file"
            - tmp/key2.pem: { get_output: secondary_key }
            - groot.txt: 'I am Groot'
          environment:
            - hosts: { get_output: server_ip }
            - key: { get_output: private_key }
            - local: localhost
          commands:
            - command: "python test/script/example.py"
              command_args:
                - "--ip"
                - { get_output: server_ip }
            - command: "./special validate"
              command_args:
                - "--ip"
                - { get_output: server_ips }