Dagsesh: Airflow DAG Session Manager
- Overview
- Prerequisites
- Getting Started
- Help
- Running the Test Harness
- Using
dagsesh
Plugin in your Project's Test Suite - FAQs
Overview
An Apache Airflow session context manager that overrides the default AIRFLOW_HOME
path with a random, ephemeral alternate.
Why is this useful? As per the Airflow configuration docs:
The first time you run Airflow, it will create a file called
airflow.cfg
in your$AIRFLOW_HOME
directory (~/airflow
by default).
Dagsesh delays the creation of AIRFLOW_HOME
using a lazy-loading facility whilst injecting it with a random alternate. Great if you want to create a pristine Airflow environment for repeatable testing.
Dagsesh can be used as a convenient pytest plugin to prime an Airflow environment for testing.
Prerequisites
- GNU make
- Python 3 Interpreter. We recommend installing pyenv
- Docker
Getting Started
Makester is used as the Integrated Developer Platform.
(macOS Users) Upgrading GNU Make
Follow these notes to get GNU make.
Creating the Local Environment
Get the code and change into the top level git
project directory:
git clone git@github.com:loum/dagsesh.git && cd dagsesh
NOTE: Run all commands from the top-level directory of the
git
repository.
For first-time setup, get the Makester project:
git submodule update --init
Initialise the environment:
make init-dev
Local Environment Maintenance
Keep Makester project up-to-date with:
git submodule update --remote --merge
Help
There should be a make
target to get most things done. Check the help for more information:
make help
Running the Test Harness
We use pytest. To run the tests:
make tests
dagsesh
Plugin in your Project's Test Suite
Using Add the dagsesh
package as a dependency to your project's development environment so that the plugin is installed and visible in your PYTHONPATH
. dagsesh
takes care of the distribution's entry point so that pytest
automatically finds the plugin module. Nothing else needs to be done.
NOTE: See Making your plugin installable by others for more information.