Web IDE for Apache Airflow workflows development.

• Phoebe by codbex
  • Description
  • Run steps
    • Start using Docker, SQLite and released image
      • Start Docker image
    • Start using Docker, PostgreSQL and released image
      • Start PostgreSQL
      • Start Docker image
    • Build the project jar
    • Start using Docker Compose and local sources
    • Java standalone application
      • Start Airflow
        • Option 1 - using SQLite
        • Option 2 - using PostgreSQL
      • Start the application
    • Multi-platform Docker build
      • Spring profiles
    • Run unit tests
    • Run integration tests
    • Run all tests
    • Format the code
  • Configurations
  • Access the application

Phoebe is a Web IDE which allows you to write Apache Airflow application in an efficient way.

It has the following perspectives:

• Workbench
• Integrated Apache Airflow instance and embedded Apache Airflow Web UI
• Git
• Database Management
• Terminal

It also helps you to easily start your work using the defined Apache Airflow starter template.

Prerequisites:

• Export the following variables before executing the steps

  export GIT_REPO_FOLDER='<path-to-the-git-repo>'
  
  export PHOEBE_CONTAINER_NAME='phoebe'
  export DEV_IMAGE='codbex-phoebe:dev'
  export PHOEBE_IMAGE='ghcr.io/codbex/codbex-phoebe:latest'
  
  export PHOEBE_AIRFLOW_POSTGRES_USER="postgres"
  export PHOEBE_AIRFLOW_POSTGRES_PASS="postgres"
  export PHOEBE_AIRFLOW_POSTGRES_DB="postgres"
  
  export AIRFLOW_WORK_DIR="$HOME/airflow_work"
  export PHOEBE_AIRFLOW_WORK_DIR="$AIRFLOW_WORK_DIR"
  
  export GITHUB_USERNAME='<your-github-user>'
  

docker rm -f "$PHOEBE_CONTAINER_NAME"

docker pull "$PHOEBE_IMAGE"

docker run --name "$PHOEBE_CONTAINER_NAME"  \
    -p 80:80 \
    $PHOEBE_IMAGE

Note: SQLite is located at path /opt/airflow/airflow.db

The instance which will be used for Airflow DB or used existing DB instance.

docker rm -f postgres

docker run --name postgres \
  -e POSTGRES_PASSWORD="$PHOEBE_AIRFLOW_POSTGRES_PASS" \
  -e POSTGRES_USER="$PHOEBE_AIRFLOW_POSTGRES_USER" \
  -e POSTGRES_DB="$PHOEBE_AIRFLOW_POSTGRES_DB" \
  -p 5432:5432 \
  -d postgres:16
  

docker rm -f "$PHOEBE_CONTAINER_NAME"

docker pull "$PHOEBE_IMAGE"

docker run --name "$PHOEBE_CONTAINER_NAME"  \
    -p 80:80 \
    -e PHOEBE_AIRFLOW_POSTGRES_USER="$PHOEBE_AIRFLOW_POSTGRES_USER" \
    -e PHOEBE_AIRFLOW_POSTGRES_PASS="$PHOEBE_AIRFLOW_POSTGRES_PASS" \
    -e PHOEBE_AIRFLOW_POSTGRES_HOST="host.docker.internal" \
    -e PHOEBE_AIRFLOW_POSTGRES_DB="$PHOEBE_AIRFLOW_POSTGRES_DB" \
    $PHOEBE_IMAGE
    

cd $GIT_REPO_FOLDER

mvn -T 1C clean install -P quick-build

Prerequisites: Build the project jar

cd "$GIT_REPO_FOLDER/application"

# cleanup
docker rm -f "$PHOEBE_CONTAINER_NAME"
docker compose down -v

# To force rebuild add --build
# Needed when you modify something in Dockerfile or in the application
docker compose up --build

• Start Airflow locally

  cd "$GIT_REPO_FOLDER"
  
  docker rm -f airflow
  
  docker run --name airflow  \
     -p 8080:8080 \
     -v "$AIRFLOW_WORK_DIR/dags:/opt/airflow/dags" \
     -v "$AIRFLOW_WORK_DIR/logs:/opt/airflow/logs" \
     -v "$AIRFLOW_WORK_DIR/config:/opt/airflow/config" \
     -e AIRFLOW__CORE__LOAD_EXAMPLES=False \
     -e _AIRFLOW_DB_MIGRATE=true \
     -e AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL=5 \
     -e AIRFLOW__CORE__EXECUTOR=LocalExecutor \
     -e AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_ALL_ADMINS=True \
     -d apache/airflow:3.0.2 standalone

• Ensure Airflow is started at http://localhost:8080

• Start PostgreSQL

• Start Airflow locally

  cd "$GIT_REPO_FOLDER"
  
  docker rm -f airflow
  
  docker run --name airflow  \
     -p 8080:8080 \
     -v "$AIRFLOW_WORK_DIR/dags:/opt/airflow/dags" \
     -v "$AIRFLOW_WORK_DIR/logs:/opt/airflow/logs" \
     -v "$AIRFLOW_WORK_DIR/config:/opt/airflow/config" \
     -e AIRFLOW__CORE__LOAD_EXAMPLES=False \
     -e _AIRFLOW_DB_MIGRATE=true \
     -e AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL=5 \
     -e AIRFLOW__CORE__EXECUTOR=LocalExecutor \
     -e AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_ALL_ADMINS=True \
     -e AIRFLOW__DATABASE__SQL_ALCHEMY_CONN="postgresql+psycopg2://$PHOEBE_AIRFLOW_POSTGRES_USER:$PHOEBE_AIRFLOW_POSTGRES_PASS@host.docker.internal:5432/$PHOEBE_AIRFLOW_POSTGRES_DB" \
     -d apache/airflow:3.0.2 standalone

• Ensure Airflow is started at http://localhost:8080

• Build the project jar

• Start the application

  cd "$GIT_REPO_FOLDER"
  
  java \
      --add-opens=java.base/java.lang=ALL-UNNAMED \
      --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \
      --add-opens=java.base/java.nio=ALL-UNNAMED \
      -jar application/target/*executable*.jar
  

• Start the application in debug with debug port 8000

  cd "$GIT_REPO_FOLDER"
  
  java \
      -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 \
      --add-opens=java.base/java.lang=ALL-UNNAMED \
      --add-opens=java.base/java.lang.reflect=ALL-UNNAMED \
      --add-opens=java.base/java.nio=ALL-UNNAMED \
      -jar application/target/*executable*.jar
  

Prerequisites: Build the project jar

cd "$GIT_REPO_FOLDER/application"

export DOCKER_CLI_EXPERIMENTAL=enabled
docker buildx create --use

# build image for linux/amd64
docker buildx build --platform linux/amd64 -t $DEV_IMAGE --load .

# build image for linux/arm64
docker buildx build --platform linux/arm64 -t $DEV_IMAGE --load .

# build images for both platforms
docker buildx build --platform=linux/arm64,linux/amd64 -t $DEV_IMAGE -o type=image .

# build multiplatform images and push them to GitHub Container Registry
docker login ghcr.io -u "$GITHUB_USERNAME"

docker buildx build \
    --platform linux/amd64,linux/arm64 \
    -t "ghcr.io/$GITHUB_USERNAME/$DEV_IMAGE" \
    --push .
    
## pull images locally

# linux/amd64
docker pull "ghcr.io/$GITHUB_USERNAME/$DEV_IMAGE" --platform linux/amd64

# linux/arm64
docker pull "ghcr.io/$GITHUB_USERNAME/$DEV_IMAGE" --platform linux/arm64

• Eclipse Dirigible profiles To activate Eclipse Dirigible, you have to add profiles common and app-default explicitly.
  Example for profile snowflake: SPRING_PROFILES_ACTIVE=common,snowflake,app-default

cd "$GIT_REPO_FOLDER"
mvn clean install -P unit-tests

cd "$GIT_REPO_FOLDER"
mvn clean install -P integration-tests

cd "$GIT_REPO_FOLDER"
mvn clean install -P tests

cd "$GIT_REPO_FOLDER"
mvn verify -P format

The following configurations are available:

Note: PHOEBE_AIRFLOW_POSTGRES_USER, PHOEBE_AIRFLOW_POSTGRES_PASS, PHOEBE_AIRFLOW_POSTGRES_DB and PHOEBE_AIRFLOW_POSTGRES_HOST are optional. If one of them is not provided, SQLite will be used for Airflow DB.

Depending on the use case these configurations could be set in different ways.

• For java standalone application they could be set as environment variables.

  export PHOEBE_AIRFLOW_URL='http://localhost:8080'
  java -jar ...

• For docker run

  docker run --name "$PHOEBE_CONTAINER_NAME"  \
      -p 80:80 \
      -e PHOEBE_AIRFLOW_POSTGRES_USER="$PHOEBE_AIRFLOW_POSTGRES_USER" \
      -e PHOEBE_AIRFLOW_POSTGRES_PASS="$PHOEBE_AIRFLOW_POSTGRES_PASS" \
      -e PHOEBE_AIRFLOW_POSTGRES_HOST="host.docker.internal" \
      -e PHOEBE_AIRFLOW_POSTGRES_DB="$PHOEBE_AIRFLOW_POSTGRES_DB" \
      $PHOEBE_IMAGE
  

• When using docker compose they could be set in the docker-compose.yml file.

  services:
    phoebe:
      environment:
        PHOEBE_AIRFLOW_POSTGRES_USER: postgres
        PHOEBE_AIRFLOW_POSTGRES_PASS: postgres
        PHOEBE_AIRFLOW_POSTGRES_HOST: host.docker.internal
        PHOEBE_AIRFLOW_POSTGRES_DB: postgres

• Open URL http://localhost in your browser
• Login with default credentials admin / admin