Deploy Airflow with the airflow template

Stand up Apache Airflow, an open-source workflow orchestration platform, on a single Quake AI instance using the validated OpenTofu template airflow. You apply the template, read the generated admin credentials, sign in to the web UI, trigger a sample DAG, and optionally serve the UI over HTTPS.

Airflow schedules and monitors data pipelines on infrastructure you own. You run it yourself; this is a self-hosted tool you operate, not a managed orchestration cloud.

Prerequisites#

You need:

• OpenTofu 1.6.0 or later (or Terraform 1.6.0 or later) installed locally.
• Your OpenStack credentials sourced into the shell (source openrc.sh). See the OpenStack CLI guide.
• An SSH keypair that already exists in your project. Record its name for the key_name variable.
• A copy of the airflow template directory from the template reference page.
• Your workstation's public IP address, so you can open the web UI port to it for first-boot setup. Find it with curl -sS https://api.ipify.org.

A domain is optional for first boot. You add it in step 4 to serve the UI over HTTPS.

Step 1: Set the variables and apply the template#

The web UI listens on port 8080 over plain HTTP. The template's security group restricts port 8080 to ui_allowed_cidr, which defaults to the private network only, so the raw UI stays off the public internet. To reach the UI from your workstation for first-boot setup, set ui_allowed_cidr to your own address.

Copy the template's example variables file and open it:

cp terraform.tfvars.example terraform.tfvars

Set key_name to the SSH keypair already in your project, and ui_allowed_cidr to your workstation's public IP with a /32 suffix:

key_name         = "YOUR_KEY_NAME"
ui_allowed_cidr  = "YOUR_IP/32"

Initialize the working directory, preview the plan, and apply:

tofu init
tofu plan
tofu apply

OpenTofu provisions a private network, a router, a security group, a block volume mounted at /var/lib/docker, an instance, and a floating IP. On first boot, cloud-init mounts the data volume, installs Docker Engine, generates secrets, and starts Airflow (webserver, scheduler, and metadata PostgreSQL) from a compose file on port 8080.

When the apply finishes, read the outputs:

tofu output

Record floating_ip and web_ui_url.

Step 2: Sign in to the web UI#

Airflow does not ship a default password. cloud-init generates the admin password on first boot and writes it to /opt/airflow/credentials.txt on the instance.

cloud-init takes several minutes after the instance reaches ACTIVE. Watch the containers start over SSH:

ssh ubuntu@YOUR_FLOATING_IP "sudo docker compose -f /opt/airflow/docker-compose.yml ps"

When the webserver and scheduler containers report healthy, read the login details:

ssh ubuntu@YOUR_FLOATING_IP "sudo cat /opt/airflow/credentials.txt"

Open web_ui_url (for example http://YOUR_FLOATING_IP:8080) in your browser. Sign in with username admin and the password from the credentials file. Store the password in your password manager and remove or restrict access to credentials.txt on the instance when you are done.

Step 3: Trigger the sample DAG#

The template ships a sample DAG at /opt/airflow/dags/hello_quake.py. New DAGs start paused.

1. In the Airflow UI, open DAGs and find hello_quake.
2. Toggle the pause switch to unpause the DAG.
3. Open the DAG and select Trigger DAG (play icon).
4. Open the Graph or Grid view and confirm the hello task reaches success.

The task runs echo "Hello from Quake AI Airflow" on the scheduler container. Check logs from the task instance detail page or over SSH:

ssh ubuntu@YOUR_FLOATING_IP "sudo docker compose -f /opt/airflow/docker-compose.yml logs airflow-scheduler --tail 20"

To add your own pipelines, copy Python DAG files into /opt/airflow/dags on the instance (or sync them from Git in a later iteration). Airflow picks up new files within its scan interval.

Step 4: Serve the UI over HTTPS with Caddy#

The template leaves ports 80 and 443 open for a reverse proxy. Caddy obtains and renews a TLS certificate automatically once a domain resolves to the instance.

1. Create a DNS A record for your domain (for example airflow.example.com) pointing at YOUR_FLOATING_IP. Follow How to point a domain at a Quake AI resource. Wait until the record resolves:

dig +short airflow.example.com

The command returns your floating IP once the record propagates.

1. SSH to the instance and add a Caddy service that proxies HTTPS to Airflow on port 8080. Create /opt/airflow/Caddyfile:

airflow.example.com {
  reverse_proxy 127.0.0.1:8080
}

1. Add Caddy to the compose file at /opt/airflow/docker-compose.yml so it runs alongside Airflow:

services:
  caddy:
    image: caddy:2
    restart: unless-stopped
    network_mode: host
    volumes:
      - /opt/airflow/Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
volumes:
  caddy_data:

1. Tell Airflow its public base URL. Append to /opt/airflow/.env:

AIRFLOW__WEBSERVER__BASE_URL=https://airflow.example.com

1. Apply the changes and confirm Caddy and the Airflow containers run:

cd /opt/airflow
sudo docker compose up -d
sudo docker compose ps

Open https://airflow.example.com and confirm the padlock. For background on certificate issuance and renewal, see How to issue and auto-renew a TLS certificate with Let's Encrypt. Once HTTPS works, close direct access to port 8080 by setting ui_allowed_cidr back to the private network in terraform.tfvars and running tofu apply.

Kubernetes executor scale path#

This deployment runs LocalExecutor on one VM. When schedules or task concurrency outgrow that shape, move to the Kubernetes executor:

1. Provision a cluster with the Kubernetes cluster template.
2. Deploy Airflow worker pods on the cluster while keeping the scheduler and webserver on a control node, or migrate the full stack into the cluster with the official Helm chart or a compose overlay you maintain.
3. Point Airflow at the cluster API and configure worker pod templates for your task images (for example a dbt container invoked from a KubernetesPodOperator).

The single-instance template in this walkthrough stays the entry pattern. The cluster template is the reuse point for scale-out execution without changing how you author DAGs.

What you built#

• Applied the airflow template to provision a network, security group, data volume, instance, and floating IP, and let cloud-init install Docker and start Airflow with LocalExecutor
• Signed in to the web UI with the admin credentials cloud-init generated on first boot
• Triggered the sample DAG and confirmed the task succeeded
• Served the UI over HTTPS by pointing a domain at the floating IP and routing it through a Caddy reverse proxy

Scope of this deployment#

This template runs a single-VM Airflow host with LocalExecutor, not a managed orchestration cloud. The instance is CPU-only and runs in one region. You operate the instance, Docker, Airflow, and the data volume yourself: back them up, patch them, and watch resource use as pipeline volume grows. Snapshot the data volume before you resize or rebuild the host. For higher concurrency, follow the Kubernetes executor path above and reuse the k8s-cluster template.

Next steps#

• Apache Airflow template: the template reference, parameters, and resource map
• self-managed PostgreSQL template: a warehouse or metadata store to query from DAG tasks
• Kubernetes cluster template: the cluster footprint for Kubernetes executor workers
• How to store application secrets and inject them at runtime: move connection strings and API keys out of plain environment variables
• Security hardening checklist: tighten SSH access and exposure before you serve real traffic

Clean up#

When you no longer need the deployment, destroy everything the template created:

tofu destroy

Then remove the DNS A record you created in step 4. Because Airflow, its metadata database, and DAG logs all live on the instance and its attached volume, tofu destroy removes them along with the infrastructure. Export any DAG files you want to keep before you destroy.