# Apache Airflow orchestration

Source: https://docs.quake.ai/resources/iac-templates/airflow
Markdown: https://docs.quake.ai/resources/iac-templates/airflow.md

---

# Apache Airflow orchestration

This pattern composes Compute, Network, and Block Storage into a self-hosted workflow orchestration host you run on infrastructure you control.

## What this template does

Provisions a single instance running [Apache Airflow](https://airflow.apache.org), an open-source workflow orchestration platform (a self-hosted alternative to AWS MWAA or Astronomer). You define pipelines as DAGs, schedule them, and track task dependencies and retries:

- Compute instance that runs Airflow in Docker with LocalExecutor (webserver, scheduler, and bundled metadata PostgreSQL), sized for moderate DAG volume (4 vCPU and 4 GiB RAM)
- Private network, subnet, router, port, and security group; a floating IP for public access
- A block volume mounted at `/var/lib/docker`, so the metadata PostgreSQL database, DAG logs, and Docker named volumes live on a volume you can grow rather than on the boot disk
- cloud-init installs Docker Engine and starts Airflow from a compose file on first boot

Airflow is the orchestration layer for data pipelines. It schedules tasks and tracks dependencies on a VM you own, which keeps DAG code, logs, and connection metadata on your infrastructure.

No credential ships with this template. cloud-init generates the metadata database password, Fernet key, and admin password on first boot and writes the login details to `/opt/airflow/credentials.txt` on the instance.

## Parameters

| Parameter | Description | Default |
| --- | --- | --- |
| `key_name` | SSH keypair name (must already exist) | No default |
| `flavor_name` | Instance size (LocalExecutor plus metadata PostgreSQL runs on 4 vCPU / 4 GiB) | `s1a.medium` |
| `image_name` | Operating system image | `Ubuntu-24.04` |
| `app_name` | Display name prefix for resources | `airflow` |
| `volume_size` | Block volume size in GiB, mounted at `/var/lib/docker` | `40` |
| `external_network` | External network for floating IP allocation | `PublicStatic` |
| `private_cidr` | CIDR for the private subnet | `10.40.0.0/24` |
| `ui_allowed_cidr` | CIDR allowed to reach the web UI on port 8080 | `10.40.0.0/24` |

## Web UI access and security

The web UI listens on port 8080 over plain HTTP. The security group restricts 8080 to `ui_allowed_cidr`, which defaults to the private network only, so the raw UI stays off the public internet. Airflow's login still gates the UI. Reach it one of three ways:

- Put a reverse proxy (Caddy or Nginx) in front of Airflow and serve the UI over HTTPS on 443. Point the domain's DNS A record at the floating IP, then set `AIRFLOW__WEBSERVER__BASE_URL` in `/opt/airflow/.env`. This is the recommended path for routine access.
- Tunnel over SSH: `ssh -L 8080:localhost:8080 user@FLOATING_IP`, then open `http://localhost:8080`.
- Set `ui_allowed_cidr` to `YOUR_IP/32` to reach port 8080 directly from one address.

Ports 80 and 443 stay open for the reverse proxy you put in front; they carry no traffic until you add one.


Airflow stores connection secrets encrypted with the Fernet key generated on first boot. The key lives in `/opt/airflow/.env` on the instance. Snapshot the data volume before you resize or rebuild the host, and back up DAG files from `/opt/airflow/dags`.


## Executor and scale path

This template runs LocalExecutor on a single instance: the scheduler and workers share the same VM, and metadata PostgreSQL runs in Docker alongside them. That shape suits getting started, moderate schedules, and pipelines that invoke external tools (for example a dbt container or SQL against a [self-managed PostgreSQL](/resources/iac-templates/self-managed-postgres) instance).

When task concurrency outgrows one VM, move to the Kubernetes executor and reuse the [Kubernetes cluster template](/resources/iac-templates/k8s-cluster). You deploy Airflow workers as pods on the cluster while keeping the scheduler and webserver on a control node or moving the full stack into the cluster. The [Deploy Airflow with the airflow template](/resources/deployments/deploy-airflow-template) walkthrough calls out that path in prose.

## When to use this pattern

Run a workflow orchestration platform with a web UI, cron-style scheduling, and Python-defined DAGs on a VM you operate. Airflow suits ETL pipelines, batch jobs, and orchestration around analytics transforms.

For lightweight API-to-API glue without DAG authoring, use the [n8n workflow template](/resources/iac-templates/n8n-workflow). For the warehouse or datastore your DAGs query, see [self-managed PostgreSQL](/resources/iac-templates/self-managed-postgres).

## Estimated cost

<PricingCompanion
  components={[
    { kind: "template", slug: "airflow", required: true },
  ]}
/>

## Template source

<TemplateSource slug="airflow" />

<TemplateResourceMap template="airflow" format="opentofu" />

## Customize this pattern

- [Customize a template's image and flavor](/docs/automation/how-to/customize-template-image-flavor)
- [Add a block volume to a template](/docs/automation/how-to/add-volume-to-template)
- [Parameterize a template with a tfvars file](/docs/automation/how-to/parameterize-template-tfvars)

## See also

- [self-managed PostgreSQL](/resources/iac-templates/self-managed-postgres)
- [Kubernetes cluster](/resources/iac-templates/k8s-cluster)
- [n8n workflow automation](/resources/iac-templates/n8n-workflow)