# Airbyte ingestion (ELT)

Source: https://docs.quake.ai/resources/iac-templates/airbyte
Markdown: https://docs.quake.ai/resources/iac-templates/airbyte.md

---

# Airbyte ingestion (ELT)

This [validated OpenTofu template](/docs/platform/validation#how-infrastructure-templates-are-checked) composes Compute, Network, and Block Storage into a self-hosted extract-and-load host you run on infrastructure you control.

## What this template does

Provisions a single instance running [Airbyte](https://airbyte.com), an open-source extract-and-load platform (a self-hosted alternative to Fivetran or Stitch). You configure sources and destinations in the web UI and run replication jobs that land raw data in Object Storage and/or a PostgreSQL warehouse:

- Multi-container Docker stack: web app, worker, metadata PostgreSQL, and Temporal on port 8000
- Default sizing: `m2a.large` (2 vCPU / 8 GiB RAM) and 50 GiB data volume at `/var/lib/docker`
- Private network, security group, floating IP; UI restricted to `ui_allowed_cidr` by default
- cloud-init installs Docker Engine and starts the Airbyte stack from docker compose on first boot

Airbyte is the extract-and-load layer in a self-operated data stack. You run replication jobs on a VM you own, which keeps connection credentials and sync history on your infrastructure.

No credential ships with this template. You configure source and destination credentials in the Airbyte UI after first boot.

## Parameters

| Parameter | Description | Default |
| --- | --- | --- |
| `key_name` | SSH keypair name (must already exist) | No default |
| `flavor_name` | Instance size (Airbyte stack on 2 vCPU / 8 GiB) | `m2a.large` |
| `image_name` | Operating system image | `Ubuntu-24.04` |
| `app_name` | Display name prefix for resources | `airbyte` |
| `volume_size` | Block volume size in GiB, mounted at `/var/lib/docker` | `50` |
| `external_network` | External network for floating IP allocation | `PublicStatic` |
| `private_cidr` | CIDR for the private subnet | `10.49.0.0/24` |
| `ui_allowed_cidr` | CIDR allowed to reach the web UI on port 8000 | `10.49.0.0/24` |
| `airbyte_version` | Pinned Airbyte platform release for first boot | `0.63.15` |

## Web UI access and security

The web UI listens on port 8000 over plain HTTP. The security group restricts 8000 to `ui_allowed_cidr`, which defaults to the private network only, so the raw UI stays off the public internet. Reach the UI one of three ways:

- Put a reverse proxy (Caddy or Nginx) in front of Airbyte and serve the UI over HTTPS on 443. Point the domain's DNS A record at the floating IP. This is the recommended path for routine access.
- Tunnel over SSH: `ssh -L 8000:localhost:8000 user@FLOATING_IP`, then open `http://localhost:8000`.
- Set `ui_allowed_cidr` to `YOUR_IP/32` to reach port 8000 directly from one address.

Ports 80 and 443 stay open for the reverse proxy you put in front; they carry no traffic until you add one.


Connector workspaces, metadata, and sync logs live on the data volume. Snapshot the volume before you resize or rebuild the host.


## Destinations

Configure destinations in the Airbyte UI after first boot:

- **Object Storage:** point a destination at your Quake AI Object Storage bucket (S3-compatible endpoint and access keys you create in the Console).
- **PostgreSQL warehouse:** point a destination at a [self-managed PostgreSQL](/resources/iac-templates/self-managed-postgres) instance on the same private network.

Connection secrets stay in the Airbyte UI; they are never set in tfvars or committed to the repo.

## When to use this pattern

Run an extract-and-load tool with hundreds of source connectors on a VM you operate. Airbyte suits SaaS-to-warehouse replication, database CDC, and landing raw files in Object Storage before downstream transforms.

For workflow orchestration around the pipeline, pair with Apache Airflow once that template lands in the data tooling catalog. For the warehouse destination, see [self-managed PostgreSQL](/resources/iac-templates/self-managed-postgres). For BI on top of the warehouse, see Metabase or Apache Superset.

## Estimated cost

<PricingCompanion
  components={[
    { kind: "template", slug: "airbyte", required: true },
  ]}
/>

## Template source

<TemplateSource slug="airbyte" />

<TemplateResourceMap template="airbyte" format="opentofu" />

## Customize this pattern

- [Customize a template's image and flavor](/docs/automation/how-to/customize-template-image-flavor)
- [Add a block volume to a template](/docs/automation/how-to/add-volume-to-template)
- [Parameterize a template with a tfvars file](/docs/automation/how-to/parameterize-template-tfvars)

## See also

- [Deploy Airbyte](/resources/deployments/deploy-airbyte-template)
- [self-managed PostgreSQL](/resources/iac-templates/self-managed-postgres)
- [S3 storage ACL template](/resources/iac-templates/s3-storage-acl)
- [data pipelines and analytics brief](/resources/solutions/data-pipelines-and-analytics)