# Redpanda Kafka-API streaming

Source: https://docs.quake.ai/resources/iac-templates/redpanda
Markdown: https://docs.quake.ai/resources/iac-templates/redpanda.md

---

# Redpanda Kafka-API streaming

This [validated OpenTofu template](/docs/platform/validation#how-infrastructure-templates-are-checked) composes Compute, Network, and Block Storage into a self-hosted Kafka-API streaming broker you run on infrastructure you control.

## What this template does

Provisions a single instance running [Redpanda](https://www.redpanda.com), an open-source streaming platform with a Kafka-compatible API in one binary (a self-hosted alternative to Confluent Cloud or Amazon MSK):

- Docker Compose stack: one Redpanda broker and optional Redpanda Console on port 8080
- Default sizing: `s1a.small` (2 vCPU / 2 GiB RAM) and a 50 GiB data volume at `/var/lib/redpanda/data` for topic logs
- Private network, security group, floating IP; Kafka (9092), Schema Registry (8081), HTTP Proxy (8082), and Console (8080) restricted to `client_allowed_cidr` by default
- cloud-init installs Docker Engine and starts the stack on first boot; the broker advertises the floating IP for external clients

Redpanda covers the streaming-ingestion path in a data pipeline: producers and consumers use standard Kafka clients against `kafka_bootstrap` from the template outputs. Wire Airbyte, Flink, or your application producers to the bootstrap address and create topics with `rpk` or your client library. No credential ships with this template; the broker starts with PLAINTEXT listeners and you add TLS or SASL when you harden the host.

## Parameters

| Parameter | Description | Default |
| --- | --- | --- |
| `key_name` | SSH keypair name (must already exist) | No default |
| `flavor_name` | Instance size (single-node broker on 2 vCPU / 2 GiB) | `s1a.small` |
| `image_name` | Operating system image | `Ubuntu-24.04` |
| `app_name` | Display name prefix for resources | `redpanda` |
| `volume_size` | Block volume size in GiB, mounted at `/var/lib/redpanda/data` | `50` |
| `external_network` | External network for floating IP allocation | `PublicStatic` |
| `private_cidr` | CIDR for the private subnet | `10.60.0.0/24` |
| `client_allowed_cidr` | CIDR allowed to reach Kafka, Schema Registry, HTTP Proxy, and Console | `10.60.0.0/24` |
| `enable_console` | Start Redpanda Console on port 8080 | `true` |

## Client access and security

The broker listens on port 9092 with PLAINTEXT. Schema Registry uses 8081 and HTTP Proxy uses 8082. Redpanda Console uses 8080 when `enable_console` is true. The security group restricts those ports to `client_allowed_cidr`, which defaults to the private network only. Reach the services one of three ways:

- Put a reverse proxy (Caddy or Nginx) in front of Console on 443 and keep Kafka on a private network or VPN.
- Tunnel over SSH: `ssh -L 9092:localhost:9092 -L 8080:localhost:8080 ubuntu@FLOATING_IP`, then point clients at `localhost:9092`.
- Set `client_allowed_cidr` to `YOUR_IP/32` to reach the ports directly from one address.

Ports 80 and 443 stay open for the reverse proxy you put in front; they carry no traffic until you add one.


Topic logs live on the data volume at `/var/lib/redpanda/data`. Snapshot the volume before you resize or rebuild the host, or you lose retained messages.


## When to use this pattern

Run a Kafka-compatible streaming broker for event ingestion, log pipelines, and microservice decoupling on a VM you operate. Redpanda suits the streaming layer in a self-hosted data stack alongside [Airflow](/resources/iac-templates/airflow), [Airbyte](/resources/iac-templates/airbyte), and a [Postgres warehouse](/resources/iac-templates/self-managed-postgres).

For object storage and lake tables rather than a message bus, see [MinIO and Iceberg](/resources/iac-templates/minio-iceberg). For federated SQL over the lake, see [Trino](/resources/iac-templates/trino).

## Estimated cost

<PricingCompanion
  components={[
    { kind: "template", slug: "redpanda", required: true },
  ]}
/>

## Template source

<TemplateSource slug="redpanda" />

<TemplateResourceMap template="redpanda" format="opentofu" />

## Customize this pattern

- [Customize a template's image and flavor](/docs/automation/how-to/customize-template-image-flavor)
- [Add a block volume to a template](/docs/automation/how-to/add-volume-to-template)
- [Parameterize a template with a tfvars file](/docs/automation/how-to/parameterize-template-tfvars)

## See also

- [Deploy Redpanda](/resources/deployments/deploy-redpanda-template)
- [Airflow orchestration](/resources/iac-templates/airflow)
- [self-managed PostgreSQL](/resources/iac-templates/self-managed-postgres)