Redpanda Kafka-API streaming
Redpanda Kafka-API streaming
This validated OpenTofu template composes Compute, Network, and Block Storage into a self-hosted Kafka-API streaming broker you run on infrastructure you control.
What this template does#
Provisions a single instance running Redpanda, an open-source streaming platform with a Kafka-compatible API in one binary (a self-hosted alternative to Confluent Cloud or Amazon MSK):
- Docker Compose stack: one Redpanda broker and optional Redpanda Console on port 8080
- Default sizing:
s1a.small(2 vCPU / 2 GiB RAM) and a 50 GiB data volume at/var/lib/redpanda/datafor topic logs - Private network, security group, floating IP; Kafka (9092), Schema Registry (8081), HTTP Proxy (8082), and Console (8080) restricted to
client_allowed_cidrby default - cloud-init installs Docker Engine and starts the stack on first boot; the broker advertises the floating IP for external clients
Redpanda covers the streaming-ingestion path in a data pipeline: producers and consumers use standard Kafka clients against kafka_bootstrap from the template outputs. Wire Airbyte, Flink, or your application producers to the bootstrap address and create topics with rpk or your client library. No credential ships with this template; the broker starts with PLAINTEXT listeners and you add TLS or SASL when you harden the host.
Parameters#
| Parameter | Description | Default |
|---|---|---|
key_name | SSH keypair name (must already exist) | No default |
flavor_name | Instance size (single-node broker on 2 vCPU / 2 GiB) | s1a.small |
image_name | Operating system image | Ubuntu-24.04 |
app_name | Display name prefix for resources | redpanda |
volume_size | Block volume size in GiB, mounted at /var/lib/redpanda/data | 50 |
external_network | External network for floating IP allocation | PublicStatic |
private_cidr | CIDR for the private subnet | 10.60.0.0/24 |
client_allowed_cidr | CIDR allowed to reach Kafka, Schema Registry, HTTP Proxy, and Console | 10.60.0.0/24 |
enable_console | Start Redpanda Console on port 8080 | true |
Client access and security#
The broker listens on port 9092 with PLAINTEXT. Schema Registry uses 8081 and HTTP Proxy uses 8082. Redpanda Console uses 8080 when enable_console is true. The security group restricts those ports to client_allowed_cidr, which defaults to the private network only. Reach the services one of three ways:
- Put a reverse proxy (Caddy or Nginx) in front of Console on 443 and keep Kafka on a private network or VPN.
- Tunnel over SSH:
ssh -L 9092:localhost:9092 -L 8080:localhost:8080 ubuntu@FLOATING_IP, then point clients atlocalhost:9092. - Set
client_allowed_cidrtoYOUR_IP/32to reach the ports directly from one address.
Ports 80 and 443 stay open for the reverse proxy you put in front; they carry no traffic until you add one.
When to use this pattern#
Run a Kafka-compatible streaming broker for event ingestion, log pipelines, and microservice decoupling on a VM you operate. Redpanda suits the streaming layer in a self-hosted data stack alongside Airflow, Airbyte, and a Postgres warehouse.
For object storage and lake tables rather than a message bus, see MinIO and Iceberg. For federated SQL over the lake, see Trino.
Estimated cost#
Monthly cost estimate
Pricing calculator ↗Sized as a custom package on shared vCPU.
Monthly total for the required template above. Use the configurator below to add optional pieces and see the total update.
What each resource is for
Redpanda broker
s1a.small · 2 shared vCPU, 2 GiB RAM, 0.5 Gbps
Runs Redpanda in Docker with Kafka-API streaming and optional Redpanda Console.
Single-node profile on 2 vCPU and 2 GiB RAM for light ingestion; size up for higher throughput or longer log retention.
Compute shown per role at custom-package rates ($29/dedicated vCPU, $7.25/shared vCPU, $1/GiB RAM). The headline above is the billed total: the cheaper of a named plan and the custom package, plus add-ons.
Included in baseline
s1a.small
2 shared vCPU, 2 GiB RAM, 0.5 Gbps
Compute + RAM rate basis
2 vCPU + 2 GiB RAM at $29/dedicated vCPU, $7.25/shared vCPU, $1/GiB RAM (regular). Totals apply the flat −$5/mo package promotion.
Block storage (80 GiB)
80 GiB at $0.08/GiB/mo
Public IP (included)
1 included with the custom package
Package promotional discount
Flat −$5.00/mo on the custom package (same promotion as named plans).
Included at no charge
These line items are zero on Quake AI. Many other providers meter them separately.
Data transfer (inbound and outbound)
Unlimited data transfer on every plan; Quake AI does not meter per-GB egress.
AWS, GCP, and Azure meter outbound transfer per GB. DigitalOcean and Hetzner include an allowance on compute plans, then charge overage.
Learn morePrivate networking
Private networks, subnets, Neutron routers, and security groups are included with the plan.
VPC objects are usually free to create elsewhere, but NAT gateways bill hourly plus per-GB processed. Quake AI uses router SNAT with no separate NAT line item.
Control-plane API requests
OpenStack API calls for provisioning and management are included.
Some managed services on other clouds meter API calls or charge for premium control-plane features.
Pricing data last validated: . For current rates, check quake.ai/pricing.
Template source#
Show source (6 files)Hide source
data "openstack_images_image_v2" "os" {
name = var.image_name
most_recent = true
}
data "openstack_networking_network_v2" "external" {
name = var.external_network
}
resource "openstack_networking_network_v2" "private" {
name = "${var.app_name}-net"
admin_state_up = true
}
resource "openstack_networking_subnet_v2" "private" {
name = "${var.app_name}-subnet"
network_id = openstack_networking_network_v2.private.id
cidr = var.private_cidr
ip_version = 4
dns_nameservers = ["1.1.1.1", "8.8.8.8"]
}
resource "openstack_networking_router_v2" "main" {
name = "${var.app_name}-router"
external_network_id = data.openstack_networking_network_v2.external.id
}
resource "openstack_networking_router_interface_v2" "private" {
router_id = openstack_networking_router_v2.main.id
subnet_id = openstack_networking_subnet_v2.private.id
}
resource "openstack_networking_secgroup_v2" "redpanda" {
name = "${var.app_name}-sg"
description = "SSH and HTTP/HTTPS for a reverse proxy; Kafka and Console ports restricted"
}
resource "openstack_networking_secgroup_rule_v2" "ssh" {
direction = "ingress"
ethertype = "IPv4"
protocol = "tcp"
port_range_min = 22
port_range_max = 22
remote_ip_prefix = "0.0.0.0/0"
security_group_id = openstack_networking_secgroup_v2.redpanda.id
}
resource "openstack_networking_secgroup_rule_v2" "http" {
direction = "ingress"
ethertype = "IPv4"
protocol = "tcp"
port_range_min = 80
port_range_max = 80
remote_ip_prefix = "0.0.0.0/0"
security_group_id = openstack_networking_secgroup_v2.redpanda.id
}
resource "openstack_networking_secgroup_rule_v2" "https" {
direction = "ingress"
ethertype = "IPv4"
protocol = "tcp"
port_range_min = 443
port_range_max = 443
remote_ip_prefix = "0.0.0.0/0"
security_group_id = openstack_networking_secgroup_v2.redpanda.id
}
resource "openstack_networking_secgroup_rule_v2" "kafka" {
direction = "ingress"
ethertype = "IPv4"
protocol = "tcp"
port_range_min = 9092
port_range_max = 9092
remote_ip_prefix = var.client_allowed_cidr
security_group_id = openstack_networking_secgroup_v2.redpanda.id
}
resource "openstack_networking_secgroup_rule_v2" "schema_registry" {
direction = "ingress"
ethertype = "IPv4"
protocol = "tcp"
port_range_min = 8081
port_range_max = 8081
remote_ip_prefix = var.client_allowed_cidr
security_group_id = openstack_networking_secgroup_v2.redpanda.id
}
resource "openstack_networking_secgroup_rule_v2" "pandaproxy" {
direction = "ingress"
ethertype = "IPv4"
protocol = "tcp"
port_range_min = 8082
port_range_max = 8082
remote_ip_prefix = var.client_allowed_cidr
security_group_id = openstack_networking_secgroup_v2.redpanda.id
}
resource "openstack_networking_secgroup_rule_v2" "console" {
count = var.enable_console ? 1 : 0
direction = "ingress"
ethertype = "IPv4"
protocol = "tcp"
port_range_min = 8080
port_range_max = 8080
remote_ip_prefix = var.client_allowed_cidr
security_group_id = openstack_networking_secgroup_v2.redpanda.id
}
resource "openstack_networking_port_v2" "redpanda" {
name = "${var.app_name}-port"
network_id = openstack_networking_network_v2.private.id
security_group_ids = [openstack_networking_secgroup_v2.redpanda.id]
fixed_ip {
subnet_id = openstack_networking_subnet_v2.private.id
}
depends_on = [openstack_networking_router_interface_v2.private]
}
resource "openstack_blockstorage_volume_v3" "data" {
name = "${var.app_name}-data"
size = var.volume_size
}
resource "openstack_networking_floatingip_v2" "redpanda" {
pool = var.external_network
}
resource "openstack_compute_instance_v2" "redpanda" {
name = var.app_name
flavor_name = var.flavor_name
key_pair = var.key_name
user_data = templatefile("${path.module}/cloud-init/redpanda.yaml.tftpl", {
app_name = var.app_name
advertise_host = openstack_networking_floatingip_v2.redpanda.address
enable_console = var.enable_console
})
block_device {
uuid = data.openstack_images_image_v2.os.id
source_type = "image"
destination_type = "volume"
volume_size = 30
boot_index = 0
delete_on_termination = true
}
network {
port = openstack_networking_port_v2.redpanda.id
}
}
resource "openstack_compute_volume_attach_v2" "data" {
instance_id = openstack_compute_instance_v2.redpanda.id
volume_id = openstack_blockstorage_volume_v3.data.id
}
resource "openstack_networking_floatingip_associate_v2" "redpanda" {
floating_ip = openstack_networking_floatingip_v2.redpanda.address
port_id = openstack_networking_port_v2.redpanda.id
}
variable "key_name" {
description = "SSH keypair name (must already exist in your project)"
type = string
}
variable "flavor_name" {
description = "Instance size. A single-node Redpanda broker runs on 2 vCPU and 2 GiB RAM for light ingestion; size up for higher throughput or longer retention."
type = string
default = "s1a.small"
}
variable "image_name" {
description = "Operating system image. Ubuntu 24.04 is the recommended base."
type = string
default = "Ubuntu-24.04"
}
variable "app_name" {
description = "Display name prefix for compute and network resources"
type = string
default = "redpanda"
}
variable "volume_size" {
description = "Block volume size in GiB, mounted at /var/lib/redpanda/data so topic logs live on a volume you can grow rather than on the boot disk."
type = number
default = 50
}
variable "external_network" {
description = "Shared external network for router gateway and floating IPs; defaults to PublicStatic (persisted FIP / production pattern). Override with PublicEphemeral for ephemeral demos."
type = string
default = "PublicStatic"
}
variable "private_cidr" {
description = "CIDR for the private tenant network the instance lives in"
type = string
default = "10.60.0.0/24"
}
variable "client_allowed_cidr" {
description = "CIDR allowed to reach Kafka (9092), Schema Registry (8081), HTTP Proxy (8082), and Redpanda Console (8080). Defaults to the private network only. Set to YOUR_IP/32 for direct workstation access, or reach services over an SSH tunnel."
type = string
default = "10.60.0.0/24"
}
variable "enable_console" {
description = "When true, cloud-init starts Redpanda Console on port 8080 alongside the broker."
type = bool
default = true
}
output "instance_id" {
description = "ID of the compute instance running Redpanda"
value = openstack_compute_instance_v2.redpanda.id
}
output "floating_ip" {
description = "Public floating IP address of the Redpanda host"
value = openstack_networking_floatingip_v2.redpanda.address
}
output "private_ip" {
description = "Private IP address of the instance"
value = openstack_compute_instance_v2.redpanda.access_ip_v4
}
output "kafka_bootstrap" {
description = "Kafka bootstrap address for external clients (PLAINTEXT). Restricted to client_allowed_cidr by default."
value = "${openstack_networking_floatingip_v2.redpanda.address}:9092"
}
output "console_url" {
description = "Redpanda Console URL on port 8080 when enable_console is true. Empty when Console is disabled."
value = var.enable_console ? "http://${openstack_networking_floatingip_v2.redpanda.address}:8080" : ""
}
output "schema_registry_url" {
description = "Schema Registry base URL on port 8081"
value = "http://${openstack_networking_floatingip_v2.redpanda.address}:8081"
}
terraform {
required_version = ">= 1.6.0"
required_providers {
openstack = {
source = "terraform-provider-openstack/openstack"
version = "~> 2.0"
}
}
}
provider "openstack" {}
# Required: SSH keypair must already exist in your project
key_name = "YOUR_KEY_NAME"
# Recommended: restrict Kafka and Console ports to your workstation IP.
# Leave unset to keep client ports reachable only from the private network
# and tunnel over SSH, or put a reverse proxy in front on 443.
# client_allowed_cidr = "203.0.113.10/32"
# enable_console = true
# flavor_name = "s1a.small"
# image_name = "Ubuntu-24.04"
# app_name = "redpanda"
# volume_size = 50
# external_network = "PublicStatic"
# private_cidr = "10.60.0.0/24"
#cloud-config
package_update: true
packages:
- ca-certificates
- curl
write_files:
- path: /opt/redpanda/docker-compose.yml
permissions: "0644"
content: |
services:
redpanda:
image: docker.redpanda.com/redpandadata/redpanda:v24.2.4
hostname: redpanda
restart: unless-stopped
command:
- redpanda
- start
- --overprovisioned
- --smp
- "1"
- --memory
- 1G
- --reserve-memory
- 0M
- --node-id
- "0"
- --check=false
- --kafka-addr
- internal://0.0.0.0:9092,external://0.0.0.0:9092
- --advertise-kafka-addr
- internal://redpanda:9092,external://${advertise_host}:9092
- --pandaproxy-addr
- internal://0.0.0.0:8082,external://0.0.0.0:8082
- --advertise-pandaproxy-addr
- internal://redpanda:8082,external://${advertise_host}:8082
- --schema-registry-addr
- internal://0.0.0.0:8081,external://0.0.0.0:8081
- --advertise-schema-registry-addr
- internal://redpanda:8081,external://${advertise_host}:8081
ports:
- "9092:9092"
- "8081:8081"
- "8082:8082"
volumes:
- /var/lib/redpanda/data:/var/lib/redpanda/data
%{ if enable_console ~}
console:
image: docker.redpanda.com/redpandadata/console:v2.7.2
restart: unless-stopped
depends_on:
- redpanda
ports:
- "8080:8080"
environment:
CONFIG_FILEPATH: /tmp/config.yml
CONSOLE_CONFIG_FILE: |
kafka:
brokers: ["redpanda:9092"]
schemaRegistry:
enabled: true
urls: ["http://redpanda:8081"]
entrypoint: /bin/sh
command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console'
%{ endif ~}
runcmd:
- |
set -e
DEV=/dev/sdb
for i in $(seq 1 30); do [ -b "$DEV" ] && break; sleep 5; done
if ! blkid "$DEV" >/dev/null 2>&1; then mkfs.ext4 -F -L redpandadata "$DEV"; fi
mkdir -p /var/lib/redpanda/data
mount "$DEV" /var/lib/redpanda/data
grep -q "$DEV" /etc/fstab || echo "$DEV /var/lib/redpanda/data ext4 defaults,nofail 0 2" >> /etc/fstab
curl -fsSL https://get.docker.com | sh
cd /opt/redpanda
docker compose up -d
Resources, parameters, and variables
key_namerequiredflavor_name="s1a.small"image_name="Ubuntu-24.04"app_name="redpanda"volume_size=50external_network="PublicStatic"private_cidr="10.60.0.0/24"client_allowed_cidr="10.60.0.0/24"enable_console=true
Customize this pattern#
- Customize a template's image and flavor
- Add a block volume to a template
- Parameterize a template with a tfvars file
See also#
Usage Guidelines
The sample code, software libraries, command line tools, proofs of concept, templates, and other related technology on this page (including any of the foregoing that is provided by Quake AI personnel) is provided to you as Quake AI Content under the Quake AI Customer Agreement, or the relevant written agreement between you and Quake AI (whichever applies). Do not use this Quake AI Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the Quake AI Content (such as sample code) as appropriate for production grade use based on your specific quality control practices and standards. Deploying Quake AI Content may incur Quake AI charges for creating or using Quake AI chargeable resources, such as running Compute instances or storing data in Object Storage. Your use is also subject to the Acceptable Use Policy.
For the full policy, see Usage Guidelines.
See Also
Terraform and OpenTofu on Quake AI
Prerequisite
Networks
Prerequisite
Authoring IaC templates for Quake AI
Shares: Volumes, Security Groups
Deploy an API gateway with the api-gateway template
Shares: Volumes, Security Groups
Deploy a regional edge cache with the edge-cache template
Shares: Volumes, Security Groups