Skip to content

Redpanda Kafka-API streaming

Template

Redpanda Kafka-API streaming

This validated OpenTofu template composes Compute, Network, and Block Storage into a self-hosted Kafka-API streaming broker you run on infrastructure you control.

What this template does#

Provisions a single instance running Redpanda, an open-source streaming platform with a Kafka-compatible API in one binary (a self-hosted alternative to Confluent Cloud or Amazon MSK):

  • Docker Compose stack: one Redpanda broker and optional Redpanda Console on port 8080
  • Default sizing: s1a.small (2 vCPU / 2 GiB RAM) and a 50 GiB data volume at /var/lib/redpanda/data for topic logs
  • Private network, security group, floating IP; Kafka (9092), Schema Registry (8081), HTTP Proxy (8082), and Console (8080) restricted to client_allowed_cidr by default
  • cloud-init installs Docker Engine and starts the stack on first boot; the broker advertises the floating IP for external clients

Redpanda covers the streaming-ingestion path in a data pipeline: producers and consumers use standard Kafka clients against kafka_bootstrap from the template outputs. Wire Airbyte, Flink, or your application producers to the bootstrap address and create topics with rpk or your client library. No credential ships with this template; the broker starts with PLAINTEXT listeners and you add TLS or SASL when you harden the host.

Parameters#

ParameterDescriptionDefault
key_nameSSH keypair name (must already exist)No default
flavor_nameInstance size (single-node broker on 2 vCPU / 2 GiB)s1a.small
image_nameOperating system imageUbuntu-24.04
app_nameDisplay name prefix for resourcesredpanda
volume_sizeBlock volume size in GiB, mounted at /var/lib/redpanda/data50
external_networkExternal network for floating IP allocationPublicStatic
private_cidrCIDR for the private subnet10.60.0.0/24
client_allowed_cidrCIDR allowed to reach Kafka, Schema Registry, HTTP Proxy, and Console10.60.0.0/24
enable_consoleStart Redpanda Console on port 8080true

Client access and security#

The broker listens on port 9092 with PLAINTEXT. Schema Registry uses 8081 and HTTP Proxy uses 8082. Redpanda Console uses 8080 when enable_console is true. The security group restricts those ports to client_allowed_cidr, which defaults to the private network only. Reach the services one of three ways:

  • Put a reverse proxy (Caddy or Nginx) in front of Console on 443 and keep Kafka on a private network or VPN.
  • Tunnel over SSH: ssh -L 9092:localhost:9092 -L 8080:localhost:8080 ubuntu@FLOATING_IP, then point clients at localhost:9092.
  • Set client_allowed_cidr to YOUR_IP/32 to reach the ports directly from one address.

Ports 80 and 443 stay open for the reverse proxy you put in front; they carry no traffic until you add one.

When to use this pattern#

Run a Kafka-compatible streaming broker for event ingestion, log pipelines, and microservice decoupling on a VM you operate. Redpanda suits the streaming layer in a self-hosted data stack alongside Airflow, Airbyte, and a Postgres warehouse.

For object storage and lake tables rather than a message bus, see MinIO and Iceberg. For federated SQL over the lake, see Trino.

Estimated cost#

Monthly cost estimate

Pricing calculator ↗

Sized as a custom package on shared vCPU.

Starting template$17.90/mo

Monthly total for the required template above. Use the configurator below to add optional pieces and see the total update.

What each resource is for

Redpanda broker

s1a.small · 2 shared vCPU, 2 GiB RAM, 0.5 Gbps

Runs Redpanda in Docker with Kafka-API streaming and optional Redpanda Console.

Single-node profile on 2 vCPU and 2 GiB RAM for light ingestion; size up for higher throughput or longer log retention.

$16.50/mo

Compute shown per role at custom-package rates ($29/dedicated vCPU, $7.25/shared vCPU, $1/GiB RAM). The headline above is the billed total: the cheaper of a named plan and the custom package, plus add-ons.

Included in baseline

s1a.small

2 shared vCPU, 2 GiB RAM, 0.5 Gbps

$16.50

Compute + RAM rate basis

2 vCPU + 2 GiB RAM at $29/dedicated vCPU, $7.25/shared vCPU, $1/GiB RAM (regular). Totals apply the flat −$5/mo package promotion.

—

Block storage (80 GiB)

80 GiB at $0.08/GiB/mo

$6.40

Public IP (included)

1 included with the custom package

$0.00

Package promotional discount

Flat −$5.00/mo on the custom package (same promotion as named plans).

$-5.00

Included at no charge

These line items are zero on Quake AI. Many other providers meter them separately.

Data transfer (inbound and outbound)

Unlimited data transfer on every plan; Quake AI does not meter per-GB egress.

AWS, GCP, and Azure meter outbound transfer per GB. DigitalOcean and Hetzner include an allowance on compute plans, then charge overage.

Learn more
$0.00

Private networking

Private networks, subnets, Neutron routers, and security groups are included with the plan.

VPC objects are usually free to create elsewhere, but NAT gateways bill hourly plus per-GB processed. Quake AI uses router SNAT with no separate NAT line item.

$0.00

Control-plane API requests

OpenStack API calls for provisioning and management are included.

Some managed services on other clouds meter API calls or charge for premium control-plane features.

$0.00

Pricing data last validated: . For current rates, check quake.ai/pricing.

Template source#

6 files. Download the zip or expand to copy any file.Download redpanda.zip
Show source (6 files)
main.tfHCL
data "openstack_images_image_v2" "os" {
  name        = var.image_name
  most_recent = true
}

data "openstack_networking_network_v2" "external" {
  name = var.external_network
}

resource "openstack_networking_network_v2" "private" {
  name           = "${var.app_name}-net"
  admin_state_up = true
}

resource "openstack_networking_subnet_v2" "private" {
  name            = "${var.app_name}-subnet"
  network_id      = openstack_networking_network_v2.private.id
  cidr            = var.private_cidr
  ip_version      = 4
  dns_nameservers = ["1.1.1.1", "8.8.8.8"]
}

resource "openstack_networking_router_v2" "main" {
  name                = "${var.app_name}-router"
  external_network_id = data.openstack_networking_network_v2.external.id
}

resource "openstack_networking_router_interface_v2" "private" {
  router_id = openstack_networking_router_v2.main.id
  subnet_id = openstack_networking_subnet_v2.private.id
}

resource "openstack_networking_secgroup_v2" "redpanda" {
  name        = "${var.app_name}-sg"
  description = "SSH and HTTP/HTTPS for a reverse proxy; Kafka and Console ports restricted"
}

resource "openstack_networking_secgroup_rule_v2" "ssh" {
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 22
  port_range_max    = 22
  remote_ip_prefix  = "0.0.0.0/0"
  security_group_id = openstack_networking_secgroup_v2.redpanda.id
}

resource "openstack_networking_secgroup_rule_v2" "http" {
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 80
  port_range_max    = 80
  remote_ip_prefix  = "0.0.0.0/0"
  security_group_id = openstack_networking_secgroup_v2.redpanda.id
}

resource "openstack_networking_secgroup_rule_v2" "https" {
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 443
  port_range_max    = 443
  remote_ip_prefix  = "0.0.0.0/0"
  security_group_id = openstack_networking_secgroup_v2.redpanda.id
}

resource "openstack_networking_secgroup_rule_v2" "kafka" {
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 9092
  port_range_max    = 9092
  remote_ip_prefix  = var.client_allowed_cidr
  security_group_id = openstack_networking_secgroup_v2.redpanda.id
}

resource "openstack_networking_secgroup_rule_v2" "schema_registry" {
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 8081
  port_range_max    = 8081
  remote_ip_prefix  = var.client_allowed_cidr
  security_group_id = openstack_networking_secgroup_v2.redpanda.id
}

resource "openstack_networking_secgroup_rule_v2" "pandaproxy" {
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 8082
  port_range_max    = 8082
  remote_ip_prefix  = var.client_allowed_cidr
  security_group_id = openstack_networking_secgroup_v2.redpanda.id
}

resource "openstack_networking_secgroup_rule_v2" "console" {
  count             = var.enable_console ? 1 : 0
  direction         = "ingress"
  ethertype         = "IPv4"
  protocol          = "tcp"
  port_range_min    = 8080
  port_range_max    = 8080
  remote_ip_prefix  = var.client_allowed_cidr
  security_group_id = openstack_networking_secgroup_v2.redpanda.id
}

resource "openstack_networking_port_v2" "redpanda" {
  name               = "${var.app_name}-port"
  network_id         = openstack_networking_network_v2.private.id
  security_group_ids = [openstack_networking_secgroup_v2.redpanda.id]

  fixed_ip {
    subnet_id = openstack_networking_subnet_v2.private.id
  }

  depends_on = [openstack_networking_router_interface_v2.private]
}

resource "openstack_blockstorage_volume_v3" "data" {
  name = "${var.app_name}-data"
  size = var.volume_size
}

resource "openstack_networking_floatingip_v2" "redpanda" {
  pool = var.external_network
}

resource "openstack_compute_instance_v2" "redpanda" {
  name        = var.app_name
  flavor_name = var.flavor_name
  key_pair    = var.key_name

  user_data = templatefile("${path.module}/cloud-init/redpanda.yaml.tftpl", {
    app_name         = var.app_name
    advertise_host   = openstack_networking_floatingip_v2.redpanda.address
    enable_console   = var.enable_console
  })

  block_device {
    uuid                  = data.openstack_images_image_v2.os.id
    source_type           = "image"
    destination_type      = "volume"
    volume_size           = 30
    boot_index            = 0
    delete_on_termination = true
  }

  network {
    port = openstack_networking_port_v2.redpanda.id
  }
}

resource "openstack_compute_volume_attach_v2" "data" {
  instance_id = openstack_compute_instance_v2.redpanda.id
  volume_id   = openstack_blockstorage_volume_v3.data.id
}

resource "openstack_networking_floatingip_associate_v2" "redpanda" {
  floating_ip = openstack_networking_floatingip_v2.redpanda.address
  port_id     = openstack_networking_port_v2.redpanda.id
}
variables.tfHCL
variable "key_name" {
  description = "SSH keypair name (must already exist in your project)"
  type        = string
}

variable "flavor_name" {
  description = "Instance size. A single-node Redpanda broker runs on 2 vCPU and 2 GiB RAM for light ingestion; size up for higher throughput or longer retention."
  type        = string
  default     = "s1a.small"
}

variable "image_name" {
  description = "Operating system image. Ubuntu 24.04 is the recommended base."
  type        = string
  default     = "Ubuntu-24.04"
}

variable "app_name" {
  description = "Display name prefix for compute and network resources"
  type        = string
  default     = "redpanda"
}

variable "volume_size" {
  description = "Block volume size in GiB, mounted at /var/lib/redpanda/data so topic logs live on a volume you can grow rather than on the boot disk."
  type        = number
  default     = 50
}

variable "external_network" {
  description = "Shared external network for router gateway and floating IPs; defaults to PublicStatic (persisted FIP / production pattern). Override with PublicEphemeral for ephemeral demos."
  type        = string
  default     = "PublicStatic"
}

variable "private_cidr" {
  description = "CIDR for the private tenant network the instance lives in"
  type        = string
  default     = "10.60.0.0/24"
}

variable "client_allowed_cidr" {
  description = "CIDR allowed to reach Kafka (9092), Schema Registry (8081), HTTP Proxy (8082), and Redpanda Console (8080). Defaults to the private network only. Set to YOUR_IP/32 for direct workstation access, or reach services over an SSH tunnel."
  type        = string
  default     = "10.60.0.0/24"
}

variable "enable_console" {
  description = "When true, cloud-init starts Redpanda Console on port 8080 alongside the broker."
  type        = bool
  default     = true
}
outputs.tfHCL
output "instance_id" {
  description = "ID of the compute instance running Redpanda"
  value       = openstack_compute_instance_v2.redpanda.id
}

output "floating_ip" {
  description = "Public floating IP address of the Redpanda host"
  value       = openstack_networking_floatingip_v2.redpanda.address
}

output "private_ip" {
  description = "Private IP address of the instance"
  value       = openstack_compute_instance_v2.redpanda.access_ip_v4
}

output "kafka_bootstrap" {
  description = "Kafka bootstrap address for external clients (PLAINTEXT). Restricted to client_allowed_cidr by default."
  value       = "${openstack_networking_floatingip_v2.redpanda.address}:9092"
}

output "console_url" {
  description = "Redpanda Console URL on port 8080 when enable_console is true. Empty when Console is disabled."
  value       = var.enable_console ? "http://${openstack_networking_floatingip_v2.redpanda.address}:8080" : ""
}

output "schema_registry_url" {
  description = "Schema Registry base URL on port 8081"
  value       = "http://${openstack_networking_floatingip_v2.redpanda.address}:8081"
}
versions.tfHCL
terraform {
  required_version = ">= 1.6.0"

  required_providers {
    openstack = {
      source  = "terraform-provider-openstack/openstack"
      version = "~> 2.0"
    }
  }
}

provider "openstack" {}
terraform.tfvars.exampleHCL
# Required: SSH keypair must already exist in your project
key_name = "YOUR_KEY_NAME"

# Recommended: restrict Kafka and Console ports to your workstation IP.
# Leave unset to keep client ports reachable only from the private network
# and tunnel over SSH, or put a reverse proxy in front on 443.
# client_allowed_cidr = "203.0.113.10/32"

# enable_console = true
# flavor_name = "s1a.small"
# image_name = "Ubuntu-24.04"
# app_name = "redpanda"
# volume_size = 50
# external_network = "PublicStatic"
# private_cidr = "10.60.0.0/24"
cloud-init/redpanda.yaml.tftpl
#cloud-config
package_update: true
packages:
  - ca-certificates
  - curl
write_files:
  - path: /opt/redpanda/docker-compose.yml
    permissions: "0644"
    content: |
      services:
        redpanda:
          image: docker.redpanda.com/redpandadata/redpanda:v24.2.4
          hostname: redpanda
          restart: unless-stopped
          command:
            - redpanda
            - start
            - --overprovisioned
            - --smp
            - "1"
            - --memory
            - 1G
            - --reserve-memory
            - 0M
            - --node-id
            - "0"
            - --check=false
            - --kafka-addr
            - internal://0.0.0.0:9092,external://0.0.0.0:9092
            - --advertise-kafka-addr
            - internal://redpanda:9092,external://${advertise_host}:9092
            - --pandaproxy-addr
            - internal://0.0.0.0:8082,external://0.0.0.0:8082
            - --advertise-pandaproxy-addr
            - internal://redpanda:8082,external://${advertise_host}:8082
            - --schema-registry-addr
            - internal://0.0.0.0:8081,external://0.0.0.0:8081
            - --advertise-schema-registry-addr
            - internal://redpanda:8081,external://${advertise_host}:8081
          ports:
            - "9092:9092"
            - "8081:8081"
            - "8082:8082"
          volumes:
            - /var/lib/redpanda/data:/var/lib/redpanda/data
%{ if enable_console ~}
        console:
          image: docker.redpanda.com/redpandadata/console:v2.7.2
          restart: unless-stopped
          depends_on:
            - redpanda
          ports:
            - "8080:8080"
          environment:
            CONFIG_FILEPATH: /tmp/config.yml
            CONSOLE_CONFIG_FILE: |
              kafka:
                brokers: ["redpanda:9092"]
              schemaRegistry:
                enabled: true
                urls: ["http://redpanda:8081"]
          entrypoint: /bin/sh
          command: -c 'echo "$$CONSOLE_CONFIG_FILE" > /tmp/config.yml; /app/console'
%{ endif ~}
runcmd:
  - |
    set -e
    DEV=/dev/sdb
    for i in $(seq 1 30); do [ -b "$DEV" ] && break; sleep 5; done
    if ! blkid "$DEV" >/dev/null 2>&1; then mkfs.ext4 -F -L redpandadata "$DEV"; fi
    mkdir -p /var/lib/redpanda/data
    mount "$DEV" /var/lib/redpanda/data
    grep -q "$DEV" /etc/fstab || echo "$DEV /var/lib/redpanda/data ext4 defaults,nofail 0 2" >> /etc/fstab
    curl -fsSL https://get.docker.com | sh
    cd /opt/redpanda
    docker compose up -d
Resources, parameters, and variables
Provisions
Parameterized by
Variables
  • key_namerequired
  • flavor_name="s1a.small"
  • image_name="Ubuntu-24.04"
  • app_name="redpanda"
  • volume_size=50
  • external_network="PublicStatic"
  • private_cidr="10.60.0.0/24"
  • client_allowed_cidr="10.60.0.0/24"
  • enable_console=true

Customize this pattern#

See also#

Usage Guidelines

The sample code, software libraries, command line tools, proofs of concept, templates, and other related technology on this page (including any of the foregoing that is provided by Quake AI personnel) is provided to you as Quake AI Content under the Quake AI Customer Agreement, or the relevant written agreement between you and Quake AI (whichever applies). Do not use this Quake AI Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the Quake AI Content (such as sample code) as appropriate for production grade use based on your specific quality control practices and standards. Deploying Quake AI Content may incur Quake AI charges for creating or using Quake AI chargeable resources, such as running Compute instances or storing data in Object Storage. Your use is also subject to the Acceptable Use Policy.

For the full policy, see Usage Guidelines.

Was this page helpful?