Deploy Redpanda with the redpanda template

Stand up Redpanda, an open-source Kafka-API streaming broker, on a single Quake AI instance using the validated OpenTofu template redpanda. You apply the template, create a topic, produce and consume a test message, and open Redpanda Console when it is enabled.

Redpanda is the streaming-ingestion layer in a data pipeline. You run it yourself; this is a self-hosted broker you operate, not a hosted streaming service.

Prerequisites#

You need:

• OpenTofu 1.6.0 or later (or Terraform 1.6.0 or later) installed locally.
• Your OpenStack credentials sourced into the shell (source openrc.sh). See the OpenStack CLI guide.
• An SSH keypair that already exists in your project. Record its name for the key_name variable.
• A copy of the redpanda template directory from the template reference page.

Step 1: Set the variables and apply the template#

Copy the template's example variables file and open it:

cp terraform.tfvars.example terraform.tfvars

Set key_name to the SSH keypair already in your project. Leave client_allowed_cidr at its default if you will run the smoke test over SSH; set it to YOUR_IP/32 only when you need direct Kafka access from your workstation.

Initialize the working directory, preview the plan, and apply:

tofu init
tofu plan
tofu apply

OpenTofu provisions a private network, a router, a security group, a block volume mounted at /var/lib/redpanda/data, an instance, and a floating IP. On first boot, cloud-init formats and mounts the data volume, installs Docker Engine, and starts Redpanda from a compose file.

When the apply finishes, read the outputs:

tofu output

Record floating_ip, kafka_bootstrap, and console_url (when Console is enabled).

Step 2: Create a topic and run a smoke test#

cloud-init takes a minute or two after the instance reaches ACTIVE. SSH to the host and confirm the containers are running:

ssh ubuntu@YOUR_FLOATING_IP "sudo docker ps --filter name=redpanda"

Create a topic, produce one message, and consume it with rpk inside the broker container:

ssh ubuntu@YOUR_FLOATING_IP <<'EOF'
cd /opt/redpanda
sudo docker compose exec redpanda rpk topic create smoke-test
echo 'hello-redpanda' | sudo docker compose exec -T redpanda rpk topic produce smoke-test -k smoke-key
sudo docker compose exec redpanda rpk topic consume smoke-test -n 1
EOF

The consume command prints the record your producer sent. External clients use the same topic name against kafka_bootstrap from the template outputs once you open client_allowed_cidr to their source addresses or place the broker behind your private network.

Step 3: Open Redpanda Console (optional)#

When enable_console is true (the default), open console_url in your browser. Console lists brokers, topics, and consumer groups. If the page does not load from your workstation, tunnel port 8080 over SSH:

ssh -L 8080:localhost:8080 ubuntu@YOUR_FLOATING_IP

Then open http://localhost:8080. Confirm the smoke-test topic appears in the Topics view.

What you built#

• Applied the redpanda template to provision a network, security group, data volume, instance, and floating IP, and let cloud-init install Docker and start Redpanda
• Created a topic and ran a produce/consume smoke test with rpk inside the broker container
• Browsed the cluster in Redpanda Console when Console is enabled

Scope of this deployment#

This template runs a single-node Redpanda broker on one VM, not a multi-broker Kafka cluster. The instance is CPU-only and runs in one region. You operate the broker, Docker, and the data volume yourself: back them up, patch them, and watch disk use as retention grows. For production throughput or fault tolerance, size the instance up, add TLS and SASL, and plan a multi-broker layout outside this single-VM template.

Next steps#

• Redpanda template: the template reference, parameters, and resource map
• Airflow template: orchestrate jobs that read from and write to your topics
• Airbyte template: sync database changelogs into Kafka-compatible topics
• Security hardening checklist: tighten SSH access and listener exposure before you serve real traffic

Clean up#

When you no longer need the deployment, destroy everything the template created:

tofu destroy

Because topic logs live on the attached volume, tofu destroy removes retained messages along with the infrastructure. Export any topics you want to keep before you destroy.