# Deploy Redpanda with the redpanda template

Source: https://docs.quake.ai/resources/deployments/deploy-redpanda-template
Markdown: https://docs.quake.ai/resources/deployments/deploy-redpanda-template.md

---

# Deploy Redpanda with the redpanda template

Stand up [Redpanda](https://www.redpanda.com), an open-source Kafka-API streaming broker, on a single Quake AI instance using the [validated OpenTofu template](/docs/platform/validation#how-infrastructure-templates-are-checked) `redpanda`. You apply the template, create a topic, produce and consume a test message, and open Redpanda Console when it is enabled.

Redpanda is the streaming-ingestion layer in a data pipeline. You run it yourself; this is a self-hosted broker you operate, not a hosted streaming service.

<Figure size="md" caption="What you will build: a single Redpanda broker on one instance, with optional Console on port 8080">

```d2
direction: right

producer: Producer app {shape: person}
consumer: Consumer app {shape: person}
fip: Floating IP
instance: Ubuntu instance {
  redpanda: Redpanda broker\nKafka API :9092
  console: Redpanda Console\n:8080
  console -> redpanda: reads cluster state
}

producer -> fip: produce
fip -> instance.redpanda
consumer -> fip: consume
fip -> instance.redpanda
```

</Figure>

<PricingCompanion
  components={[
    { kind: "template", slug: "redpanda", required: true },
  ]}
/>

## Prerequisites

You need:

- OpenTofu 1.6.0 or later (or Terraform 1.6.0 or later) installed locally.
- Your OpenStack credentials sourced into the shell (`source openrc.sh`). See [the OpenStack CLI guide](/docs/tools/openstack-cli).
- An SSH keypair that already exists in your project. Record its name for the `key_name` variable.
- A copy of the `redpanda` template directory from [the template reference page](/resources/iac-templates/redpanda).

## Step 1: Set the variables and apply the template

Copy the template's example variables file and open it:

```bash
cp terraform.tfvars.example terraform.tfvars
```

Set `key_name` to the SSH keypair already in your project. Leave `client_allowed_cidr` at its default if you will run the smoke test over SSH; set it to `YOUR_IP/32` only when you need direct Kafka access from your workstation.

Initialize the working directory, preview the plan, and apply:

```bash
tofu init
tofu plan
tofu apply
```

OpenTofu provisions a private network, a router, a security group, a block volume mounted at `/var/lib/redpanda/data`, an instance, and a floating IP. On first boot, cloud-init formats and mounts the data volume, installs Docker Engine, and starts Redpanda from a compose file.

When the apply finishes, read the outputs:

```bash
tofu output
```

Record `floating_ip`, `kafka_bootstrap`, and `console_url` (when Console is enabled).

## Step 2: Create a topic and run a smoke test

cloud-init takes a minute or two after the instance reaches `ACTIVE`. SSH to the host and confirm the containers are running:

```bash
ssh ubuntu@YOUR_FLOATING_IP "sudo docker ps --filter name=redpanda"
```

Create a topic, produce one message, and consume it with `rpk` inside the broker container:

```bash
ssh ubuntu@YOUR_FLOATING_IP <<'EOF'
cd /opt/redpanda
sudo docker compose exec redpanda rpk topic create smoke-test
echo 'hello-redpanda' | sudo docker compose exec -T redpanda rpk topic produce smoke-test -k smoke-key
sudo docker compose exec redpanda rpk topic consume smoke-test -n 1
EOF
```

The consume command prints the record your producer sent. External clients use the same topic name against `kafka_bootstrap` from the template outputs once you open `client_allowed_cidr` to their source addresses or place the broker behind your private network.

## Step 3: Open Redpanda Console (optional)

When `enable_console` is true (the default), open `console_url` in your browser. Console lists brokers, topics, and consumer groups. If the page does not load from your workstation, tunnel port 8080 over SSH:

```bash
ssh -L 8080:localhost:8080 ubuntu@YOUR_FLOATING_IP
```

Then open `http://localhost:8080`. Confirm the `smoke-test` topic appears in the Topics view.

## What you built

- **Applied the `redpanda` template** to provision a network, security group, data volume, instance, and floating IP, and let cloud-init install Docker and start Redpanda
- **Created a topic and ran a produce/consume smoke test** with `rpk` inside the broker container
- **Browsed the cluster in Redpanda Console** when Console is enabled

## Scope of this deployment

This template runs a single-node Redpanda broker on one VM, not a multi-broker Kafka cluster. The instance is CPU-only and runs in one region. You operate the broker, Docker, and the data volume yourself: back them up, patch them, and watch disk use as retention grows. For production throughput or fault tolerance, size the instance up, add TLS and SASL, and plan a multi-broker layout outside this single-VM template.

## Next steps

- [Redpanda template](/resources/iac-templates/redpanda): the template reference, parameters, and resource map
- [Airflow template](/resources/iac-templates/airflow): orchestrate jobs that read from and write to your topics
- [Airbyte template](/resources/iac-templates/airbyte): sync database changelogs into Kafka-compatible topics
- [Security hardening checklist](/docs/security/hardening-checklist): tighten SSH access and listener exposure before you serve real traffic

## Clean up

When you no longer need the deployment, destroy everything the template created:

```bash
tofu destroy
```

Because topic logs live on the attached volume, `tofu destroy` removes retained messages along with the infrastructure. Export any topics you want to keep before you destroy.