# How to run blue/green or canary deployments on Quake AI

Source: https://docs.quake.ai/docs/network/how-to/blue-green-canary-deploys
Markdown: https://docs.quake.ai/docs/network/how-to/blue-green-canary-deploys.md

---

# How to run blue/green or canary deployments on Quake AI

Cut over to a new application version by routing traffic through a reverse proxy that you operate. This guide uses HAProxy because its runtime API can change backend weights without restarting the proxy. Nginx, Caddy, Traefik, and Envoy can implement the same pattern with their own configuration and control interfaces.

<PrerequisiteBlock methods={["cli"]}>

- A reverse proxy instance with a [floating IP](/docs/network/how-to/allocate-floating-ips)
- Running instances for the current ("blue") version and the new ("green") version on a private network
- A health endpoint, such as `/healthz`, on both versions
- Security group rules that allow the proxy instance to reach the application port

</PrerequisiteBlock>

## Configure the proxy backends

Install HAProxy on the reverse proxy instance:

```bash
sudo apt update
sudo apt install -y haproxy socat
```

Add blue and green servers to one backend in `/etc/haproxy/haproxy.cfg`:

```text
frontend public_https
    bind :443 ssl crt /etc/haproxy/certs/example.com.pem
    default_backend application

backend application
    option httpchk GET /healthz
    http-check expect status 200
    server blue-1 10.0.0.11:8080 check weight 100
    server green-1 10.0.0.21:8080 check weight 0
```

Enable the HAProxy runtime socket in the `global` section:

```text
global
    stats socket /run/haproxy/admin.sock mode 660 level admin
```

Validate and reload the configuration:

```bash
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy
```

## Verify the green version

Test the green instance from the proxy before sending public traffic to it:

```bash
curl -fsS http://10.0.0.21:8080/healthz
```

Check HAProxy's view of both backends:

```bash
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock
```

The green server must report `UP` before you change its weight.

## Run a blue/green cutover

Move traffic to green by setting green to full weight and blue to zero:

```bash
echo "set weight application/green-1 100%" | sudo socat stdio /run/haproxy/admin.sock
echo "set weight application/blue-1 0%" | sudo socat stdio /run/haproxy/admin.sock
```

Keep the blue version running during the verification window. The proxy's floating IP and DNS records stay unchanged.

To roll back:

```bash
echo "set weight application/blue-1 100%" | sudo socat stdio /run/haproxy/admin.sock
echo "set weight application/green-1 0%" | sudo socat stdio /run/haproxy/admin.sock
```

## Run a canary rollout

Start green at 10%:

```bash
echo "set weight application/blue-1 90%" | sudo socat stdio /run/haproxy/admin.sock
echo "set weight application/green-1 10%" | sudo socat stdio /run/haproxy/admin.sock
```

Watch application errors, latency, and business metrics. Increase green in stages after each observation window:

```bash
echo "set weight application/blue-1 50%" | sudo socat stdio /run/haproxy/admin.sock
echo "set weight application/green-1 50%" | sudo socat stdio /run/haproxy/admin.sock
```

Complete the rollout by setting green to 100% and blue to zero. Roll back at any stage by restoring blue to 100%.

## Persist the final state

Runtime weight changes do not survive a proxy restart. After the rollout, update the server weights in `/etc/haproxy/haproxy.cfg`, validate the file, and reload HAProxy:

```bash
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy
```

## See also

- [How to allocate floating IPs](/docs/network/how-to/allocate-floating-ips)
- [How to issue and auto-renew a TLS certificate with Let's Encrypt](/docs/network/how-to/lets-encrypt-certificate)
- [Edge reverse proxy template](/resources/iac-templates/edge-reverse-proxy)
- [How to put a CDN in front of a Quake AI workload](/docs/network/how-to/front-with-cdn)
- [How to put a WAF in front of a Quake AI workload](/docs/network/how-to/front-with-waf)