# Understanding limitations and advanced features of Quake AI Kubernetes clusters

Source: https://docs.quake.ai/docs/operate/troubleshooting/advanced-kubernetes
Markdown: https://docs.quake.ai/docs/operate/troubleshooting/advanced-kubernetes.md

---

# Understanding limitations and advanced features of Quake AI Kubernetes clusters

Floating IPs and Kubernetes `LoadBalancer` services both translate addresses. When you assign a floating IP to a cluster node and also expose a LoadBalancer service, the two NAT paths can conflict. You then see failed health checks and unreachable services.

## The problem: IP conflict between floating IPs and load balancers

Assigning floating IPs directly to cluster nodes while you also deploy LoadBalancer-type services creates overlapping address translation:

- A floating IP translates public traffic to the node's internal IP.
- A load balancer performs NAT toward service endpoints (pods or nodes).

When both run at once, NAT rules can collide and the load balancer stops passing traffic.

## Recommended workarounds

### Avoid assigning floating IPs to nodes at create time

Quake AI Magnum templates leave per-node floating IPs off by default (`floating_ip_enabled: false`). Magnum still exposes the create-time switches:

- CLI: `--floating-ip-enabled` and `--floating-ip-disabled` on `openstack coe cluster create`
- API: `floating_ip_enabled`

Do not pass `--floating-ip-enabled`, and do not set `floating_ip_enabled` to true. Use `--floating-ip-disabled`, or leave `floating_ip_enabled` false, so Magnum does not assign a floating IP to each node during installation.

A template may still set `master_lb_floating_ip_enabled: true` so the cluster API load balancer has a public address. That setting is separate from per-node floating IPs.

### Remove floating IPs from nodes on existing clusters

If an existing cluster already has floating IPs on Kubernetes nodes, disassociate those floating IPs from the nodes. Use the Quake AI web interface or the OpenStack CLI.

Disassociating node floating IPs also removes:

- Direct SSH to nodes over a public IP
- NodePort access to Kubernetes services from the public internet

### Avoid re-associating floating IPs

Temporarily re-associating and then disassociating floating IPs can look like it clears the conflict. The conflict can return after you re-associate them.

### Exclude specific nodes from load balancers

If some nodes must keep a floating IP, exclude those nodes from external load balancers. Apply the `node.kubernetes.io/exclude-from-external-load-balancers=true` label. See the [Kubernetes labels reference for `node.kubernetes.io/exclude-from-external-load-balancers`](https://kubernetes.io/docs/reference/labels-annotations-taints/#node-kubernetes-io-exclude-from-external-load-balancers) for semantics and when to use the label.