# Troubleshooting

Source: https://docs.quake.ai/docs/operate/troubleshooting
Markdown: https://docs.quake.ai/docs/operate/troubleshooting.md

---

# Troubleshooting

Troubleshooting guides help you diagnose and resolve common problems with your Quake AI workloads. Each guide follows a symptom-first structure: start with what you observe, narrow down the cause, and apply a targeted fix.

When something goes wrong, follow this general diagnostic pattern:

1. **Identify the symptom**: what exactly is failing? An SSH connection timeout, an API error code, an instance stuck in a build state?
2. **Check recent changes**: did you modify a security group, rotate credentials, resize an instance, or update a configuration?
3. **Isolate the layer**: is the problem at the network level (security groups, floating IPs), the instance level (OS, application), or the platform level (API errors, quota limits)?
4. **Gather evidence**: collect logs, error messages, and resource status before making changes. Use `openstack server show`, `openstack console log show`, and instance system logs.
5. **Apply and verify**: make one change at a time and verify whether it resolves the issue before proceeding.

<DocsSectionLinks section="operate/troubleshooting" />

<Figure
  size="lg"
  caption="Symptom-first triage. Follow the edge that matches your observed symptom to the matching guide in the table below."
>

```d2
direction: right

start: What went wrong? {
  shape: diamond
}

api_err: API error code
connect: Cannot connect to instance
stuck: Resource stuck in a state
quota: Resource creation rejected
storage: Object storage / S3 error
iac: Stack or Terraform failure

start -> api_err: HTTP 4xx/5xx
start -> connect: SSH timeout
start -> stuck: BUILD or attaching
start -> quota: Quota exceeded
start -> storage: S3 403 or upload
start -> iac: Stack FAILED
```

</Figure>

## Common symptoms quick reference

| Symptom | First steps | Detailed guide |
|---|---|---|
| Cannot SSH into instance | Check security group rules (port 22 open?), verify floating IP is assigned, confirm key pair matches | [Instance connectivity](/docs/operate/runbooks/instance-connectivity) |
| Instance stuck in BUILD | Wait 5 minutes; if still building, check quota limits | [Instance lifecycle](/docs/operate/runbooks/instance-lifecycle) |
| API returns 401 Unauthorized | Re-source your `openrc.sh`; token may have expired | [Auth token diagnostics](/docs/operate/troubleshooting/auth-token-diagnostics) |
| Object upload fails with 403 | Verify EC2 credentials (not app credentials) are valid | [Object storage access](/docs/operate/runbooks/object-storage-access) |
| Instance has no network | Check that the instance is attached to a network and the subnet has DHCP enabled | [Instance connectivity](/docs/operate/runbooks/instance-connectivity) |
| Volume stuck in attaching/detaching | Check volume status; may need state reset | [Volume troubleshooting](/docs/operate/runbooks/volume-troubleshooting) |
| Quota exceeded error | Check usage with `openstack quota show --usage`; clean up unused resources | [Quota and limits](/docs/operate/troubleshooting/quota-and-limits) |
| Stack creation/update failed | Check stack events: `openstack stack event list YOUR_STACK` | [Automation API errors](/reference/automation/api-errors) |
| Terraform apply error | Enable debug: `TF_LOG=DEBUG terraform apply` | [Terraform error handling](/docs/automation/how-to/terraform-error-handling) |
| CLI command fails | Add `--debug` to see full HTTP request/response | [CLI debugging](/docs/tools/cli-debugging) |

## See also

- [Error response format](/reference/api-conventions/error-response-format): JSON and XML error structure, fault objects, request correlation, parsing examples
- [Retry and resilience patterns](/reference/api-conventions/retry-and-resilience): exponential backoff, jitter, idempotency, and which errors are retryable
- [API rate limits](/docs/tools/api-rate-limits): quotas vs. rate limits, 429 handling
- [Operate overview](/docs/operate)
- [Runbooks](/docs/operate/runbooks)
- [Monitoring](/docs/operate/monitoring)