# Object storage access troubleshooting

Source: https://docs.quake.ai/docs/operate/runbooks/object-storage-access
Markdown: https://docs.quake.ai/docs/operate/runbooks/object-storage-access.md
> Diagnose S3 403 errors, credential confusion, and upload failures for Quake AI object storage.

---

# Object storage access troubleshooting

This runbook covers S3-compatible access to Quake AI object storage (the Object Storage service, [OpenStack Swift](/resources/migration/openstack)): credential and signing failures (401/403), and large object uploads that require multipart.

<Figure size="md" caption="Where to start: pick the branch that matches your symptom, then jump to that section below.">

```d2
direction: right

start: "Object storage\nS3 API issue"

q_kind: "What's failing?" {shape: diamond}

leaf_auth: "S3 API access denied\n(credential issues)"
leaf_size: "Large file upload\nfailure (multipart)"

start -> q_kind
q_kind -> leaf_auth: 401 / 403 / signature error
q_kind -> leaf_size: file > 5 GB or timeout
```

</Figure>

## Quick triage

| Symptom | Most likely cause | Jump to |
| --- | --- | --- |
| S3 API returns HTTP 401/403, or errors like `InvalidAccessKeyId`, `SignatureDoesNotMatch`, or `Access Denied` | EC2 credentials missing or wrong type, wrong project scope, SigV4 misconfiguration, or invalid container name for S3 | [S3 API access denied (credential issues)](#s3-api-access-denied-credential-issues) |
| Files over 5 GB fail, `EntityTooLarge`, or uploads stall or time out | Multipart upload not used or not completed, network timeout, or Swift S3 gateway segment limits | [Large file upload failure (multipart)](#large-file-upload-failure-multipart) |

---

## S3 API access denied (credential issues)

### Symptoms

- S3 API calls return HTTP **401** or **403**.
- Error strings include `InvalidAccessKeyId`, `SignatureDoesNotMatch`, or `Access Denied`.



On the Quake AI S3 gateway, AWS CLI v2 can surface an authentication failure as a client-side `argument of type 'NoneType' is not iterable` stack trace instead of one of the error strings above. The gateway returns the error `<Code>` with an empty `<Message>` element, and AWS CLI v2 fails while parsing the empty message. If you hit this stack trace, rerun the command with `--debug` (Resolution step 7) and read the `<Code>` element from the response body to recover the underlying error, such as `SignatureDoesNotMatch` or `InvalidAccessKeyId`.



### Diagnosis

Check each cause below; more than one can apply.

**Credential type and existence.** EC2 credentials are separate from application credentials. Application credentials authenticate the OpenStack API; they do **not** work as S3 access keys. You must create EC2 credentials for the S3 API.

**Project scope.** EC2 credentials are project-scoped. A key from one project cannot access containers in another.

**SigV4 signing.** Wrong region, wrong service name in the signer, or clock skew produces `SignatureDoesNotMatch` even when the key pair is valid.

**Container naming.** Invalid or unsupported container (bucket) names for the S3 compatibility layer cause requests to fail or hit the wrong resource.

**Endpoint configuration.** The client must use the S3-compatible endpoint URL your environment defines: read it from configuration or environment variables at runtime, never hardcode endpoint URLs in application source.

### Resolution

1. List existing EC2 credentials:

```bash
openstack ec2 credentials list
```

2. If none exist, create a pair:

```bash
openstack ec2 credentials create
```

3. Confirm `project_id` on the credential matches the project that owns your containers. If you use the wrong project, set the active project and create a new pair:

```bash
export OS_PROJECT_NAME=YOUR_PROJECT_NAME
openstack ec2 credentials create
```

4. In your S3 client, set the region to the value Quake AI documents for SigV4. Set the endpoint from your environment (for example `AWS_ENDPOINT_URL` or your app's config), not a literal URL in code.

5. Test listing:

```bash
aws s3 ls --endpoint-url "${YOUR_ENDPOINT}"
```

Use the same endpoint variable you configure in production.

6. If `SignatureDoesNotMatch` remains after you verify region, endpoint (from env), system time (UTC), and simple bucket or key names, regenerate EC2 credentials and update `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.

7. For deep CLI debugging:

```bash
aws s3 ls --endpoint-url "${YOUR_ENDPOINT}" --debug
```

Redact keys and signatures before you share output.

On Quake AI you use **both** credential types when you mix APIs: application credentials for OpenStack CLI/API, EC2 credentials for the S3 API.

### Verification

- `openstack ec2 credentials list` shows the active key with the correct `project_id`.
- `aws s3 ls --endpoint-url "${YOUR_ENDPOINT}"` returns expected containers, or object operations succeed against `YOUR_BUCKET`.

### Prevention

- Create EC2 credentials right after account or project setup; store the secret securely; you cannot retrieve it again after creation.
- Standardize on environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_ENDPOINT_URL` or equivalent) so endpoint and keys never live in source control.
- Document which project each key pair belongs to.

### When to escalate

Escalate if regenerated credentials, correct project, verified SigV4 settings, and a minimal bucket or key test still return 401/403, or if multiple projects fail the same way at the same time (possible platform identity or gateway issue).

---

## Large file upload failure (multipart)

### Symptoms (Large file upload failure (multipart))

- Objects larger than 5 GB fail to upload.
- The AWS CLI or SDK returns `EntityTooLarge`, or the transfer stalls and times out.

### Diagnosis (Large file upload failure (multipart))

Objects over 5 GB require multipart upload. Some clients do not enable multipart by default for your code path. Other failures come from a multipart upload that started but never completed (stale uploads), network timeouts, or segment or part-size limits on the Swift S3 gateway that differ from default AWS S3 behavior.

### Resolution (Large file upload failure (multipart))

1. **AWS CLI**: Multipart kicks in automatically for objects larger than 8 MB. Confirm what the client does:

```bash
aws s3 cp YOUR_LOCAL_PATH s3://YOUR_BUCKET/YOUR_OBJECT_KEY --endpoint-url "${YOUR_ENDPOINT}" --debug
```

2. **boto3**: Set transfer limits explicitly. Read the endpoint from the environment; do not hardcode it.

```python
import os

import boto3
from boto3.s3.transfer import TransferConfig

s3 = boto3.client("s3", endpoint_url=os.environ["AWS_ENDPOINT_URL"])
config = TransferConfig(
    multipart_threshold=8 * 1024 * 1024,
    multipart_chunksize=16 * 1024 * 1024,
)
s3.upload_file("YOUR_LOCAL_PATH", "YOUR_BUCKET", "YOUR_OBJECT_KEY", Config=config)
```

3. List incomplete multipart uploads:

```bash
aws s3api list-multipart-uploads --bucket YOUR_BUCKET --endpoint-url "${YOUR_ENDPOINT}"
```

4. Abort a stalled upload when you have the upload ID:

```bash
aws s3api abort-multipart-upload \
  --bucket YOUR_BUCKET \
  --key YOUR_OBJECT_KEY \
  --upload-id YOUR_UPLOAD_ID \
  --endpoint-url "${YOUR_ENDPOINT}"
```

5. Retry with a larger part size if timeouts caused partial failures. The Swift S3 gateway enforces segment size limits that differ from AWS S3's default 5 GB per-part maximum; reduce part size when you see limit or entity-size errors.

### Verification (Large file upload failure (multipart))

- The object appears in `YOUR_BUCKET` with the expected size.
- `aws s3api list-multipart-uploads --bucket YOUR_BUCKET --endpoint-url "${YOUR_ENDPOINT}"` shows no stray in-progress uploads after a successful complete.

### Prevention (Large file upload failure (multipart))

- Use **rclone** (or similar) for large or resumable transfers; it handles multipart and retries.
- Set client timeouts for your bandwidth and object size.
- For objects well above 50 GB, split into smaller objects or use a dedicated bulk transfer tool.

### When to escalate (Large file upload failure (multipart))

Escalate if abort and retry with conservative part sizes still fails, or if failures cluster at a specific size across multiple clients (possible platform limit or gateway defect).

---

## Collecting evidence for support tickets

Gather the following before you open a ticket. Redact secrets and access keys; never paste live credentials.

| Artifact | Command or action |
| --- | --- |
| EC2 credential rows (redact secrets) | `openstack ec2 credentials list` |
| Active project | `openstack project show YOUR_PROJECT_NAME -c id` |
| S3 list test (redacted debug tail) | `aws s3 ls --endpoint-url "${YOUR_ENDPOINT}" --debug` |
| Incomplete multipart uploads | `aws s3api list-multipart-uploads --bucket YOUR_BUCKET --endpoint-url "${YOUR_ENDPOINT}"` |
| Failure time (UTC) | `date -u` |

Example redacted debug capture:

```bash
aws s3 ls --endpoint-url "${YOUR_ENDPOINT}" --debug 2>&1 | tail -20
```

---

## See also

- [Object storage API error reference](/reference/object-storage/api-errors): S3 authentication errors, credential confusion, and upload failures
- [Create S3 credentials](/docs/object/how-to/create-s3-credentials)
- [Upload a file](/docs/object/how-to/upload-file)
- [Auth token diagnostics](/docs/operate/troubleshooting/auth-token-diagnostics)
- [Support ticket evidence](/docs/operate/troubleshooting/support-ticket-evidence)