Object storage access troubleshooting

This runbook covers S3-compatible access to Quake AI object storage (the Object Storage service, OpenStack Swift): credential and signing failures (401/403), and large object uploads that require multipart.

Quick triage#

S3 API access denied (credential issues)#

Symptoms#

• S3 API calls return HTTP 401 or 403.
• Error strings include InvalidAccessKeyId, SignatureDoesNotMatch, or Access Denied.

Diagnosis#

Check each cause below; more than one can apply.

Credential type and existence. EC2 credentials are separate from application credentials. Application credentials authenticate the OpenStack API; they do not work as S3 access keys. You must create EC2 credentials for the S3 API.

Project scope. EC2 credentials are project-scoped. A key from one project cannot access containers in another.

SigV4 signing. Wrong region, wrong service name in the signer, or clock skew produces SignatureDoesNotMatch even when the key pair is valid.

Container naming. Invalid or unsupported container (bucket) names for the S3 compatibility layer cause requests to fail or hit the wrong resource.

Endpoint configuration. The client must use the S3-compatible endpoint URL your environment defines: read it from configuration or environment variables at runtime, never hardcode endpoint URLs in application source.

Resolution#

1. List existing EC2 credentials:

openstack ec2 credentials list

1. If none exist, create a pair:

openstack ec2 credentials create

1. Confirm project_id on the credential matches the project that owns your containers. If you use the wrong project, set the active project and create a new pair:

export OS_PROJECT_NAME=YOUR_PROJECT_NAME
openstack ec2 credentials create

1. In your S3 client, set the region to the value Quake AI documents for SigV4. Set the endpoint from your environment (for example AWS_ENDPOINT_URL or your app's config), not a literal URL in code.

2. Test listing:

aws s3 ls --endpoint-url "${YOUR_ENDPOINT}"

Use the same endpoint variable you configure in production.

1. If SignatureDoesNotMatch remains after you verify region, endpoint (from env), system time (UTC), and simple bucket or key names, regenerate EC2 credentials and update AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

2. For deep CLI debugging:

aws s3 ls --endpoint-url "${YOUR_ENDPOINT}" --debug

Redact keys and signatures before you share output.

On Quake AI you use both credential types when you mix APIs: application credentials for OpenStack CLI/API, EC2 credentials for the S3 API.

Verification#

• openstack ec2 credentials list shows the active key with the correct project_id.
• aws s3 ls --endpoint-url "${YOUR_ENDPOINT}" returns expected containers, or object operations succeed against YOUR_BUCKET.

Prevention#

• Create EC2 credentials right after account or project setup; store the secret securely; you cannot retrieve it again after creation.
• Standardize on environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_ENDPOINT_URL or equivalent) so endpoint and keys never live in source control.
• Document which project each key pair belongs to.

When to escalate#

Escalate if regenerated credentials, correct project, verified SigV4 settings, and a minimal bucket or key test still return 401/403, or if multiple projects fail the same way at the same time (possible platform identity or gateway issue).

Large file upload failure (multipart)#

Symptoms (Large file upload failure (multipart))#

• Objects larger than 5 GB fail to upload.
• The AWS CLI or SDK returns EntityTooLarge, or the transfer stalls and times out.

Diagnosis (Large file upload failure (multipart))#

Objects over 5 GB require multipart upload. Some clients do not enable multipart by default for your code path. Other failures come from a multipart upload that started but never completed (stale uploads), network timeouts, or segment or part-size limits on the Swift S3 gateway that differ from default AWS S3 behavior.

Resolution (Large file upload failure (multipart))#

1. AWS CLI: Multipart kicks in automatically for objects larger than 8 MB. Confirm what the client does:

aws s3 cp YOUR_LOCAL_PATH s3://YOUR_BUCKET/YOUR_OBJECT_KEY --endpoint-url "${YOUR_ENDPOINT}" --debug

1. boto3: Set transfer limits explicitly. Read the endpoint from the environment; do not hardcode it.

import os

import boto3
from boto3.s3.transfer import TransferConfig

s3 = boto3.client("s3", endpoint_url=os.environ["AWS_ENDPOINT_URL"])
config = TransferConfig(
    multipart_threshold=8 * 1024 * 1024,
    multipart_chunksize=16 * 1024 * 1024,
)
s3.upload_file("YOUR_LOCAL_PATH", "YOUR_BUCKET", "YOUR_OBJECT_KEY", Config=config)

1. List incomplete multipart uploads:

aws s3api list-multipart-uploads --bucket YOUR_BUCKET --endpoint-url "${YOUR_ENDPOINT}"

1. Abort a stalled upload when you have the upload ID:

aws s3api abort-multipart-upload \
  --bucket YOUR_BUCKET \
  --key YOUR_OBJECT_KEY \
  --upload-id YOUR_UPLOAD_ID \
  --endpoint-url "${YOUR_ENDPOINT}"

1. Retry with a larger part size if timeouts caused partial failures. The Swift S3 gateway enforces segment size limits that differ from AWS S3's default 5 GB per-part maximum; reduce part size when you see limit or entity-size errors.

Verification (Large file upload failure (multipart))#

• The object appears in YOUR_BUCKET with the expected size.
• aws s3api list-multipart-uploads --bucket YOUR_BUCKET --endpoint-url "${YOUR_ENDPOINT}" shows no stray in-progress uploads after a successful complete.

Prevention (Large file upload failure (multipart))#

• Use rclone (or similar) for large or resumable transfers; it handles multipart and retries.
• Set client timeouts for your bandwidth and object size.
• For objects well above 50 GB, split into smaller objects or use a dedicated bulk transfer tool.

When to escalate (Large file upload failure (multipart))#

Escalate if abort and retry with conservative part sizes still fails, or if failures cluster at a specific size across multiple clients (possible platform limit or gateway defect).

Collecting evidence for support tickets#

Gather the following before you open a ticket. Redact secrets and access keys; never paste live credentials.

Example redacted debug capture:

aws s3 ls --endpoint-url "${YOUR_ENDPOINT}" --debug 2>&1 | tail -20

See also#

• Object storage API error reference: S3 authentication errors, credential confusion, and upload failures
• Create S3 credentials
• Upload a file
• Auth token diagnostics
• Support ticket evidence