Authentication and token diagnostics

This page covers diagnosis and recovery for API authentication failures: HTTP 401 Unauthorized, HTTP 403 Forbidden, token expiry, credential scope mismatches, and the common confusion between application credentials and EC2 (S3) credentials. Use this when API calls fail with authentication or authorization errors.

Quick diagnosis: 401 vs 403#

Start with the 401 path if you cannot authenticate at all. Start with the 403 path if you can authenticate but specific operations fail.

401 Unauthorized: token and credential failures#

Symptom#

Any API call returns 401. The OpenStack CLI shows:

Unauthorized (HTTP 401)

Or SDK/HTTP calls return:

{"error": {"message": "The request you have made requires authentication.", "code": 401}}

Diagnosis#

Step 1: Test token issuance

openstack token issue

If this succeeds, your credentials are valid and a fresh token was issued. The original failure was likely a stale token in your environment.

If this fails with 401, your credentials are invalid or expired.

Step 2: Verify your environment

env | grep OS_

Check that:

• OS_AUTH_URL points to the correct Keystone endpoint
• OS_PROJECT_NAME or OS_PROJECT_ID is set
• OS_APPLICATION_CREDENTIAL_ID and OS_APPLICATION_CREDENTIAL_SECRET are set (if using app credentials)
• No stale OS_TOKEN variable overrides the credential flow

Step 3: Re-source your credentials

source openrc.sh
openstack token issue

Resolution#

Expired token: tokens have a limited lifetime (typically 1 hour). Re-source your credentials and retry:

source openrc.sh

Stale environment: if you have multiple terminal sessions, each may have different credentials loaded. Source the correct openrc.sh in the session where you are working.

Invalid credentials: if openstack token issue fails even after re-sourcing:

1. Verify the openrc.sh file has not been modified or corrupted
2. Re-download your application credentials from the Quake AI console under Identity > Application Credentials
3. Create new application credentials if the existing ones are compromised

Cached token override: if OS_TOKEN is set in your environment, unset it:

unset OS_TOKEN
openstack token issue

403 Forbidden: scope and role failures#

Symptom (403 Forbidden: scope and role failures)#

Authentication succeeds (openstack token issue works) but specific operations fail with 403.

Diagnosis (403 Forbidden: scope and role failures)#

Step 1: Verify project scope

openstack token issue -c project_id

Compare the returned project ID with the project that owns the resources you are trying to access.

Step 2: Check your assigned roles

The openstack role assignment list command requires the identity:list_role_assignments policy, which is reserved for identity administrators. Under an ordinary application credential it returns its own 403, so it cannot diagnose the error you are troubleshooting:

You are not authorized to perform the requested action: identity:list_role_assignments. (HTTP 403)

To see the roles your credential carries, open the Quake AI console under Identity > Application Credentials. The console lists the roles each credential was created with, and the same roles appear when you create a credential.

Common roles and their permissions:

Step 3: Check quota. Some 403 errors indicate quota exhaustion:

openstack quota show --usage

If a resource is at its limit, the API returns 403 with a quota message rather than a role/scope message.

Resolution (403 Forbidden: scope and role failures)#

Wrong project scope: re-source with the correct project:

Verify your openrc.sh specifies the correct OS_PROJECT_NAME. If you work with multiple projects, create separate openrc.sh files for each.

Insufficient role: if you need a role upgrade, contact your project administrator.

Quota exhaustion: see quota and limits troubleshooting.

Credential type confusion: application credentials vs EC2 credentials#

This is the most common authentication issue for users working with both the OpenStack API and the S3-compatible object storage API.

Symptoms of using the wrong credential type#

Resolution (credential type confusion: application credentials vs EC2 credentials)#

For OpenStack API access:

source openrc.sh
openstack token issue

For S3-compatible API access:

openstack ec2 credentials list

If no credentials appear, create them:

openstack ec2 credentials create

Then configure your S3 client:

export AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY"
export AWS_ENDPOINT_URL="RUMBLE_S3_ENDPOINT"

Verification#

After resolving an authentication issue, verify with these tests:

OpenStack API:

openstack token issue
openstack server list

S3-compatible API:

aws s3 ls --endpoint-url $RUMBLE_S3_ENDPOINT

Both commands should succeed without authentication errors.

Escalation signals#

Escalate to support when:

• openstack token issue fails with 401 after downloading fresh application credentials from the console
• A valid token (verified via openstack token issue) is rejected by a specific service endpoint
• EC2 credentials created moments ago fail with InvalidAccessKeyId immediately
• Your assigned roles (shown in the console under Identity > Application Credentials) look correct but 403 persists for operations the role should allow

Include in your ticket:

• The exact error message
• Output of openstack token issue (the token ID, not the credential secret)
• Project ID and user ID
• The specific operation that fails

See also#

• How to generate application credentials
• API tokens
• How to create S3 credentials
• Object storage access troubleshooting
• Object storage API error reference
• Troubleshooting overview