# Authentication and token diagnostics

Source: https://docs.quake.ai/docs/operate/troubleshooting/auth-token-diagnostics
Markdown: https://docs.quake.ai/docs/operate/troubleshooting/auth-token-diagnostics.md
> Diagnose HTTP 401/403 errors, expired tokens, credential scope mismatches, and app credential vs EC2 credential confusion.

---

# Authentication and token diagnostics

This page covers diagnosis and recovery for API authentication failures: HTTP 401 Unauthorized, HTTP 403 Forbidden, token expiry, credential scope mismatches, and the common confusion between application credentials and EC2 (S3) credentials. Use this when API calls fail with authentication or authorization errors.

## Quick diagnosis: 401 vs 403

| HTTP status | Meaning | Most likely cause |
|---|---|---|
| **401 Unauthorized** | Identity not established | Expired token, missing `X-Auth-Token` header, invalid credentials |
| **403 Forbidden** | Identity established but action not permitted | Wrong project scope, insufficient role, quota exceeded |

Start with the 401 path if you cannot authenticate at all. Start with the 403 path if you can authenticate but specific operations fail.

---

## 401 Unauthorized: token and credential failures

### Symptom

Any API call returns 401. The OpenStack CLI shows:

```text
Unauthorized (HTTP 401)
```

Or SDK/HTTP calls return:

```json
{"error": {"message": "The request you have made requires authentication.", "code": 401}}
```

### Diagnosis

**Step 1: Test token issuance**

```bash
openstack token issue
```

If this succeeds, your credentials are valid and a fresh token was issued. The original failure was likely a stale token in your environment.

If this fails with 401, your credentials are invalid or expired.

**Step 2: Verify your environment**

```bash
env | grep OS_
```

Check that:
- `OS_AUTH_URL` points to the correct Keystone endpoint
- `OS_PROJECT_NAME` or `OS_PROJECT_ID` is set
- `OS_APPLICATION_CREDENTIAL_ID` and `OS_APPLICATION_CREDENTIAL_SECRET` are set (if using app credentials)
- No stale `OS_TOKEN` variable overrides the credential flow

**Step 3: Re-source your credentials**

```bash
source openrc.sh
openstack token issue
```

### Resolution

**Expired token**: tokens have a limited lifetime (typically 1 hour). Re-source your credentials and retry:

```bash
source openrc.sh
```

**Stale environment**: if you have multiple terminal sessions, each may have different credentials loaded. Source the correct `openrc.sh` in the session where you are working.

**Invalid credentials**: if `openstack token issue` fails even after re-sourcing:

1. Verify the `openrc.sh` file has not been modified or corrupted
2. Re-download your application credentials from the [Quake AI console](https://cloud.rumble.cloud) under **Identity** > **Application Credentials**
3. Create new application credentials if the existing ones are compromised

**Cached token override**: if `OS_TOKEN` is set in your environment, unset it:

```bash
unset OS_TOKEN
openstack token issue
```

---

## 403 Forbidden: scope and role failures

### Symptom (403 Forbidden: scope and role failures)

Authentication succeeds (`openstack token issue` works) but specific operations fail with 403.

### Diagnosis (403 Forbidden: scope and role failures)

**Step 1: Verify project scope**

```bash
openstack token issue -c project_id
```

Compare the returned project ID with the project that owns the resources you are trying to access.

**Step 2: Check your assigned roles**

The `openstack role assignment list` command requires the `identity:list_role_assignments` policy, which is reserved for identity administrators. Under an ordinary application credential it returns its own 403, so it cannot diagnose the error you are troubleshooting:

```text
You are not authorized to perform the requested action: identity:list_role_assignments. (HTTP 403)
```

To see the roles your credential carries, open the [Quake AI console](https://cloud.rumble.cloud) under **Identity** > **Application Credentials**. The console lists the roles each credential was created with, and the same roles appear when you create a credential.

Common roles and their permissions:

| Role | Permissions |
|---|---|
| `member` | Create, read, update, delete project resources |
| `reader` | Read-only access to project resources |
| `admin` | Full administrative access (not assigned to regular users in normal operation) |

**Step 3: Check quota.** Some 403 errors indicate quota exhaustion:

```bash
openstack quota show --usage
```

If a resource is at its limit, the API returns 403 with a quota message rather than a role/scope message.

### Resolution (403 Forbidden: scope and role failures)

**Wrong project scope**: re-source with the correct project:

Verify your `openrc.sh` specifies the correct `OS_PROJECT_NAME`. If you work with multiple projects, create separate `openrc.sh` files for each.

**Insufficient role**: if you need a role upgrade, contact your project administrator.

**Quota exhaustion**: see [quota and limits troubleshooting](/docs/operate/troubleshooting/quota-and-limits).

---

## Credential type confusion: application credentials vs EC2 credentials

This is the most common authentication issue for users working with both the OpenStack API and the S3-compatible object storage API.

| | Application credentials | EC2 credentials |
|---|---|---|
| **Purpose** | OpenStack API authentication (CLI, SDK, API) | S3-compatible object storage API |
| **Create command** | `openstack application credential create` | `openstack ec2 credentials create` |
| **Stored in** | `openrc.sh` or environment variables | AWS-style config or environment variables |
| **Project scope** | Yes, scoped to one project | Yes, scoped to one project |
| **Works with `openstack` CLI** | Yes | No |
| **Works with `aws s3` / boto3** | No | Yes |

### Symptoms of using the wrong credential type

| Scenario | Error |
|---|---|
| Using application credentials with `aws s3` | `InvalidAccessKeyId` (403) |
| Using EC2 credentials with `openstack` CLI | `Unauthorized` (401) |
| Using EC2 credentials from project A to access containers in project B | `AccessDenied` (403) |

### Resolution (credential type confusion: application credentials vs EC2 credentials)

**For OpenStack API access:**

```bash
source openrc.sh
openstack token issue
```

**For S3-compatible API access:**

```bash
openstack ec2 credentials list
```

If no credentials appear, create them:

```bash
openstack ec2 credentials create
```

Then configure your S3 client:

```bash
export AWS_ACCESS_KEY_ID="YOUR_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="YOUR_SECRET_KEY"
export AWS_ENDPOINT_URL="RUMBLE_S3_ENDPOINT"
```

---

## Verification

After resolving an authentication issue, verify with these tests:

**OpenStack API:**

```bash
openstack token issue
openstack server list
```

**S3-compatible API:**

```bash
aws s3 ls --endpoint-url $RUMBLE_S3_ENDPOINT
```

Both commands should succeed without authentication errors.

## Escalation signals

Escalate to support when:

- `openstack token issue` fails with 401 after downloading fresh application credentials from the console
- A valid token (verified via `openstack token issue`) is rejected by a specific service endpoint
- EC2 credentials created moments ago fail with `InvalidAccessKeyId` immediately
- Your assigned roles (shown in the console under **Identity** > **Application Credentials**) look correct but 403 persists for operations the role should allow

Include in your ticket:
- The exact error message
- Output of `openstack token issue` (the token ID, not the credential secret)
- Project ID and user ID
- The specific operation that fails

## See also

- [How to generate application credentials](/docs/tools/generate-app-credentials)
- [API tokens](/docs/tools/api-tokens)
- [How to create S3 credentials](/docs/object/how-to/create-s3-credentials)
- [Object storage access troubleshooting](/docs/operate/runbooks/object-storage-access)
- [Object storage API error reference](/reference/object-storage/api-errors)
- [Troubleshooting overview](/docs/operate/troubleshooting)