Store and retrieve files with S3-compatible object storage

In this tutorial, we create S3 credentials, configure the AWS CLI to point at Quake AI's object storage endpoint, create a storage container, upload and download files, and apply a basic access policy. By the end, you will have a working object storage workflow that you can use from any S3-compatible tool or SDK.

What you will learn:

• What S3 credentials are and how they differ from your Quake AI login
• How to configure the AWS CLI to work with Quake AI's S3-compatible endpoint
• How containers (buckets) work in Quake AI object storage
• How to upload, list, and download files using both the console and the AWS CLI
• How to apply a read-only access policy to a container

Time estimate: 20–30 minutes

Prerequisites#

You need:

• A Quake AI account with an active project
• The AWS CLI installed on your local machine:

pip install awscli

Verify the installation:

aws --version

You do not need an AWS account. The AWS CLI works with any S3-compatible storage endpoint, including Quake AI's.

Why S3 compatibility matters#

Quake AI Object Storage implements the S3 API. This means every tool built for Amazon S3, the AWS CLI, s3cmd, rclone, boto3, and dozens of backup and sync utilities, works with Quake AI storage without modification. You point the tool at a different endpoint URL and use Quake AI's credentials instead of AWS credentials.

This compatibility is deliberate. Migrating data to and from Quake AI does not require rewriting integrations, only reconfiguring endpoints.

Step 1: Create S3 credentials#

S3 credentials are separate from your Quake AI console login. They consist of an access key and a secret key, scoped to your project, used exclusively for S3-compatible storage access.

1. In the Quake AI console, go to Storage > Object Storage > S3 Credentials.
2. Select Create S3 Credential.
3. Enter a name for the credential set, for example tutorial-credentials.
4. Select OK.
5. The console displays the Access Key and Secret Key. Copy both values and store them somewhere secure. The secret key is shown only once and cannot be retrieved again.
6. Select Close.

Step 2: Configure the AWS CLI#

Configure the AWS CLI with your Quake AI S3 credentials and endpoint. We use a named profile (rumble) to keep these settings separate from any existing AWS configuration on your machine.

aws configure --profile rumble

At each prompt, enter:

AWS Access Key ID [None]: YOUR_ACCESS_KEY
AWS Secret Access Key [None]: YOUR_SECRET_KEY
Default region name [None]: us-east-1
Default output format [None]: json

Replace YOUR_ACCESS_KEY and YOUR_SECRET_KEY with the values from Step 1.

Now set the S3 endpoint for this profile. The AWS CLI reads a per-profile endpoint_url from ~/.aws/config. The endpoint URL is specific to the region your project is in:

Open ~/.aws/config and add the endpoint_url line to the [profile rumble] section that aws configure created:

[profile rumble]
region = us-east-1
output = json
endpoint_url = https://object.us-east-1.rumble.cloud

Verify the configuration by listing your containers (the list is empty, which is expected):

aws s3 ls --profile rumble

No error means the credentials and endpoint are configured correctly.

Step 3: Create a container#

Object storage organizes files into containers. Containers are the equivalent of S3 buckets: a flat namespace for storing objects. Container names must be unique within your project.

Console:

1. Go to Storage > Object Storage > Create Container.
2. Enter the name tutorial-bucket.
3. Leave the access set to private (not publicly accessible).
4. Select OK.

The container appears in the list immediately.

CLI equivalent:

aws s3 mb s3://tutorial-bucket --profile rumble

Both methods create the same result. The console is easier for one-off creation; the CLI is better for scripting and automation.

Step 4: Upload a file#

Console:

1. Go to Storage > Object Storage and select tutorial-bucket.
2. Select Upload File.
3. Choose a file from your local machine (any small file works: a text file or image is fine).
4. Select OK.

The file appears in the container listing.

CLI:

Upload a file with the AWS CLI:

echo "hello from Quake AI" > /tmp/hello.txt
aws s3 cp /tmp/hello.txt s3://tutorial-bucket/hello.txt --profile rumble

Upload an entire local directory:

aws s3 sync /tmp/my-folder s3://tutorial-bucket/my-folder --profile rumble

The sync command uploads only files that are new or changed, making it efficient for incremental backups and deployments.

Step 5: List and download files#

List the contents of the container:

aws s3 ls s3://tutorial-bucket --profile rumble

Download a file:

aws s3 cp s3://tutorial-bucket/hello.txt /tmp/hello-downloaded.txt --profile rumble
cat /tmp/hello-downloaded.txt

The downloaded file contains the same content you uploaded. Files in object storage are retrieved via key (their path within the container) and are stored with full consistency: what you put in is exactly what you get back.

Step 6: Set a basic access policy#

By default, objects in your container are private: only requests authenticated with your S3 credentials can access them. You can make a container or individual objects publicly readable using a bucket policy.

The following policy makes all objects in tutorial-bucket publicly readable via HTTPS, without credentials:

aws s3api put-bucket-policy \
  --bucket tutorial-bucket \
  --policy '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": "*",
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::tutorial-bucket/*"
      }
    ]
  }' \
  --profile rumble

After applying the policy, objects in tutorial-bucket are accessible at:

https://object.us-east-1.rumble.cloud/YOUR_PROJECT_ID:tutorial-bucket/OBJECT_KEY

Replace YOUR_PROJECT_ID with your cloud project ID and OBJECT_KEY with the filename or path of the object. For hello.txt, the URL is:

https://object.us-east-1.rumble.cloud/YOUR_PROJECT_ID:tutorial-bucket/hello.txt

The YOUR_PROJECT_ID: prefix is required for anonymous reads on Quake AI's S3-compatible endpoint. The plain tutorial-bucket/OBJECT_KEY form is reserved for authenticated S3 API access and returns NoSuchBucket to anonymous callers.

For more policy patterns, see Bucket policy examples.

What you learned#

In this tutorial, we:

• Created S3 credentials scoped to our Quake AI project
• Configured the AWS CLI to use Quake AI's S3-compatible endpoint with a named profile
• Created a container via both the console and the CLI
• Uploaded, listed, and downloaded files using aws s3 cp and aws s3 ls
• Applied a bucket policy to make objects publicly readable

Any tool or library that speaks S3 works with this setup. The endpoint_url configuration is the only Quake AI-specific change required.

Next steps#

• Create a volume and add persistent storage to your server: the previous tutorial in this series, if you have not completed it
• Mount S3 storage as a filesystem: make object storage browseable on Linux with s3fs
• Back up to Quake AI: 3-2-1 backups with rclone and restic
• Object storage concepts: deeper background on the storage model, consistency guarantees, and pricing

Clean up#

To remove the resources we created:

Delete all objects in the container:

aws s3 rm s3://tutorial-bucket --recursive --profile rumble

Delete the container:

aws s3 rb s3://tutorial-bucket --profile rumble

Optionally revoke the S3 credentials:

1. In the Quake AI console, go to Storage > Object Storage > S3 Credentials.
2. Select tutorial-credentials and select Delete.

After deletion, any tools configured with those credentials will no longer have access to your object storage.