Migrate your data from AWS S3 to Quake AI

In this migration guide, you'll move data from an AWS S3 bucket to Quake AI object storage using rclone. By the end, you will have migrated a complete bucket with integrity verification, identified where Quake AI's S3 compatibility has limits, and have a tested procedure you can apply to the rest of your buckets.

You'll walk through the full migration methodology: inventory your source data, check for compatibility gaps, run a pilot migration on one bucket, validate that every object arrived intact, and then update your application to use the new endpoint. Each step explains why it matters so you can adapt the process to your own situation.

What you will learn:

How to inventory an AWS S3 bucket and estimate migration time
Which S3 features Quake AI supports and which it does not (and what to do about gaps)
How to configure rclone to copy data between two S3-compatible providers
How to validate a migration with checksum verification
How to update your application's S3 endpoint with zero data loss
How to run both endpoints in parallel during a transition period

Time estimate: 45–60 minutes (excluding data transfer time, which depends on bucket size)

Click to zoom

What you'll build: a five-step migration: inventory the source, check compatibility, pilot one bucket, validate checksums, then cut the application over to Quake AI.

Prerequisites#

You need:

An AWS account with an S3 bucket containing data you want to migrate
An AWS IAM user or role with s3:GetObject, s3:ListBucket, and s3:GetBucketLocation permissions on the source bucket
A Quake AI account with S3 credentials created
rclone v1.65+ installed on your local machine
A stable internet connection (migration speed depends on your upload bandwidth)

Why migrate#

With AWS S3 pricing, you pay for storage, requests, data retrieval, and egress separately, with different rates for different storage classes. Quake AI charges a flat rate per TB with unlimited data transfer included. No egress fees, no request-tier pricing.

Step 1: Inventory your source bucket#

Before copying anything, we need to know what we are migrating. Counting objects and measuring total size upfront helps you estimate transfer time and catch surprises (like a bucket that is larger than expected).

List the bucket contents and count objects:

bash

aws s3 ls s3://YOUR_SOURCE_BUCKET --recursive --summarize

The --summarize flag prints total object count and total size at the end. Note both numbers.

For a more detailed breakdown by prefix (folder), use:

bash

aws s3 ls s3://YOUR_SOURCE_BUCKET --recursive | awk '{sum += $3; count++} END {printf "Objects: %d\nTotal size: %.2f GB\n", count, sum/1024/1024/1024}'

Record these numbers. We use them after the migration to verify nothing was lost.

Estimate transfer time: At a typical home connection (50 Mbps upload), 100 GB takes about 4.5 hours. At a datacenter connection (1 Gbps), the same transfer takes about 15 minutes. Plan your migration window accordingly.

Step 2: Check for compatibility gaps#

Quake AI's object storage is S3-compatible, but not S3-identical. It runs on OpenStack Swift with an S3 gateway. Most S3 operations work without modification, but some AWS-specific features are not supported.

What works:

s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket: all standard CRUD operations
Multipart uploads (rclone handles this automatically for large files)
Bucket policies for public-read access
Pre-signed URLs for time-limited access
Server-side encryption with SSE-S3
Object metadata (Content-Type, Cache-Control, custom headers)

What does not work:

AWS feature	Quake AI status	What to do
S3 Lifecycle policies	Not supported	Manage retention manually or with rclone scripts
S3 Event Notifications (SNS/SQS/Lambda)	Not supported	Use application-level polling or cron jobs
S3 Object Lock / WORM	Not supported	Use bucket policies for access control
IAM-style bucket policies (complex conditions)	Partial	Basic Allow/Deny works; condition keys are limited
S3 Storage Classes (Glacier, IA)	Not supported	All objects use standard storage
S3 Inventory reports	Not supported	Use `rclone ls` or `aws s3 ls` for inventory
Cross-region replication	Not supported	Use rclone sync between regions if needed

If your application depends on lifecycle policies or event notifications. You will need to replace those features before switching over. For most applications (file storage, static assets, backups, media hosting), the compatibility is sufficient.

For the complete compatibility matrix, see the migration planning guide.

Step 3: Configure rclone for both providers#

rclone acts as a bridge between AWS and Quake AI. We configure a remote for each provider so rclone can read from one and write to the other.

Edit your rclone configuration:

bash

nano ~/.config/rclone/rclone.conf

Add both remotes:

ini

[aws]
type = s3
provider = AWS
access_key_id = YOUR_AWS_ACCESS_KEY
secret_access_key = YOUR_AWS_SECRET_KEY
region = us-east-1

[quakeai]
type = s3
provider = Ceph
access_key_id = YOUR_RUMBLE_ACCESS_KEY
secret_access_key = YOUR_RUMBLE_SECRET_KEY
endpoint = object.us-east-1.rumble.cloud

Replace the placeholder values with your actual credentials.

Quake AI's object storage runs on OpenStack Swift with an S3 gateway. The provider = Ceph setting is a hint that adjusts how rclone shapes its requests (region handling, list-buckets behavior, retry semantics) for non-AWS S3-compatible services, and it is the recommended setting for Quake AI because it avoids a handful of AWS-specific edge cases. provider = AWS against the same endpoint = object.us-east-1.rumble.cloud also works, so either value will list buckets and copy objects.

Protect the config file. The block you pasted contains plaintext AWS and Quake AI S3 secrets, and a default Linux file mode leaves the config world-readable to other local accounts.

bash

chmod 600 ~/.config/rclone/rclone.conf

⚠️ Protect your credentials

The rclone config file stores plaintext AWS and Quake AI S3 secret keys. Set its mode to 0600 so other local users cannot read it:

bash

chmod 600 ~/.config/rclone/rclone.conf

On a multi-user host or a shared migration VM, this step is mandatory. For production deployments, load credentials from environment variables or a secret manager and remove the long-lived secrets from disk entirely.

Verify both remotes work:

bash

rclone lsd aws:
rclone lsd quakeai:

Each command should list your buckets on the respective provider. If either fails, check the credentials and endpoint.

Step 4: Create the destination bucket#

Create a bucket on Quake AI to receive the migrated data. We use the same name as the source bucket when possible: this simplifies endpoint switching later.

bash

rclone mkdir quakeai:YOUR_SOURCE_BUCKET

Quake AI bucket names are unique within your project, not globally unique like AWS S3. Unless a bucket with the source name already exists inside your own Quake AI project, the name is available and you can keep it. Reusing the source name makes the endpoint swap in Step 8 a one-line change. If you do hit a name collision in your own project (for example, you created a test bucket with that name earlier), pick a different name and record the mapping:

bash

rclone mkdir quakeai:YOUR_NEW_BUCKET_NAME

Step 5: Run a pilot migration#

We start with a small subset of data, not the entire bucket. This validates that the configuration works, metadata transfers correctly, and the transfer speed meets expectations.

Copy a single prefix (folder) from AWS to Quake AI:

bash

rclone copy aws:YOUR_SOURCE_BUCKET/sample-folder quakeai:YOUR_SOURCE_BUCKET/sample-folder \
  --transfers 8 \
  --checkers 16 \
  --progress \
  --metadata

What each flag does:

copy transfers files without deleting anything at the destination
--transfers 8 runs 8 parallel file uploads
--checkers 16 runs 16 parallel integrity checkers
--progress shows real-time transfer statistics
--metadata preserves object metadata (Content-Type, timestamps, custom headers)

Watch the output. You should see files transferring with speed and progress indicators. When complete, verify the pilot:

bash

echo "Source:"
rclone ls aws:YOUR_SOURCE_BUCKET/sample-folder | wc -l

echo "Destination:"
rclone ls quakeai:YOUR_SOURCE_BUCKET/sample-folder | wc -l

If the counts match, the pilot succeeded.

Step 6: Run the full migration#

With the pilot verified, copy the entire bucket:

bash

rclone copy aws:YOUR_SOURCE_BUCKET quakeai:YOUR_SOURCE_BUCKET \
  --transfers 16 \
  --checkers 32 \
  --progress \
  --metadata \
  --fast-list \
  --log-file ~/migration.log \
  --log-level INFO

New flags for the full migration:

--transfers 16 and --checkers 32 increase parallelism for faster throughput
--fast-list uses fewer API calls to list objects (faster for large buckets, uses more memory)
--log-file writes a detailed log for post-migration review
--log-level INFO logs each transferred file

For buckets over 100 GB, consider running the migration from a VM in the same region as either the source or destination. This avoids routing data through your local machine and can be 10x faster.

⚠️ Large migrations

For buckets over 1 TB, run the migration in stages by prefix. This lets you resume without re-transferring everything if the connection drops:

bash

rclone copy aws:YOUR_SOURCE_BUCKET/prefix-a quakeai:YOUR_SOURCE_BUCKET/prefix-a ...
rclone copy aws:YOUR_SOURCE_BUCKET/prefix-b quakeai:YOUR_SOURCE_BUCKET/prefix-b ...

rclone's copy command is idempotent: running it again only transfers files that are missing or changed at the destination.

Step 7: Validate the migration#

Counting files is a sanity check. Checksum verification is proof. rclone can compare the source and destination byte-for-byte:

bash

rclone check aws:YOUR_SOURCE_BUCKET quakeai:YOUR_SOURCE_BUCKET \
  --one-way \
  --fast-list

rclone check --one-way verifies that every file in the source exists at the destination with matching size and checksum. It does not transfer any data; it only compares.

The output reports:

Files that match (good)
Files that differ in size or checksum (investigate)
Files that exist in the source but not the destination (re-transfer)

If all files match, the migration is verified. If some files differ, re-run the rclone copy command: it will only transfer the mismatched files.

For a final count comparison:

bash

echo "AWS objects:"
rclone size aws:YOUR_SOURCE_BUCKET

echo "Quake AI objects:"
rclone size quakeai:YOUR_SOURCE_BUCKET

Both should report the same object count and total size.

Step 8: Update your application endpoint#

With the data verified on Quake AI, the last step is pointing your application at the new endpoint. The beauty of S3 compatibility is that this is a configuration change, not a code change.

Before (AWS):

endpoint: s3.us-east-1.amazonaws.com
bucket: my-app-assets
access_key: AKIA...
secret_key: ...

After (Quake AI):

endpoint: object.us-east-1.rumble.cloud
bucket: my-app-assets
access_key: YOUR_RUMBLE_ACCESS_KEY
secret_key: YOUR_RUMBLE_SECRET_KEY

The S3 API calls (GetObject, PutObject, ListBucket, etc.) remain identical. Only the endpoint and credentials change.

For applications using the AWS SDK, set the endpoint URL explicitly:

Python

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="https://object.us-east-1.rumble.cloud",
    aws_access_key_id="YOUR_RUMBLE_ACCESS_KEY",
    aws_secret_access_key="YOUR_RUMBLE_SECRET_KEY",
)

JavaScript

const { S3Client } = require("@aws-sdk/client-s3");

const s3 = new S3Client({
  endpoint: "https://object.us-east-1.rumble.cloud",
  region: "us-east-1",
  credentials: {
    accessKeyId: "YOUR_RUMBLE_ACCESS_KEY",
    secretAccessKey: "YOUR_RUMBLE_SECRET_KEY",
  },
  forcePathStyle: true,
});

💡 Run both endpoints in parallel

For zero-downtime migrations, run a dual-write pattern during the transition: write new objects to both AWS and Quake AI, fetch from Quake AI. Once you confirm fetches are stable, stop writing to AWS. This gives you a rollback path if anything unexpected surfaces.

For retrieval, switch your application to fetch from Quake AI first and fall back to AWS:

Python

def get_object(key):
    try:
        return rumble_s3.get_object(Bucket=BUCKET, Key=key)
    except Exception:
        return aws_s3.get_object(Bucket=BUCKET, Key=key)

After a week with zero AWS fallbacks. You can remove the dual-read logic.

What you learned#

In this migration guide, you:

Inventoried a source bucket to understand what you were migrating and estimate transfer time
Checked S3 compatibility to identify features that work differently on Quake AI (lifecycle policies, event notifications)
Configured rclone as a bridge between AWS S3 and Quake AI
Ran a pilot migration on a single prefix to validate the configuration before committing to the full transfer
Migrated the entire bucket with parallel transfers, metadata preservation, and logging
Validated integrity with checksum comparison to prove every object arrived intact
Updated the application endpoint: a configuration change, not a code rewrite

This process works for any S3-compatible source, not only AWS. Replace the [aws] remote with any provider rclone supports: DigitalOcean Spaces, Backblaze B2, Google Cloud Storage, Hetzner, Wasabi. The same steps apply.

Next steps#

Set up automated backups to Quake AI: protect your migrated data with rclone and restic
S3 feature compatibility reference: the complete compatibility matrix
Multi-cloud sync: keep data synchronized across providers during a phased migration
Update your application endpoint: the how-to reference for endpoint switching across languages and frameworks

Clean up#

This migration moves real data that you likely want to keep. Nothing needs to be torn down on the Quake AI side.

On AWS, once you are confident the migration is complete and your application is running on Quake AI:

Keep the AWS bucket in read-only mode for 30 days as a safety net.
After 30 days with no issues, delete the AWS bucket contents and then the bucket itself:

bash

aws s3 rm s3://YOUR_SOURCE_BUCKET --recursive
aws s3 rb s3://YOUR_SOURCE_BUCKET

Do not delete the source bucket until you are certain the migration is complete and your application is fully operational on Quake AI.

Migrate your data from AWS S3 to Quake AI

Migrate your data from AWS S3 to Quake AI

Prerequisites#

Why migrate#

Step 1: Inventory your source bucket#

Step 2: Check for compatibility gaps#

Step 3: Configure rclone for both providers#

Step 4: Create the destination bucket#

Step 5: Run a pilot migration#

Step 6: Run the full migration#

Step 7: Validate the migration#

Step 8: Update your application endpoint#

What you learned#

Next steps#

Clean up#

Usage Guidelines

See Also