How to migrate from AWS S3 to Quake AI

This guide walks through a complete migration of objects from an AWS S3 bucket to Quake AI object storage. The recommended tool is rclone, which copies data directly between providers without staging files locally.

Service mapping#

Prerequisites#

• A Quake AI account with S3 credentials
• An AWS IAM user or role with s3:GetObject, s3:ListBucket, and s3:GetBucketLocation permissions on the source bucket
• rclone installed (v1.65+)
• Optionally: MinIO Client if your objects have custom metadata that must be preserved

Review the planning guide#

Before starting, read through the migration planning guide to:

• Check S3 feature compatibility (lifecycle policies, bucket policies, and event notifications are not supported on Quake AI)
• Inventory your data (total size, object count, custom metadata usage)
• Choose a migration strategy (big-bang, phased, or continuous sync)

Configure rclone remotes#

Add both your AWS source and Quake AI destination to rclone's config. You can do this interactively with rclone config or edit ~/.config/rclone/rclone.conf directly:

[aws]
type = s3
provider = AWS
access_key_id = YOUR_AWS_ACCESS_KEY
secret_access_key = YOUR_AWS_SECRET_KEY
region = us-east-1

[quakeai]
type = s3
provider = Ceph
access_key_id = YOUR_RUMBLE_ACCESS_KEY
secret_access_key = YOUR_RUMBLE_SECRET_KEY
endpoint = object.us-east-2.rumble.cloud
acl = private

Replace object.us-east-2.rumble.cloud with your Quake AI region: us-east-1, us-east-2, or us-west-1.

Verify both remotes:

rclone lsd aws:
rclone lsd quakeai:

Create the destination bucket#

If the target bucket does not already exist on Quake AI:

rclone mkdir quakeai:my-bucket

Or use the Quake AI console: Storage > Object Storage > Create Bucket.

Run the initial copy#

For data sets under 100 GB, a single sync is sufficient:

rclone sync aws:my-source-bucket quakeai:my-bucket \
  --transfers 16 \
  --checkers 32 \
  --s3-chunk-size 64M \
  --progress

For larger data sets, use copy instead of sync for the initial transfer (safer: does not delete anything on the destination), then switch to sync for delta passes:

rclone copy aws:my-source-bucket quakeai:my-bucket \
  --transfers 16 \
  --checkers 32 \
  --s3-chunk-size 64M \
  --progress \
  --log-file migration.log \
  --log-level INFO

Bandwidth limiting#

To avoid saturating your network during business hours:

rclone copy aws:my-source-bucket quakeai:my-bucket \
  --bwlimit "08:00,10M 18:00,off" \
  --transfers 16 \
  --progress

This limits bandwidth to 10 MB/s between 08:00 and 18:00, and removes the limit outside those hours.

Run delta syncs#

After the initial copy, schedule incremental syncs to catch objects added or modified since the last run:

rclone sync aws:my-source-bucket quakeai:my-bucket \
  --transfers 16 \
  --checkers 32 \
  --progress

rclone sync compares file sizes and modification times, transferring only changed objects. Run this daily (or more frequently) until you are ready to cut over.

Verify the migration#

Compare object counts and sizes#

rclone size aws:my-source-bucket
rclone size quakeai:my-bucket

Both outputs should show the same object count and total size (minor byte differences from metadata are normal).

Check file integrity#

rclone check aws:my-source-bucket quakeai:my-bucket \
  --one-way \
  --log-file check.log

--one-way checks that every object on the source exists on the destination with matching size. Review check.log for any mismatches.

Spot-check a sample#

Download a few objects from both sides and compare checksums:

rclone cat aws:my-source-bucket/path/to/file.dat | md5sum
rclone cat quakeai:my-bucket/path/to/file.dat | md5sum

Cut over your application#

Once verification passes:

1. Run a final rclone sync to capture any last-minute changes.
2. Update your application's S3 endpoint configuration. See update application endpoint for SDK-specific instructions.
3. Confirm the application reads and writes successfully against Quake AI.
4. Keep the AWS source bucket for 30 days as a rollback target. Do not delete it immediately.

Alternative: MinIO Client (metadata-preserving)#

If your objects have custom x-amz-meta-* headers that must be preserved, use MinIO Client instead of rclone:

mc alias set aws https://s3.amazonaws.com \
  YOUR_AWS_ACCESS_KEY YOUR_AWS_SECRET_KEY

mc alias set rumble https://object.us-east-2.rumble.cloud \
  YOUR_RUMBLE_ACCESS_KEY YOUR_RUMBLE_SECRET_KEY

mc mirror aws/my-source-bucket rumble/my-bucket

mc mirror copies objects including custom metadata. Verify with:

mc stat rumble/my-bucket/path/to/file.dat

Check that X-Amz-Meta-* headers appear in the output.

Alternative: AWS CLI (local staging)#

The AWS CLI cannot copy directly between two different S3-compatible endpoints. Use this approach only if you are already invested in AWS CLI workflows and the data set is small enough to stage locally:

aws s3 sync s3://my-source-bucket ./local-staging/ \
  --profile aws-source

aws s3 sync ./local-staging/ s3://my-bucket \
  --profile rumble \
  --endpoint-url https://object.us-east-2.rumble.cloud

What does not transfer#

The following AWS S3 features have no equivalent on Quake AI and must be handled separately:

See also#

• Plan your migration: compatibility matrix, cost estimation, validation checklist
• Migration tools comparison: rclone vs AWS CLI vs s3cmd vs MinIO Client
• Update application endpoint: swap your SDK config to point at Quake AI
• Set up Quake AI as a backup target: keep AWS as primary, back up to Quake AI