# How to Migrate from AWS S3 to Quake AI Source: https://docs.quake.ai/docs/object/migration/migrate-from-s3 Markdown: https://docs.quake.ai/docs/object/migration/migrate-from-s3.md --- # How to migrate from AWS S3 to Quake AI This guide walks through a complete migration of objects from an AWS S3 bucket to Quake AI object storage. The recommended tool is rclone, which copies data directly between providers without staging files locally. ## Service mapping ## Prerequisites - A Quake AI account with [S3 credentials](/docs/object/how-to/create-s3-credentials) - An AWS IAM user or role with `s3:GetObject`, `s3:ListBucket`, and `s3:GetBucketLocation` permissions on the source bucket - [rclone](https://rclone.org/install/) installed (v1.65+) - Optionally: [MinIO Client](https://min.io/docs/minio/linux/reference/minio-mc.html) if your objects have custom metadata that must be preserved AWS can waive data transfer out charges when you move off AWS, but the waiver is not automatic. You request it through AWS Support, AWS approves credits case by case, and once approved you have a limited window (90 days as of the [AWS announcement updated on 30.09.2025](https://aws.amazon.com/blogs/aws/free-data-transfer-out-to-internet-when-moving-out-of-aws/)) to finish the migration and delete the remaining data and workloads from your account. The waiver does not cover transfer through CloudFront, Direct Connect, Snow Family, or Global Accelerator. If you store under 100 GB, the standard AWS free tier already covers the transfer. See the [AWS free data transfer FAQ](https://aws.amazon.com/about-aws/global-infrastructure/global-network/faqs/) for current eligibility and exclusions. ## Review the planning guide Before starting, read through the [migration planning guide](/docs/object/migration/plan-your-migration) to: - Check S3 feature compatibility (lifecycle policies, bucket policies, and event notifications are not supported on Quake AI) - Inventory your data (total size, object count, custom metadata usage) - Choose a migration strategy (big-bang, phased, or continuous sync) ## Configure rclone remotes Add both your AWS source and Quake AI destination to rclone's config. You can do this interactively with `rclone config` or edit `~/.config/rclone/rclone.conf` directly: ```ini [aws] type = s3 provider = AWS access_key_id = YOUR_AWS_ACCESS_KEY secret_access_key = YOUR_AWS_SECRET_KEY region = us-east-1 [quakeai] type = s3 provider = Ceph access_key_id = YOUR_RUMBLE_ACCESS_KEY secret_access_key = YOUR_RUMBLE_SECRET_KEY endpoint = object.us-east-2.rumble.cloud acl = private ``` Replace `object.us-east-2.rumble.cloud` with your Quake AI region: `us-east-1`, `us-east-2`, or `us-west-1`. Verify both remotes: ```bash rclone lsd aws: rclone lsd quakeai: ``` ## Create the destination bucket If the target bucket does not already exist on Quake AI: ```bash rclone mkdir quakeai:my-bucket ``` Or use the Quake AI console: **Storage** > **Object Storage** > **Create Bucket**. ## Run the initial copy For data sets under 100 GB, a single sync is sufficient: ```bash rclone sync aws:my-source-bucket quakeai:my-bucket \ --transfers 16 \ --checkers 32 \ --s3-chunk-size 64M \ --progress ``` For larger data sets, use `copy` instead of `sync` for the initial transfer (safer: does not delete anything on the destination), then switch to `sync` for delta passes: ```bash rclone copy aws:my-source-bucket quakeai:my-bucket \ --transfers 16 \ --checkers 32 \ --s3-chunk-size 64M \ --progress \ --log-file migration.log \ --log-level INFO ``` For buckets with millions of small files, increase `--checkers` to 64 and consider `--fast-list` to reduce API calls (uses more memory). ### Bandwidth limiting To avoid saturating your network during business hours: ```bash rclone copy aws:my-source-bucket quakeai:my-bucket \ --bwlimit "08:00,10M 18:00,off" \ --transfers 16 \ --progress ``` This limits bandwidth to 10 MB/s between 08:00 and 18:00, and removes the limit outside those hours. ## Run delta syncs After the initial copy, schedule incremental syncs to catch objects added or modified since the last run: ```bash rclone sync aws:my-source-bucket quakeai:my-bucket \ --transfers 16 \ --checkers 32 \ --progress ``` `rclone sync` compares file sizes and modification times, transferring only changed objects. Run this daily (or more frequently) until you are ready to cut over. ## Verify the migration ### Compare object counts and sizes ```bash rclone size aws:my-source-bucket rclone size quakeai:my-bucket ``` Both outputs should show the same object count and total size (minor byte differences from metadata are normal). ### Check file integrity ```bash rclone check aws:my-source-bucket quakeai:my-bucket \ --one-way \ --log-file check.log ``` `--one-way` checks that every object on the source exists on the destination with matching size. Review `check.log` for any mismatches. ### Spot-check a sample Download a few objects from both sides and compare checksums: ```bash rclone cat aws:my-source-bucket/path/to/file.dat | md5sum rclone cat quakeai:my-bucket/path/to/file.dat | md5sum ``` ## Cut over your application Once verification passes: 1. Run a final `rclone sync` to capture any last-minute changes. 2. Update your application's S3 endpoint configuration. See [update application endpoint](/docs/object/migration/update-application-endpoint) for SDK-specific instructions. 3. Confirm the application reads and writes successfully against Quake AI. 4. Keep the AWS source bucket for 30 days as a rollback target. Do not delete it immediately. ## Alternative: MinIO Client (metadata-preserving) If your objects have custom `x-amz-meta-*` headers that must be preserved, use MinIO Client instead of rclone: ```bash mc alias set aws https://s3.amazonaws.com \ YOUR_AWS_ACCESS_KEY YOUR_AWS_SECRET_KEY mc alias set rumble https://object.us-east-2.rumble.cloud \ YOUR_RUMBLE_ACCESS_KEY YOUR_RUMBLE_SECRET_KEY mc mirror aws/my-source-bucket rumble/my-bucket ``` `mc mirror` copies objects including custom metadata. Verify with: ```bash mc stat rumble/my-bucket/path/to/file.dat ``` Check that `X-Amz-Meta-*` headers appear in the output. ## Alternative: AWS CLI (local staging) The AWS CLI cannot copy directly between two different S3-compatible endpoints. Use this approach only if you are already invested in AWS CLI workflows and the data set is small enough to stage locally: ```bash aws s3 sync s3://my-source-bucket ./local-staging/ \ --profile aws-source aws s3 sync ./local-staging/ s3://my-bucket \ --profile rumble \ --endpoint-url https://object.us-east-2.rumble.cloud ``` Local staging requires disk space equal to the data set size and doubles the transfer time. Use rclone for direct cloud-to-cloud transfers whenever possible. ## What does not transfer The following AWS S3 features have no equivalent on Quake AI and must be handled separately: | Feature | Action required | |---|---| | Lifecycle policies | Implement with cron + `rclone delete --min-age`. See [plan your migration](/docs/object/migration/plan-your-migration#retention-without-lifecycle-policies). | | Event notifications | Rebuild with application-level polling or webhooks. | | Bucket policies | Translate to Swift container ACLs. See [grant access control](/docs/object/how-to/grant-access-control). | | S3 Replication rules | Replace with scheduled `rclone sync`. See [multi-cloud sync](/docs/object/migration/multi-cloud-sync). | | Storage class transitions | All data is stored on a single tier on Quake AI. No action needed. | ## See also - [Plan your migration](/docs/object/migration/plan-your-migration): compatibility matrix, cost estimation, validation checklist - [Migration tools comparison](/docs/object/migration/tools-comparison): rclone vs AWS CLI vs s3cmd vs MinIO Client - [Update application endpoint](/docs/object/migration/update-application-endpoint): swap your SDK config to point at Quake AI - [Set up Quake AI as a backup target](/docs/object/migration/backup-to-quake-ai): keep AWS as primary, back up to Quake AI