How to migrate from Google Cloud storage to Quake AI

This guide walks through migrating objects from a Google Cloud Storage (GCS) bucket to Quake AI object storage using rclone for direct cloud-to-cloud transfer.

Service mapping#

Prerequisites#

• A Quake AI account with S3 credentials
• A GCS service account with the Storage Object Viewer role on the source bucket (or Storage Admin if you plan to delete after migration)
• A service account JSON key file downloaded from the Google Cloud Console
• rclone installed (v1.65+)

Before you start#

Read the migration planning guide to review S3/Swift feature compatibility and build a validation checklist.

GCS-specific considerations:

• Storage classes: GCS objects may be stored in Nearline, Coldline, or Archive classes. These classes have minimum storage duration charges (30, 90, and 365 days respectively). If you delete objects from GCS before the minimum duration, you are charged an early deletion fee. Factor this into your migration timeline.
• IAM permissions: GCS uses IAM roles (Storage Admin, Object Viewer). These do not translate to Swift ACLs. Re-apply access controls on Quake AI after migration using grant access control.
• Object Lifecycle rules: GCS lifecycle rules (SetStorageClass transitions, Delete actions) are not supported on Quake AI. Implement retention with external tooling. See retention without lifecycle policies.

Configure rclone remotes#

Add both GCS and Quake AI to ~/.config/rclone/rclone.conf:

[gcs]
type = google cloud storage
service_account_file = /path/to/service-account-key.json
project_number = YOUR_GCP_PROJECT_NUMBER
object_acl = private
bucket_acl = private

[quakeai]
type = s3
provider = Ceph
access_key_id = YOUR_RUMBLE_ACCESS_KEY
secret_access_key = YOUR_RUMBLE_SECRET_KEY
endpoint = object.us-east-2.rumble.cloud
acl = private

Replace object.us-east-2.rumble.cloud with your Quake AI region endpoint.

Verify both remotes:

rclone lsd gcs:
rclone lsd quakeai:

Create the destination bucket#

rclone mkdir quakeai:my-bucket

Run the migration#

Initial copy#

rclone copy gcs:my-source-bucket quakeai:my-bucket \
  --transfers 16 \
  --checkers 32 \
  --s3-chunk-size 64M \
  --progress \
  --log-file migration.log \
  --log-level INFO

Delta syncs#

Run incremental syncs to catch changes after the initial copy:

rclone sync gcs:my-source-bucket quakeai:my-bucket \
  --transfers 16 \
  --checkers 32 \
  --progress

Schedule this daily until you are ready to cut over.

Estimate egress cost#

GCS charges per-GB for egress, with rates that vary by region and destination. Google runs a free egress program for customers who close their Google Cloud account and complete the move within 60 days. A migration that keeps your GCP account active pays standard egress rates, so the egress meter dominates the bill for large data sets. Egress and exit terms change, so confirm the current policy on the Google Cloud Storage pricing page before you plan your timeline.

For large data sets, consider a phased migration to spread the cost across billing periods.

Verify the migration#

rclone size gcs:my-source-bucket
rclone size quakeai:my-bucket

Check file integrity on a sample:

rclone check gcs:my-source-bucket quakeai:my-bucket \
  --one-way \
  --log-file check.log

Review check.log for any mismatches. See the validation checklist for the full procedure.

Cut over#

1. Run a final rclone sync to capture last-minute changes.
2. Update your application's storage endpoint. See update application endpoint.
3. Keep the GCS source bucket for 30 days as a rollback target.
4. Account for minimum storage duration charges before deleting Nearline/Coldline/Archive objects.

See also#

• Plan your migration: compatibility matrix, cost estimation, validation checklist
• Migration tools comparison: rclone configuration and alternatives
• Update application endpoint: swap SDK config to Quake AI