Object storage

An object in object storage is a blob of bytes plus metadata plus a unique identifier, stored in a flat namespace inside a bucket. Applications read and write objects over HTTP APIs.

Object storage scales to large capacity without the operational burden of traditional file servers. Common uses include backups, media assets, data lakes, static website content, and durable remote-access storage.

Quake AI object storage runs on OpenStack Swift with a Ceph backend and exposes an S3-compatible API. Each subscription includes up to 1 TB of storage, with additional capacity available on request.

How object storage works#

Objects are made up of three parts:

• Data: the file content itself (an image, a log file, a database dump, anything)
• Metadata: key-value pairs describing the object (content type, creation date, custom attributes)
• Unique identifier: a URL that gives you direct access to the object

Objects live inside buckets (also called containers). Each bucket holds many objects, and you can create folders within buckets to organize objects logically. Folder paths are a naming convention on object keys, not a directory tree on disk.

Objects are immutable; you do not edit an object in place. Instead, you overwrite it with a new version or use versioning to retain previous versions of an object for audit trails or rollback.

The storage layer replicates data across multiple nodes automatically. If a disk or server fails, replicas on other nodes keep the object available. You do not manage RAID arrays or backup jobs for platform-level durability.

When to use object storage#

Object storage and block storage serve different purposes. Choosing the right one depends on how your application accesses data.

Use object storage when your application treats files as whole units: upload once, read many times. Use block storage when your application needs a filesystem with random read/write access, like a database or operating system disk.

S3 API compatibility#

Quake AI object storage speaks the S3 protocol. You generate S3 credentials through the Console and then use S3-compatible clients (the AWS CLI, s3cmd, Boto3, or application SDKs) to interact with your buckets and objects. Existing tools and libraries built for S3 work with Quake AI without code changes.

The S3 API covers bucket creation, object upload and download, access control policies, versioning, multipart uploads for large files, and presigned URLs for temporary access.

Access control#

You can control who sees your data at two levels:

• Bucket-level: set a bucket to private (requires authentication for access) or public (objects are readable by anyone with the URL). Manage this through the Console or via S3 bucket policies.
• Object-level: S3 ACLs and bucket policies let you grant fine-grained permissions to specific users or credential sets. Use this when different teams or applications need different access levels within the same bucket.

Versioning#

Versioning retains prior versions of objects in a bucket. When you overwrite or delete an object, the previous version is retained. You can list, retrieve, or restore any historical version. Enable versioning through the S3 API when you need audit trails, accidental deletion protection, or the ability to roll back content changes.

Object Storage on Quake AI#

The object storage service uses OpenStack Swift with a Ceph storage backend, exposed through both the native Swift API and the S3-compatible API. Data is replicated across multiple storage nodes for durability.

The platform imposes default limits on the number of objects per bucket, but you can request increases through support if your use case requires it. Large objects are handled through multipart upload, which splits files into segments stored as separate objects and reassembled on retrieval.

Further reading#

On this platform:

• Create a bucket: set up your first storage bucket via the Console
• How to host a static site on object storage: publish HTML and assets with optional CDN fronting
• How to put a CDN in front of a Quake AI workload: partner CDN patterns for buckets and VMs
• Create S3 credentials: generate credentials for S3-compatible tools
• Upload a file: upload objects through the Console
• Grant access control: configure bucket permissions
• Enable versioning: turn on object version history
• Block storage concepts: when block storage is the better fit

External resources:

• OpenStack Swift documentation: upstream architecture and administration reference
• Amazon S3 API reference: the S3 protocol specification that Quake AI implements
• Ceph documentation: the storage backend that powers Quake AI object storage