AI inference and RAG pipelines

Run the durable control plane for an inference or RAG workload on Quake AI: the serving API, the embedding and job workers, the data layer, the model router, and the credentials all live on Compute you operate, and the heavy model inference is a call out to a model backend you choose. Quake AI provides CPU-only Compute, Block Storage, Object Storage, and private networking. Quake AI flavors are AMD EPYC with no GPU option, which fits smaller models, CPU inference, and calling an external model API; see the compute FAQ for flavor details.

What this is for#

Teams building retrieval-augmented generation or self-hosted inference need a serving API, an embedding pipeline, a vector store, and the credentials that tie them to a model, all under their control. Quake AI provides CPU compute for serving and embedding, Block Storage for the database, and object storage for model artifacts and corpora; you operate the model runtime, the index, and the routing to whatever model backend serves the heavy inference. The outcome is a self-managed RAG and inference stack for smaller models, deployable from a validated OpenTofu template and its companion tutorial.

The control plane runs on Quake AI#

An inference or RAG workload is more than the model call. The parts that run continuously and hold state fit a flat-priced, always-on CPU tier, and they stay the same whichever model backend you point them at:

• Serving and agent runtime. The long-running API or agent loop that accepts a request, retrieves context, and issues the model call.
• Data gravity. Postgres with pgvector, Qdrant, Object Storage for model artifacts and corpora, and Redis for queues. The data a job reads and writes lives here.
• Job dispatcher. A queue and workers that submit work to a model backend, poll, retry, and land results in Object Storage.
• Model router. One stable endpoint that fans out to several model backends with auth, caching, fallback, and logging. The inference gateway template builds this tier.
• Identity and secrets. Provider credentials and the spend policy stay on the instance you operate, inside your private network and security groups.
• Observability. The monitoring stack watches latency, throughput, and cost from one place.

The heavy model inference is the one piece you place anywhere: serve a small model on the CPU instances above, or call an external model API (OpenAI, Anthropic, Google, or an open-weight host) from the same control plane. Re-pointing the model backend does not change the control plane around it.

OpenClaw, a one-click Console App, is this shape running today: a persistent AI agent on a Quake AI CPU VM that connects to any model provider, or to a local model through Ollama, with your API keys and conversation data staying on the VM you own.

Reference architecture#

Download diagram: SVG, PNG, and PDF.

The pipeline is the full-stack app template plus a vector variation. Each tier maps to the diagram and to the template that builds it.

1. API clients. Applications send queries to the inference API over HTTPS.

2. Inference API. A serving process runs on Compute (CPU), accepts queries, retrieves context, and calls the model. The full-stack app template covers the API and database tier.

3. Embedding workers. Compute workers turn source documents into embeddings, write them to the vector index, and read corpora and model artifacts from storage.

4. Object Storage. Model weights, tokenizer files, and document corpora live in S3-compatible Object Storage, the bucket the template provisions.

5. Vector variation. Embeddings are stored and searched in Postgres with pgvector on Block Storage. The vector variation adds the pgvector index to the database tier; the self-managed PostgreSQL template covers the same tier as a standalone stack.

Worked example: AI-assist for creator pipelines#

This worked example applies the same CPU-only stack to a creator media pipeline. It is a different shape of the same story, not a stronger CPU case: every component runs on the AMD EPYC compute described above, with no GPU. The components are whisper.cpp for transcription on a CPU worker, Piper or Coqui TTS for voice synthesis on a CPU worker, a small open-weight LLM served through Ollama (Llama 3.1 8B, Mistral 7B, or Qwen 2.5 7B) for caption and metadata generation, and Qdrant for the asset-search vector store. The self-hosted AI explainer documents all of these.

Three creator workflows run on this stack:

1. Transcription. When an asset lands in Object Storage, a whisper.cpp worker transcribes a podcast or video back catalog into SRT or VTT captions.

2. Caption and metadata generation. A small LLM generates a title, description, chapter markers, and social-post drafts from the transcript. The output writes back to the asset's metadata or to a webhook target.

3. Asset search. Embedding workers index transcripts and visual descriptions in Qdrant. The creator's CMS queries Qdrant to find every clip that covers a given topic.

The CPU ceiling from the rest of this page applies without softening. This stack transcribes, generates captions, and indexes assets; it does not deliver GPU-quality voice cloning, image generation, or video generation. For the broader creator back end, see Creator businesses; for the audio pipeline this example often extends, see Podcast and audio publishing.

Services involved#

Get started#

• Full-stack app template and its deploy tutorial: single-stack OpenTofu pattern with app server, database, and networking for an inference API and retrieval layer.
• Simple VM template and its deploy tutorial: single Compute instance to host a model or retrieval service.
• Self-managed PostgreSQL template and its deploy tutorial: Postgres on Block Storage for metadata, session state, or vector-indexed Postgres indexes you operate.
• Inference gateway template and its deploy walkthrough: one stable endpoint that routes to several model backends with auth, caching, and fallback.
• Upload objects to Object Storage: store model artifacts, document corpora, and embedding exports in the object bucket your template provisions.
• OpenTofu template library: browse validated IaC starting points for inference and RAG layouts.

Estimate the cost#

Migrating an existing AI inference / RAG service?#

Move an inference API or RAG stack that already runs on AWS EC2 and S3, Azure VMs and Blob, Google Compute Engine and Cloud Storage, or another hyperscaler. The outcome is the same serving and retrieval pipeline on Quake AI CPU compute with model artifacts, corpora, and vector indexes cut over in a controlled order.

Follow this cutover path. Each step links an existing migration page; this section composes those pages into a workload-shaped sequence rather than duplicating their steps.

1. Map your source provider. Start with the concept-translation page for your current cloud: Coming from AWS, Coming from Azure, Coming from GCP, Coming from DigitalOcean, or Coming from Hetzner.

2. Stand up the target shape on Quake AI. Pick the greenfield template that matches your serving layout: Full-stack app for an API plus database tier, or Self-managed PostgreSQL when vector-indexed Postgres is the primary data store.

3. Move compute workloads. Rebuild or migrate inference and embedding workers with Migrate from EC2 (or the matching compute migration page for your source provider).

4. Move object and artifact data. Sync model weights, document corpora, and embedding exports with Migrate from S3 (or the matching object migration page for your source provider).

5. Cut over the inference endpoint. Stage the new API behind an inference gateway, API gateway, or private hostname. Run shadow traffic or canary requests against the Quake AI stack, then switch clients to the new base URL.

Workload-specific cutover callouts#

• Model artifacts and corpora. Copy checkpoints, tokenizer files, and source documents before you switch production traffic. Validate checksums on large weight files; confirm the runtime loads the artifact from the Quake AI path you provisioned.
• Vector store and index rebuild. Decide whether to transfer a vector-indexed Postgres or external index snapshot or rebuild embeddings from the migrated corpus. Rebuilds take longer but avoid version skew between embedding models and stored vectors.
• CPU-only workloads. CPU inference fits smaller models and batch embedding jobs on Quake AI (compute FAQ). GPU-trained models that require GPU inference at production scale need accelerators outside this flavor catalog.
• Endpoint cutover and rollback. Keep the source inference endpoint warm until latency and error rates stabilize on Quake AI. Roll back by switching clients back to the old base URL if quality checks or health probes fail.

Considerations and limits#

• You operate serving and the index. Quake AI provides compute and storage; the model runtime, embedding pipeline, and vector index are yours under the shared responsibility model.
• CPU-only compute. Compute is AMD EPYC with no GPU option (compute FAQ). CPU inference fits smaller models and batch embedding; GPU-required models at production scale need a different hosting path.
• No managed database or vector service. Postgres with pgvector or another vector store runs on Compute and Block Storage you operate.
• Flat egress. Quake AI applies a no-egress-fee policy for outbound transfer.
• Three US regions. All current regions are in the United States.
• Compliance posture. Quake AI holds SOC 2 Type I and Type II attestations and SOC 3. See Compliance and certifications for the platform scope.