Model resource requirements

Reference for selecting a Quake AI instance flavor based on the language model you intend to run. All figures assume Ollama with 4-bit quantization (the default), CPU-only inference, and no other significant processes running on the instance.

How to read this table#

• RAM needed is the minimum to load and run the model. Add 2 GB headroom for the OS and Ollama overhead.
• Recommended flavor uses {family}.{size} naming (for example m2a.large, r2a.4xlarge) and targets comfortable headroom for inference at typical conversational throughput. See the flavor reference for vCPU and RAM per size.
• Disk for weights is the storage required for the downloaded model weights. Ollama stores models in /usr/share/ollama/.ollama/models by default.
• Tokens/sec (est.) is a rough estimate on the recommended CPU flavor. GPU inference is 10–50× faster.

Small models (1–4 B parameters)#

Suitable for summarization, classification, simple Q&A, and code completion on constrained budgets.

Mid-size models (7–9 B parameters)#

The practical sweet spot for CPU inference: strong general capability with tolerable throughput.

Large models (13–14 B parameters)#

Better reasoning and instruction-following at the cost of throughput and RAM.

Large models (30–70 B parameters)#

Frontier-class open-weight models. Require substantial RAM; practical throughput is low without GPU.

Embedding models#

Embedding models are much lighter than generative models. They can run on the same Ollama instance as a generative model without significant performance impact.

Quantization variants#

Ollama pulls 4-bit quantized models by default (Q4_K_M or similar). Higher quantization means better output quality at the cost of more RAM and slower inference.

Pull a specific quantization with ollama pull MODEL:TAG.

Storage configuration#

For large model collections, attach a block volume and set the Ollama model directory:

sudo mkdir -p /mnt/models
sudo chown ollama:ollama /mnt/models

sudo tee /etc/systemd/system/ollama.service.d/storage.conf <<EOF
[Service]
Environment="OLLAMA_MODELS=/mnt/models"
EOF

sudo systemctl daemon-reload
sudo systemctl restart ollama

See How to attach a block volume for instructions on provisioning and mounting storage.

Inference hardware#

The Quake AI flavor catalog provides CPU-backed compute. Models in the 34–70 B parameter range are not practical for interactive inference on CPU; for those workloads, run inference on hardware outside Quake AI or against a managed inference provider.