Running local AI models
without cloud
dependency
Ollama and local LLMs as upstream for the OpenClaw Gateway: VRAM, quantization, inference latency, and routing — with the same approvals and logging as the cloud.
direct answer
What are local models
within OpenClaw?
Short answer: LLM inference on your own hardware (typically Ollama on GPU or CPU) that uses the OpenClaw Gateway as an upstream provider — tokens are generated locally, not via the Anthropic/OpenAI cloud.
Nuance: Local means control over data egress, latency, and model version — nothing less approvals on tools. VRAM, quantization, and routing determine whether it is production-feasible.
docker · Monitoring · Hub
Consideration
When local models are useful
No “always local” dogma — operational trade-offs explicit.
Privacy & data residency
Prompts and context remain on your network — no upstream tenant with the cloud provider. Compliance ≠ automatic; logging and backups remain your responsibility.
Offline / air-gapped
Inference without internet after model pull — useful for isolated VLANs; updates and patches are self-managed.
Lower network latency
LAN call to Ollama at :11434 can be faster than round-trip cloud — If GPU/CPU can handle it; otherwise slower than API.
Sensitive data in agents
Governance & audit
You check which model, which version, which quant — audit trail on Gateway + Monitoring.
Reverse side: maintenance
Model lifecycle, GPU drivers, VRAM tuning, rollback — operationally heavier than “API key in .env”.
Counterpart: hardware
GPU costs, RAM/VRAM ceilings, cooling — no free scale like pay-per-token cloud.
infrastructure
Ollama & local LLM stack
No installation tutorial — how it hangs alongside the Gateway in production.
Ollama as inference endpoint
OpenClaw Gateway points upstream to OLLAMA_HOST (sidecar or host). No replacement of Gateway — routing and tools remain central.
Model pulling & lifecycle
Pin tags (llama3.2:3b not “latest” on prod); test new weights on staging before Gateway switch.
Quantization (Q4, Q8, …)
Trade-off: lower VRAM vs. quality/hallucinations. Overly aggressive quant → unstable behavior in workflows.
CPU vs GPU inference
CPU-only can be sufficient for small models/demos; production agents with browser tools often want a GPU or cloud fallback.
Model switching
Multiple models = multiple VRAM footprints; unload policy needed to prevent OOM.
Local routing
Gateway selects provider per agent/channel — same governance as cloud upstream.
Hardware
VRAM, RAM & performance
Why VRAM counts
Weights and KV cache for the context window are stored in GPU RAM. Model too small for VRAM → load fail; context too large → OOM mid-request.
Example: 7B Q4 on 8 GB GPU is possible, 13B+ long context often not without offload.
Context window
Longer context = more memory and latency. Browser/mail dumps in a single prompt are VRAM killers.
Example: Truncate/summarize before inference; separate small model for triage.
OOM & Exit 137
Docker Ollama sidecar without memory limit can wreck entire VPS — see docker RAM section.
Example: Docker stats + cgroup limit on ollama service.
Swap & memory pressure
CPU swap with GPU OOM → inference in seconds per token — looks like a “hanging agent”.
Example: Disable swap for inference nodes or monitor vmstat.
Inference latency
Tokens/s dependent on GPU, quant, batch size. Browser automation waits for LLM — slow local = slow workflow.
Example: Measure p95 latency before prod; set timeouts on tool calls.
Comparison
Cloud vs local models
Neutral — hybrid stacks are normal in SME automation.
Privacy / egress
Local: Data remains LAN/VPC
Cloud: Upstream tenant & DPA
Latency (typical)
Local: Low on LAN if GPU OK
Cloud: Internet + queue
Costs (SME)
Local: CapEx GPU + power + management
Cloud: By usage / subscription
governance
Local: You manage the model + logs
Cloud: Provider policies + your Gateway
Monitoring
Local: VRAM, GPU temp, Ollama health
Cloud: API errors, rate limits
compliance
Local: Residency possible
Cloud: Certificate provider
Reliability
Local: Your hardware/SPOF
Cloud: Provider SLA
Scalability
Local: VRAM/GPU bound
Cloud: Elastic (with costs)
routing
Multi-model
routing
Short answer: Not one model for everything — Gateway policies per agent/channel with fallback and latency awareness.
- → Small vs large model: Triage with 3B/7B on-premises; strong reasoning towards a larger model or cloud — separation in routing policy.
- → Browser agents + local: DOM steps wait for LLM — slow local inference = slow automation; consider cloud for interactive paths.
- → Fallback to cloud: In case of OOM, timeout or overload: controlled fallback to Anthropic/OpenAI compatible — no silent dual-write.
- → Workload splitting: Read-only local summarization; mutating tool planning with cloud — see Agent Tools.
- → Routing policies: Per channel/agent in Gateway config; document which model and why — audit upon incident.
Document: for OOM/timeout → which cloud model, max tokens, which data may egress. No silent fallback without RBAC-review.
Failures
When local models go wrong
GPU memory exhaustion
Two models loaded or context spike → CUDA OOM; Gateway sees upstream 500/timeout.
Slow inference
CPU-only or thermal throttle — approval queue looks like “broken AI”.
Instability & corruption
Incomplete pull, disk full on model store — restart does not resolve without integrity check.
Bad quantization
Low quality → more hallucinations in tool plans — not “model is stupid”, but a wrong trade-off.
Browser agent delay
Every DOM decision waits for tokens — timeouts on the browser tool chain.
Upstream errors: problems · check OLLAMA_HOST and VRAM before Gateway reinstall.
Production
Local inference
checklist
Same governance as cloud: Agent Tools and approvals remain active.
Short answer: LLM inference on your hardware (Ollama, llama.cpp, GPU stack) as upstream for the OpenClaw Gateway — no cloud API for every token.
Nuance: Local = you manage weights, VRAM, and updates; Gateway remains the orchestrator for agents and tools.
Follow-up: docker sidecar setup
Short answer: Yes with pinning, monitoring, memory limits, staging tests, and fallback — no as “latest pull on a small VPS without observability”.
Cause: OOM and silent slowdown are production incidents.
Follow-up: logs & monitoring
Short answer: Not strictly — small models can run on the CPU for triage; interactive agents + browser workflows usually require the GPU or cloud.
Cause: tokens/s on CPU vs CUDA.
Advice: measure latency before prod-commit.
Short answer: GPU memory for model weights and inference cache — bottleneck alongside RAM for Ollama process.
Nuance: context window growth consumes VRAM; quantization reduces footprint with quality trade-off.
Follow-up: VRAM section on this page.
Short answer: Hardware limit (GPU/CPU), quant, context length, concurrent agents — no “local = always faster”.
Cause: swap, thermal throttle, two models loaded simultaneously.
Fix: routing, smaller model, cloud fallback for hot path.
Short answer: In case of VRAM shortage, high quality requirements, elastic peak load, or if GPU management does not suit the team.
Nuance: Hybrid is common: local for privacy triage, cloud for heavy reasoning.
Follow-up: multi-model routing above.
Self-hosted AI
Local AI infrastructure
& architecture
We help with Ollama sidecars, VRAM sizing, model routing, fallback policies, and Gateway/Docker hardening — deployment guidance for teams that consciously combine cloud and on-premises.