Running local AI models
without cloud
dependency

Ollama and local LLMs as upstream for the OpenClaw Gateway: VRAM, quantization, inference latency, and routing — with the same approvals and logging as the cloud.

✓ Ollama upstream
✓ VRAM / quant
✓ model routing
✓ Governance remains

direct answer

What are local models
within OpenClaw?

Short answer: LLM inference on your own hardware (typically Ollama on GPU or CPU) that uses the OpenClaw Gateway as an upstream provider — tokens are generated locally, not via the Anthropic/OpenAI cloud.

Nuance: Local means control over data egress, latency, and model version — nothing less approvals on tools. VRAM, quantization, and routing determine whether it is production-feasible.

docker · Monitoring · Hub

Local
inference
VRAM
bound
Route
policy
audit
logs

Consideration

When local models are useful

No “always local” dogma — operational trade-offs explicit.

Privacy & data residency

Prompts and context remain on your network — no upstream tenant with the cloud provider. Compliance ≠ automatic; logging and backups remain your responsibility.

Offline / air-gapped

Inference without internet after model pull — useful for isolated VLANs; updates and patches are self-managed.

Lower network latency

LAN call to Ollama at :11434 can be faster than round-trip cloud — If GPU/CPU can handle it; otherwise slower than API.

Sensitive data in agents

Mail/browser context to local model — less egress; still safety and approvals on tools.

Governance & audit

You check which model, which version, which quant — audit trail on Gateway + Monitoring.

Reverse side: maintenance

Model lifecycle, GPU drivers, VRAM tuning, rollback — operationally heavier than “API key in .env”.

Counterpart: hardware

GPU costs, RAM/VRAM ceilings, cooling — no free scale like pay-per-token cloud.

infrastructure

Ollama & local LLM stack

No installation tutorial — how it hangs alongside the Gateway in production.

Ollama as inference endpoint

OpenClaw Gateway points upstream to OLLAMA_HOST (sidecar or host). No replacement of Gateway — routing and tools remain central.

Model pulling & lifecycle

Pin tags (llama3.2:3b not “latest” on prod); test new weights on staging before Gateway switch.

Quantization (Q4, Q8, …)

Trade-off: lower VRAM vs. quality/hallucinations. Overly aggressive quant → unstable behavior in workflows.

CPU vs GPU inference

CPU-only can be sufficient for small models/demos; production agents with browser tools often want a GPU or cloud fallback.

Model switching

Multiple models = multiple VRAM footprints; unload policy needed to prevent OOM.

Local routing

Gateway selects provider per agent/channel — same governance as cloud upstream.

Hardware

VRAM, RAM & performance

VRAM

Why VRAM counts

Weights and KV cache for the context window are stored in GPU RAM. Model too small for VRAM → load fail; context too large → OOM mid-request.

Example: 7B Q4 on 8 GB GPU is possible, 13B+ long context often not without offload.

CTX

Context window

Longer context = more memory and latency. Browser/mail dumps in a single prompt are VRAM killers.

Example: Truncate/summarize before inference; separate small model for triage.

UNCLE

OOM & Exit 137

Docker Ollama sidecar without memory limit can wreck entire VPS — see docker RAM section.

Example: Docker stats + cgroup limit on ollama service.

Swap

Swap & memory pressure

CPU swap with GPU OOM → inference in seconds per token — looks like a “hanging agent”.

Example: Disable swap for inference nodes or monitor vmstat.

LAT

Inference latency

Tokens/s dependent on GPU, quant, batch size. Browser automation waits for LLM — slow local = slow workflow.

Example: Measure p95 latency before prod; set timeouts on tool calls.

Comparison

Cloud vs local models

Neutral — hybrid stacks are normal in SME automation.

Privacy / egress

Local: Data remains LAN/VPC

Cloud: Upstream tenant & DPA

Latency (typical)

Local: Low on LAN if GPU OK

Cloud: Internet + queue

Costs (SME)

Local: CapEx GPU + power + management

Cloud: By usage / subscription

governance

Local: You manage the model + logs

Cloud: Provider policies + your Gateway

Monitoring

Local: VRAM, GPU temp, Ollama health

Cloud: API errors, rate limits

compliance

Local: Residency possible

Cloud: Certificate provider

Reliability

Local: Your hardware/SPOF

Cloud: Provider SLA

Scalability

Local: VRAM/GPU bound

Cloud: Elastic (with costs)

routing

Multi-model
routing

Short answer: Not one model for everything — Gateway policies per agent/channel with fallback and latency awareness.

  • → Small vs large model: Triage with 3B/7B on-premises; strong reasoning towards a larger model or cloud — separation in routing policy.
  • → Browser agents + local: DOM steps wait for LLM — slow local inference = slow automation; consider cloud for interactive paths.
  • → Fallback to cloud: In case of OOM, timeout or overload: controlled fallback to Anthropic/OpenAI compatible — no silent dual-write.
  • → Workload splitting: Read-only local summarization; mutating tool planning with cloud — see Agent Tools.
  • → Routing policies: Per channel/agent in Gateway config; document which model and why — audit upon incident.
🔀 Fallback policy

Document: for OOM/timeout → which cloud model, max tokens, which data may egress. No silent fallback without RBAC-review.

Failures

When local models go wrong

VRAM

GPU memory exhaustion

Two models loaded or context spike → CUDA OOM; Gateway sees upstream 500/timeout.

Perfect

Slow inference

CPU-only or thermal throttle — approval queue looks like “broken AI”.

Stab

Instability & corruption

Incomplete pull, disk full on model store — restart does not resolve without integrity check.

Quant

Bad quantization

Low quality → more hallucinations in tool plans — not “model is stupid”, but a wrong trade-off.

Agent

Browser agent delay

Every DOM decision waits for tokens — timeouts on the browser tool chain.

Upstream errors: problems · check OLLAMA_HOST and VRAM before Gateway reinstall.

Production

Local inference
checklist

Same governance as cloud: Agent Tools and approvals remain active.

📋 Local models production

FAQ

Local models —
direct answer

Ollama, GPU, VRAM, cloud fallback — briefly first.

Infrastructure question →

Short answer: LLM inference on your hardware (Ollama, llama.cpp, GPU stack) as upstream for the OpenClaw Gateway — no cloud API for every token.

Nuance: Local = you manage weights, VRAM, and updates; Gateway remains the orchestrator for agents and tools.

Follow-up: docker sidecar setup

Short answer: Yes with pinning, monitoring, memory limits, staging tests, and fallback — no as “latest pull on a small VPS without observability”.

Cause: OOM and silent slowdown are production incidents.

Follow-up: logs & monitoring

Short answer: Not strictly — small models can run on the CPU for triage; interactive agents + browser workflows usually require the GPU or cloud.

Cause: tokens/s on CPU vs CUDA.

Advice: measure latency before prod-commit.

Short answer: GPU memory for model weights and inference cache — bottleneck alongside RAM for Ollama process.

Nuance: context window growth consumes VRAM; quantization reduces footprint with quality trade-off.

Follow-up: VRAM section on this page.

Short answer: Hardware limit (GPU/CPU), quant, context length, concurrent agents — no “local = always faster”.

Cause: swap, thermal throttle, two models loaded simultaneously.

Fix: routing, smaller model, cloud fallback for hot path.

Short answer: In case of VRAM shortage, high quality requirements, elastic peak load, or if GPU management does not suit the team.

Nuance: Hybrid is common: local for privacy triage, cloud for heavy reasoning.

Follow-up: multi-model routing above.

Short answer: Safer for Data egress, not automatically for actions — Agents with shell/browser remain risky.

Cause: local inference ≠ local governance.

Follow-up: safety · RBAC

Self-hosted AI

Local AI infrastructure
& architecture

We help with Ollama sidecars, VRAM sizing, model routing, fallback policies, and Gateway/Docker hardening — deployment guidance for teams that consciously combine cloud and on-premises.

✓ GPU / VRAM review
✓ model routing
✓ Hybrid cloud/local