How AI agent workloads
scales
in production

Workers, queues, concurrency, and browser pools on the OpenClaw Gateway — with real bottlenecks, retry limits, and observability. No “infinite scaling,” but measurable caps.

✓ Queue metrics
✓ Retry caps
✓ Browser pools
✓ Workload isolation

direct answer

What are workers and queues
within OpenClaw?

Short answer: The Gateway places work (tool calls, workflow steps, browser jobs) in queues; workers are executive processes that claim jobs within concurrency limits — not automatically “more AI agents”.

Nuance: separate Browser workers, Orchestration between coordinator and workers, retries with caps, and hard scaling limits due to API rate limits and RAM.

Monitoring · docker · workflows

queue
buffer
Pool
workers
Cap
Retries
Watch
depth

Bottlenecks

Why scaling becomes complex

AI agent infrastructure does not scale linearly — browser and upstream quotas are usually the first wall.

Browser automation

Chromium is heavy — pool size ≠ linearly more throughput; CAPTCHA and sessions limit before the CPU.

Retries

Exponential backoff without a cap → retry storms that fill queues and hit upstream rate limits.

Queue pressure

Approval-pending jobs count — depth increases while workers appear idle.

API limits

LLM and SaaS quotas — more workers multiply 429s, not success rate.

Orchestration

Coordinator/worker splits in Multi-agent — state sync becomes bottleneck before raw worker count.

Concurrent workflows

Parallel runs on shared CRM/browser state → race conditions and duplicate actions.

State synchronization

Distributed tasks without task locking → duplicate email send or duplicate ticket update.

Architecture

Workers, queues & concurrency

Worker pools

Separate pools for Gateway jobs, browser steps, and inference — not one global “agent count”.

Queue systems

FIFO with priority for escalations; dead-letter queue for jobs after max retries.

Retries

Max attempts + jitter; idempotency keys on mutating steps in workflows.

Distributed tasks

Job ID + correlation ID across workers; no implicit shared memory between containers.

Browser workers

Dedicated Chromium per pool slot — see Browser automation.

Job isolation

Docker service per worker type or cgroup limits — workload isolation on host, no magic autoscale.

Task locking

Lock on resource (ticket-id, thread-id) during mutating tool — prevents parallel duplicate actions.

Failures

When workers go wrong

Retry

Retry storms

Scenario: 429 from upstream → 10 workers retry at the same time without a backoff cap.

Impact: Queue congestion; provider ban; cost spike.

Duplicate

Duplicate actions

Scenario: Two workers take the same job after a visibility timeout.

Impact: Duplicate CRM update or duplicate email — audit trail unclear.

queue

Queue congestion

Scenario: Browser pool full; inference queue empty — operators see “everything slow”.

Impact: SLA miss; approvals piling up.

Worker

Stuck workers

Scenario: Chromium hangs after DOM timeout; worker claims lock without release.

Impact: Pool shrink to restart; zombie sessions.

Browser

Zombie browser sessions

Scenario: Crash without profile cleanup — next job inherits cookies/state.

Impact: Incorrect tenant data in workflow.

Race

Race conditions

Scenario: Parallel workflow steps on the same record without lock.

Impact: Inconsistent state; rollback required.

Browser pressure

Rate limits & browser automation

Browser pools

Fixed pool size + queue — not unlimited tabs; RAM and file descriptors are hard limits.

CAPTCHA pressure

More parallel logins → more challenges; scaling often worsens here first.

Throttling

Per-domain rate limits in workflow config; global cap on browser tool calls/minute.

Shared browser sessions

One profile for multiple workers = implicit concurrency without locking — avoid.

Proxy rotation

Only where necessary; rotation without observability masks failures as a “flaky site”.

observability

Monitoring &
queue visibility

Short answer: Scaling without metrics is a gamble — measure queue depth, retry rate, and browser pool utilization before you double your workers.

  • → Queue metrics: Depth, age p95, dead-letter count — alert to sustained growth, not just an empty queue.
  • → Retries: Retry rate per workflow type; plot against upstream 429/5xx.
  • → Worker health: Heartbeat, last job duration, pool utilization — see logs & monitoring.
  • → Browser worker monitoring: Active sessions, OOM kills, average step latency.
  • → Orchestration visibility: Coordinator queue vs. worker queues — where is the backlog really?
📊 Alert examples

queue depth > p95 15 min · retry rate > 10% · browser pool 100% > 5 min · dead-letter > 0 sustained.

Production

Scaling & workload
checklist

Combine with approvals on mutating distributed steps and Local Models when inference is the bottleneck.

📋 Production caps

Pragmatism

When scaling becomes overkill

In short: More workers ≠ faster business — sometimes you lower concurrency and fix retries.

Small workflows

One mail triage per hour — extra workers only add operational overhead.

Limited teams

No 24/7 on-call → complex orchestration is harder to debug than waiting.

Debugging complexity

Missing distributed traces → more workers = more unreproducible failures.

Unnecessary orchestration

Coordinator/worker split without parallel load — start with single Gateway + queue.

FAQ

Workers & queues —
direct answer

Operational — measure first, then scale with hard caps.

Scaling question →

Short answer: Processes or containers that execute jobs from a queue — Gateway dispatch, browser steps, or inference workers — are no longer “agents” as a marketing term.

Nuance: Pools are per workload type; concurrency is a policy, not a slider to infinity.

Follow-up: multi-agent orchestration

Short answer: They buffer peaks, separate approval waiting, and prevent all tools from going upstream simultaneously.

Cause of chaos: fire-and-forget without queue at browser/LLM limits.

Follow-up: Monitoring on queue depth

Short answer: Too many parallel retries after 429/timeout without max attempts, jitter or circuit breaker.

Symptom: Queue depth explodes, success rate drops.

Fix: cap retries, backoff, temporarily lower worker concurrency.

Short answer: Fixed browser worker pool + queue; scale horizontally to RAM/CAPTCHA limit — not to “one agent per tab”.

Nuance: session isolation per job; no shared admin profile.

Follow-up: Browser automation

Short answer: Double job delivery, missing idempotency, or race without task locking on the same resource.

Cause: Visibility timeout too short with slow browser steps.

Advice: idempotency keys + locks on record id.

Short answer: Workers claiming a queue slot but not completing — often hanging Chromium or blocking I/O without a timeout.

Fix: worker timeout, health check, pool restart with audit of ongoing jobs.

Follow-up: problems

Short answer: As a coordinator, multiple worker types and state sync take more time than the saved latency — typically at low volumes or with a small team.

Advice: One queue + limited pool first; split only at measured bottleneck.

Follow-up: workflows · Orchestration

Infrastructure hardening

Scaling audits &
orchestration governance

We review queue depth, retry policy, browser pool sizing, worker isolation, and deployment architecture on your OpenClaw stack — with caps and observability, not with a Kubernetes slide.

✓ Queue & retry metrics
✓ Browser pool caps
✓ docker Limits
✓ Workflow governance