Capacity

Sizing questions usually arrive as “how many users per machine”. The honest answer depends on one number you control and one you do not.

What a turn costs

A turn is a question and everything the agent does to answer it — searches, document reads, model calls. Measured on a mid-size agent with memory search enabled:

wall time	~75–150 s for a typical question
memory	~450 MB resident per concurrent turn
tokens	100 K–200 K input for a well-scoped mission

The spread matters more than the average. A well-scoped mission over a curated corpus answered in 11 turns and 118 K tokens. The same engine against a 512 K-document corpus, on a question whose vocabulary did not match the documents, took 39 turns and 1.8 M tokens — fifteen times the cost for a worse answer.

Scope is the lever. A fence that narrows what a mission may search does more for capacity than a bigger machine.

Concurrency

Little’s Law: concurrent turns = arrival rate × turn duration. Eight users asking one question a minute each, at 120 s a turn, is 16 concurrent turns — not 8.

At ~450 MB per concurrent turn that is ~7 GB of turn memory, before the agent, the router, and the operating system.

What the model does not grow with

Two things that are commonly assumed and are not true:

• Memory does not grow with conversation length. Measured across a 30-turn session: flat.
• Tokens do not grow unboundedly on a session with memory enabled — a measured 30-turn run fell to 0.55× the opening turn as summarisation engaged.

The cost driver is the shape of the work, not the length of the conversation.

The embedding model is shared

One ONNX model is loaded per process however many memories it opens — about 180 kB private per additional engine. Do not budget a copy per agent.

Sizing a machine

Start from concurrent turns, not users:

RAM  ≈  (concurrent turns × 0.45 GB)  +  2 GB agent/router  +  OS

RAM is the binding constraint, not CPU — a turn spends most of its life waiting on a model call, so vCPU rarely decides the ceiling. As a rough floor, allow 1 vCPU per 2 concurrent turns.

On a 16 vCPU / 32 GB machine, RAM allows roughly (32 - 2 - 4) / 0.45 ≈ 57 turns in theory; in practice budget 20–25, because turn memory spikes during document reads and you do not want to be near the limit when it does. At 120 s a turn that is roughly 600 questions an hour, which serves several hundred people asking a handful of questions a day.

Model throughput is usually the real ceiling. Rate limits and per-model concurrency at the router bite long before RAM does.

Ingest is a different workload

Building a corpus is not serving one, and sizing for the first does not size the second.

A merge of 8 shards and 24 M chunks needed more than 58 GB and was killed by the kernel twice on a 61 GB machine — after concatenation had already succeeded. A group over the same shards took 5 MB and about ten minutes.

Prefer the group unless you can prove you need a merge.