Scale: what an agent costs

The question

SAM’s design makes a claim that sounds expensive. Every agent is a principal, and every agent reaches the network through its own boundary that resolves names and enforces policy on each flow. Nothing an agent does escapes that boundary, because in a properly built sandbox there is no other route out.

Architecture documents can argue for that. They cannot tell you whether it is affordable. If a boundary costs a hundred megabytes and adds ten milliseconds, the design is a nice idea for a dozen agents and useless for a thousand. So this page is not an argument. It is a measurement, with the method written down so you can disagree with it, and a script so you can re-run it.

Four questions, each phrased so that a bad answer would be visible:

1. Density. What does the Nth boundary cost in memory?
2. Startup. How long from launching a sandbox to it being able to call the mesh, and does that grow with the population?
3. Overhead. What does an agent pay in latency for reaching the mesh through a policy instead of a socket?
4. Enforcement. Does policy still hold under load, and what does refusing cost?

Method

Apparatus

One real mesh: a control plane, a router and a node, all running as they do in the end-to-end suite, on a kind cluster. Not a simulation and not a mock — the node enrols, mints real tokens and answers real requests.

Sandbox boundaries are then attached to that node one at a time. They run as host processes rather than containers, deliberately. The question is what sam-box costs; wrapping each one in a container would measure the container runtime instead.

The workload

The load generator (sam-bench) issues exactly the same request every time. This repo also ships a chaos agent driven by a real language model, and that is the right tool for asking whether the mesh survives an autonomous caller — but it is the wrong tool for asking what anything costs, because it never issues the same request twice, so a difference between two runs cannot be attributed to the mesh.

The generator reaches the mesh the way an agent does: over the boundary’s SOCKS5 socket. What it measures is therefore what an agent experiences, not what an operator with host access would.

Conditions

One number would have been misleading, so every step of the sweep is measured four ways:

The baseline reaches the node over its own Unix socket rather than through a relay, so the comparison is not charged for a hop the mesh path does not make.

What is recorded

Client-side latency says what an agent experienced. The servers’ own counters say what they did to produce it, including the flows they refused, which leave no latency behind. Both are recorded, along with the machine, the commit and whether the tree was dirty, because a latency with no machine attached to it is not a result.

Percentiles are computed from the full sorted sample by nearest rank, so every figure quoted below was actually observed rather than interpolated between two figures that were. Failed requests never enter the latency distribution: a mesh that refuses quickly must not be able to look like a mesh that answers quickly.

Warmup requests are issued and reported separately rather than discarded, since a first call that pays for provider discovery is a real cost and hiding it would flatter the result.

Environment

Results

Recorded 2026-08-21 at commit e5b2966. Tables are generated by sam-bench report from the raw observations, which are kept alongside them.

The measurements

Latency in milliseconds, memory in MB, per population size.

Every allowed step served 500 of 500 requests. Every denied step served 0 of 500. “Admission” is the boundary’s own mean time to classify a destination and open it; “leaked” is the number of requests that reached a destination policy forbade.

Density

A boundary settles at 17–20 MB of resident memory once idle. Sixty-four of them, attached to one node with one under continuous load, came to 1.37 GiB in total.

The per-sandbox mean falls as the population grows, which is an artefact worth naming rather than a discovery: the same boundary carries the load at every step, so its heap reflects the traffic while the others sit idle, and the mean is diluted towards the idle cost as the population grows. The floor, not the mean, is the marginal cost of one more agent.

Startup

Across 64 sandboxes: median 59 ms, min 58 ms, p95 59 ms, max 70 ms.

Readiness here means the socket is accepting connections, not that the process exists — a boundary that has not bound its socket yet is of no use to the agent waiting on it.

The spread is the notable feature: the 64th boundary started as quickly as the first, so attaching an agent is a constant-cost operation over this range rather than one that degrades as the host fills up.

Overhead

This is the result that came out differently from what the method expected, so it is worth being precise about.

Baseline, no boundary in the path: p50 1.39 ms. Through the boundary on a reused flow, p50 ranged from 1.30 to 1.39 ms across the sweep — that is, at or below the baseline at every population size. Taken literally the boundary made things faster, which it obviously did not. The honest reading is that the end-to-end overhead of the boundary is smaller than the run-to-run variation of the measurement itself, so this method cannot resolve it. A figure like “the boundary costs 60 µs” would be reading noise.

Opening a fresh flow for every request, which pays the SOCKS5 handshake and the policy decision each time, is resolvable but small: p50 ran +9 to +157 µs against the baseline.

The reliable number comes from the boundary’s own instrumentation, which is measured in-process and excludes everything downstream. Classifying a destination and opening it took a mean of 18–20 µs, and that figure moved by less than 2 µs across the entire sweep, from one sandbox to sixty-four.

So: enforcement costs tens of microseconds, and whatever the boundary adds end to end is lost in the noise of a millisecond-scale request.

Enforcement

Every request to a destination policy did not allow was refused. 3,500 attempts, zero successes, at every population size, and the boundary’s own counter agreed with the client’s on the exact number refused at every step.

Refusal ran at 24,000–28,000 per second, roughly ten times the rate of allowed traffic, which is what should happen: a denied flow is decided from the name and never dialled.

Conclusions

The per-agent boundary cost is small relative to the agent it contains. At ~18 MB and ~59 ms to attach, the constraint on how many agents a host carries is the agents themselves rather than the thing that contains them. Extrapolating the floor, the boundaries for a thousand agents would be roughly 18 GB. That extrapolation is precisely what the next experiment should test rather than assume.

Enforcement is not a latency problem. 18–20 µs to classify and open a flow, and an end-to-end cost too small for this method to separate from noise. The design’s insistence that every flow be checked does not have to be traded against performance, which removes the usual reason such checks get quietly disabled.

Nothing degraded across the range tested. Median latency was flat from 1 to 64 sandboxes, startup was constant, and the boundary’s own admission cost moved by under 2 µs across a 64-fold increase in population.

10,500 requests. No failures where success was expected, and no successes where it was not.

Threats to validity

Written down because a result without them is advocacy.

• One machine, one run. These are not confidence intervals over repeated trials. Treat single figures as indicative and the shape of the trend as the claim. The overhead result above is precisely a case where a single figure would have been over-read.
• A trivial request. /v1/models is a small response from a warm node. It isolates mesh overhead well precisely because it does almost no work, which also means it says nothing about large payloads or streaming completions.
• Load through one boundary. Every step drives traffic through the first sandbox while the rest are attached but idle. This answers “do the others get in the way”, which is the density question, but it is not the same experiment as all N being busy at once.
• A reproducible tail anomaly. The single-sandbox, new-flow-per-request step showed p99 excursions of 60–80 ms in every run, and did not recur at higher populations. It is reproducible and unexplained; it is left in the record rather than smoothed out.
• Host processes, not sandboxes. This measures the boundary, not the isolation around it. A Firecracker microVM per agent costs far more than 18 MB, and none of that cost is counted here.

Reproducing this

# The boundary measurements: requires docker and kind, stands up a real mesh.
./tests/scale/run-density.sh --steps 1,2,4,8,16,32,64 --requests 500

# The thousand-agent run: needs a host with KVM, a guest kernel with
# CONFIG_TUN, and a bootstrap token for a control plane.
./scripts/provision-scale-vm.sh --no-spot --local-binaries gs://your-bucket/sam
# then, on the host:
SANDBOX_LINGER=2400 /opt/microvm/launch-microvms.sh 1000
/opt/microvm/collect-fleet.sh --node-socket /var/run/sam-node.sock --duration 1500

Results land in tests/scale/results/, including environment.json and the raw per-step observations, with a rendered table.md.

Individual measurements can be taken by hand:

# Through a boundary, as an agent would
sam-bench run --socket /run/agent.sock --target http://mesh.sam.alt/v1/models \
  --requests 500 --concurrency 4 --scrape http://127.0.0.1:9600/metrics

# The same request with no boundary in the path, for comparison
sam-bench run --target-unix /run/node.sock --target http://localhost/v1/models \
  --requests 500 --concurrency 4

sam-box serves metrics only when asked, with --metrics-addr. It sits between a sandbox and the mesh, so any listener it opens is one more thing reachable from wherever it is bound.

What this does not answer

This measures a boundary on a host. It says nothing yet about a thousand agents in real sandboxes against a public testnet, where microVM memory, control-plane enrolment and DHT behaviour all start to matter and none of them are exercised above. That experiment is below.

A thousand agents on one host

The measurements above are of a boundary, not of a sandbox: host processes, no kernel per agent, no mesh beyond the machine. The obvious objection is that none of it survives contact with the real arrangement, where every agent is a Firecracker microVM with its own kernel and the node has to be a real member of a real mesh.

So: one n2-standard-64, one sam-node enrolled in the public bananas testnet, and a thousand agents. Each agent is a microVM with no network device at all, reaching the mesh through its own sam-box over vsock.

Recorded 2026-08-21. Raw samples in tests/scale/results/fleet-1k.

What a thousand agents cost

The node figure is the one that bears on the design. One sam-node served a thousand distinct agent principals in 386 MB, because an agent is not a peer: it has no enrolment, no key and no place in the DHT. The mesh gained a thousand principals and no members.

How fast they came up

A thousand sandboxes launched in 149 seconds — 6.7 per second, median 153 ms each, p95 245 ms. The population became reachable as fast as it was started:

Agents trail microVMs by a few seconds and never by more than about fifty — that gap is a guest booting and its agent resolving a mesh name, not a queue forming. Nothing degraded as the population grew, and the last hundred came up as quickly as the first.

sam_node_agents_untracked_total was 0: every agent that ran was counted, so the thousand is a thousand and not a ceiling.

What they can push

Density is not throughput, so with the thousand still resident, traffic was driven through 200 of them at once — each through its own boundary, so the node saw 200 distinct principals rather than 200 connections from one client. 200 requests per agent, two in flight each.

What the following number is, and is not. It is the aggregate rate at which a locally-served endpoint was reached through 200 concurrent policy boundaries, with the load generated on the host. It is not an end-to-end figure for an agent workload. The request is answered by the node itself, so no upstream provider is in the path; and the generator enters at the boundary’s SOCKS5 socket rather than from inside a guest, so the in-guest network stack and the vsock hop are not either. It should be read as an upper bound on what the enforcement path permits, not as a throughput claim for the system as deployed.

Under those conditions: 40,000 requests, none failed, in 0.43 s — an aggregate of roughly 92,000 requests per second, while the other 800 sandboxes sat resident. Per agent that is a median of 534 req/s, with a minimum of 490 and a maximum of 652, so no principal was starved. Client-observed time to first byte was a median of 2.0 ms across agents, 13 ms at p95, 21 ms at p99, and 98 ms for the single worst request in the run.

The node’s own counters record 200 for all 41,000:

That last row is the interesting one, and it is not jitter. Those 400 requests took almost exactly 5.04 seconds each, which is dhtLookupTimeout to the millisecond. /v1/models resolves providers through the DHT, and the node’s own log shows that walk returning four peers — so it was not waiting on an empty mesh. It was waiting because the walk has no reason to stop early: it asks for up to twenty providers, four is not twenty, and nothing tells it that four is all there are. So it runs out the clock, then caches the answer for thirty seconds.

The cost is therefore not per agent and not per request. It is one slow lookup per cache expiry, paid by whichever caller happens to arrive on a cold cache; everything behind the cache is the sub-millisecond mass above.

It is still a fair thing to dislike. A lookup that already has its answers should not wait five seconds to admit it, and the two hundred callers who arrived together each paid it instead of sharing one lookup.

What an agent actually gets

Everything above is a cost. None of it says what the agent receives in return, and a boundary that is cheap but useless is not a result worth publishing. So, separately from the thousand-agent run: a sandbox was booted and asked what the mesh would give it.

It is a Firecracker microVM with no network device — no NIC, no bridge, no route to anywhere. Its only path out is a vsock to its own sam-box. Inside it there is no API key, no model endpoint, no tool server address and no configuration naming any of them. The agent asks for mesh.sam.alt, which is not in DNS and has no route to it, and the boundary decides what that means for this particular agent.

What came back:

Both models answer. google/gemma-2-2b-it is served by a vLLM instance on a TPU elsewhere in the mesh and replied mesh ok, reporting its own vllm-0.26.1rc1 fingerprint. openrouter/auto resolved outward and was routed to deepseek/deepseek-v4-flash-0731. The agent asked for neither by address; it asked for a model, and the mesh chose the provider.

The LangChain agent then ran inside that sandbox and produced model output from the TPU-hosted vLLM, having taken its model name from the catalog rather than being told one. Two ordinary SDKs, the OpenAI client and the MCP client, used exactly as they are used anywhere — nothing in the sandbox is SAM-specific, which is the point: an agent that needed a special client would be an agent nobody could port onto this.

The honest limit of this demonstration: it establishes that tools and both providers are reachable and answering from inside a network-less sandbox. It is not a claim about how well a 2B model drives a fifteen-tool loop, which is a question about the model and not about the mesh.

What this run says

An agent costs about 192 MiB and 150 ms, nearly all of it the microVM. The boundary is 25 MB of that and the node’s share is under half a megabyte. A host’s agent count is decided by how much memory the agents themselves want.

The mesh does not grow when the agent population does. A thousand agents arrived as one member with one enrolment. That is the reason for separating the principal from the peer, and it is the difference between a mesh that can hold this population and one that cannot.

It ran against a real, shared control plane — not a fixture — with one bootstrap enrolment.

The enforcement path did not limit throughput under these conditions. 200 agents issued 40,000 requests in under half a second with no failures, and the node served most of them in under half a millisecond while holding a thousand principals. Since the endpoint was local and the load host-generated, this bounds the enforcement path rather than characterising a deployed workload.

Inference was not exercised in this run. The agents’ model calls returned 404, 998 of them: they asked for a model named default, and a name no catalog contains resolves to no provider. The run demonstrates reach, admission and MCP at a thousand agents; the inference path is demonstrated separately, and the two should not be confused.

Threats to validity

• One host, one run. No repetitions, no confidence intervals.
• The agents are the example harness, which connects, discovers tools, attempts a model call and stops. A heavier agent needs more guest memory, and 167 MiB is a statement about this workload rather than about agents in general.
• Memory was measured with the population idle. The throughput numbers came from a separate pass over the same resident fleet, so the 192 MiB/agent figure and the 92k req/s figure are not from the same instant.
• The load was driven from the host, into each boundary’s SOCKS5 socket — the same entry point the guest uses, but skipping the in-guest netstack and the vsock hop. It measures the mesh datapath, not the sandbox’s own overhead, and the real per-agent ceiling will be lower.
• The load target is served by the local node. /v1/models answers from the node itself, so no upstream provider is in the path. That is deliberate — it isolates SAM’s cost from a backend’s — but it is not an end-to-end figure.
• Only 200 of the thousand pushed traffic. The rest were resident and idle. Nothing here says what all thousand at full rate would do.
• A thousand is not a limit. It is what fits comfortably in 251 GiB with this agent. Nothing here found a ceiling, which also means nothing here located one.
• Spot instances were preempted twice before this run, mid-measurement. The final run used a standard instance. Worth knowing if you reproduce it: an experiment that takes ten minutes and a machine that lasts ten minutes do not mix.