Batching without missing a deadline

A model answers one request at a time only if you let it. Loaded on an accelerator, the same forward pass that answers one caller can answer several, at close to the cost of one. That is batching, and it is the reason a facility can share a pool rather than buy compute for every machine on the floor.

The difficulty is that batching means waiting. To fill a group you hold the first request until others arrive. On a floor where some callers are robots mid-motion, waiting is not free. Substrate's rule is simple to state: group as much as possible, and never past the tightest budget in the group.

The group and its bound

When a request arrives for a model, the serving layer looks at the group currently forming for that model on the chosen node. Each request in the group carries its own latency budget, and the group inherits the shortest one. That shortest budget, less the time the pass itself will take, is the latest moment the group can be released.

New requests can join until that moment. If a request arrives whose budget is shorter than the group's current bound, the bound tightens. If a request arrives whose budget is too short even for the group's next release, it does not join; it goes to a group that is about to run, or runs alone.

The effect is that the pool is always asking the same question: can I fit one more without failing anyone already here. When the answer is no, the group runs.

Tolerant work fills the gaps

A floor's workload is uneven. Perception requests from mobile robots arrive in bursts as the fleet moves. Language requests from a planning application come in a steady trickle. Embedding jobs from the Retrieval service indexing a new procedure manual arrive as a long, patient stream.

The patient stream is what keeps the pool efficient. Those requests have long budgets, so they can sit in a forming group and wait for the bursty work to pass. When an urgent perception request arrives, it finds a group that already has members and is about to release, and it rides along. The urgent caller pays almost nothing for the efficiency the tolerant callers provide.

This only works if budgets are honest. A floor where every caller declares urgency gives the pool no tolerant work to batch, and throughput falls for everyone. The budget article covers this trade in more detail.

Batching across a Czar Grid

On a single Czar, grouping happens on the node. On a Czar Grid, the routing layer has a hand in it too. Same-model requests are steered toward nodes where that model is loaded and a group is forming, so that work arriving from different parts of the building can share a pass.

This is one reason model placement on a Grid is a deliberate decision rather than an accident of which node was free. A model spread across every node batches poorly, because each node's group is small. A model concentrated where its callers' budgets allow batches well. The placement article discusses how the pool balances this against the need to keep urgent callers close to a warm copy.

What the caller experiences

The caller never sees a group. It sends a request with a budget and receives an answer inside that budget, or a refusal early enough to fall back. Whether its request ran alone or alongside many others is invisible, and that is the point. Batching is a property of the pool, not a contract the caller has to understand.

An integrator does not tune batch sizes or windows. The pool derives them from the budgets it is given. If the floor's behavior changes, the grouping adapts, because the budgets arriving with the requests have changed.

Seeing it in the record

The Evaluation trace records, for each request, which group it ran in, how long it waited before the pass, how long the pass took, and how much of its budget remained when the answer left. From that an engineer can answer questions that matter during bring-up: whether the tolerant work is actually filling the pool, whether one caller is declaring a budget that fragments the groups, and whether a model would batch better placed differently on the Grid.

The same record is what an auditor reads when asked whether a given request was served on time. Batching never hides behind an average. Each request's wait is its own line.

The budget that bounds every group is explained in the latency budget each caller declares. How a Grid steers same-model work together is in routing work across a Czar Grid. The whole path from floor to rack is in how Substrate serves a model to the floor.