Skip to content
ELEMENT 31

Latency budget

A latency budget is the time a caller declares it can wait for an answer from the pool. Substrate routes and batches to meet it, and the caller falls back if the pool cannot.

· 2 min read

IN SHORT

A latency budget is the time a caller declares it can wait for an answer from the Substrate pool before it must act on its own. The pool uses the budget to route the request to a node that can meet it and to decide how much batching is safe. If the budget cannot be met, the caller falls back to what it carries locally, and the miss is recorded.

A latency budget is the time a caller declares it can wait for an answer from the Substrate pool before it must act without one. It is set per request by the caller, and the pool routes and batches to meet it.

Why it matters

A robot that has to decide whether to proceed cannot wait indefinitely for a model. Neither can a pick arm with an item in its gripper or a vehicle approaching an intersection on the floor. Each has a moment by which it needs an answer, and the moment is different for each. The latency budget is how a caller tells the pool what that moment is.

Declaring the budget explicitly changes how the pool behaves. Instead of maximizing throughput and hoping every caller is satisfied, Substrate schedules against deadlines. A request with a tight budget is routed and served ahead of one that can wait. Batching, which improves efficiency, is used only where it will not push any accepted request past its deadline.

How it works here

A caller sends a request through the one API and includes its budget. The Inference service picks a node in the pool, on a single Czar or across a Czar Grid, that can answer inside that budget given its current load. It groups the request with others where grouping is safe and serves it.

If the pool determines it cannot meet the budget, it says so promptly rather than answering late. The caller then falls back to whatever it carries locally: an onboard model, a conservative default, or simply stopping and waiting for a person. Safety functions never depend on the pool in the first place, so a missed budget degrades capability and never safety.

Every request and every miss is written to the Evaluation trace. Aggregated by caller class, budget misses are the clearest signal that a pool is approaching its limit, and the usual response is to add a Czar to the Grid before the next fleet expansion.

  • Inference service: the service that routes and batches against the budget.
  • Substrate: the platform the budget is declared to.
  • Adapter: site-trained modifications served within the same budgets.
  • Watchman: the integrity service that runs alongside serving.

Questions this raises

Who sets the latency budget?
The caller does, when it makes the request. A robot vendor sets budgets for its robots based on what each decision needs; an integrator sets budgets for applications. The pool does not impose one.
Does a short budget get priority over a long one?
The pool routes so that every budget it accepts is met, which in practice means time-critical work is scheduled ahead of deferrable work. A request with a long budget may wait while shorter ones are served, without missing its own deadline.
What happens to a request that misses its budget?
The pool tells the caller it cannot answer in time and the caller falls back to its local behavior. The miss is written to the Evaluation trace, where the site can see how often it happens and for which callers.

KEEP READING

Related.