When the pool cannot answer in time
Substrate is built on the assumption that the pool will sometimes miss. When it cannot meet a caller's budget it refuses early, the caller falls back to what it carries, and safety never depended on the rack.
· 4 min read
IN SHORT
When Substrate cannot meet a caller's declared latency budget it refuses the request early rather than answering late. The caller then falls back to whatever it carries locally, such as an on-board model, a conservative default, or a controlled stop. Safety functions never depend on the pool, so a missed answer degrades capability, not safety.
Every shared resource is sometimes busy. A facility pool is no exception, and Substrate does not pretend otherwise. Instead of designing for a pool that always answers, it designs for the moment the pool cannot, and makes that moment safe, fast, and visible.
This article is about that moment: what the pool does, what the caller does, and what the record shows afterward.
The pool refuses early
A request arrives with a latency budget. The routing layer looks for a node that holds the model warm and can run the request, alone or in a forming batch, before the budget expires. If it finds none, it does not accept the request and hope. It refuses immediately.
The refusal is the useful outcome. It arrives while most of the budget is still unspent, which means the caller has time to do something else. An answer that arrived after the budget would have been discarded anyway, and would have consumed pool capacity that another caller could have used.
The same logic applies mid-flight. If a node leaves the Grid while a request is queued and no other node can meet the remaining budget, the request is refused rather than silently delayed.
The caller falls back
What happens next belongs to the caller. Substrate does not decide what a robot does without an answer; the robot vendor and the integrator do, because they know the machine.
For a mobile robot asking for perception help, fallback might be an on-board model that is less capable but fast enough to continue at reduced speed. For a pick arm that asked the pool to identify an unfamiliar item, fallback might be to set the item aside and move to the next. For a planning application, fallback might be to show the user the last good result and retry. For a humanoid pilot guided by a vision-language-action model, fallback might be a controlled stop and a request for a person.
None of these are failures of the system. They are the system behaving as designed, with the pool as an improvement over the baseline rather than a dependency the baseline cannot survive.
Safety never depended on the pool
This is the rule that makes everything else acceptable. A robot's safety functions, the ones that keep it from striking a person or exceeding a limit, stay on the machine. They are not routed through Substrate, they do not wait on the facility network, and they do not change when a pool is added to the building.
The pool therefore sits above the safety layer, not inside it. When the pool is absent, the robot is less capable. It is not less safe. A safety officer evaluating a Substrate deployment should find that the vendor's safety case reads the same with the pool as without it, and that is exactly what the platform intends.
Why the pool misses, and what to do about it
A refusal has a cause, and the Evaluation trace records it. The common causes during bring-up are predictable.
Capacity is the obvious one. A floor that has grown its fleet since the pool was sized will see more refusals at peak. The fix is another node in the Grid, and the traces show when and for which models the shortfall occurs.
Placement is subtler. A model that is warm on only one node leaves callers with short budgets waiting on that node's queue while others sit idle. Spreading the model, or moving it closer to its callers' budgets, reduces refusals without adding capacity.
Dishonest budgets are the quietest cause. A caller declaring a tighter budget than it needs forces itself to run alone and crowds out others, and it refuses more often than it should. The trace shows how much budget each caller leaves unused, which is the tell.
What the record shows
Every refusal is a line in the trace: the caller, the model, the budget, the node states considered, and the reason none sufficed. Every served request shows how much of its budget remained. From these an operator can answer the question that follows any slow afternoon on the floor: was that the pool missing, or the robot falling back as it should.
The answer is often the latter, and the record is what lets the site say so with confidence rather than guess. It is also what an auditor reads when asked whether a system that influences machines behaved within its declared limits.
The budget that defines a miss is explained in the latency budget each caller declares. The normal serving path is in how Substrate serves a model to the floor. How node departures are handled is in routing work across a Czar Grid.
Questions this raises
- Does a robot stop when the pool cannot answer?
- It does whatever its vendor designed it to do without the pool, which may be slowing, using an on-board model, or stopping. The pool improves what the robot can do; it is not what keeps the robot safe, and the vendor's safety system is unchanged by its presence.
- Why refuse instead of answering as soon as possible?
- An answer that arrives after the budget is useless to the caller and may be worse than none, because the caller has already committed to its fallback. Refusing early returns the remaining budget to the caller so it can act with time to spare.
- How does the site know how often the pool misses?
- Every refusal is written to the Evaluation trace with the model, the budget, and the node state at the time. The site can see which callers miss, when, and why, and whether the fix is placement, capacity, or a more honest budget.
KEEP READING