Capacity planning for a serving pool
Sizing a Substrate pool starts with the callers and their latency budgets, not with hardware. Count the callers, know their budgets, and let the pool tell you when it is near its edge.
· 4 min read
IN SHORT
Capacity planning for a Substrate pool begins with the callers: how many applications and robots will call the API, which models they need, and what latency budget each declares. The pool's own records then show whether budgets are being met. When fallback rises for a class of caller, the pool is near its edge, and the answer is to add a Czar to the Grid.
The usual way to size an inference system is to start from hardware: how many accelerators, how much memory, what the benchmarks say. That approach answers the wrong question for a facility. The floor does not care how much hardware is in the rack. It cares whether each robot and each application gets an answer inside the time it has to act. Substrate is sized from that side.
Start with the callers. Know their budgets. Then let the pool's own records tell you when it is getting close to its edge.
Begin with the callers
Every caller on a Substrate pool declares a latency budget: the time within which it needs an answer or it will fall back to whatever it carries locally. A mobile robot deciding whether to proceed has a short budget. A pick arm evaluating an unfamiliar item has a longer one. An application drafting a shift summary has a long one. A VLA policy steering a humanoid through a task has a budget somewhere in between, and it matters more than most.
Capacity planning is the exercise of listing those callers, the models each one needs, and the budget each one declares. The list is the plan. It tells you the mix of work the pool will see, which models must be resident at the same time, and how much of the work is time-critical versus deferrable. A pool serving mostly perception for a robot fleet looks very different from one serving mostly retrieval for people, even if the hardware were identical.
What the pool tells you
Once the pool is serving, Evaluation records every call: who asked, which model answered, how long it took, and whether the answer arrived inside the budget or the caller fell back. Aggregated by caller class, these traces are the capacity signal.
A healthy pool meets its budgets during normal operation and may see occasional fallback at peak. A pool near its edge shows fallback creeping into ordinary shifts for one class of caller, usually the one with the tightest budget. That is the signal to add capacity, and because it comes from the record, it arrives before the floor notices.
The site's monitoring picks this up from the facility network, in the same dashboards it uses for everything else. No one outside the building has to tell the site it is running out of room, and no one could, because the platform does not report to anyone.
Growing the Grid
A Czar Grid grows by adding a Czar. The new node attests what it is running, joins the Grid, and begins taking work. The existing nodes keep serving while this happens. Substrate routes and batches across the pool to meet budgets, and a node that is joining is treated the same as one that is busy: work goes around it until it is ready.
This changes the rhythm of planning. Instead of a single large sizing decision at the start, a site makes a reasonable first estimate from its caller list and then grows the Grid as the record shows the need. The first Czar might serve a pilot and one fleet. The second arrives when a second vendor's robots do. The third arrives when retrieval and language workloads for the site's people are added alongside perception for its robots.
Planning for the fleet you have not bought
The pool outlives any one vendor's fleet. A site that buys robots from one vendor this year and another next year serves both from the same pool. The second vendor's robots will have their own models and their own budgets, and they will arrive on the pool's doorstep whether or not the site planned for them.
Sensible planning therefore looks at the site's intentions, not just its inventory: the kinds of robots the site is likely to run in the next phase, whether people will start using retrieval and language models on the same pool, and whether Adaptation will be running on captured footage alongside serving. Each of these adds to the caller list, and a site that has thought about them can add nodes ahead of need rather than behind it.
Where the hardware question goes
None of this means hardware does not matter. It means the hardware question is answered by Element 31 when the appliance is configured for the subscription, based on the caller list the site provides, and revisited when the Grid grows. The site's job is to know its callers and read its record. The platform's job is to meet the budgets and say so when it cannot. Between the two, a pool stays sized to the floor it serves, and the floor gets its answers in time. The platform overview at /platform describes how routing and batching do their part.
Questions this raises
- How do we know when to add a node?
- Watch the fallback rate and the budget misses in the Evaluation traces, broken out by caller class. When a class of callers starts falling back during normal operation rather than only at peak, the pool needs another node before the next fleet expansion, not after.
- Does adding a Czar to a Grid interrupt the floor?
- No. A new node attests what it is running, joins the Grid, and begins taking work while the existing nodes keep serving. The pool routes around a node that is joining or leaving the same way it routes around one that is busy.
- Should we plan for robots we have not bought yet?
- Yes. The pool is a facility asset that will serve several vendors' robots over its life. Planning for the next fleet rather than the current one is how sites avoid a scramble when the second vendor arrives.
KEEP READING