Inference service
The Inference service is the part of Substrate that serves models to callers on the facility network, behind a stable API that does not change when the hardware underneath does.
· 2 min read
IN SHORT
The Inference service is the Substrate service that serves models to applications and robots over the facility network. It presents a stable, hardware-independent API, loads open-weight models of several kinds, and routes and batches requests across the pool to meet each caller's declared latency budget. When it cannot answer in time, the caller falls back to what it carries locally.
The Inference service is the Substrate service that serves models to callers on the facility network. It presents a stable, hardware-independent API and answers requests from applications and robots within the latency budgets they declare.
Why it matters
Most of what a floor asks of a serving pool is inference: read this label, segment this scene, transcribe this instruction, decide how to grasp this item. The Inference service is where those requests land. If it is fast enough and predictable enough, robots can be built lighter and applications can be built simpler, because the heavy models live in the rack rather than on every device.
The stability of the API is the other half of why it matters. A robot vendor or an integrator writes against the API once. When the appliance is updated, when a Czar is added to the Grid, or when a node is replaced, the caller does not change. That is what hardware-independent means from the floor's point of view.
How it works here
The service loads open-weight models onto the appliance: language models, vision and perception models, speech models, embedding models for Retrieval, and vision-language-action models for robots. Several are resident at once, and adapters trained on site through Adaptation are served alongside the base models they modify.
A caller sends a request through the one API and declares a latency budget. The Inference service routes the request to a node in the pool that can meet it and batches work where batching will not miss a deadline. On a Czar Grid, routing spans every node, so the caller addresses the pool and not any particular machine.
When the service cannot answer within budget, it says so and the caller falls back to whatever it carries locally. Safety functions never depend on the pool. Every request, whether answered or fallen back, is recorded by Evaluation, so the site can see how well the service is keeping to the budgets its callers declared.
Related terms
- Substrate: the platform the Inference service belongs to.
- Latency budget: what each caller declares and the service routes against.
- Adapter: a site-trained modification served alongside a base model.
- Local Bridge: how results reach on-premise systems when the site declares a route.
Questions this raises
- Does a caller need to know which Czar in a Grid will answer?
- No. The caller addresses the pool through the one API and the Inference service routes the request to a node that can meet the budget. Which node answered is recorded in the Evaluation trace, but the caller never has to choose.
- Can several models be served at once?
- Yes. A pool typically has several models resident at the same time: perception models for robots, a language model for people, embedding models for Retrieval, and a VLA model for a humanoid pilot. The service manages which are loaded where.
- What does hardware-independent mean for the API?
- The caller's interface does not change when the appliance underneath is updated, when a node is replaced, or when a Grid grows. An application written against the API at install continues to work as the pool changes.
KEEP READING