Vision-language-action models served from the rack

A robot that can see, understand an instruction, and decide what to do next is running a vision-language-action model. These models are what let a humanoid pick up an unfamiliar object or a mobile manipulator follow a spoken request. They are also among the heaviest models a facility will ever run, and the machine that most needs one is often the machine least able to carry it.

Substrate's answer is to serve them from the rack. This article describes how that works, what it asks of the robot, and what never changes.

Why the rack and not the backpack

A vision-language-action model wants memory and compute that a mobile machine pays for in weight, heat, and battery. A fleet of such machines, each carrying its own copy, multiplies that cost and freezes the model at whatever version shipped with the machine.

A pool in the rack carries one copy, or a few, for the whole fleet. The machines become lighter and simpler. The site can update the model in one place, adapt it on its own captures, and serve the result to every machine that calls, regardless of vendor. The Grid grows when the fleet grows, rather than each new machine arriving with its own compute.

The cost is that the machine depends on the facility network and the pool for its hardest decisions. Substrate is built to make that dependency safe.

The request and the budget

The robot sends what it sees, usually a frame or a short sequence, together with the instruction it is working on and its current state. It declares a latency budget: the time it can hold before it must act on its own. For a machine mid-motion that budget is short.

The pool treats the request accordingly. It routes to a node holding the model warm with queue room, joins a batch only if one is about to release, and otherwise runs alone. If no node can answer in time it refuses at once. The robot, having received either an action or a refusal well inside its budget, continues.

The answer is a proposed next action. It is not a command the robot must obey. That distinction is the subject of the next section.

What stays on the machine

The robot's safety functions are the vendor's and they remain on the machine. Speed limits near people, force limits at contact, protective stops, and workspace boundaries do not route through Substrate. They do not wait on the network. They evaluate whatever the robot is about to do, whether the idea came from the rack or from an on-board model, and they permit it or they do not.

The pool therefore sits above the safety layer. A good VLA answer makes the robot more capable. A missing one makes it less capable. Neither makes it less safe, and the vendor's safety case reads the same with the pool as without it. A safety officer reviewing a deployment should look for exactly this separation, and the architecture is designed to show it.

Placing VLA models on a Grid

Because the callers cannot wait and the model is heavy, placement is deliberate. The site holds more than one warm copy so that a node leaving for maintenance does not take the model offline. It places copies on nodes that serve the cells where the robots work, so that a short budget is met from a nearby queue rather than a distant one. It avoids crowding those nodes with patient workloads that would deepen the queue.

The Evaluation traces show whether the placement is right: refusals per cell, budget remaining on each answer, and batch sizes. A cell with rising refusals during shift changes is telling the site where the next copy, or the next node, belongs.

Fallback and the record it leaves

The robot vendor decides what the machine does without an answer. A common pattern is a lighter model on board that handles the ordinary cases at reduced capability and leaves the hard cases to the rack. When the rack refuses, the on-board model answers or the machine pauses and asks for a person.

Substrate's part of this is to make the refusal fast and the record complete. A machine that paused on the floor yesterday either received a refusal, which the trace shows with its cause, or received an answer its safety layer declined, which the robot's own log shows. The two records together explain the pause.

Every VLA request, the frame it carried, the action proposed, and what happened next is recorded on the appliance. After a season, that record is a training set made of the site's own items, lighting, and layout. The Adaptation service can fine-tune the open base model on it, producing an adapter that belongs to the site and serves every robot that calls.

That loop, serve from the rack, record on the rack, adapt on the rack, is what turns a general model into one that knows this floor. None of it requires anything to leave the building.

The refusal path is detailed in when the pool cannot answer in time. Placement decisions are covered in which model runs where on a Grid. Turning the floor's record into an adapter is in adapting a model on the site's own data. The use-case page for robots is at /use-cases.