Routing work across a Czar Grid

One Czar serves a floor. A Czar Grid serves a facility. The difference, from the floor's point of view, is nothing at all: the same API, the same endpoint on the facility network, the same budget declared with each request. The difference inside the rack is that there is now a decision to make about which node does the work.

That decision is routing. Substrate makes it for every request, and this article describes how.

One pool, many nodes

A Grid is several Czars joined into one pool. Each node carries the same sealed, signed software image and has attested what it is running before it was admitted. Each node holds some set of models warm in memory. Together they present one serving surface to the building.

The routing layer sits in front of that surface. When a request arrives it knows three things: the model the caller wants, the budget the caller declared, and the current state of every node, including which models are loaded, how deep each queue is, and which batches are forming.

Three questions for every request

The first question is where the model is warm. A model loaded on a node answers quickly; a model that must be loaded first does not. Routing strongly prefers nodes already holding the requested model, and the placement decisions described in the model placement article determine how many such nodes there are.

The second question is which of those nodes can meet the budget. A node may hold the model but be deep in queue with work that will not clear in time. Routing looks at what each candidate has already accepted and the budgets attached to that work, and discards nodes that cannot fit the new request without failing it or something ahead of it.

The third question is where this request batches best. Among the nodes that can meet the budget, routing prefers one where a group for the same model is already forming and has room before its bound. Sending the request there lets it share a forward pass and keeps the pool efficient. If the budget is too short to wait for any forming group, routing picks the node that can run it soonest alone.

When no node satisfies all three, the pool refuses promptly, inside the budget, so the caller can act on its own.

Routing and integrity are one list

A node is eligible for routing only after it has attested its software image and been admitted to the Grid. Watchman verifies integrity continuously after that. If a node's attestation fails or Watchman records a deviation, the node leaves the eligible set. Routing does not have a separate notion of health that could disagree with the integrity record; the list of nodes that may take work is the list of nodes that have proved what they are running.

This matters to the safety officer and the auditor as much as to the engineer. A request's trace names the node that served it, and that node's provenance and attestation record are on the appliance. The chain from a robot's decision back to a verified piece of software is unbroken.

When a node leaves

Nodes leave for ordinary reasons: a scheduled replacement, a rollback to the previous signed release, a hardware fault. When one does, routing stops placing new work on it immediately. Requests already queued there and not yet run are moved to other nodes if their budgets still allow. Requests that cannot be met in time are refused so their callers fall back.

The floor experiences this as a brief reduction in capacity, not an outage. Callers with short budgets may see more refusals until the remaining nodes absorb the load; callers with long budgets see slightly longer waits. The Evaluation trace records every request that changed nodes and every refusal, so the operator can see exactly what the event cost.

Adding a node is the reverse. Once it attests and is admitted, it begins receiving work according to the models placed on it. No caller is reconfigured.

What the integrator does and does not do

The integrator does not write routing rules. There is no table of which cell talks to which node. The integrator's work is to make sure callers declare honest budgets and that the models they need are placed on the Grid in a way that suits those budgets. Routing takes it from there.

The integrator does read the traces. During bring-up, the routing decisions for each request are the fastest way to see whether a model should be concentrated for batching or spread for proximity, and whether a particular caller's budget is forcing it to run alone more often than necessary.

How models are distributed across nodes is in which model runs where on a Grid. The batching that routing tries to enable is in batching without missing a deadline. The admission step that makes a node eligible is in attestation before a node takes work. The appliance itself is described at e31.io/czar-grid.