Calling Inference from the facility network

Most of what a facility asks of Substrate goes through the Inference service. Robots asking for perception, applications asking for language, consoles asking for transcription: all of them make the same kind of call over the facility network. This article is written for the integrator or vendor engineer about to make that call for the first time.

Where the service is

The Inference service is reached through Substrate's single API at an endpoint on the facility network. The site provisions the endpoint when it brings up the appliance, and it is the same endpoint whether the site has one Czar or a Czar Grid. Callers do not address nodes. They address the API and the pool decides the rest.

The endpoint is inside the building. There is no cloud endpoint to fall back to, because there is no path to one. A caller that can reach the facility network can reach the pool, and a caller that cannot reach the facility network cannot reach anything.

What a caller sends

Four things define an Inference call.

The identity is the caller's credential, provisioned by the site. It determines which tasks and models the caller may use and is recorded with every call. A robot's identity, a planning application's identity, and an operator console's identity are distinct and have distinct permissions.

The task names what the caller wants done: identify this item, transcribe this audio, propose the next action for this robot state. The site maps tasks to models and adapters, and can change the mapping without the caller knowing.

The input is the data: a frame, an audio clip, a prompt, a robot's observation. It travels over the facility network to the appliance and no further.

The budget is how long the caller can wait. It is the most consequential field in the request, and the article on latency budgets explains why. A caller should declare the budget that matches what it will do without an answer.

What comes back

One of two things. An answer, with the output of the model, the identity of the model and version that produced it, and how much of the budget remained. Or a refusal, which arrives early and says that no placement could meet the budget.

Callers should treat the refusal as a normal outcome from the first integration, not an exception to be handled later. A robot that receives a refusal falls back to its on-board capability. An application shows the last good result and retries. The pool is designed to refuse early so that this is always possible.

Answers are returned to the caller and nowhere else. The response, like the request, stays on the facility network.

Some tasks produce output incrementally, such as a language model drafting a long document. The API supports returning partial output as it is produced, so an application can show progress rather than wait for the whole. The budget still governs the whole operation, and a caller with a long budget can accept a long stream.

Tasks that are not interactive at all, such as indexing a document library for Retrieval or running an adaptation job, are submitted with long budgets and run in the gaps around urgent work. The caller checks the result rather than waiting on the connection.

What the trace records

Every Inference call is written by the Evaluation service to the record on the appliance. The trace includes the caller's identity, the task, the model and adapter that answered, the budget declared and the time used, the node that ran it and the batch it joined, and the outcome. The input and output are retained according to the site's own policy.

For an integrator this is the primary debugging tool. A caller that is refused more than expected, a model that is close to its budget, a batch that never fills: all of these are visible in the traces without instrumenting the caller. For an auditor it is the record of what the platform did on behalf of whom.

Mistakes to avoid

The common first-integration mistakes are few and recognizable. Declaring a budget tighter than the caller needs, which costs priority the caller did not require. Treating a refusal as an error to retry immediately, which only adds load to a pool that has already said it is full. Hard-coding a model name instead of a task, which breaks when the site changes the model behind the task. And assuming the pool is always there, which the design never assumes.

A vendor who avoids these ships an integration that works on the first Czar it meets and on every Grid after it.

The budget field is explained in the latency budget each caller declares. How Inference fits with the other services is in one API for seven services. What happens on refusal is in when the pool cannot answer in time.