Tracing every operation with Evaluation
Substrate's Evaluation service writes a trace of every AI operation on the appliance: who called, which model answered, how long it took, and what came back. The record stays on site for tuning and audit.
· 4 min read
IN SHORT
Substrate's Evaluation service writes a trace for every operation on the platform: the caller's identity, the service and model used, the latency budget and time consumed, the node and batch, and the outcome, with inputs and outputs retained under the site's policy. The record stays on the Czar appliance, serves engineers during tuning, and is something an auditor can read.
A model that influences what a robot does, what a technician is told, or what an agent decides should be able to account for itself. On Substrate that accounting is the job of the Evaluation service. It writes a trace of every operation on the platform and keeps the record on the appliance, where the site's engineers tune from it and the site's auditors read it.
This article describes what a trace contains, how it is used, and why it stays where it is written.
What a trace contains
For every operation on every service, Evaluation records who, what, how, and what happened.
Who is the caller's identity on the API: the robot, the application, the agent, or the person. What is the service and the task, and the model and version that fulfilled it, including any adapter trained on site. How is the latency budget declared, the time consumed, the node that did the work, and the batch the request joined. What happened is the outcome: an answer, a refusal with its cause, or an error.
Inputs and outputs are retained according to the site's policy. A site may keep every frame a perception model saw, keep summaries, or keep nothing beyond the metadata. The policy is the site's, and it is applied on the appliance.
Nothing is untraced
Because all seven services sit behind a single API, every operation crosses one surface and Evaluation sees it. An inference, a retrieval, a memory write, an adaptation job, an integrity check by Watchman, a route opened by Local Bridge: each is a line in the same record. There is no side path through which the platform could act without a trace.
Reading the trace is itself an operation and is traced. The record shows who looked at it and when, which is what an auditor expects of a record that matters.
The trace during bring-up and after
An integrator bringing up a floor lives in the traces. They are the fastest way to answer the questions that come up in the first weeks.
Which callers are close to their budget, and which are declaring budgets far tighter than they use. Which models are refused most, and on which nodes. Whether batches are forming or whether urgent callers are fragmenting them. Whether a model placed on one node should be spread, or a model spread across many should be concentrated. Whether a new adapter answers better than the base model it layers on, measured on the site's own requests.
None of this requires instrumenting the callers. The pool already knows, and the trace is where it says so.
Day to day, the trace is the site's view of a platform that never phones home. Operations staff watch the same record the engineers tuned from: refusal rates by model and by cell, budget consumption trends, and the health of each node. A floor that slowed yesterday afternoon can be explained from the trace this morning, and the explanation names the cause rather than guessing at it.
The trace is also how drift shows up. A perception model whose confidence declines as suppliers change their packaging, or a retrieval library whose answers increasingly cite superseded documents, appears in the record before anyone on the floor names the problem.
The trace for the auditor
When a safety officer or an auditor asks what the system did, the trace is the answer. For a given request they can see the caller, the model and its version, the data it was given if retained, the answer, how long it took, and whether it was served or refused. For a given model they can see every request it answered over a period. For a given adapter they can see when it was trained, from what, when it was loaded, and what it influenced.
This is a record an auditor can read without any party outside the building. It is not a report prepared for them; it is the operational record itself, kept as written.
Why the record stays on the appliance
The trace is a detailed picture of the site: what its robots see, what its technicians ask, what its agents decide. It is written on the appliance and stays there. No default path exists to the public internet, the appliance never connects to the customer portal, and nothing in Evaluation sends the record anywhere.
A site that wants to move traces to its own systems, for long-term archive or for its own analytics, can do so over a route it declares through Local Bridge. That is a decision the site makes, and the trace records that it was made.
The surface that makes universal tracing possible is described in one API for seven services. The most common traced operation is covered in calling Inference from the facility network. The integrity record that Evaluation sits alongside is the subject of what Watchman watches.
Questions this raises
- Does tracing slow the pool down?
- Tracing is part of the serving path and is budgeted for. The record is written on the appliance alongside the operation, and the latency budget a caller declares already accounts for it. A site does not trade visibility for speed.
- Who can read the traces?
- Roles the site grants on the API. Typically engineers and integrators during bring-up, operations staff day to day, and auditors or safety officers on request. Reading the trace is itself a traced operation.
- How long are traces kept?
- As long as the site's retention policy says. The site chooses what to retain, including whether inputs and outputs are kept in full, summarized, or dropped after a period, and the policy is applied on the appliance.
KEEP READING