Attestation before a node takes work

A Czar Grid is only as trustworthy as its least trustworthy node. A machine that has been tampered with, or that is running an old release by mistake, or that is not what it claims to be at all, should never be handed a robot's perception request. Attestation is how Substrate ensures it is not.

This article describes what a node proves, when, to whom, and what happens when the proof fails.

What a node attests

A node attests the software image it is running. Because the image is sealed, signed, and read-only, the node can present a verifiable statement: this is the release I booted, here is the evidence that it matches the signature, and here is the evidence that it has not been altered. The statement is checked against the release's signature on site.

The attestation names the exact signed release. Not a version string that could be edited, but a verified identity for the image. This is the same identity that appears in the node's provenance record and in every trace the node later writes, so the chain from a served request back to a verified image is unbroken.

When it happens

At boot. A node verifies its own image before it brings up any service.

Before joining a Grid. A node that wants to become part of a pool presents its attestation to the Grid, and the Grid admits it only if the attestation verifies. A new node arriving at a site, or a node returning from a repair, goes through this step.

Before resuming work. A node that has been excluded, whether for a rollback, for maintenance, or because Watchman recorded a deviation, attests again before it is allowed back.

Between these moments, Watchman verifies the image continuously. Attestation covers the point of admission; Watchman covers everything after. The two together mean there is no interval during which an unverified node serves work.

Attestation and routing are one list

This is the design decision that matters most. The routing layer on a Grid places requests on nodes from a set of eligible machines. That set is the set that has attested and has not since been flagged. There is no separate list of healthy nodes maintained by a different mechanism that could drift from the integrity record.

The practical consequence is that a question an operator might ask, whether a given request could have been served by an unverified node, has a structural answer rather than a procedural one. The request was placed by routing; routing only places on attested nodes; therefore no.

When attestation fails

A node whose attestation does not verify is not admitted. It does not receive work. The failure is written to the append-only record with what the node presented and why it did not match. The site's operators see it in the same monitoring they use for everything else on the appliance, from inside the site.

The failure is not retried silently in the hope it clears. A node that fails attestation has something wrong with it that a person should look at: an incomplete update, a storage fault, or something worse. The record is there for that person.

Meanwhile the Grid continues on its remaining nodes. Models held on the excluded node are served from other copies if the site placed them, or refused if it did not, and the traces show which. This is one reason the placement article recommends more than one copy of models the floor cannot do without.

No outside party, and the record

Attestation is verified on site. The node presents its evidence to the appliance or the Grid, and verification happens there against the release's signature. Element 31 is not in the loop. The appliance never connects to the portal, there is no attestation server somewhere else, and an air-gapped site attests exactly as a connected one does.

This is consistent with the rest of the platform's posture. Integrity is something the site can verify for itself, from inside the building, with the record to hand. It is not a service the site receives from outside.

Every attestation, successful or not, is a line in the append-only record: the node, the release it attested, the time, and the outcome. Over the life of a Grid this becomes a history of exactly which verified software ran on which machine and when. Combined with the provenance record for each node and the Evaluation traces of what each node served, it is the material an auditor needs to answer the question of what software influenced a given decision on a given day.

The image property that makes attestation possible is in what the signed image guarantees. How the attested set is used request by request is in routing work across a Czar Grid. The per-node history that attestation feeds is in the provenance record for a node.