Rolling back to the previous signed release
A Substrate update can be undone. The previous signed release stays on the node, and a site that needs to step back applies it, attests, and resumes, with the rollback recorded like any other software load.
· 4 min read
IN SHORT
Rolling back returns a Czar node to the previous signed release of Substrate, which remains on the node after an update. The site initiates it, the node applies the prior image, attests, and rejoins the Grid. Models, adapters, traces, and site data are outside the image and are preserved. The rollback is written to the provenance record as a signed software load.
An update that cannot be undone is a commitment the site did not ask to make. Substrate treats every update as reversible. When a node is updated to a new signed release, the previous signed release remains on the node, and the site can return to it whenever it decides to. This article describes how that works and what the site should expect.
Why rollback is ordinary
A new release is tested before it ships, but it is tested in general. A site's floor is particular. A release that improves batching for most workloads might shift refusal patterns for one site's unusual mix of callers. A site should be able to find that out on one node, step back, and report it, rather than live with it or race to fix it.
Rollback is therefore part of the normal update practice rather than a last resort. The sensible sequence is to update one node, run it for a while, read the Evaluation traces for that node against the others, and either proceed across the Grid or roll back. Nothing about rolling back is exceptional.
What rollback does
The site initiates the rollback for a node. The node applies the previous signed release, which has remained on the node since the update. It verifies that image's signature exactly as it would verify a new one. It boots, attests the release it is now running, and presents that attestation to the Grid, which readmits it. Routing begins placing work on it again.
The rollback is written to the node's provenance record as a signed software load, naming the release returned to and the time. Looking at the record afterward, an auditor sees an update entry followed by a rollback entry, both verified, with no gap in which the node's software was undocumented.
What is preserved
The signed image is the platform. Everything the site has shaped lives outside it: the models loaded, the adapters trained on site, the policies on the API, the document indexes Retrieval built, the memories agents hold, and every Evaluation trace. A rollback changes the image and leaves all of this where it was.
The node resumes serving the same models to the same callers under the same policies. The traces it wrote while on the newer release remain, which is useful: they are the evidence for why the site rolled back, and the record of what the newer release did differently on this floor.
The Grid during a rollback
A rollback is done one node at a time. While a node is applying the prior release and attesting, it is out of the eligible set and the rest of the Grid serves the floor. Models with more than one warm copy remain available; models held only on the node being rolled back are briefly unavailable, and the traces show which callers were affected.
A Grid can run mixed releases while the site evaluates. Each node attests its own release and the provenance record shows which node is on which. The site brings them into line when it is ready, forward or back. There is no requirement that every node change at once, and no state in which a node is running something it has not attested.
Watchman, and what rollback does not do
If Watchman records a deviation on a node after an update, rollback is one of the responses available to the operator reviewing the record. It is not automatic; a deviation is a reason for a person to look, and the record is there for that person. If the operator concludes the update did not apply cleanly, rolling back to the previous verified release and attesting again is the direct remedy, and the Watchman entry, the rollback entry, and the subsequent attestation all appear in order in the node's history.
Rollback returns the platform software to the previous signed release. It does not undo changes the site made to its own content in the meantime: an adapter promoted last week stays promoted, a policy changed yesterday stays changed. Those are the site's decisions, made outside the image, and reversing them is a separate action the site takes if it wants to.
It also does not reach further back than the previous release. A site that wants a release older than that applies it as a signed bundle in the usual way, downloaded from the portal on connected sites or carried in by media on air-gapped ones.
The image property that makes a verified rollback possible is in what the signed image guarantees. Where the rollback is recorded is in the provenance record for a node. The step a rolled-back node takes before resuming work is in attestation before a node takes work.
Questions this raises
- When would a site roll back?
- When a new release behaves in a way the site did not expect on its floor, when Watchman flags a problem after an update, or when the site wants more time to validate a release before committing the whole Grid. Rollback is an ordinary tool, not an emergency measure.
- Does rollback lose traces or adapters?
- No. Traces, adapters, models, policies, document indexes, and memories live outside the signed image and are untouched by changing it. A rollback changes the platform software and nothing else.
- Can the site roll back one node and leave the rest updated?
- Yes, for as long as it chooses. Each node attests its own release, and a Grid can run mixed releases while the site evaluates. The provenance record shows which node is on which release, and the site brings them into line when it is ready.
KEEP READING