Monitoring a platform that never phones home

Most hosted platforms are monitored by their vendor. The customer gets a status page and a support email; the vendor gets the telemetry. Substrate inverts this. The appliance sends nothing out, so the only party who can see how it is doing is the site that runs it. That is deliberate, and it changes what monitoring looks like.

The platform exposes everything a site needs to watch it, on the facility network, in forms that ordinary monitoring tools can read. The site's existing stack becomes the console. Nothing new is adopted and nothing is sent anywhere.

Three kinds of signal

The first signal is serving health. Each Czar reports whether it is up, which models and services are loaded, and how the pool is keeping to the latency budgets its callers declared. On a Czar Grid this is visible per node and for the pool as a whole, so a site can tell the difference between one node being slow and the whole floor being under-provisioned.

The second signal is the trace. Evaluation records every AI operation: which caller asked, which model answered, over what route if any, and whether the answer arrived within budget or the caller fell back. These traces are the raw material for understanding what the pool actually did during a shift, and they are written locally.

The third signal is integrity. Watchman continuously verifies the signed software image and the enclosure's tamper indicators and writes any deviation to an append-only record. This is the signal a security team cares about most, and it sits beside the others on the facility network rather than in a separate system.

Folding it into what the site already runs

Sites do not need a new monitoring product for Substrate. The signals are exposed in forms the site's existing collectors can scrape or subscribe to. A plant that already graphs its historians and alerts on its network gear adds the appliance as one more source. The alerting rules are the site's: which latency misses matter, which fallback rates are acceptable for which callers, which Watchman events page someone at night.

This is where an integrator often earns their keep. They know the site's monitoring conventions and can map the platform's signals onto them in the first week, so that by the time the pool is serving the floor, the operations team sees it in the same dashboards they already trust.

Reading the pool during a shift

In practice, a site watches a few things. Whether callers are getting answers inside their budgets, or fallback is rising for a particular class of robot or application. Whether one node in the Grid is behaving differently from the others. Whether a model has been swapped or an adapter loaded, and whether the traces changed afterward. Whether Watchman has recorded anything since the last review.

Because every one of these questions is answered from local data, the answer is available immediately and does not depend on anyone outside the building. A site at the end of a shift can review what the pool did without filing a request or waiting on a vendor portal.

Reconstructing an incident

When something goes wrong on the floor, the record is how the site finds out what the pool contributed. The traces show which decisions the pool returned and when. The serving health shows whether the pool was under load. The Watchman record shows whether the platform was running the image it was supposed to be running. The provenance record shows which release each node had loaded.

None of this requires Element 31 to be involved. The site has the full picture, locally, in records that cannot be edited after the fact. If the site wants outside help interpreting them, it opens a customer-initiated support session through the portal, and on a sovereign site that session requires attestation and carries a hard expiry. The record is the site's; the help is optional.

Notice what the site is not doing. It is not checking a vendor status page, because there is none. It is not wondering whether telemetry is leaving, because the platform has no route to send it on. It is not comparing a vendor's view of the pool with its own, because there is only one view. Monitoring a platform that never phones home turns out to be simpler than monitoring one that does: everything is in one place, and the place is yours. The platform overview at /platform describes the services that produce each signal.