We ran Nervous Machine on the one thing you can rarely see: a real multi-vendor supercomputer's full cooling telemetry; IT compute, air cooling, liquid cooling, and weather. Five independent owners, 2.5 years, the CINECA Marconi100 archive. The premise it tests is simple: autonomy you can trust has to know what it doesn't know, and direct its learning there. On this archive it sorted the cooling relationships into the ones safe to automate and the ones that silently decay, surfaced one risk no single vendor can observe, and found a variable missing from its own model and corrected it.
We watched each relationship in the cooling system across ten operating regimes over 2.5 years and asked one question per edge: can you trust it? The answers fell into four classes; safe to learn once, decays over time, missing a variable, or unlearnable. Each one calls for a different response. An edge's variability across regimes predicts how badly it fails on an unseen regime (ρ = 0.76), and the pattern survives the strict version of the test, on the well-specified edges only (ρ = 0.80).
| Edge | Class | CV | Held-out relRMSE | Coef drift → 2022 | Fix |
|---|---|---|---|---|---|
| node_power → gpu_core_temp | INVARIANT | 0.08 | 0.079 | 0.09 | None needed, twin-safe |
| rack_coupling | INVARIANT | 0.08 | 0.057 | 0.02 | None needed, twin-safe |
| cooling_to_ambient | DRIFTING | 0.21 | 0.086 | 0.2 | Online recalibration (coef decays) |
| node_power → fan_speed | DRIFTING | 0.7 | 0.091 | 0.69 | Online recalibration (coef decays) |
| node_power → dimm_temp | DRIFTING | 0.86 | 0.091 | 1.16 | Online recalibration (coef decays) |
| ambient_inlet → cpu_core_temp | WEAK | 0.28 | 0.172 | 0.38 | Add feature (cpu_user: −25% err) |
| fan_speed → cpu_core_temp | WEAK | 0.85 | 0.18 | 1.65 | Add feature (cpu_user: −25% err) |
| node_power → cpu_core_temp | WEAK | 0.85 | 0.177 | 0.11 | Add feature (cpu_user: −25% err) |
INVARIANT: stable physics across regimes; learn once, trust everywhere. DRIFTING: pointwise predictions accurate but coefficients drift. Safe to forecast with, unsafe to invert as a control law without continual recalibration. WEAK: missing a feature (cpu_user); the decomposition prescribes the fix and cuts held-out error 25%. INVARIANT edges are the negative control that proves the audit isn't flagging everything.
The DRIFTING class is the finding most likely to bite an operator, because it never shows up on a dashboard. Predictions on these edges stay accurate across the whole 2.5-year window. Their error is on par with the rock-solid INVARIANT edges. By every health metric most teams watch, they look fine.
The coefficients tell a different story. The starkest case is node_power → fan_speed: its slope fell from approximately 1.3 in winter 2021 to approximately 0.16 in winter 2022, an 8× monotone decline, almost certainly real control-firmware evolution rather than noise. relRMSE is normalized by target magnitude, so it under-weights exactly this failure mode: the prediction can track the target while the relationship underneath it changes.
A twin used for prediction looks fine and is fine. The same twin used as a control law inverting the relationship to choose an actuator setting is silently stale, acting on a coefficient that no longer holds.
No feature fixes a temporal drift of this kind. Continual recalibration is necessary. The audit identifies which edges have this property before deployment; the online recalibration loop is what closes them.
The Marconi100 cooling system has five telemetry owners: two IT-side (compute, thermal) and three facility-side (Vertiv air cooling, Schneider liquid cooling, external weather). Typically, in data centers each owner runs their own inner loop; their own controller, their own model, their own data tap. DCIM platforms aggregate polled summary metrics across vendors for dashboards and PUE / capacity reporting. The high-rate, closed-loop control telemetry needed to learn, or even observe, the edges between vendors typically isn't centralized across vendors at that granularity. It resides in each unit's own controller and is exported only as coarse polled summaries. No outer loop exists at that granularity. The CINECA M100 archive is structurally rare in the field: the Borghesi et al. data paper assembled it from independent per-subsystem collectors; IPMI, Vertiv, and Schneider each gathered separately. It is referenced as the largest public dataset of its kind, and notes that public data in this field is mostly kept on premises. That separate-tap collection is itself the evidence of the siloing, and it's what makes this one of the few archives where the cross-owner edge is even visible.
A digital twin built inside one silo treats the other silos the way a self-driving car treats a stoplight: as a point observation of the environment, not as an autonomously controlled system with its own logic. Both views can be wrong in the same way, at the same time, on the edge between them, and neither system can see it.
We tested an early Phase-0 hypothesis that the two cooling loops coordinate complementarily under load. The pattern held for one month and did not generalize: the two winter regime reps disagree in sign, spring is complementary throughout, summer is decoupled. There is no stable coordination policy for any single model to learn in this data. Two claims follow, kept separate because they have different evidential standing:
Across the instrumented regimes, coordination between the two loops is non-stationary. The one-month flip is not a learnable invariant. We tested and rejected the obvious artifact explanation; the pattern survives excluding non-cooling units.
This coordination lives in the one edge no single owner instruments. Neither vendor sees the other loop's ΔT, so even detecting whether the two loops have drifted out of their jointly-validated regime is impossible from inside either silo. It requires a cross-owner layer to observe at all.
The non-stationarity itself isn't novel; it's the well-known central problem of multi-agent learning, where each adapting agent makes the environment non-stationary for the others. The standard mitigations (multi-agent RL, federated digital twins) all assume you either control the agents or they cooperate. Hyperscalers handle this by collapsing the stakeholders into one organization that owns the full stack. Everyone else — supercomputer centers, enterprise data centers, colocation operators — lives with the cross-vendor reality. M100 is the rare archive where the non-owned edge is even observable; in the typical multi-vendor environment it isn't, which means twins built inside today's silos are structurally blind there.
Knowing what it doesn't know isn't a slogan here; it's a running list. Every relationship carries a live certainty, and the ones that stay uncertain or carry residual error, collect into what we call the curiosity surface: the boundary of what the system can't yet explain. On this archive, one relationship sat there flagged — node_power → dimm_temp predicted memory temperature confidently, but with a small, stubborn bias. Confident and consistently off by a hair: the signature of a missing variable.
So the system formed a new hypothesis; the missing factor is the air coming into the rack, andtested it against the telemetry. It held: adding inlet temperature lifted that relationship's certainty from 0.76 to 1.0 and cut its error 65%. The new edge was added to the model. No one told it a variable was missing; it noticed, proposed, checked, and patched.
It also proposed two fixes that failed; a cooling-zone boundary to explain a collapsed edge, and hot-aisle recirculation between neighboring racks. Certainty failed to rise in both cases, and the system correctly rejected them.
This isn't a one-off tuned to cooling. The same compose → learn → escalate loop has run autonomously across five unrelated public domains; battery aging, telescope photometry, power-grid frequency, river flow, and urban air quality. Adaptation of priors was successful in four (the fifth correctly refused, and we report it as a negative). On NASA Mars surface-radiation data, the same machinery learned two physical effects that were never written into its prior. Same primitives, different worlds.