01 Why not just use a graphing tool
Grafana and its relatives are excellent at here is a time series. They are much weaker at the question an operator asks first: what is the estate, and what in it is unhappy right now?
Answering that normally means four or five tabs — the hypervisor for guests, the monitor for problems, the network controller for clients, a terminal for the accelerators. This puts all of it into one spatial view legible from across a room.
Nothing in the scene is decorative. Planet colour is the guest's role, size is its memory allocation, dimming means stopped, and a red flare means the monitor has an active problem against that specific guest. If something looks wrong, something is wrong.
02 Architecture
Backends are polled by background threads into a single in-memory cache; the HTTP layer only ever reads that cache. Three properties fall out of that shape, and all three matter more than they sound.
- Requests never block on a backend. A monitoring server that has gone away makes its poller log and retry. It does not make the dashboard slow.
- Failures are isolated. Each poller writes only its own key of the cache, wrapped end to end in try/except. One dead backend removes one layer from the scene rather than the scene.
- Last-good data survives. Pollers overwrite their key only on success, so a transient failure leaves the previous reading on screen instead of blanking the panel.
03 The session that lies
A network controller can return an empty client list rather than an error when its session expires. Handled naively, the client count silently drops to zero and every graph shows a cliff that never happened.
The poller treats an empty list as session expiry: keep the last-good reading, re-authenticate on the next cycle. It is two lines of code and the difference between a display you trust and one you learn to second-guess.
04 Degrading gracefully
A wall display that shows an error page is worse than no wall display, because nobody notices it has stopped being true.
Three render paths exist. The full WebGL scene when the browser supports it; a plain HUD rendering the same data as panels when a script fails or the scene has not reported ready within 4.5 seconds; and optionally a server-side offscreen render on the host GPU, for clients that cannot run WebGL at all.
The same principle governs configuration. Every backend except the hypervisor is optional — leave its URL unset and that layer is simply absent. The whole thing runs against nothing but the hypervisor and still draws the estate.
05 Seven planets out of sixty-four
Pointed at two hypervisors that were not clustered together, it first showed only 7 of 64 guests. The cause is an API detail that is easy to miss: /cluster/resources on a non-clustered node reports only itself, and guest IDs are not unique across standalone hypervisors, so a naive merge keyed on ID silently overwrote one node's guests with the other's.
A correct merge has to key on node and ID together. The lesson is broader than this display: an inventory that looks complete is the hardest kind of wrong to notice, which is the same failure this whole practice exists to catch.
06 Taking an internal tool public
Internal tools grow the way internal tools do: addresses and credentials inline, panels wired to services only one estate has, a utility carrying a live API token.
Opening it meant moving every setting into an environment-driven configuration module, replacing site-specific panels with generic equivalents, making the GPU telemetry and the offscreen renderer optional imports so it runs on hosts without an accelerator, and adding a startup preflight that reports exactly what configuration is missing instead of failing deep inside a poller.
There are no credentials and no private addresses anywhere in the application code. That work is part of any handover here, not a one-off.