Your system is set to reduce motion, so the compute hall is paused. This screen has a slowly orbiting camera, moving data trails, and a gentle red pulse at the edges of the screen during incidents.
This page is the operations screen of an AI supercomputer. In the middle, a camera orbits a compute hall: two rings of GPU racks in four pods, with the model being trained floating above them as a glowing lattice of layers. Each training step sends a forward pass up the lattice and a wave of gradients back down, then the ring of racks synchronises in an all-reduce. Around the hall, panels track the training run, GPU temperatures, power and cooling, models in service, and a scrolling event stream. A single training run fills the whole ten-hour cycle, so the loss curve falls from the first minute to the last. About every five minutes something goes wrong, the camera dives onto the failing rack, and the incident is worked through to recovery. Click ⛶ FULLSCREEN and it fills the screen.
No. It is an animation, and both the bar under it and the corner of the screen say SIMULATION. The company, the compute hall, the model called HELIX-70B, the rack and incident numbers, the deployments, and every line of the event stream are invented and drawn in your browser. The clock in the header is an invented operations time, not your local time. The page makes no connections while it runs, trains nothing, and shows no real telemetry.
The things it depicts are real, though: GPU clusters, data, tensor, and pipeline parallelism, all-reduce, loss curves, checkpoints, and the steady stream of hardware faults that large training runs have to survive. The section on how large AI models are trained below explains them.
The ten hours are one training run. Training advances one step about every 2.6 seconds and reaches step 12,590 at the end of the cycle, having seen about 78.6 billion tokens. The loss starts at about 4.7 and finishes at about 1.64. The learning rate warms up over the first few minutes and then eases down along a cosine curve.
The cycle is divided into 120 five-minute slots, and each slot holds one incident somewhere inside it. Every incident follows the same four stages:
The seven incident types, and how many of each fall in the ten hours:
| Incident | Count | On screen | Training |
|---|---|---|---|
| Node failure | 24 | GPU fell off the bus; restoring from checkpoint | Paused |
| GPU thermal throttle | 22 | Junction temperature at 94 °C; raising coolant flow | Slowed |
| Interconnect fault | 20 | Link errors on the fabric; rerouting the all-reduce ring | Paused |
| Power cap event | 18 | Feed B at 98% capacity; shifting load to feed A | Continues |
| Memory ECC errors | 16 | Uncorrectable ECC errors; swapping in a spare node | Paused |
| Loss spike | 11 | Gradient norm 40× normal; rolling back 200 steps | Paused |
| Inference surge | 9 | Traffic at 3× baseline; scaling out replicas | Continues |
Moments worth seeing in the first part of the cycle (seek with the bar or use ?t=):
| Time | Seconds | What happens |
|---|---|---|
| 00:00:00 | 0 | Boot-up: the panels wipe on, the racks rise from the floor, and the lattice lights up |
| 00:00:12 | 12 | Node failure on rack B-04, node 4; training pauses and the camera dives in |
| 00:00:28 | 28 | Restoring from checkpoint: the restore stream runs to the rack until about 0:44 |
| 00:06:48 | 408 | Power cap event on rack A-05; training carries on |
| 00:10:36 | 636 | GPU thermal throttle on rack C-05; the heat map and coolant figures climb |
| 00:17:41 | 1061 | Interconnect fault on rack D-05; training pauses while the ring is rerouted |
| 00:27:39 | 1659 | Memory ECC errors on rack C-04; a spare node is swapped in |
| 00:30:00 | 1800 | The first checkpoint diamond appears on the loss chart |
| 01:27:50 | 5270 | First loss spike: the loss jumps on the chart and training pauses |
| 03:21:21 | 12081 | First inference surge: traffic triples and response times climb |
Over the full ten-hour cycle there are 120 incidents, 71 of which pause training. Incidents last between 46 seconds and about two minutes, with a mean time to recovery of 1 minute 30 seconds. Training is paused for about 54 minutes in total, which is why the screen gives a goodput of 90.9%.
?t= and the number of seconds to the address, for example /ai-data-center?t=636 for the first thermal throttle. Minutes and hours also work: ?t=27m39s or ?t=1h27m50s.The page draws at the full resolution of your screen, up to 4K. On a slow computer it lowers the resolution by itself to keep the motion smooth.
Background screens for films, series, and adverts set in an AI lab, a tech company, or a near-future control room; a wall display for a talk, lecture, or meetup about machine learning; a stream backdrop for coding, AI, or hardware channels; an ambient screen for an office, a hackathon, or a server-room-themed event; and a prop for escape rooms and tabletop games with a supercomputer in the story. It is calm for long stretches and busy for a minute or two every five, so it suits a screen that should look like work is going on. For OBS setup as a browser source, see the streaming toolkit.
A large language model is a very long list of numbers, called parameters or weights, arranged in layers. Training sets those numbers by showing the model text and asking it to predict the next piece, the next token, again and again. Each training step has the same parts. In the forward pass, a batch of text goes through the layers from bottom to top and the model makes its predictions. The predictions are compared with the real text to give the loss. In the backward pass, the error is traced back down through the layers to work out the gradient: how much each weight should move to make the loss smaller. An optimiser then nudges every weight a little in that direction. The lattice on this screen draws those first two parts as the green wave going up and the lime wave coming down.
GPUs are good at this work because a step is mostly the multiplication of large tables of numbers, which they do in parallel. But a big model does not fit on one. A model with 70 billion parameters needs about 140 GB just to hold its weights at 16-bit precision, and training also keeps gradients and optimiser state that take several times as much again. And the amount of text involved, often trillions of tokens, would take a single GPU far too long. So the work is spread across thousands of GPUs at once.
GPUs are grouped into nodes, typically eight to a server, linked inside the node by very fast connections. Nodes sit in racks, and racks are tied together by a high-bandwidth network called the fabric. AI racks draw several times the power of an ordinary server rack, often tens of kilowatts each, so many are cooled by liquid piped to the chips rather than by air alone.
There are three main ways to divide a training job, and large runs combine all three. The model label on this screen, TP8 × PP8 × DP40, is one such combination: 8 × 8 × 40 = 2,560 GPUs, exactly the size of the hall.
With data parallelism, every copy of the model computes its own gradients from its own data. Before the weights can be updated, those gradients have to be averaged across all the copies, so that every copy makes the same change. The operation that does this is called an all-reduce. A common way to run it is a ring: each GPU passes a slice of its gradients to its neighbour, adds what it receives, and passes it on, until every GPU holds the full total. The useful property of the ring is that each GPU sends about twice the size of its gradients however many GPUs are in the ring, so it scales well. On this screen the green trails hopping around the inner ring are all-reduce traffic, and the ring of light on the floor at the end of each step is the all-reduce completing. Real systems overlap much of this communication with computation so the GPUs are not left waiting.
The loss is usually the cross-entropy of the model's predictions: a measure of how surprised the model is by the real next token. It falls fast early in training, as the model picks up common words and grammar, and then more and more slowly as the remaining gains get harder. That is the shape drawn in the training panel. The learning rate, the size of each nudge to the weights, is usually ramped up over a short warm-up and then lowered gradually; this screen warms up over the first 3% of the run and then follows a cosine curve down to a tenth of its peak.
Sometimes the loss jumps suddenly, a loss spike, when the training becomes numerically unstable. Published accounts of large runs describe responses such as going back to an earlier checkpoint, skipping the batches that triggered it, or lowering the learning rate. The screen's loss-spike incidents show the first of these.
A checkpoint is a saved copy of everything needed to resume training: the weights, the optimiser state, and where the run had got to in the data. For a large model it is split into many shards written in parallel to fast storage; the event stream here reports shards out of 64.
Checkpoints matter because, at this scale, hardware fails all the time. A run on thousands of GPUs depends on all of them, their memory, the cables between them, and the power and cooling around them. Each part is reliable on its own, but with tens of thousands of parts something goes wrong often. One published account of a run on about 16,000 GPUs recorded over 400 interruptions in under two months, most of them put down to hardware. Because the GPUs move in lockstep, one failure stops the whole job: it has to be detected, the bad node replaced with a spare, and the job restarted from the last checkpoint, losing whatever was done since. Operators track goodput, the share of time spent on useful training, and work to raise it with health checks that catch failing parts early, spare nodes kept ready, and faster checkpointing.
A hall like this one draws megawatts. Power usage effectiveness (PUE) compares the total power the building uses with the power that reaches the computers; a value of 1.0 would mean nothing was spent on cooling or losses. Coolant distribution units pump water or another coolant through cold plates on the chips; the facility panel shows the supply and return temperatures and the flow. Two independent power feeds, A and B, let equipment keep running if one fails.
Training is only half the job. Once a model is trained it is deployed for inference, answering requests. New versions are usually rolled out as a canary first, to a small share of traffic, before going fully live. Two common measures of how the service is doing are queries per second and time to first token, the delay before a reply starts to appear.
The whole ten-hour cycle is worked out in advance from a fixed random seed: when each incident starts, how long it lasts, which rack and node it hits, and what kind it is. Training progress is the time elapsed minus the time spent paused by earlier incidents, so the step counter, token count, and loss all freeze during a stall and pick up afterwards. The loss is a smooth falling curve with a little noise added. The hall is drawn with a simple 3D projection: racks are boxes, the lattice is a set of rotating rings, and packets follow paths along the floor, all blended so that overlapping light glows. The camera makes a quarter turn every 90 seconds, a full orbit every six minutes, and moves in to lock onto a failing rack. Because every frame is a function of the time alone, the page can jump anywhere in the cycle at once, and the same moment always looks the same.
No. It is an animation. The compute hall, the HELIX-70B model, the company, the racks, the incidents, and every line in the event stream are invented and drawn in your browser. The page connects to nothing, trains nothing, and shows no real telemetry.
Loss measures how wrong the model's predictions are on its training data, so lower is better. On this screen it falls from about 4.7 at the start of the cycle to about 1.64 at the end: steeply at first, then more and more slowly. The dashed line is the projected path. A loss spike pushes it up for a minute or two, as can happen in real training runs.
Because the GPUs work in lockstep. Every training step needs results from all of them, so if one node drops out the rest have to wait. Real clusters handle this much as the screen shows: the faulty node is swapped for a spare, the job reloads the last checkpoint, and training carries on.
Ten hours. The whole cycle is one training run of 12,590 steps, with 120 incidents along the way, about one every five minutes. After ten hours the hall boots up again and the run starts from step zero. The screen is calculated frame by frame rather than played from a recording, so it needs no download and never buffers.
Yes. Drag the seek bar, or press the left and right arrow keys to move ten seconds at a time. To open the page at a given moment, add ?t= to the address with the number of seconds, for example ?t=5270 for the first loss spike.
Yes. It is free to use for videos, films, streams, music visuals, presentations, and events. Click FULLSCREEN, or add the page as an OBS browser source at 1920 × 1080 or 3840 × 2160; the scene is drawn at the full resolution of the source.
No. There are no white or full-screen flashes and no strobing. While an incident is active, a red tint at the edges of the screen pulses gently, once every three seconds, and fades away as the incident recovers. A thin ring on the floor of the hall glows briefly at the end of every training step, about every 2.6 seconds. If your system is set to reduce motion, the page opens paused. See accessibility and visual effects for more.
Once the page and its fonts have loaded, the simulation makes no further network requests, so it keeps running if the connection drops. The optional VIDEO view streams from YouTube and needs a connection.
The data centre simulator shows a whole isometric server city under load, and the SOC screen is the security operations centre that watches over one. The tech dashboard is a quieter wall of server and network metrics, and the reactor control room is another control screen with alarms to work through. For a rogue AI taking over a computer, as a prank rather than a training run, see the AI hacker screen.