AI DATA CENTER SIMULATION
00:00:00 / 10:00:00
← EXIT

AI Data Center — GPU Cluster Training Screen

This page is the operations screen of an AI supercomputer. In the middle, a camera orbits a compute hall: two rings of GPU racks in four pods, with the model being trained floating above them as a glowing lattice of layers. Each training step sends a forward pass up the lattice and a wave of gradients back down, then the ring of racks synchronises in an all-reduce. Around the hall, panels track the training run, GPU temperatures, power and cooling, models in service, and a scrolling event stream. A single training run fills the whole ten-hour cycle, so the loss curve falls from the first minute to the last. About every five minutes something goes wrong, the camera dives onto the failing rack, and the incident is worked through to recovery. Click ⛶ FULLSCREEN and it fills the screen.

Is any of it real?

No. It is an animation, and both the bar under it and the corner of the screen say SIMULATION. The company, the compute hall, the model called HELIX-70B, the rack and incident numbers, the deployments, and every line of the event stream are invented and drawn in your browser. The clock in the header is an invented operations time, not your local time. The page makes no connections while it runs, trains nothing, and shows no real telemetry.

The things it depicts are real, though: GPU clusters, data, tensor, and pipeline parallelism, all-reduce, loss curves, checkpoints, and the steady stream of hardware faults that large training runs have to survive. The section on how large AI models are trained below explains them.

What's on the screen

What happens

The ten hours are one training run. Training advances one step about every 2.6 seconds and reaches step 12,590 at the end of the cycle, having seen about 78.6 billion tokens. The loss starts at about 4.7 and finishes at about 1.64. The learning rate warms up over the first few minutes and then eases down along a cosine curve.

The cycle is divided into 120 five-minute slots, and each slot holds one incident somewhere inside it. Every incident follows the same four stages:

  1. Detected. The alert fires. The header status changes, the red tint appears at the edges of the screen, sparks rise from the failing rack, a red beam rises from it (amber once recovery starts), and the camera dives in from its orbit and zooms onto it (CAMERA · FAULT LOCK).
  2. Diagnosing. About the first third of the incident. If the fault stops training, the step counter freezes, the lattice goes still, data stops moving, tokens per second and GPU utilisation fall, and power draw drops to idle.
  3. Recovering. A stream of light runs from the checkpoint store at the centre to the rack, and a ring around the rack fills as recovery progresses. Training resumes partway through this stage.
  4. Resolved. A green ring spreads out from the rack, the panel turns green, and about 15 seconds later the camera pulls back to its orbit.

The seven incident types, and how many of each fall in the ten hours:

IncidentCountOn screenTraining
Node failure24GPU fell off the bus; restoring from checkpointPaused
GPU thermal throttle22Junction temperature at 94 °C; raising coolant flowSlowed
Interconnect fault20Link errors on the fabric; rerouting the all-reduce ringPaused
Power cap event18Feed B at 98% capacity; shifting load to feed AContinues
Memory ECC errors16Uncorrectable ECC errors; swapping in a spare nodePaused
Loss spike11Gradient norm 40× normal; rolling back 200 stepsPaused
Inference surge9Traffic at 3× baseline; scaling out replicasContinues

Moments worth seeing in the first part of the cycle (seek with the bar or use ?t=):

TimeSecondsWhat happens
00:00:000Boot-up: the panels wipe on, the racks rise from the floor, and the lattice lights up
00:00:1212Node failure on rack B-04, node 4; training pauses and the camera dives in
00:00:2828Restoring from checkpoint: the restore stream runs to the rack until about 0:44
00:06:48408Power cap event on rack A-05; training carries on
00:10:36636GPU thermal throttle on rack C-05; the heat map and coolant figures climb
00:17:411061Interconnect fault on rack D-05; training pauses while the ring is rerouted
00:27:391659Memory ECC errors on rack C-04; a spare node is swapped in
00:30:001800The first checkpoint diamond appears on the loss chart
01:27:505270First loss spike: the loss jumps on the chart and training pauses
03:21:2112081First inference surge: traffic triples and response times climb

Over the full ten-hour cycle there are 120 incidents, 71 of which pause training. Incidents last between 46 seconds and about two minutes, with a mean time to recovery of 1 minute 30 seconds. Training is paused for about 54 minutes in total, which is why the screen gives a goodput of 90.9%.

How to use it

The page draws at the full resolution of your screen, up to 4K. On a slow computer it lowers the resolution by itself to keep the motion smooth.

Keyboard shortcuts

Good uses

Background screens for films, series, and adverts set in an AI lab, a tech company, or a near-future control room; a wall display for a talk, lecture, or meetup about machine learning; a stream backdrop for coding, AI, or hardware channels; an ambient screen for an office, a hackathon, or a server-room-themed event; and a prop for escape rooms and tabletop games with a supercomputer in the story. It is calm for long stretches and busy for a minute or two every five, so it suits a screen that should look like work is going on. For OBS setup as a browser source, see the streaming toolkit.

How large AI models are trained

A large language model is a very long list of numbers, called parameters or weights, arranged in layers. Training sets those numbers by showing the model text and asking it to predict the next piece, the next token, again and again. Each training step has the same parts. In the forward pass, a batch of text goes through the layers from bottom to top and the model makes its predictions. The predictions are compared with the real text to give the loss. In the backward pass, the error is traced back down through the layers to work out the gradient: how much each weight should move to make the loss smaller. An optimiser then nudges every weight a little in that direction. The lattice on this screen draws those first two parts as the green wave going up and the lime wave coming down.

Why it takes thousands of GPUs

GPUs are good at this work because a step is mostly the multiplication of large tables of numbers, which they do in parallel. But a big model does not fit on one. A model with 70 billion parameters needs about 140 GB just to hold its weights at 16-bit precision, and training also keeps gradients and optimiser state that take several times as much again. And the amount of text involved, often trillions of tokens, would take a single GPU far too long. So the work is spread across thousands of GPUs at once.

GPUs are grouped into nodes, typically eight to a server, linked inside the node by very fast connections. Nodes sit in racks, and racks are tied together by a high-bandwidth network called the fabric. AI racks draw several times the power of an ordinary server rack, often tens of kilowatts each, so many are cooled by liquid piped to the chips rather than by air alone.

Splitting the work

There are three main ways to divide a training job, and large runs combine all three. The model label on this screen, TP8 × PP8 × DP40, is one such combination: 8 × 8 × 40 = 2,560 GPUs, exactly the size of the hall.

All-reduce

With data parallelism, every copy of the model computes its own gradients from its own data. Before the weights can be updated, those gradients have to be averaged across all the copies, so that every copy makes the same change. The operation that does this is called an all-reduce. A common way to run it is a ring: each GPU passes a slice of its gradients to its neighbour, adds what it receives, and passes it on, until every GPU holds the full total. The useful property of the ring is that each GPU sends about twice the size of its gradients however many GPUs are in the ring, so it scales well. On this screen the green trails hopping around the inner ring are all-reduce traffic, and the ring of light on the floor at the end of each step is the all-reduce completing. Real systems overlap much of this communication with computation so the GPUs are not left waiting.

Reading a loss curve

The loss is usually the cross-entropy of the model's predictions: a measure of how surprised the model is by the real next token. It falls fast early in training, as the model picks up common words and grammar, and then more and more slowly as the remaining gains get harder. That is the shape drawn in the training panel. The learning rate, the size of each nudge to the weights, is usually ramped up over a short warm-up and then lowered gradually; this screen warms up over the first 3% of the run and then follows a cosine curve down to a tenth of its peak.

Sometimes the loss jumps suddenly, a loss spike, when the training becomes numerically unstable. Published accounts of large runs describe responses such as going back to an earlier checkpoint, skipping the batches that triggered it, or lowering the learning rate. The screen's loss-spike incidents show the first of these.

Checkpoints and failures

A checkpoint is a saved copy of everything needed to resume training: the weights, the optimiser state, and where the run had got to in the data. For a large model it is split into many shards written in parallel to fast storage; the event stream here reports shards out of 64.

Checkpoints matter because, at this scale, hardware fails all the time. A run on thousands of GPUs depends on all of them, their memory, the cables between them, and the power and cooling around them. Each part is reliable on its own, but with tens of thousands of parts something goes wrong often. One published account of a run on about 16,000 GPUs recorded over 400 interruptions in under two months, most of them put down to hardware. Because the GPUs move in lockstep, one failure stops the whole job: it has to be detected, the bad node replaced with a spare, and the job restarted from the last checkpoint, losing whatever was done since. Operators track goodput, the share of time spent on useful training, and work to raise it with health checks that catch failing parts early, spare nodes kept ready, and faster checkpointing.

Power, cooling, and serving

A hall like this one draws megawatts. Power usage effectiveness (PUE) compares the total power the building uses with the power that reaches the computers; a value of 1.0 would mean nothing was spent on cooling or losses. Coolant distribution units pump water or another coolant through cold plates on the chips; the facility panel shows the supply and return temperatures and the flow. Two independent power feeds, A and B, let equipment keep running if one fails.

Training is only half the job. Once a model is trained it is deployed for inference, answering requests. New versions are usually rolled out as a canary first, to a small share of traffic, before going fully live. Two common measures of how the service is doing are queries per second and time to first token, the delay before a reply starts to appear.

The vocabulary

GPU
Graphics processing unit, the chip that does the arithmetic of training. Other kinds of AI accelerator do the same job.
Node and pod
A node is one server, here with eight GPUs. A pod is a group of racks managed together; this hall has four pods of 640 GPUs.
Token
A small piece of text, often part of a word, that the model reads and predicts. Training throughput is measured in tokens per second.
Step
One round of forward pass, backward pass, all-reduce, and weight update on one batch of data.
Gradient
For each weight, the direction and size of the change that would reduce the loss. A gradient norm far above normal is a warning sign of instability.
Checkpoint
A saved snapshot of the run that it can be restarted from.
Goodput
The share of time a cluster spends on useful training, after failures, restarts, and lost work are taken out.
ECC
Error-correcting code memory, which detects and fixes small memory errors. An uncorrectable error means the memory can no longer be trusted, and the node is taken out of service.
Thermal throttling
A chip slowing itself down to protect itself when it gets too hot.
TTFT P95
Time to first token at the 95th percentile: 95% of requests start their reply faster than this.

How the simulation works

The whole ten-hour cycle is worked out in advance from a fixed random seed: when each incident starts, how long it lasts, which rack and node it hits, and what kind it is. Training progress is the time elapsed minus the time spent paused by earlier incidents, so the step counter, token count, and loss all freeze during a stall and pick up afterwards. The loss is a smooth falling curve with a little noise added. The hall is drawn with a simple 3D projection: racks are boxes, the lattice is a set of rotating rings, and packets follow paths along the floor, all blended so that overlapping light glows. The camera makes a quarter turn every 90 seconds, a full orbit every six minutes, and moves in to lock onto a failing rack. Because every frame is a function of the time alone, the page can jump anywhere in the cycle at once, and the same moment always looks the same.

Frequently asked questions

Is this a real AI data centre?

No. It is an animation. The compute hall, the HELIX-70B model, the company, the racks, the incidents, and every line in the event stream are invented and drawn in your browser. The page connects to nothing, trains nothing, and shows no real telemetry.

What does the loss curve show?

Loss measures how wrong the model's predictions are on its training data, so lower is better. On this screen it falls from about 4.7 at the start of the cycle to about 1.64 at the end: steeply at first, then more and more slowly. The dashed line is the projected path. A loss spike pushes it up for a minute or two, as can happen in real training runs.

Why does training stop when one GPU fails?

Because the GPUs work in lockstep. Every training step needs results from all of them, so if one node drops out the rest have to wait. Real clusters handle this much as the screen shows: the faulty node is swapped for a spare, the job reloads the last checkpoint, and training carries on.

How long does it run before it repeats?

Ten hours. The whole cycle is one training run of 12,590 steps, with 120 incidents along the way, about one every five minutes. After ten hours the hall boots up again and the run starts from step zero. The screen is calculated frame by frame rather than played from a recording, so it needs no download and never buffers.

Can I jump to a particular moment?

Yes. Drag the seek bar, or press the left and right arrow keys to move ten seconds at a time. To open the page at a given moment, add ?t= to the address with the number of seconds, for example ?t=5270 for the first loss spike.

Can I use it in a video or on a stream?

Yes. It is free to use for videos, films, streams, music visuals, presentations, and events. Click FULLSCREEN, or add the page as an OBS browser source at 1920 × 1080 or 3840 × 2160; the scene is drawn at the full resolution of the source.

Does it flash?

No. There are no white or full-screen flashes and no strobing. While an incident is active, a red tint at the edges of the screen pulses gently, once every three seconds, and fades away as the incident recovers. A thin ring on the floor of the hall glows briefly at the end of every training step, about every 2.6 seconds. If your system is set to reduce motion, the page opens paused. See accessibility and visual effects for more.

Does it work offline?

Once the page and its fonts have loaded, the simulation makes no further network requests, so it keeps running if the connection drops. The optional VIDEO view streams from YouTube and needs a connection.

Related tools

The data centre simulator shows a whole isometric server city under load, and the SOC screen is the security operations centre that watches over one. The tech dashboard is a quieter wall of server and network metrics, and the reactor control room is another control screen with alarms to work through. For a rogue AI taking over a computer, as a prank rather than a training run, see the AI hacker screen.