Predicts failing GPUs and network links in training clusters and drains them before a job crashes
On a 4,000-GPU training run, one failing GPU, memory error or flapping network link can crash the whole job, and teams lose hours rolling back to the last checkpoint. Operators at that scale report interruptions every few hours, and the warning signs, rising ECC error counts, thermal throttling, link retries, sit unread in separate logs.
Wardline runs a lightweight agent on every node that streams hardware telemetry to a model trained on more than 9,000 recorded failures. When a node's failure risk crosses a threshold, it triggers a checkpoint, cordons the node in the scheduler and swaps in a spare, so the job carries on instead of restarting from an old checkpoint.
AI labs, GPU cloud providers and enterprises running clusters above 500 GPUs; roughly 1,200 such operators worldwide, managing an estimated 6M accelerators by 2027.
Subscription priced per monitored GPU, about US$12 per GPU per month, with discounts above 10,000 GPUs and an on-premise edition for air-gapped sites.
Jakob ReinhardtCEO and Co-founder
Mira HoffmannCTO and Co-founder
Emre YildizHead of Machine Learning
Margaux DuboisVP Customer SuccessWardline is not trying to train models; it keeps other people's training runs alive. The rack-level finding in the fleet view, four risky nodes sharing one overheating coolant loop, is the kind of thing an operator would otherwise spot only after a failure. The interface is dense and clearly built for SREs rather than managers.
Anyone who has babysat a large training run knows the 3 a.m. crash and the rollback to a checkpoint from hours earlier. Wardline's incident timeline showed a 41-minute warning from rising ECC errors and a clean swap to a spare node. Network failures still get only about 12 minutes of warning, which the team openly acknowledges.
Customers can put a number on the value: 61,000 GPU-hours recovered over 90 days translates directly into money. Mira's team has the largest labelled set of hardware failures I have seen at this stage. The risk is that hardware vendors build similar prediction into their own tooling, so the switch and storage roadmap matters.
We run a 640-GPU cluster for fraud and risk models, on older hardware than most Wardline customers. Getting the node agent and the Slurm integration working took three weeks, and for the first month older nodes produced a steady stream of false alarms my team learned to ignore. After per-generation retuning it has caught two real failures, but at US$12 per GPU the value is thin at our scale.
Reply from Jakob Reinhardt, CEO
Agreed on the setup time, Lucas. Per-generation baselines now ship on day one, and the new Slurm installer takes about a day.
Coming soon. Testimonials are invited by the founder and shown separately from open reviews.
Coming soon. Follow Wardline to hear about product and funding updates when this launches.
