Are they busy? Are they healthy? Are they worth what you pay? Union shows GPU utilization, health and cost, from the whole fleet down to a single GPU in a single run.
Try for free
GPUs
What’s running on each GPU in this cluster, and which nodes need attention.
To your scheduler, these three GPUs look identical. One is throttling at 86 °C, one just threw Xid 79 and one is mostly idle. Each costs the same per hour.
Meta’s Llama 3 team logged 419 unexpected interruptions in 54 days of training on up to 16,000 GPUs. GPU issues caused more than half.
Source: The Llama 3 Herd of Models
Union tracks utilization, tensor activity, clocks and temperature for each GPU in a run. On a multi-node job, every node shares the same charts. When one node throttles, the other three spend the rest of the run waiting on it, and the clocks show which one.
import flyte
from flyte.clustered import ClusteredTaskEnvironment
env = ClusteredTaskEnvironment(
name="llama3-pretrain",
resources=flyte.Resources(gpu="H100:8"),
replicas=4,
nproc_per_node=8,
)Click a pod to show only its GPUs in all three charts. Click it again to show all pods.
A throttling node slows a run. A failing one ends it. When a GPU throws a hardware Xid, Union cordons its node and moves your run to a healthy GPU, without using up a retry. Nothing to change in your code.
A health agent on every GPU node reads NVIDIA Xid errors and separates hardware faults from errors in your code.
Union cordons the node as soon as the fault shows up. Jobs on healthy GPUs finish, and your team decides when the node comes back. Quarantine pulls only a few nodes at a time, so a bad driver update can’t take down your cluster.
The fault is recorded as a system error, so your retry budget stays intact, and the next attempt lands on a healthy GPU. It also covers distributed PyTorch jobs and Ray clusters started for a run.
Cloud node auto-repair
Union
GPU-hours, utilization and unused spend for each project, run and task. The hours three nodes spent waiting on a hot one show up here as unused spend, and so does a sandbox that keeps its GPUs mostly idle.
| Project | Cost | Unused | GPU hours | GPU utilization |
|---|---|---|---|---|
| llm-pretrain | $141,696 | $17,004 | 29,520 | 88% |
| fine-tuning | $44,928 | $8,536 | 9,360 | 81% |
| evals | $28,224 | $10,161 | 5,880 | 64% |
| research-sandbox | $20,448 | $15,745 | 4,260 | 23% |
Jobs in research-sandbox keep their GPUs about a quarter busy. Give them a MIG slice in one line, and the rest of each GPU goes back to the pool.
import flyte
env = flyte.TaskEnvironment(
"research-sandbox",
resources=flyte.Resources(
gpu=flyte.GPU("H100", 1, partition="1g.10gb"),
),
)Run metrics, multi-node charts and MIG requests work today. The GPUs view, cost by project and fault recovery are coming soon.