Union.ai achieves 9.8x ROI according to analysts.

View the report
GPU-utilization study · FSG-26-04

Reuse or Reload

A GPU-utilization study of batch LLM inference: keep the worker warm, or spin a fresh container per call.

Qwen2.5-7B on one NVIDIA L4 | 500 GSM8K questions, ten 50-question chunks | measured with nvidia-smi every 2 s
AbstractServing large-model batch inference under an orchestrator forces a choice: keep a GPU worker warm across many calls, or spin a fresh container per call. We run one identical vLLM workload — 500 GSM8K questions through Qwen2.5-7B on an L4 — two ways. With container reuse on a managed Union cluster the job finishes in 404 s; a no-reuse variant on open-source Flyte takes 1,665 s — 4.1× slower — and, measured with nvidia-smi, keeps the GPU actually computing only 16% of the run while a model sits resident 86% of it, where the reuse GPU, warm, holds a steady 100%.

Read the full paper (PDF) ↗

4.1×
slower without container reuse (1,665s vs 404s)
100%
sustained GPU utilization with reuse, once warm
10
model loads without reuse — vs. just 2 with it
3.3×
higher peak decode throughput with reuse
Key results

What the numbers show

Every chart below is drawn straight from the measured data in the paper — hover any bar or point for the exact value.

500 GSM8K questions — total wall-clockFig. 1 · wall-clock
0 s 463 s 925 s 1,388 s 1,850 s Union · reuse 404 s Flyte · no-reuse 1,665 s

Loading the model twice instead of ten times, and packing concurrent callers into 256-prompt batches rather than isolated 50-prompt calls, completes the identical job 4.1× faster.

Compute utilization + GPU memory over the runFig. 2a · Union · reuse

compute util GPU memory

One model load, then a steady 100% through batched inference — mean utilization over the whole run is 35%, rising toward 100% as more work amortizes that single load.

Compute utilization + GPU memory over the runFig. 2b · Flyte · no-reuse

compute util GPU memory

Each chunk loads its own copy of the weights, fires one ~33-second 100% burst, then releases the GPU — ten loads for ten chunks. Integrated, Flyte computes for only 16% of the run while holding a model resident 86% of it: almost always occupied, almost never working.

Same workload, two execution modelsTable 1
Metric Union · reuse Flyte · no-reuse
Wall-clock, 500 questions 404 s 1,665 s (4.1×)
Model loads for the job 2 10
Cold start, amortized / question ~0.5 s ~3.7 s
Effective batch size up to 256 50
Peak decode throughput 1,424 tok/s 433 tok/s
Time computing (util > 50%) 35% 16.1%
Time holding a model, idle 61% 86.5%
Flyte · no-reuse  1,665 s

See it for yourself

Spin up a real Flyte cluster in minutes and run the same workloads on your own hardware. No infra to provision, no YAML to write first.

Try the Devbox