Painless, Reproducible Evaluation of LLM-Generated GPU Kernels

Evaluating LLM-Generated GPU Kernels Is Hard. Isolation Makes It Easier.

Anyone who has evaluated LLM-generated GPU kernels at scale knows the pain: one kernel crashes, corrupts CUDA state, and suddenly nothing after it is trustworthy. Your kernel isn't broken — the one before it was.

We built an evaluation pipeline using union.sandbox on Flyte that gives each kernel its own isolated process — clean GPU state, no contamination from previous runs. To stress-test it, we ran 1,000 kernels from Makora's triton-gpu-latency dataset and found that evaluation results can be surprisingly sensitive to the environment they run in.

The Problem: Shared Process, Shared Fate

The standard approach for evaluating GPU kernels is to run them sequentially in a single Python process. This is fast but fragile:

1. CUDA context corruption — A kernel that exhausts shared memory or triggers a device error can leave the CUDA context in a broken state. Subsequent kernels fail even if their code is perfectly valid.
2. Memory allocator state — PyTorch's caching allocator retains memory across kernel runs. A kernel that fragments GPU memory can cause the next kernel to OOM despite having enough total memory available.
3. Process-level crashes — Some malformed kernels cause segfaults or heap corruption (KernelBench#147). In a shared process, this kills the entire evaluation pipeline.

This is a well-known problem in the kernel generation community. Anyone running hundreds or thousands of untrusted Triton kernels has hit it — a crash in kernel N makes everything after it untrustworthy.

The Fix: One Kernel, One Sandbox

We built a simple evaluation pipeline where each kernel gets:

• Its own subprocess via `union.sandbox` with bubblewrap isolation
• Clean GPU state — no carried-over CUDA context, no stale allocator caches
• Network and filesystem isolation — the kernel can't interfere with anything outside its sandbox
• Timeout enforcement — hangs are killed, not waited on forever

The pipeline fans out across GPU pods using `flyte.map`, so thousands of kernels can evaluate in parallel:

# Extract kernels from the dataset
python scripts/extract_makora_kernels.py --n-pass 0 --n-fail 1000 --out scripts/makora_sample.json

# Run on Flyte with sandbox isolation
flyte run examples/eval_kernelbench_on_union.py eval_kernelbench \
    --kernels_file scripts/makora_sample.json \
    --gpu "L40s:1" --num_gpus 4

Each kernel goes through a three-gate evaluation:

1. Compile — `exec()` the kernel file, find `Model`, `ModelNew`, `get_inputs`
2. Verify — run both models on the same inputs, assert `torch.allclose` (atol=1e-2)
3. Benchmark — time `ModelNew` with CUDA events (5 warmup, 20 timed iterations)

Stress-Testing With Real Data

To see how much the evaluation environment matters, we took 1,000 kernels labeled as failed (`y=NaN`) from Makora's triton-gpu-latency dataset — a collection of 544,000 LLM-generated Triton kernels in the KernelBench format. Each entry has a reference PyTorch `Model`, a candidate Triton `ModelNew`, and a latency label `y` (float for measured latency, `NaN` for failures).

We re-evaluated these 1,000 "failed" kernels on L40s GPUs with full process isolation:

259 kernels compiled, passed numerical verification against the PyTorch reference, and benchmarked successfully. There are many possible reasons for the difference — different hardware, different software stack, different evaluation methodology — and we can't isolate the cause since the original benchmarking code isn't open sourced. The point isn't about any particular dataset. It's that kernel evaluation results are sensitive to the environment they run in, and isolation removes a major source of variance.

The other 741 are genuinely broken: bad Triton code, configs that exceed hardware shared memory limits, or kernels that produce numerically wrong outputs. Isolation doesn't fix broken code — but it stops broken code from breaking everything else. 2 kernels crashed hard enough to take down their sandbox, but the pipeline contained them and kept going.

Validating Our Pipeline

To make sure our pipeline itself is producing valid results, we ran a separate validation: 10 kernels (5 labeled pass, 5 labeled fail) through the same pipeline. The passing kernels' latencies closely matched the dataset's labels despite running on different hardware:

Same order of magnitude, same relative ranking. This gives us confidence that the evaluation pipeline is producing valid, comparable results.

Why This Matters: Better Labels, Better Models

Datasets like these aren't just benchmarks — they're training data. Teams use kernel evaluation results to finetune LLMs for code generation, build reward models for RL-based kernel optimization, and train performance predictors. The quality of those downstream models depends on the quality of the labels.

Evaluation environments that can introduce noise into pass/fail labels create real downstream problems:

• RL reward signals — A reward model trained on noisy labels will penalize valid kernel patterns. The LLM learns to avoid configs that actually work.
• Finetuning data — Filtering out "failed" kernels from training data throws away valid examples. The model sees less diversity in working Triton patterns.
• Performance predictors — Missing latency values for kernels that actually run means gaps in the latency distribution.

Beyond cleaner labels, isolation also gives you failure categorization for free. A flat `NaN` doesn't tell you whether a kernel had a syntax error, exceeded shared memory limits, or produced numerically wrong outputs. Our pipeline captures the failure stage and error message for every kernel, turning opaque failures into actionable categories.

Process isolation addresses these problems at the source. Every kernel sees the same clean environment — results don't depend on evaluation order, and a crash in kernel N can't corrupt kernel N+1's result.

• Reproducibility — Same kernel, same result, every time. No ordering effects, no shared-state contamination.
• Reliability — A kernel that segfaults or corrupts the heap doesn't take down the pipeline.
• Scalability — `flyte.map` fans out across GPU pods in parallel. Our 1,000-kernel evaluation completed in 1 hour on 4 L40s GPUs.

Reproduce It

The evaluation pipeline is open source here. Try it on your own kernels or on the Makora dataset:

# Clone the repo
git clone https://github.com/unionai/tuney.git
cd tuney

# Extract sample kernels from the Makora dataset
pip install datasets
python scripts/extract_makora_kernels.py --n-pass 0 --n-fail 1000 --out makora_fail_kernels.json

# Run locally on a GPU host with sandbox isolation
python examples/eval_kernelbench.py --kernels scripts/makora_fail_kernels.json

# Run on Flyte — fans out across GPU pods with sandbox isolation
flyte run examples/eval_kernelbench_on_union.py eval_kernelbench \\
    --kernels_file scripts/makora_fail_kernels.json \\
    --gpu "L40s:1" --num_gpus 4