PopcornBench

PopcornBench is an extension of KernelBench with a multi-model agentic sweep harness for evaluating how well LLMs can write fast, numerically equivalent CUDA / Triton / Tilelang kernels against a PyTorch reference. Each run pairs a model with a level and problem set, drives a tool-using agent loop with compile / correctness / submit tools, and records per-problem speedups, fusion ratios, energy, and roofline metrics.

Plan

We score each correct kernel on three signals and compare submissions on the Pareto frontier across all three:

• Runtime. Wall-clock per call relative to the PyTorch reference. Reported as speedup = ref_runtime / runtime.
• SOL. max(dram_utilization, compute_utilization) / 100 from Nsight counters, bounded in [0, 1]. Heuristic fallback when ncu is unavailable.
• Energy. Joules per call from NVML (energy_per_run_mj). Meaningful only when per-call runtime exceeds the NVML sampling window (~10 ms), so reported on L3 and L4 only.

A kernel is considered to dominate another only if it is at least as good on all three signals and strictly better on one. Models are then ranked by the size of their Pareto-optimal subset within each (level, variant).

Eval architecture

The harness decouples agent oversubscription from GPU contention via a per-GPU FIFO eval queue. Each GPU runs one dedicated eval-server process that drains a Manager-backed queue in arrival order; agents push submit_kernel requests and block on the response while their next-turn LLM call is free to run in parallel for other problems. With agents_per_gpu at 4-6 (vs. the previous 1-3) the GPU stays busy without locks starving any agent past worker_timeout_s.

Multi-GPU communication kernels (NCCL collectives, real tensor parallelism, pipeline parallelism — six problems in L2 popcorn) need all GPUs visible to one worker, which is incompatible with per-GPU pinning, so they are skipped from the broad sweeps and run separately via sweep.comm.toml.

Sweep matrix

Two variants (original, popcorn), four levels (L1-L4), three tool tiers (single = no tools, default, all). Each sweep covers both variants. We split L1+L2 from L3+L4 because the heavy levels need fewer perf trials and lower agent oversubscription to stay inside worker timeouts. Plus a separate comm sweep for the multi-GPU L2 popcorn problems and two AlphaEvolve focus runs.

Run order

Each sweep is GPU-bound and contests the full 8xH100 box, so the primary sweeps run sequentially. The aeproxy server and the gh-pages publisher run as background daemons throughout. Each step runs in a detached tmux session so SSH disconnects don't kill it.

Total wall-clock for all 10 steps is roughly 50-65 hours. If time is tight, the priority order to drop is: aeorig (step 10), l34_all (step 7, the heaviest cell), l12_all (step 4), comm (step 8). Single-turn baselines and default-tier sweeps are most important since they ground the comparison.

Planned figures

1. Pareto frontier in (speedup, SOL). One panel per level. Source: l12_default + l34_default.
2. Pareto frontier in (speedup, 1 / energy_ratio). L3 and L4 only. Source: l34_default.
3. Speedup vs SOL scatter, all problems. One dot per (model, problem), colored by level, faceted by variant. Source: l12_default + l34_default.
4. Tier comparison: single vs default vs all. Per (model, level), correctness rate and speedup distribution across the three tool tiers. Source: l12_* + l34_*.
5. Correctness rate stacked by tier on L2-popcorn. Source: existing cuda_l2pop_* sweeps.
6. SOL distribution by tier on L2-popcorn. Source: existing L2-pop sweeps after the SOL clamp fix.
7. AlphaEvolve speedup vs best non-AE speedup, per problem. Source: ae_focus_popcorn + ae_focus_original joined to l12_default / l34_default.
8. AlphaEvolve original-vs-popcorn paired bars, L1 and L2 only. Source: ae_focus_popcorn + ae_focus_original, restricted to problems 1, 2, 10.
9. Failure-mode breakdown per (model, level). Stacked correct / wrong / compile_fail / timeout / error. Source: l12_default + l34_default.
10. Communication kernels (separate panel). Per-model speedup on the L2 popcorn comm subset, reported separately because they run in serial multi-GPU mode. Source: comm_l2_popcorn.