PopcornBench docs

Contents

Getting started
Running a sweep
Adding a new model
Reading a report
AlphaEvolve
Troubleshooting

Getting started

The full reference for the sweep harness is in scripts/SWEEP.md in the repo. This page is the short version.

Install

git clone <repo>
cd PopcornBench
uv sync

Credentials

Drop API keys in .env at the repo root. The names referenced from sweep configs:

TEJAS_AZURE_KEY — gpt-5.x family (Azure OpenAI v1 preview).
POPCORN_AZURE_KEY — Foundry-hosted partner models (FW-GLM-5-1).
THAVA_AZURE_KEY — Azure AI Inference (Llama-Maverick, Kimi-K2.6).
XAI_API_KEY — xAI Grok (https://api.x.ai/v1); see configs/sweep.grok.toml.
AEPROXY_API_KEY — only if you're running the AlphaEvolve proxy. See the AlphaEvolve section below.

Sanity check

uv run python scripts/probe_endpoints.py
# Grok / xAI only:
# uv run python scripts/probe_endpoints.py --xai-only

One ping per configured model. Run it before any long sweep so you don't burn an hour on a stale key.

Running a sweep

uv run python scripts/run_sweep.py configs/sweep.example.toml

The runner is resumable: trajectories already on disk under runs/<name>/<variant>/<model>/ are skipped. Re-running the same TOML picks up where you left off.

For a fast smoke test, copy configs/sweep.example.toml to a new file and tighten:

problem_subset    = [1, 19, 23]   # only three problems
num_correct_trials = 3
num_perf_trials   = 20
[agent]
max_turns         = 4
max_tool_calls    = 8

Live report

While the sweep runs, the harness serves the report at http://<host>:8765/index.html. From a laptop:

ssh -L 8765:localhost:8765 you@host
# then: open http://localhost:8765/index.html

Adding a new model

Each model is one [[models]] block in the sweep TOML. Two API shapes are supported today:

`api_kind`	Endpoint	Use for
`openai`	Responses API	gpt-5.x; use `omit_responses_reasoning` only if a gateway rejects `reasoning` kwargs
`openai_chat`	Chat Completions	GLM, Llama-Maverick, Kimi, xAI Grok (see `configs/sweep.grok.toml`), generic servers

Minimal stanza

[[models]]
name        = "my-new-model"
api_kind    = "openai_chat"
base_url    = "https://example.com/openai/v1/"
api_key_env = "MY_API_KEY"
rpm         = 250
tpm         = 250000

Optional knobs

Field	What it does
`deployment_name`	Override the model id sent on the wire (when it differs from `name`).
`reasoning_effort`	Per-model override for `[agent].reasoning_effort`. Only used when `api_kind = "openai"`.
`omit_responses_reasoning`	Omit `reasoning` on `responses.create` for picky gateways. Grok: use `openai_chat`, not this flag.
`max_concurrency`	Hard cap on in-flight LLM calls. Defaults to `floor(tpm / 25000)`.
`request_timeout_s`	Per-request HTTP timeout passed to the OpenAI client. Default is 600s. Bump for slow models (AlphaEvolve, Grok reasoning).

What happens at runtime

The runner spawns one worker per num_gpu_devices × agents_per_gpu slot.
Each worker pulls (model, level, problem) tuples off the work queue.
Per-model concurrency is capped by an mp.Semaphore sized from tpm or max_concurrency.
Per-GPU timing is serialized by an fcntl.flock file lock so submit_kernel measurements aren't taken concurrently on the same device.
The trajectory JSON, the final kernel, and a compile cache land under runs/<name>/<variant>/<model>/.

Reading a report

The report has three levels:

Top-level — one card per variant (original, popcorn, …) with rollup counts.
Per-variant / per-model — grid of problems, color-coded by outcome (correct, incorrect, compile_fail, error, skipped, in_progress).
Per-trajectory — TLDR card on top (outcome, runtime, ref runtime, speedup), then each turn with its LLM latency, expandable reasoning block, assistant text, every tool call, and every tool result. Failed tool calls are expanded by default.

Extended metrics on the trajectory page (memory ratio, kernel-launch fusion ratio, energy per run, SOL score, roofline) come from submit_kernel when those subsystems are available; missing values report -1 rather than blocking the eval.

AlphaEvolve

AlphaEvolve is integrated through a separate proxy server (aeproxy) that lives in the sp26-ae-llm repo. The proxy exposes an OpenAI-compatible HTTP endpoint, so PopcornBench treats AlphaEvolve like any other api_kind = "openai_chat" model.

The AlphaEvolve repo is not public to the group. Whoever has it stands up the proxy on the shared H100 box; everyone else just points their base_url at http://localhost:<port>/v1/. No GCP credentials needed on caller machines.

Bringing up the proxy (proxy host only)

cd /scratch/tejas/sp26-ae-llm
gcloud auth application-default login
uv pip install fastapi uvicorn
uv run python -m aeproxy.server

The proxy auto-loads sp26-ae-llm/.env on startup and falls back to example.env for any unset values, so the GCP project / engine / assistant fields don't need to be exported by hand. POPCORNBENCH_PATH defaults to /scratch/tejas/PopcornBench (used to import eval_kernel_against_ref so AlphaEvolve optimizes the same metric the harness reports).

The shared bearer token defaults to prabhune. Override by setting AEPROXY_API_KEY in .env if needed.

Caller side (everyone else)

Set the shared token in your shell or PopcornBench's .env:

export AEPROXY_API_KEY=prabhune

Add a model stanza pointing at the proxy. request_timeout_s is mandatory — AlphaEvolve runs are minutes long.

[[models]]
name              = "alphaevolve-gemini-3-mixture"
api_kind          = "openai_chat"
base_url          = "http://localhost:8765/v1/"
api_key_env       = "AEPROXY_API_KEY"
deployment_name   = "alphaevolve-gemini-3-mixture"
rpm               = 4
tpm               = 1000000
max_concurrency   = 2
request_timeout_s = 2400

A ready-to-run smoke config is checked in at configs/sweep.alphaevolve.toml. Run with:

uv run python scripts/run_sweep.py configs/sweep.alphaevolve.toml

Available models

`name` in TOML	Underlying generation model
`alphaevolve-gemini-3-mixture`	Gemini 3.0 Flash 90% / Pro 10%
`alphaevolve-gemini-2.5-mixture`	Gemini 2.5 Flash 90% / Pro 10%
`alphaevolve-gemini-2.5-flash`	Gemini 2.5 Flash
`alphaevolve-default`	Server's default (`MODEL_UNSPECIFIED`)

How a request maps to an AlphaEvolve experiment

The proxy parses the chat messages, pulls the PyTorch reference module out of the last ```python fenced block, and builds an experiment config.
The reference module is registered as the initial program. The agent's task description becomes the AE problem description.
run_controller_loop drives sampling + evaluation until max_programs evaluations complete (default 20, capped at AEPROXY_MAX_PROGRAMS_CAP, default 60).
Each candidate is scored by eval_kernel_against_ref; the proxy tracks best speedup.
The response is a single OpenAI-style assistant message with a synthetic submit_kernel tool call carrying the best-evolved kernel. The PopcornBench harness then runs submit_kernel as usual to record extended metrics, and the trajectory looks like a one-turn agent run.
Streaming responses emit periodic reasoning_content deltas with the search progress (programs evaluated, best speedup so far). The HTML report's reasoning block renders this evolution log.

Per-request overrides

Pass via the OpenAI client's extra_body=:

extra_body = { "max_programs": 40, "concurrency": 4 }

Field	Default	Cap
`max_programs`	20	`AEPROXY_MAX_PROGRAMS_CAP` (60)
`concurrency`	4	—
`backend`	cuda	—
`precision`	fp32	—

Caveats

AlphaEvolve runs are non-deterministic. Use attempts_per_problem (when available) or repeat the sweep for variance estimates.
Each chat-completion request triggers a real GCP-billed evolutionary search. Keep problem_subset small until you've calibrated cost.
The proxy must run on a host with at least one CUDA GPU, because the evaluator runs candidates locally.
Server-wide concurrency is gated by AEPROXY_MAX_CONCURRENT_RUNS (default 2); excess callers get queued by the OpenAI client's request timeout.

Troubleshooting

Probe fails with 401 / 403

Rotated key or wrong env var. Check that api_key_env in the TOML matches an entry in .env, and that uv run is loading .env.

Sweep stalls on one model

Most often the per-model semaphore is throttling; check tpm and max_concurrency on that model's stanza. The Azure 429 path retries automatically up to a few times before the turn is recorded as failed.

Trajectory shows `in_progress` forever

The agent never called submit_kernel within max_turns. Either bump max_turns, lower num_perf_trials (so submit_kernel doesn't time out per call), or inspect the trajectory page to see where the agent got stuck.

AlphaEvolve calls time out at exactly 600s

The request_timeout_s field on the AE model stanza isn't set or isn't being read. Confirm you're on a recent run_sweep.py (the field was added alongside the proxy) and that the value is in seconds.