{ "schema_version": "1.0", "title": "Performance optimization benchmarks", "methodology": "Benchmarks and evaluation environments in which agents must make an existing system faster or use fewer resources without changing what it does. Each record describes what the agent is given and can do (environment), what it must accomplish (tasks), how correctness and speed are judged (grading), how tasks were built (construction) and a short summary of reported results. Facts come from original papers, official repositories, leaderboards and later audits of the benchmarks. Results summarize each source's own experiments and do not form a common leaderboard. Evidence cutoff: 29 October 2026; collection is ongoing.", "targets": [ { "id": "kernels", "label": "GPU and accelerator kernels", "description": "Low-level programs that run on GPUs, TPUs or other AI chips, usually written to replace a reference implementation in PyTorch or JAX." }, { "id": "repositories", "label": "Repositories", "description": "Real software projects, where the agent must find and change the slow code within a larger codebase." }, { "id": "programs", "label": "Programs and functions", "description": "Self-contained programs or single functions, such as contest solutions, numerical routines or small systems components." }, { "id": "ml-systems", "label": "ML training and serving", "description": "End-to-end model training scripts and language-model serving systems, judged on training time, throughput or latency." }, { "id": "compilers", "label": "Compilers", "description": "Compiler output such as assembly or LLVM intermediate code, or the compiler itself and its optimization settings." }, { "id": "queries", "label": "Database queries", "description": "SQL queries that must return the same results faster." } ], "records": [ { "id": "perfcodebench", "name": "PerfCodeBench", "summary": "A set of 1,854 systems-programming tasks, mostly in C++, in which a model rewrites a slow implementation to approach the speed of an expert-optimized one.", "targets": [ "programs", "kernels" ], "publishDate": "2026-05-13", "publicationType": "Preprint", "publisher": "Hong Kong University of Science and Technology; New York University; Southwest University of Political Science and Law; MODEIO.AI", "environment": "A single attempt without tools or feedback. The prompt gives the task goal, correctness rule, the function interface the harness calls and the full slow source file; the model returns a replacement file. It cannot compile, run or profile code and sees no hardware details or reference solution. Timing used Intel Xeon Platinum 8480C processors and NVIDIA H800 GPUs.", "tasks": "Produce a faster version with the same results, interface and dependencies. About 59% of tasks are C++; the rest are Java, C, Go, Python and CUDA (NVIDIA's GPU language). Domains include GPU code, CPU code that depends on vector instructions and cache use, parallel computing, data processing and memory allocators. Example: joining two sorted streams, where the expert replaces repeated binary search with one forward-moving pointer.", "grading": "A rewrite must compile, run and pass a task-specific check (exact output, a checksum or a floating-point tolerance). Speed is the median of repeated runs (count not stated), in the same harness as the slow original (baseline) and expert version (reference). Metrics include the shares of rewrites that are correct, beat the baseline, or match the reference, and how much of the baseline-to-reference gap correct rewrites close. No defenses against gaming are described beyond these checks.", "construction": "Slow and optimized code was taken from projects and benchmark suites such as simdjson, DuckDB, RocksDB, mimalloc, Redis, vLLM and KernelBench, with variants made by changing input size or shape. From over 8,000 candidates, the authors kept runnable tasks with a correctness check and a faster reference, capped near-duplicates, and dropped tasks with unstable timing. Some references are upstream code; others were written for the benchmark.", "results": "Of 20 models, GPT-5.4 produced the most correct rewrites (71%), while GPT-5 most often matched or beat the expert reference (62% of comparable tasks). GPU code was far harder: GPT-5.4 was correct on 82% of CPU tasks but 2.4% of GPU tasks. The authors conclude that models often speed code up but rarely reach expert efficiency.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2605.15222" }, { "title": "Code and data", "url": "https://anonymous.4open.science/r/perfcodebench-7CDE" } ] }, { "id": "swe-fficiency", "name": "SWE-fficiency", "summary": "A benchmark of 498 tasks in which a coding agent must speed up a given workload in a large Python scientific library without being told which code is slow.", "targets": [ "repositories" ], "publishDate": "2025-11-08", "publicationType": "Peer-reviewed paper", "publisher": "Harvard University; Google DeepMind; Princeton University; Xi'an Jiaotong University; Google", "environment": "The agent gets a container with one of nine libraries, such as NumPy, pandas or scikit-learn, and a script that times a slow workload. It may edit any code except that script and is not told which functions are slow; the developers' fix and grading tests are hidden. The authors used the OpenHands and SWE-agent frameworks (shell and file editing), with up to 100 actions, 3 hours and four dedicated processor cores per task.", "tasks": "Make the workload faster without changing the library's behavior, mostly in Python and Cython (a Python variant compiled to C). Example: developers sped up a pandas operation 20.5 times by avoiding a slow data-type conversion; one agent instead added a shortcut for data without missing values and reached 2.3 times.", "grading": "Correctness uses the repository's unit tests that exercise the developers' changed code, a median of about 7,700 per task. The agent's speedup on the workload is divided by the developers' to give a speedup ratio (1.0 matches the experts), and ratios are combined with a harmonic mean, which weights poor results heavily. Failing patches score as unchanged code. The harness rejects caching results between timing runs and code that detects it is being benchmarked, tricks agents tried during development.", "construction": "From about 96,000 merged pull requests, the authors kept those described with performance terms such as \"speedup\", changing code but not tests, and covered by at least one existing test, leaving about 1,000. Two authors hand-wrote a workload for each from the pull request and its discussion. The final 498 are those where the developers' change cut runtime by more than twice the run-to-run timing variation.", "results": "In the ICML 2026 version, Claude 4.5 Opus led with a speedup ratio of 0.225, followed by GPT-5 (0.157); Claude 3.7 Sonnet reached 0.024. Agents often edited the right files but missed the functions behind most of the experts' gains, stopped at the first improvement, and broke tests in 9–45% of patches. A July 2026 audit found the scoring formula gives the ten worst tasks 58–83% of a submission's score.", "sources": [ { "title": "Paper (ICML 2026)", "url": "https://proceedings.mlr.press/v306/ma26l.html" }, { "title": "Preprint", "url": "https://arxiv.org/abs/2511.06090" }, { "title": "Audit (preprint)", "url": "https://arxiv.org/abs/2607.01211" }, { "title": "Leaderboard", "url": "https://swefficiency.com" } ] }, { "id": "gso", "name": "GSO", "summary": "A benchmark of 102 tasks in which a coding agent must speed up a real open-source codebase, such as NumPy, Pillow or llama.cpp, as much as the project's developers once did.", "targets": [ "repositories" ], "publishDate": "2025-05-29", "publicationType": "Peer-reviewed paper", "publisher": "UC Berkeley", "environment": "Each task is a container with the codebase as it stood before a developer's speedup, a build script and a performance test. The agent can edit files and run shell commands to build, test and profile, but does not see the developers' change. The paper used the OpenHands framework with 3 hours per task; the maintainers later blocked GitHub and GitLab after some models downloaded the original changes.", "tasks": "Make the test workload faster without breaking it. Tasks span 10 codebases and five languages: Python, C, C++, Cython and hand-written SIMD code (instructions that process several values at once); about 59% need changes outside Python. Example: speed up text generation in llama.cpp, a C++ program for running language models.", "grading": "A patch must pass the test's assertions that outputs match the original code's. The agent's speedup is compared with that of the developers' change, and a task is solved at 95% or more of it; the headline metric, Opt@1, is the share solved in one attempt. Since November 2025 the leaderboard also reports a hack-adjusted score, for which a language model reviews patches and penalizes tricks such as caching results or tampering with timing.", "construction": "A language model and code-change rules flagged performance commits in popular GitHub repositories. Given each commit's message, files and linked issues, a language model wrote many tests that run a realistic workload and check outputs match before and after; commits were kept if consistently faster across tests. The authors then removed tasks with weak tests or unreliable timing or where simple caching matched the developers' speedup, leaving 102.", "results": "On the official leaderboard, which runs all models in OpenHands, Claude Fable 5.1 leads at 88.2% Opt@1 (87.3% after the hack penalty), ahead of GPT-6 Astra (79.4%) and Claude Fable 5 (78.4%); in the original 2025 paper no model solved more than 5%. A July 2026 audit found only 39 of the 102 developers' changes met GSO's validity criteria on all four cloud machine types it tested.", "sources": [ { "title": "Paper", "url": "https://arxiv.org/abs/2505.23671" }, { "title": "Repository", "url": "https://github.com/gso-bench/gso" }, { "title": "Leaderboard", "url": "https://gso-bench.github.io/" }, { "title": "Audit (preprint)", "url": "https://arxiv.org/abs/2607.01211" } ] }, { "id": "supercoder", "name": "SuperCoder", "summary": "A benchmark of 200 C programs in which a model must rewrite the assembly code produced by an optimizing compiler so that it runs faster.", "targets": [ "compilers", "programs" ], "publishDate": "2025-05-16", "publicationType": "Preprint", "publisher": "Stanford University; University of Illinois Urbana-Champaign; Carnegie Mellon University; Nanjing University", "environment": "Mainly a single attempt without tools. The model receives a C program and the x86-64 assembly that the GCC compiler produces at its standard highest optimization (-O3), and returns rewritten assembly; tests are hidden. The paper also tests best-of-several sampling, up to six revision rounds with compiler, test and timing feedback, and a small model trained with reinforcement learning.", "tasks": "Rewrite the assembly, the processor's low-level instructions, to give the same results faster, a task known as superoptimization. Programs average about 130 lines of assembly and contain loops. Example: replacing a loop that counts a number's 1-bits with the single popcount instruction.", "grading": "A rewrite must assemble and match the expected output on all tests, about nine per program. Each input is timed ten times after three warm-up runs, and speedup is the compiler output's runtime divided by the rewrite's; incorrect or slower rewrites count as no speedup. Tests cannot prove equivalence, so a passing rewrite may still be wrong on other inputs. No other defenses against gaming are described.", "construction": "C programs were sampled from CodeNet, IBM's collection of competitive-programming submissions (selection criteria not stated), compiled with GCC -O3, and split into 7,872 training and 200 evaluation programs. Test inputs come from AlphaCode's dataset (Li et al., 2022). Because many submissions are not accepted solutions, expected outputs come from running each original program, so rewrites are judged on matching it, not on solving the problem.", "results": "Of 23 models, Claude Opus 4 achieved the highest average speedup over the compiler, 1.43 times, with 51.5% of rewrites passing all tests; GPT-5's passed 6% of the time. The authors' 7-billion-parameter model trained with reinforcement learning passed 95% with a 1.46 times speedup. Common changes included restructuring loops, choosing faster instructions and removing branches.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2505.11480" } ] }, { "id": "kernelbench", "name": "KernelBench", "summary": "A benchmark of 250 problems in which a model must replace neural-network code written in PyTorch with faster custom GPU code.", "targets": [ "kernels" ], "publishDate": "2024-12-03", "publicationType": "Peer-reviewed paper", "publisher": "Stanford University; Princeton University", "environment": "The model receives a PyTorch module, a small neural-network component in Python, with input generators, and writes a replacement using its own GPU kernels (programs that run in parallel on the graphics processor). The paper tested single attempts, many samples, and up to 10 revision rounds with compiler, correctness and profiler feedback, on an NVIDIA L40S GPU; the July 2025 revision uses an H100.", "tasks": "Level 1 has 100 single operations such as matrix multiplication; Level 2 has 100 short chains of operations that can be merged into one kernel; Level 3 has 50 complete models such as MobileNet. Example: multiplying by a diagonal matrix, where models gained over 12 times by not building the full matrix.", "grading": "Code must match PyTorch's output within 0.01 on five random inputs, then is timed against standard PyTorch. The main metric, fast_p, is the share of problems solved correctly and more than p times faster. After exploits such as reusing the memory holding PyTorch's answer, a July 2025 revision added varied input distributions, fixed problems with trivially constant output, and added a check against the hardware's theoretical speed limit.", "construction": "For Level 1, the authors hand-picked common operations, wrote examples in PyTorch, and had a language model write variants such as convolutions of other dimensions. For Level 2, a script combined one main operation, such as matrix multiplication, with two to five follow-up operations, such as activations, and a language model wrote the code. Level 3 mixes language-model-written classic models with real GitHub models stripped to their core. Input sizes were chosen by hand, randomness such as dropout removed, and every task reviewed.", "results": "In the original paper, the best single-attempt models produced correct, faster code for at most 12% of Level 1 and 36% of Level 2 problems; with 10 feedback rounds, DeepSeek R1 reached 72% on Level 2. KernelBench-Verified (June 2026) compared against PyTorch with TF32, a faster math mode on modern NVIDIA GPUs, and added hidden tests; the best model, GPT-5.5, fell from a 1.43 to a 0.88 times average speedup.", "sources": [ { "title": "Blog post", "url": "https://scalingintelligence.stanford.edu/blogs/kernelbench/" }, { "title": "Paper (ICML 2025)", "url": "https://arxiv.org/abs/2502.10517" }, { "title": "Repository", "url": "https://github.com/ScalingIntelligence/KernelBench" }, { "title": "Revision (v0.1)", "url": "https://scalingintelligence.stanford.edu/blogs/kernelbenchv01/" }, { "title": "METR analysis", "url": "https://metr.org/blog/2025-02-14-measuring-automated-kernel-engineering" }, { "title": "KernelBench-Verified (preprint)", "url": "https://arxiv.org/abs/2607.16241" } ] }, { "id": "agentkernelarena", "name": "AgentKernelArena", "summary": "A benchmark of 196 GPU kernel tasks on AMD hardware in which coding agents speed up existing kernels, with hidden input shapes testing generalization.", "targets": [ "kernels" ], "publishDate": "2026-05-16", "publicationType": "Preprint", "publisher": "Advanced Micro Devices (AMD)", "environment": "Each agent gets an isolated workspace with shell access, the source files to modify, commands to check correctness and speed, and optional guides to AMD's GPU architecture. It may compile, test and time freely. The paper allowed one hour per task, asked for up to three successive kernel versions, and used an AMD Instinct MI300X GPU.", "tasks": "24 tasks speed up a kernel in HIP, AMD's CUDA-like GPU language, such as LayerNorm; 148 speed up a kernel in Triton, a Python-like GPU language, such as flash attention; and 24 replace a PyTorch module, such as Softmax, with a faster HIP kernel.", "grading": "A central evaluator recompiles and reruns each kernel, ignoring the agent's own reports. Outputs must match the original kernel or PyTorch module within tolerances that depend on the number format, and speed is timed over 100 runs against the original. Each task scores 20 points for compiling, 100 for correctness and 100 times the speedup. Eight unreleased input shapes per task test whether gains carry over.", "construction": "A language-model agent extracted 118 Triton tasks from the vLLM inference engine, isolating each kernel and writing test scripts; 30 more come from the ROCmBench collection. HIP and PyTorch tasks start from community PyTorch modules, with HIP kernels drafted by language models and reviewed by hand. A checking agent ran every task's scripts; failing tasks were fixed or dropped.", "results": "Cursor Agent, Claude Code and Codex, running Claude and GPT models, were nearly always correct; best mean speedups were 6.89 times for PyTorch-to-HIP, 6.69 times for HIP and 2.13 times for Triton. On unseen shapes, PyTorch-to-HIP correctness fell to 60–90% because agents hardcoded shape-specific constants. In a July 2026 AMD rerun on 44 tasks, AMD's GEAKv3 agent with Claude Opus 4.6 reached 9.04 times on HIP.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2605.16819" }, { "title": "Repository", "url": "https://github.com/AMD-AGI/AgentKernelArena" }, { "title": "AMD blog post", "url": "https://rocm.blogs.amd.com/software-tools-optimization/agent-kernel-arena/README.html" } ] }, { "id": "algotune", "name": "AlgoTune", "summary": "A benchmark of 154 algorithmic Python tasks in which a model must write code that matches a library-based reference solver's outputs but runs faster.", "targets": [ "programs" ], "publishDate": "2025-07-19", "publicationType": "Peer-reviewed paper", "publisher": "University of Tübingen; Princeton University; Meta; TU Braunschweig; Cradle Bio; LG Electronics Canada; Seoul National University; Tel Aviv University; Quansight; Stanford University; Norwegian University of Science and Technology; EPFL; Harvard University", "environment": "Each task gives a description, an example input and output, and the reference solver's code. In the paper's baseline agent, AlgoTuner, the model repeatedly edits, compiles, runs, profiles and times its code on development inputs, keeping the fastest valid version. Each model has a fixed budget per task and can use tools that compile Python into fast machine code, such as Numba and Cython.", "tasks": "Write a solve function that returns valid answers faster than the reference. Tasks span 13 categories, including matrix operations, convex optimization, graphs, compression and cryptography. References mostly call standard scientific Python libraries such as NumPy, SciPy and NetworkX. Examples: gzip compression, graph isomorphism or discrete logarithm.", "grading": "Each task's verifier checks every output on held-out test inputs, not the development inputs the agent saw. Each solution is timed 10 times on one CPU core, keeping the fastest; problem sizes are set so the reference takes about 100 milliseconds. Task score is reference time divided by candidate time, with invalid or slower code set to 1; the AlgoTune score is the harmonic mean over tasks.", "construction": "21 contributors, including library committers and academics, each wrote tasks as a problem generator, a reference solver and a verifier; two others reviewed each task. Automated tests check that runtime grows with problem size and that verifiers accept reference outputs across random seeds. Verifiers were tightened when loopholes appeared, for example adding an optimality check to vector quantization.", "results": "In the paper, AlgoTuner's average speedup ranged from 1.33 times with Claude Opus 4 to 1.72 times with o4-mini. The authors saw no new algorithms, only surface-level optimizations. The public leaderboard, which is undated, lists GPT-5.2 at 2.05 times.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2507.15887" }, { "title": "Paper (NeurIPS 2025 Datasets and Benchmarks)", "url": "https://papers.neurips.cc/paper_files/paper/2025/hash/9a648e8e1014c2427156dcb5465cd488-Abstract-Datasets_and_Benchmarks_Track.html" }, { "title": "Repository", "url": "https://github.com/oripress/AlgoTune" }, { "title": "Leaderboard", "url": "https://algotune.io" } ] }, { "id": "atrex-bench", "name": "Atrex-Bench", "summary": "A benchmark of 30 GPU operators, with 440 input shapes taken from Alibaba's production inference traffic, in which agents write kernels scored against hardware limits.", "targets": [ "kernels" ], "publishDate": "2026-07-16", "publicationType": "Preprint", "publisher": "Alibaba Group", "environment": "The agent sees a PyTorch reference, an input generator and the list of shapes, but not production timings or hardware-limit figures. It can edit its workspace and build and test in a shell, without web access. Models ran in Claude Code, or Codex for GPT-5.5, from a single prompt; budgets are not stated. Experiments used an unnamed non-NVIDIA accelerator.", "tasks": "Write kernels in FlyDSL, a kernel language chosen because it is nearly absent from model training data, for operators used in serving language models, such as attention and 8-bit mixture-of-experts layers. Shapes range from generating a single token to processing long prompts.", "grading": "A shape is correct if outputs match the PyTorch reference within tolerances on 5 random seeds. Speed is the median forward-pass time, with caches flushed. The score is the ideal time allowed by the hardware's compute and memory bandwidth divided by measured time, weighted by each operator's share of production GPU time. A separate metric shows how often kernels fall back to PyTorch or vendor libraries.", "construction": "A tracer attached to services running on more than 10,000 accelerators collected 1,303 profiles from about 20 deployed models. Equivalent kernels were grouped into one operator task, weighted by GPU time. The authors rewrote each operator as a PyTorch reference and checked it against the production kernel, and grouped similar token counts to remove near-duplicate shapes.", "results": "Of six models, GPT-5.5 scored highest, at about 10.7% of the hardware limit; it was 3.06 times faster than torch.compile, PyTorch's built-in compiler, but only 0.85 times the speed of production kernels. Many correct answers fell back to PyTorch: Qwen3.7-Max was 84.8% correct but spent only 43.8% of time in FlyDSL kernels.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2607.14541" }, { "title": "Repository", "url": "https://github.com/alibaba/atrex-bench" }, { "title": "Project page", "url": "https://alibaba.github.io/atrex-bench/" } ] }, { "id": "automated-llm-speedrunning-benchmark", "name": "Automated LLM Speedrunning Benchmark", "summary": "A benchmark of 19 tasks in which an agent must reproduce successive human records in a community contest to train a small GPT-2 model fastest.", "targets": [ "ml-systems" ], "publishDate": "2025-06-27", "publicationType": "Peer-reviewed paper", "publisher": "Meta; University of Edinburgh", "environment": "The agent gets the previous record's training script, a task description and optionally a hint, and edits the code inside a search scaffold that drafts, debugs and improves solutions as a tree. Each solution trains on one node of 8 H100 GPUs for up to 60 minutes; a run explores 20 solutions over about 10 hours.", "tasks": "Make GPT-2 training reach the target loss faster, matching the next human record's improvement. Changes range from algorithmic, such as a new optimizer, to hardware-aware, such as mixed-precision training or faster attention. Code is Python with PyTorch; hints come as pseudocode, a text description or a mini-paper.", "grading": "A solution must reach a validation loss of 3.28 on held-out FineWeb text. The headline metric, fraction of speedup recovered, is the share of the time reduction between two consecutive human records that the agent achieves, averaged over tasks; human records were rerun on the same hardware. The paper also measures code similarity to the human solution. No safeguards against gaming are described.", "construction": "The tasks come from the contest's record history: 21 records as of May 2025 give consecutive pairs, and the authors dropped one record whose speedup came only from a PyTorch upgrade, leaving 19. DeepSeek-R1 drafted three hint levels per record from its code changes and notes, and the authors checked and edited them by hand.", "results": "Across DeepSeek-R1, o3-mini, Gemini 2.5 Pro and Claude 3.7 Sonnet, no agent recovered more than about 20% of the human speedup on average without hints; o3-mini reached about 46% with all hints combined. Hints sometimes hurt DeepSeek-R1, and later records were harder for agents.", "sources": [ { "title": "Paper (NeurIPS 2025 Datasets and Benchmarks)", "url": "https://arxiv.org/abs/2506.22419" }, { "title": "Repository", "url": "https://github.com/facebookresearch/llm-speedrunner" } ] }, { "id": "autotunebench", "name": "AutoTuneBench", "summary": "A measurement protocol and about 17 tasks in which an agent tunes GPU kernels or language-model serving-engine settings, with safeguards against inflated speedups.", "targets": [ "kernels", "ml-systems" ], "publishDate": "2026-09-16", "publicationType": "Preprint", "publisher": "harnets.ai", "environment": "An agent loop edits one permitted file per task, either a kernel or an engine's launch settings, and gets measured results back each iteration. Proposers were DeepSeek language models, a fixed search over settings or a random control, with 300 queries per task. Measurement runs on an NVIDIA DGX Spark, a compact desktop AI computer, in a sandbox without network.", "tasks": "Make a kernel or serving engine faster than its baseline without changing outputs. Tasks include a GELU activation kernel, KernelBench's 100 single-operator tasks, four SGLang serving-engine kernels, and searches over SGLang and vLLM settings such as speculative decoding, which drafts tokens ahead and checks them together.", "grading": "Kernel outputs must match a protected reference within 1% across at least five input shapes. Kernels are timed over 100 runs; engines are compared on throughput and latency, and runs must vary by at most 5%. Speedup is measured against PyTorch or the engine's defaults. Scripts are read-only, kernels may not call the operator they replace, and speedups above 10 times are held for review.", "construction": "The author built the tasks during a four-day pilot of 619 model calls, reusing KernelBench and kernels from the earlier Astra study so results could be checked against published speedups. Engine versions are pinned. Every iteration is logged to a database that rejects results measured outside the protocol, and over 290 automated tests check that protocol settings stay fixed.", "results": "A DeepSeek-tuned GELU kernel was 10.6 times faster than a naive draft but only 2.03 times faster than a proper baseline, and one certified at 11.03 times on the DGX Spark ran at 0.79 times on an Apple M5 Max machine. On KernelBench, 51 of 100 tasks were correct, with a median speedup of 1.0 times.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2609.18123" }, { "title": "Repository", "url": "https://github.com/li-ch/autotunebench" } ] }, { "id": "backendbench", "name": "BackendBench", "summary": "An evaluation suite in which models write replacement GPU kernels for PyTorch's built-in operators, tested for correctness on 271 operators and for speed on 124.", "targets": [ "kernels" ], "publishDate": null, "publicationType": "Repository-only", "publisher": "Meta", "environment": "A script creates one folder per PyTorch operator; a model or person fills each with a kernel, which then replaces the built-in operator at runtime. Built-in modes either prompt a language model up to 5 times with test feedback or run an agent for up to 10 refinement rounds. Hardware is not specified; timing uses a GPU benchmarking utility from Triton, a Python-like language for GPU kernels.", "tasks": "Reimplement core PyTorch operators, typically in Triton, so they are correct and fast. Memory-allocation, random-number and niche operators are excluded. Example: a replacement for relu, run on tensor shapes taken from popular Hugging Face models.", "grading": "Correctness uses PyTorch's own operator tests, about 13 per operator, covering empty tensors, NaN and infinity, mixed data types and broadcasting. About 110 performance tests per operator time PyTorch and the candidate; failed tests count as speedup 1. Scores are mean pass rate, geometric mean speedup over PyTorch, and the share of correct tests faster than a chosen multiple of PyTorch. A replacement that simply calls the PyTorch operator recurses infinitely, so such fallbacks fail.", "construction": "Correctness tests come from PyTorch's existing operator test suite, which the authors say reflects about nine years of user bug reports. Performance inputs are operator shapes recorded from popular Hugging Face models, enlarged until memory runs out and then reduced, because small shapes mostly measure overhead. How the models were chosen is not stated.", "results": "The repository's launch note reports few numbers. In an earlier version, ChatGPT wrote Triton kernels for 84 operators; most ran at 70% to 100% of PyTorch speed and a few were up to about 1.2 times faster. Training nanoGPT with generated forward kernels changed numerical results and ran about twice as slow as standard PyTorch.", "sources": [ { "title": "Repository", "url": "https://github.com/meta-pytorch/BackendBench" }, { "title": "Blog post", "url": "https://github.com/meta-pytorch/BackendBench/blob/main/docs/correctness.md" }, { "title": "Dataset", "url": "https://huggingface.co/datasets/GPUMODE/backendbench_tests" } ] }, { "id": "cirbench", "name": "CIRBench", "summary": "A benchmark of 800 tasks on LLVM compiler intermediate code; in 300, a model must rewrite unoptimized code to beat LLVM's strongest standard optimization level.", "targets": [ "compilers" ], "publishDate": "2026-05-15", "publicationType": "Peer-reviewed paper", "publisher": "Not stated in the sources", "environment": "Each instance is a single prompt containing unoptimized LLVM IR, the compiler's low-level intermediate code, for one function or a whole module; the model replies with optimized IR. There are no tools or feedback, and up to five independent attempts are allowed. In a second mode, LLVM's standard -O3 optimizations are applied to the model's output before timing.", "tasks": "Make the IR run faster without changing behaviour. The 300 instances, a third each rated small, medium and large, form five groups of 60: loops, operations, code, whole modules and complete contest programs. Example: optimize a loop computing w[k] = x[k] / (exp(u[k]/v[k]) - 1) over arrays.", "grading": "Output must pass LLVM's validity checker; then Alive2, a tool that checks two IR programs are equivalent, compares it with the input, and where Alive2 is inconclusive, program outputs must match. The IR is compiled and timed, repeating runs until about 500 milliseconds accumulate, and speedup is reported against both the unoptimized input and the -O3 version. No anti-gaming measures are described.", "construction": "Loop instances come from classic loop benchmark suites (the Livermore loops, TSVC and PolyBench), each wrapped in a timing harness with seeded random inputs; the 60 whole programs are Project CodeNet contest submissions with stored inputs and outputs. The repository does not say where the other 180 C programs came from, and the paper's selection steps could not be checked.", "results": "The paper tests six mainstream language models, which the repository suggests include GPT-5, Claude Sonnet 4.5 and DeepSeek-V3.2. According to the abstract, models often fail at rewriting and usually fall short of -O3, though the best case ran 4.96 times faster than it. Other tracks cover analysing IR, repairing invalid IR and applying named LLVM optimization passes.", "sources": [ { "title": "Paper (ICML 2026)", "url": "https://icml.cc/virtual/2026/poster/65497" }, { "title": "Paper (OpenReview)", "url": "https://openreview.net/forum?id=D1ahrfGJ5e" }, { "title": "Repository", "url": "https://github.com/yilingqinghan/CIRBench" } ] }, { "id": "compilergym-llvm", "name": "CompilerGym (LLVM pass ordering)", "summary": "A reinforcement-learning environment in which an agent orders the LLVM compiler's optimization passes to make programs smaller than LLVM's own size-minimizing setting does.", "targets": [ "compilers" ], "publishDate": "2020-12-21", "publicationType": "Peer-reviewed paper", "publisher": "Facebook AI Research", "environment": "The agent receives a program in LLVM's intermediate code through a Python interface for reinforcement learning. Each step applies one of about 124 optimization passes and returns a reward and observations such as the code text, feature vectors or a program graph. The agent decides when to stop; time budgets and hardware are set by the user.", "tasks": "Choose the order of compiler passes for each program so the result is smaller. Programs are C, C++, OpenCL and Fortran, from 14 datasets totaling over 1.1 million programs. Example: find a pass sequence that leaves one of 23 standard test programs with fewer instructions than LLVM's -Oz setting.", "grading": "The main reward counts instructions in the intermediate code, scaled so 1.0 equals the reduction LLVM's size-minimizing -Oz setting achieves; alternatives use compiled code size or measured runtime. The leaderboard reports the geometric mean of this ratio over 23 programs from the cBench suite, plus average wall time. Passes are meant to preserve behavior, and runnable programs can be checked by running them. No anti-gaming checks are described.", "construction": "The authors packaged existing collections: standard benchmark suites, code from GitHub, the Linux kernel, OpenCV and TensorFlow, programming-contest solutions, and two random-program generators. The list of available passes was extracted automatically from LLVM. The repository was archived in May 2026; its last release was in November 2022.", "results": "In the paper, with 1 hour per program, a general-purpose search algorithm reached 1.083 times -Oz's code-size reduction and greedy search 1.053 times, while a reinforcement-learning agent trained on random programs reached 0.964 times. The leaderboard's best entry, reinforcement learning plus guided search, reaches 1.070 times in about 70 seconds per program.", "sources": [ { "title": "Paper (CGO 2022)", "url": "https://arxiv.org/abs/2109.08267" }, { "title": "Repository and leaderboard", "url": "https://github.com/facebookresearch/CompilerGym" }, { "title": "Documentation", "url": "https://compilergym.com/llvm/index.html" } ] }, { "id": "ecco", "name": "ECCO", "summary": "A Python benchmark in which a model must make a correct but slow program faster or leaner, tested on 794 programs from 48 problems.", "targets": [ "programs" ], "publishDate": "2024-07-19", "publicationType": "Peer-reviewed paper", "publisher": "Carnegie Mellon University", "environment": "In the editing mode described here, the model receives only a slow Python program and is asked for a more efficient one; ECCO also has a mode that writes code from a problem description. The paper tried plain instructions, examples, self-critique, visible test results and fine-tuning. Code runs in a sandboxed execution server on a fixed cloud machine.", "tasks": "Make the given program faster or use less memory while keeping outputs correct. All programs are Python 3 solutions to programming-contest problems from IBM's CodeNet collection. Example: given the slower of two accepted solutions to the same AtCoder problem, produce a faster version that still passes the hidden tests.", "grading": "The edited program must pass hidden test cases; pass@1 is the share of single attempts that do. Speedup is the input program's runtime divided by the edited program's, and memory reduction the same ratio for memory. The paper does not state how incorrect programs enter the speedup average. Beyond hidden tests, sandboxing and unseen test problems, no anti-gaming checks are described.", "construction": "Following the earlier PIE benchmark, the authors turned CodeNet into about 187,000 pairs of slow and fast Python solutions to the same problem. They re-ran pairs and dropped those where the fast code was slower on their machine, plus duplicate, non-compiling or test-failing programs. Tests add AlphaCode's extra tests to CodeNet's. Training, validation and test splits share no problems.", "results": "Among six models, GPT-4o with a plain instruction had the highest pass@1, 66.6%, at a 1.64 times speedup; DeepSeekCoder-7B refined with critique and test results reached 3.54 times but only 19.6% pass@1. The authors conclude that no tested method improved efficiency without losing correctness.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2407.14044" }, { "title": "Paper (EMNLP 2024)", "url": "https://aclanthology.org/2024.emnlp-main.859" }, { "title": "Repository", "url": "https://github.com/CodeEff/ECCO" } ] }, { "id": "fastkernels", "name": "FastKernels", "summary": "A benchmark of 384 GPU kernel tasks from 47 model architectures, where agents must beat production frameworks' kernels both in isolation and inside the full models.", "targets": [ "kernels" ], "publishDate": "2026-05-22", "publicationType": "Preprint", "publisher": "Snowflake AI Research; Carnegie Mellon University; UC San Diego", "environment": "Each task is a standalone module with the same interface as a production operator, plus a runner that swaps in candidate kernels, checks and times them, with NVIDIA's profiling tools available. In the paper, agents ran on an 8-GPU NVIDIA B200 machine with locked clocks, the longer-running ones for a median of 6 to 12 hours per operator. The paper does not detail each agent's tools or prompts.", "tasks": "Write faster CUDA, Triton or PyTorch implementations at four levels: 108 single operators, 155 fused operators, 74 layers or blocks and 47 complete models, covering language, vision, audio, diffusion, recommendation and robotics models. Examples include RMS normalization, mixture-of-experts token routing and attention decoding as used by vLLM.", "grading": "Each case runs three rounds of fresh inputs, and at least 99% of output elements must be within per-data-type tolerances. Candidates are then placed into the full models and compared with a reference run, with a noise run setting thresholds. Baselines are kernels from production serving libraries such as vLLM and SGLang. The leaderboard score multiplies average speedup, calibrated correctness and coverage. The harness isolates processes, detects tampering with timers or harness functions, and screens out all-zero outputs.", "construction": "For each architecture, a pipeline loaded the Hugging Face model configuration, traced its forward pass and produced standalone tasks with the configuration constants written in; data-dependent operators replay captured real tensors. Each task was audited for correct semantics, real shapes and data types, and a matching production interface. Together the operators reproduce 472 of 499 Hugging Face Transformers architectures with matching outputs.", "results": "Five agents, including Claude Code and Codex, used 6,900 agent-hours. Kernel-level average speedups of 1.6 to 6.6 times shrank to at most 1.25 times end to end, and only 22 of 110 sets of winning kernels ran correctly in the full models without fixes. Claude Code matched or beat the KDA agent at every level in isolation, yet KDA scored about 3 times higher end to end.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2605.23215" }, { "title": "Repository", "url": "https://github.com/Snowflake-AI-Research/fastkernels" } ] }, { "id": "flashinfer-bench", "name": "FlashInfer-Bench", "summary": "A benchmark and public leaderboard of 41 GPU kernel definitions from real language-model serving, where agents write kernels to beat FlashInfer, a production kernel library.", "targets": [ "kernels" ], "publishDate": "2025-10-21", "publicationType": "Peer-reviewed paper", "publisher": "University of Washington; Carnegie Mellon University; NVIDIA; UC Berkeley", "environment": "For each kernel the agent gets a definition with a plain PyTorch reference, input shapes and data types, the target language and hardware documentation. In the paper's agent loop it writes a kernel, runs the benchmark, reads the results and revises, keeping the best passing version; the number of rounds is not stated. Timing used an NVIDIA B200 GPU.", "tasks": "Write faster kernels for language-model inference operations: matrix multiplication, attention over cached keys and values, mixture-of-experts layers, normalization and token sampling, in Triton (a Python-like GPU kernel language) or CUDA. Example: a matrix multiplication with two fixed dimensions of 4096, run across the batch sizes seen in serving.", "grading": "Outputs must match the PyTorch reference within tolerances with no invalid values; low-precision kernels need a set share of elements, such as 95%, to match, and sampling kernels are checked on output frequencies. Mean latency is compared with FlashInfer, or PyTorch where FlashInfer lacks a kernel. The fast_p metric is the share of workloads that are correct and more than p times faster. The leaderboard uses hidden workloads.", "construction": "The authors ran three open models in the SGLang serving engine on real chat prompts and recorded every kernel call. Calls with identical inputs, outputs and fixed dimensions were grouped into 41 definitions across 8 kernel types, keeping diverse batch sizes and sequence lengths. Real tensors were saved where values matter, such as sampling; otherwise inputs are seeded random.", "results": "The paper evaluated 240 solutions from four models in CUDA and Triton. GPT-5 was correct 83.9% of the time, o3 71.3% and Gemini 2.5 Pro 48.8%; 30 of 32 analysed errors were compilation failures. Triton solutions were more often correct and faster than CUDA ones, but for matrix multiplication and attention most reached under half of state-of-the-art speed.", "sources": [ { "title": "Paper (MLSys 2026)", "url": "https://proceedings.mlsys.org/paper_files/paper/2026/hash/37e44c4b5321605735be9761f9b758fc-Abstract-Conference.html" }, { "title": "Preprint", "url": "https://arxiv.org/abs/2601.00227" }, { "title": "Repository", "url": "https://github.com/flashinfer-ai/flashinfer-bench" }, { "title": "Leaderboard", "url": "https://bench.flashinfer.ai/" }, { "title": "Blog post", "url": "https://flashinfer.ai/2025/10/21/flashinfer-bench.html" }, { "title": "Dataset", "url": "https://huggingface.co/datasets/flashinfer-ai/flashinfer-trace" } ] }, { "id": "formulacode", "name": "FormulaCode", "summary": "A continually updated benchmark of 957 tasks in which a coding agent must speed up scientific Python libraries, measured against the developers' own speedups.", "targets": [ "repositories" ], "publishDate": "2026-03-16", "publicationType": "Peer-reviewed paper", "publisher": "University of Texas at Austin; California Institute of Technology; Cornell University", "environment": "Each task is a container holding the repository before a developer's speedup, a description of the performance problem from GitHub, limited to what was public when the fix was proposed, and the project's timing benchmarks, which the agent can run. Agents ran in two existing frameworks, Terminus 2 and OpenHands; the main text does not state time or step limits.", "tasks": "Make a library's benchmark workloads, on average 265 per task, faster without breaking its tests. Tasks come from 70 scientific Python repositories; solutions may involve Python, parallelization, vectorized operations or C extensions. Example: an Astropy issue about slow angle and coordinate objects, measured by 59 workloads.", "grading": "A patch is valid only if it passes every test the original code passes; snapshot tests also compare workload outputs with reference values, and a failing workload counts as no speedup. Speedup per workload is old time divided by new time. The headline advantage subtracts the developers' speedup from the agent's per workload: 0 means matching the developers, negative means worse. No specific anti-gaming checks are described.", "construction": "From 766 GitHub repositories with timing benchmarks, the authors scraped 105,074 merged pull requests. Rules such as at least 100 stars, a working build and performance keywords left 26,717; a language model judging performance intent left 3,181. An agent wrote build scripts, giving 1,232 reproducible tasks, and 957 kept a statistically significant developer speedup. New tasks are added monthly.", "results": "On a 108-task subset with manually checked tests, GPT-5, Claude 4.0 Sonnet, Gemini 2.5 Pro and Qwen 3 Coder all sped up the workloads but none matched the developers: the best speedup was 1.099 times against the developers' 1.104 times. Agents did better on function-level changes and parallelization than on C extensions and vectorized code.", "sources": [ { "title": "Paper (ICML 2026)", "url": "https://arxiv.org/abs/2603.16011" }, { "title": "Project page", "url": "https://formulacode.org/" }, { "title": "Repository (evaluation harness)", "url": "https://github.com/formula-code/fc-eval" }, { "title": "Repository (task mining)", "url": "https://github.com/formula-code/datasmith" }, { "title": "Dataset", "url": "https://huggingface.co/datasets/formulacode/formulacode-all" } ] }, { "id": "inferencebench", "name": "InferenceBench", "summary": "A benchmark of 4 scenarios in which an agent must deploy a fast language-model server on one H100 GPU, beating a naive PyTorch server.", "targets": [ "ml-systems" ], "publishDate": "2026-05-20", "publicationType": "Preprint", "publisher": "ELLIS Institute Tübingen; Max Planck Institute for Intelligent Systems; Tübingen AI Center", "environment": "A command-line coding agent gets a container with root access, internet, build tools, cached model weights and one H100 GPU for 2 hours, but no starter code. It may install any serving framework and run the evaluation script on a development seed. It must deliver a launch script for a server that answers OpenAI-style API requests.", "tasks": "Make serving fast for one workload per scenario: time to first token on 8,192-token inputs, time per output token on 8,192-token outputs, throughput with up to 64 concurrent requests, and a balanced mix. The main model is Mistral-7B-Instruct, and prompts come from long documents in the LongBench v2 dataset.", "grading": "The harness relaunches the server in a fresh container and measures a held-out evaluation seed. Answers must keep 95% of the baseline's accuracy on 500 MMLU-Pro multiple-choice questions, and a language-model judge reviews transcripts for cheating such as pre-generated text, a swapped-in smaller model or outside APIs. The score is the geometric mean of speedups over the naive server, with failed runs counted as 1 times.", "construction": "Request lengths vary between 80% and 100% of a set length and are fixed by content hashing, so every run sees identical requests. The baseline is a simple web server calling Hugging Face's generate function. A manual audit of 50 runs found no false cheating flags and one missed case.", "results": "On the official leaderboard (October 2026), Claude Opus 5.5 (Max) in Claude Code leads at 12.08 times the plain PyTorch server, the only agent ahead of a simple settings search on existing engines (11.53 times); Claude Fable 5.1 follows at 9.83 times and GPT-6 Astra at 7.90 times. Agents often stopped at their first valid server; 6.1% of runs in the paper were flagged for cheating.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2607.20468" }, { "title": "Repository", "url": "https://github.com/aisa-group/InferenceBench" }, { "title": "Leaderboard", "url": "https://inferencebench.ai/" } ] }, { "id": "iso-bench", "name": "ISO-Bench", "summary": "A benchmark of 54 tasks in which a coding agent must speed up vLLM or SGLang, two open-source language-model servers, matching optimizations their developers once merged.", "targets": [ "repositories", "ml-systems" ], "publishDate": "2026-02-23", "publicationType": "Preprint", "publisher": "Lossfunk", "environment": "Each task is a container with the repository just before a developer's speedup, a description of the bottleneck that does not reveal the fix, and the original benchmark command. Agents can edit non-test files and run shell commands, tests and benchmarks on one NVIDIA H100 GPU for 120 minutes. The number of attempts per agent is not stated.", "tasks": "Reduce latency or raise throughput of serving code without changing outputs; 39 tasks come from vLLM and 15 from SGLang. Example: reduce the CPU overhead of an attention operator so it can be replayed as a recorded sequence of GPU work; the developers moved tensor reshapes out of the operator.", "grading": "Each project's benchmark tool measures time to first token or throughput for the original code, the developers' patch and the agent's. An agent matches the developers if within 5% and beats them if more than 5% better. Because timing alone can be gamed, a language model judges whether the patch targets the same bottleneck; True Success requires both. Successful patches are also checked for unchanged model accuracy. The judge was not validated against human ratings.", "construction": "From 15,234 vLLM and 8,421 SGLang commits, the authors kept those whose messages or diffs contained words such as optim, perf, latency or kernel (1,439), dropped those touching 10 or more files, and had GPT-5-mini label whether each was a GPU inference optimization (387). Manual review removed non-reproducible, multi-GPU, old or hard-to-containerize commits and wrote the descriptions, leaving 54.", "results": "Claude Code with Claude Sonnet 4.5, Codex CLI with GPT-5, and the TRAE agent with each model were tested. True Success ranged from 17.9% to 46.2% on vLLM, where Claude Code led, and 26.7% to 86.7% on SGLang, where it came last. Timing-only scores overstated success by about 10 to 20 points, and open-source models produced no working optimization.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2602.19594" }, { "title": "Repository", "url": "https://github.com/Lossfunk/ISO-Bench" }, { "title": "Dataset", "url": "https://huggingface.co/datasets/Lossfunk/ISO-Bench" } ] }, { "id": "jaxbench", "name": "JAXBench", "summary": "A benchmark of 50 JAX workloads that agents must rewrite as faster custom kernels for Google's TPU chips, compared with the standard compiled version.", "targets": [ "kernels" ], "publishDate": "2026-05-05", "publicationType": "Preprint", "publisher": "Google; Google DeepMind; Harvard University; UC Berkeley", "environment": "The model receives the workload's JAX source and a description of the TPU v6e chip and returns one Python file with the same function signature. The paper compared four methods with about 144 samples each: independent samples, 8-turn refinement with compiler, correctness, speed and profiler feedback, the same plus TPU documentation, and Autocomp, a search-based method.", "tasks": "Make each workload faster on TPU, usually by writing a kernel in Pallas, JAX's language for custom kernels. 17 are production operators from MaxText, Google's language-model library, such as attention and mixture-of-experts layers; 33 are chains of fused operations from KernelBench, such as a matrix multiply followed by activations.", "grading": "The candidate must compile and match the original output on random inputs within a tolerance of 0.01. Device time is the median of 50 profiled runs. Speedup is relative to the original code compiled by XLA, Google's default compiler; failures count as 1 times, and the headline is the geometric mean. Eight workloads also have expert-written kernels as an upper bound. Nothing checks that Pallas is actually used.", "construction": "The 17 production operators keep their source models' sizes. The 33 others are one task per fusion pattern from KernelBench Level 2, translated from PyTorch to JAX by Gemini, checked against the PyTorch original and regenerated or hand-fixed on failure. Each problem size is the smallest at which the compiled baseline uses at least 60% of the chip's matrix units.", "results": "With Gemini 3 Flash, speedups over XLA, TPU's default compiler, rose from 1.01 times for plain samples to 1.36 times with the Autocomp search method; documentation lifted correctness from 5.8% to 37.3%, and misuse of Pallas caused most failures. Experts reached 2.08 times on eight workloads. A September 2026 follow-up by overlapping authors, MaxKernel with Gemini 3.1 Pro, solved all 50 tasks at 1.58 times.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2607.20466" }, { "title": "Repository", "url": "https://github.com/AI-Hypercomputer/accelerator-agents/tree/main/JAXBench" }, { "title": "Follow-up (MaxKernel preprint)", "url": "https://arxiv.org/abs/2609.04523" } ] }, { "id": "jeto-bench", "name": "JETO-Bench", "summary": "A benchmark of 91 real speedups in open-source Java projects, mined from 1.8 million commits, in which an agent must fix a reported performance issue.", "targets": [ "repositories" ], "publishDate": "2026-06-30", "publicationType": "Preprint", "publisher": "ETH Zurich", "environment": "Each task gives the project at the commit before the fix, the GitHub issue describing the slowness, and a Docker image with the right Java version and dependencies. In the paper, OpenHands agents worked inside this image with up to 100 iterations per task on a 64-core server. The paper does not state what performance feedback the agent could see.", "tasks": "Change Java source code, not tests, so the project runs faster without changing behaviour; 37 tasks need multi-file changes. Example: one project loaded every class in a Java archive just to list their names; the developers instead read the names directly from the archive.", "grading": "The harness applies the patch, builds it, and times the changed modules' existing tests 30 times on both versions; a statistical test checks for at least a 5% improvement. Because few existing tests detect even the developers' speedups, headline results instead come from manual comparison with the developers' patch: correct means an exact or semantically equivalent fix.", "construction": "A mining tool scanned 1,769,958 commits in 3,686 Java repositories, kept those changing only non-test code and linked to an issue, and had a language model flag 660 issues about execution time. Of these, 102 built and passed all tests in every timed round; one author and GPT-5.5 independently confirmed each was a general, behaviour-preserving speedup, leaving 91.", "results": "OpenHands with GPT-5-mini correctly fixed 13 of 91 issues (14.3%); GLM-5.1 and Mimo-pro-2.5 each fixed 9. Most incorrect patches still passed the existing tests: only 19 tasks have a test whose timing changes significantly, 6 of them a slowdown. GPT-5-mini wrote a test detecting the improvement for only 1 task.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2606.31767" }, { "title": "Repository", "url": "https://github.com/khesoem/JETO-Bench" } ] }, { "id": "kernelbenchx", "name": "KernelBenchX", "summary": "A benchmark of 176 GPU operations in which generated Triton kernels must match PyTorch's output and are timed against PyTorch on several NVIDIA GPUs.", "targets": [ "kernels" ], "publishDate": "2026-05-06", "publicationType": "Preprint", "publisher": "Not stated in the sources", "environment": "Each task gives a function interface, a PyTorch reference, a description and constraints; the exact prompt is not published. Generators write Triton, a Python-like language for GPU kernels. The paper ran three models once per task and two agents, GEAK and KernelAgent, over several refinement rounds. The released harness runs on any NVIDIA GPU.", "tasks": "Implement a Triton kernel that reproduces a PyTorch function and runs faster. Fused operation chains (60) and math functions (36) dominate; others include activations and optimizer steps, plus six quantization tasks that forbid built-in quantization calls. Example: batched matrix multiply followed by RMSNorm, GELU, dropout and a subtraction.", "grading": "Code must import, compile and expose the expected function. Outputs must match the reference in shape, type and value within tolerances, on normal inputs and on inputs with injected large outliers; quantization tasks use error thresholds instead. Correct kernels are timed, taking the median of 100 runs, against PyTorch re-measured on the same GPU, and their memory bandwidth and compute rate are compared with hardware peaks.", "construction": "The tasks extend TritonBench-T, an existing set of operators paired with PyTorch documentation, adding a 15-category grouping by computational structure, lower-precision variants and six quantization tasks. The paper does not say how many tasks were new, who wrote them or how they were checked beyond the test protocol. 110 before-and-after kernel pairs from GEAK's refinement are also released.", "results": "GEAK produced correct kernels for 30.7% of tasks and Claude for 22.7%; no method solved a quantization task. Of all correct kernels, 46.6% were slower than PyTorch, the median speedup was about 1.0 times, and one kernel's speedup varied by up to 21.4 times across six GPUs. GEAK's refinement raised correctness, but rescued kernels were slower.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2605.04956" }, { "title": "Repository", "url": "https://github.com/BonnieW05/KernelBenchX" }, { "title": "Dataset", "url": "https://huggingface.co/datasets/BonnieWang/KernelBenchX" } ] }, { "id": "kernelgenbench", "name": "KernelGenBench", "summary": "A benchmark of 210 operators from PyTorch, vLLM and NVIDIA's cuBLAS library that agents must rewrite in Triton, a Python-like GPU language, across six hardware platforms.", "targets": [ "kernels" ], "publishDate": "2026-07-22", "publicationType": "Preprint", "publisher": "Beijing Academy of Artificial Intelligence; Beijing Normal University; Beijing Jiaotong University; Institute of Automation, Chinese Academy of Sciences; Peking University", "environment": "The prompt gives the operator's exact function signature and a reference: PyTorch code, or a short wrapper calling the native vLLM or cuBLAS routine, whose source is not shown. One track samples code once or five times without feedback; the agent track lets coding agents run and repair code for 30 minutes per operator. Reference hardware is an NVIDIA A100; five other platforms are anonymized.", "tasks": "Write a drop-in Triton replacement that computes the same result as an existing operator. Sources: 110 core PyTorch operators, 50 vLLM GPU kernels for attention, caching and quantization, and 50 cuBLAS linear-algebra routines. Example: re-implement vLLM's repetition-penalty kernel used during text generation.", "grading": "Each operator runs on a grid of test cases combining parameters, shapes, data types and memory layouts, and must match the native output within tolerance on all of them. Speedup is native time divided by Triton time, averaged geometrically over test cases and then operators. Anti-cheating checks scan code for calls to the native library, rerun with the Triton kernel disabled to catch kernels that never run, and on NVIDIA check profiler traces for Triton kernels.", "construction": "PyTorch operators: the 50 most frequent in training traces of 2,907 open-source models plus 60 sampled from the rest. vLLM: 50 operators with standalone CUDA implementations. cuBLAS: the 10 most-invoked routines in profiling traces, expanded across precisions and batching modes, plus other routines. A 110-operator subset is used across platforms.", "results": "Models such as Claude Opus 4.6, GLM-5.0 and Qwen3.5-27B were tested alone and in agents. On the A100, Claude Code with Opus 4.6 was correct on 92% of PyTorch, 68% of vLLM and 94% of cuBLAS operators, at 0.86, 1.02 and 0.51 times native speed. One agent's accuracy fell from 87% on NVIDIA to 25% on another platform.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2607.27231" }, { "title": "Repository", "url": "https://github.com/flagos-ai/KernelGenBench" } ] }, { "id": "kernelgym", "name": "KernelGYM", "summary": "A distributed GPU environment for training and evaluating models that rewrite PyTorch code as faster Triton kernels, with about 72,000 training tasks and reward-hacking checks.", "targets": [ "kernels" ], "publishDate": "2026-02-05", "publicationType": "Peer-reviewed paper", "publisher": "Hong Kong University of Science and Technology; Chinese University of Hong Kong, Shenzhen; TikTok; Nanyang Technological University", "environment": "A server queues submissions to GPU workers, each running one in a fresh process to contain crashes. The model receives PyTorch reference code and returns a kernel in Triton, a Python-like GPU language. Over several turns, 3 in the paper's training, it gets feedback on errors, correctness, speedup and a profiling summary. The paper used NVIDIA H100 GPUs.", "tasks": "Replace a PyTorch module with faster Triton kernels, in the style of KernelBench, an earlier benchmark of single operators to whole networks. Example: for a module ending in a channel sum, fuse the work into Triton kernels rather than replacing only the sum, a shortcut the paper calls lazy optimization.", "grading": "Outputs must match the PyTorch reference on several random inputs; only correct kernels are timed. Reward is correctness plus correctness times speedup, capped at 3 times. To stop reward hacking, the environment records which kernels actually run: a submission that runs no Triton kernel, for example by copying the reference or skipping the kernel when in training mode, is marked incorrect. An optional reward credits time spent in generated kernels.", "construction": "Training queries come from the existing cudaLLM dataset; GPT-5 wrote multi-turn Triton solutions for about 8,000 of them as starter data. The public release has about 72,000 reinforcement-learning tasks and 100 validation tasks; the paper does not describe how they were filtered or checked. Evaluation used KernelBench levels 1 to 3.", "results": "Dr. Kernel-14B, trained in this environment, was compared with Claude 4.5 Sonnet and GPT-5 on KernelBench, counting reward-hacked samples as wrong. On level 2, 31.6% of its kernels were at least 1.2 times faster than PyTorch, versus 26.7% and 28.6% respectively. Without profiling-based rewards, models tended to make trivial correct changes with little speedup.", "sources": [ { "title": "Paper (ICML 2026)", "url": "https://icml.cc/virtual/2026/poster/65609" }, { "title": "Preprint", "url": "https://arxiv.org/abs/2602.05885" }, { "title": "Repository", "url": "https://github.com/hkust-nlp/KernelGYM" }, { "title": "Datasets and models", "url": "https://huggingface.co/collections/hkust-nlp/drkernel" } ] }, { "id": "multikernelbench", "name": "MultiKernelBench", "summary": "A benchmark of 285 deep learning operations that a model must rewrite from PyTorch as faster kernels for NVIDIA GPUs, Huawei NPUs or Google TPUs.", "targets": [ "kernels" ], "publishDate": "2025-07-18", "publicationType": "Preprint", "publisher": "Nanjing University; Zhejiang University", "environment": "The model gets a prompt with a platform-specific role and output format, one worked example kernel, and the task's PyTorch code with its inputs. It returns a kernel plus PyTorch code that calls it, in CUDA, AscendC (Huawei's kernel language) or Pallas (Google's TPU kernel interface). The paper uses single attempts without feedback or tools.", "tasks": "Write a kernel that reproduces a PyTorch module's output and runs faster, across 14 categories such as convolution, matrix multiply, normalization and fused operations. Example: multiplying a diagonal matrix by a matrix, where one model's kernel skipped the zero entries. The repository has since added attention tasks and more platforms.", "grading": "The generated code and the reference run on 5 random inputs; outputs must match within a tolerance of 0.01. Runtime is compared with standard PyTorch on the same hardware. Metrics are the shares of tasks where one of k samples compiles, is correct, or is correct and at least as fast as PyTorch. Since May 2026 the harness flags code that falls back to PyTorch instead of a custom kernel.", "construction": "The authors started from the 250 tasks of KernelBench, an earlier GPU-kernel benchmark, sorted them by computation pattern, and compared them with PyTorch's own CUDA kernel library to find missing kinds. Three experienced kernel developers agreed the categories, adding broadcast, resize and optimizer tasks and expanding indexing, for 285 in all. New tasks were checked by compiling and running them.", "results": "Of seven models tested, the best on CUDA, DeepSeek-R1, was correct on 52.6% of tasks in one attempt and faster than PyTorch on 26.0%. No model exceeded 2.5% correct on Huawei's chips or 8.4% on Google's. Showing an example from the same category instead of a simple addition kernel more than doubled pass rates on Google's chips.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2507.17773" }, { "title": "Repository", "url": "https://github.com/wzzll123/MultiKernelBench" } ] }, { "id": "nanogpt-bench", "name": "NanoGPT-Bench", "summary": "An open-ended task: a coding agent must speed up the September 2025 record in the NanoGPT speedrun, a community race to train small GPT-2 models.", "targets": [ "ml-systems" ], "publishDate": "2026-05-19", "publicationType": "Repository-only", "publisher": "Intology", "environment": "The agent works alone in a Docker container holding the 3 September 2025 record script and training data, with 8-GPU H100 training runs and no internet. It experiments within a fixed GPU budget, 512 H100 GPU-hours in the published runs, and checks candidates with a submit command. Ready-made setups are provided for Claude Code and Codex.", "tasks": "Reduce the end-to-end training time of the script while still reaching the target validation loss. Rules forbid changing the data, training objective, validation, timing or compiler settings, installing packages, and brute-force hyperparameter tuning against the validation set. The task can be refreshed with a newer starting record to avoid contamination.", "grading": "The submit command first has a language model, GPT-5.4, compare the code with the starting record and flag rule violations, such as moving work outside the timed section. It then trains 10 times and requires statistical evidence, a one-sided p-value below 0.01, that validation loss is under 3.28. The headline metric is the share of the speedup humans achieved between 3 September 2025 and 19 January 2026.", "construction": "The starting point and human reference come from the community NanoGPT speedrun leaderboard, built on Keller Jordan's modded-nanogpt code; the repository includes snapshots of later human record submissions. The judge's rules mirror the speedrun's own human review. The repository does not describe further validation of the task.", "results": "With 512 H100 GPU-hours each, Claude Code with Opus 4.6 recovered 8.2% to 9.3% of the human progress, depending on the prompt, and Codex with GPT-5.4 recovered 8.6%. Agents spent most of their compute tuning hyperparameters, whereas about 77% of human records introduced algorithmic changes.", "sources": [ { "title": "Repository", "url": "https://github.com/IntologyAI/NanoGPT-Bench" }, { "title": "Blog post", "url": "https://www.intology.ai/blog/nanogpt-bench" } ] }, { "id": "nkibench", "name": "NKIBench", "summary": "A benchmark of 14 language-model kernels on AWS Trainium chips, in which a system must speed up a starting kernel written in NKI, AWS's Python-based kernel language.", "targets": [ "kernels" ], "publishDate": "2025-11-19", "publicationType": "Peer-reviewed paper", "publisher": "Stanford University; Amazon Web Services; University of Toronto", "environment": "The system receives the operator, a starting kernel in NKI, a correctness test and profiler metrics such as memory traffic and how busy each compute unit is. In the paper, the AccelOpt agent keeps the best few candidates each round: a planner proposes changes, an executor rewrites the kernel, and a memory stores lessons from pairs of slow and fast kernels. Hardware: Trainium 1 and 2.", "tasks": "Make a working NKI kernel faster while keeping its output. The 14 kernels come from transformer language models and one Mamba model, ranging from single operators such as matrix multiply, through fused chains such as normalization plus matrix multiply, to larger blocks such as attention and an optimizer step.", "grading": "Outputs are compared with a CPU reference over several random seeds and must stay within a per-task relative error. Running time excludes compilation and averages repeated runs until stable. Results are reported as speedup over the starting kernel and as percent of peak: a theoretical best time from memory and compute limits divided by measured time. No anti-gaming safeguards are described, and models were seen exploiting the random-input checker.", "construction": "Most starting kernels were generated by the official AWS Neuron compiler; for four problems it produced none, so the authors wrote kernels by hand, splitting work into blocks and merging steps, or adapted official examples. Each kernel uses one fixed input configuration. No further selection or validation steps are described.", "results": "AccelOpt with the open models gpt-oss-120b and Qwen3-Coder-480B raised average percent of peak from 49% to 61% on Trainium 1 and from 45% to 59% on Trainium 2. With gpt-oss-120b alone it reached about 1.24 times average speedup, matching repeated sampling of Claude Sonnet 4 at about 26 times lower cost.", "sources": [ { "title": "Paper (MLSys 2026)", "url": "https://proceedings.mlsys.org/paper_files/paper/2026/hash/0f8426558905746fc38da5e335700aec-Abstract-Conference.html" }, { "title": "Preprint", "url": "https://arxiv.org/abs/2511.15915" }, { "title": "Repository", "url": "https://github.com/zhang677/AccelOpt" } ] }, { "id": "peacexec", "name": "PEACExec", "summary": "A benchmark of 146 tasks from 47 popular Python projects, in which a method must make a given function and related functions more efficient.", "targets": [ "repositories" ], "publishDate": "2025-10-20", "publicationType": "Peer-reviewed paper", "publisher": "Zhejiang University; The Chinese University of Hong Kong; Ant Group", "environment": "Each task is a Python project in a container at the version before a developer's efficiency commit, with the target function, project history and tests; the commit message is an optional hint. The benchmark fixes no interaction mode; the paper's own method edits functions in turn, retrieving related code up to 10 times. Budgets and hardware are not stated.", "tasks": "Rewrite a target function, and where needed related functions in other files, so the project's tests use less computation while still passing. 80 tasks involve a single function and 66 require editing several. The paper gives no full worked example.", "grading": "Correctness is the share of solutions passing the task's tests on the first try (pass@1). Efficiency is counted in CPU instructions executed, which varies less than wall-clock time. Speedup is the developer patch's instruction count divided by the candidate's, so values above 1 beat the developer. Opt rate is the instruction reduction relative to GPT-4o simply prompted to optimize, not to the original code. No anti-gaming safeguards are described.", "construction": "From about 440,000 recent commits in over 2,000 top-starred Python projects, keyword matching on commit messages kept about 16,000; dropping changes under 5 or over 150 lines, or touching 5 or more files, left 10,998. A language model ranked these for efficiency relevance, keeping 509, and the authors removed by hand those with too little test coverage, leaving 146.", "results": "The paper compared its own method, PEACE, with prompting GPT-4o and five published optimization methods. PEACE reached 69.2% pass@1, a 46.9% opt rate and a speedup of 0.84 times, still slower than developers' patches; the strongest baseline, SBLLM, reached 58.2%, 28.5% and 0.62 times. The repository says some released data is missing pending an update.", "sources": [ { "title": "Paper (ASE 2025)", "url": "https://conf.researchr.org/details/ase-2025/ase-2025-papers/189/PEACE-Towards-Efficient-Project-Level-Performance-Optimization-via-Hybrid-Code-Editi" }, { "title": "Preprint", "url": "https://arxiv.org/abs/2510.17142" }, { "title": "Repository", "url": "https://github.com/linrenmeng/Peace" } ] }, { "id": "peepholebench", "name": "PeepholeBench", "summary": "A benchmark of 19 missed optimizations in the LLVM compiler, from issue reports, in which a coding agent must patch it to produce better code.", "targets": [ "compilers" ], "publishDate": "2026-07-02", "publicationType": "Preprint", "publisher": "University of Waterloo", "environment": "The agent gets the LLVM repository at the commit before the human fix and a prompt with the issue text, example input and output code, excerpts of LLVM's contributor guide, and steps to generalize, add tests, implement and verify. The paper ran Claude Code, Codex CLI and Gemini CLI with default settings, 2 hours per task and three runs each.", "tasks": "Implement a missed peephole optimization, a local rewrite of a few instructions into cheaper ones, in C++ in LLVM's instruction-combining pass. Tasks correspond to 19 pull requests merged from September 2025 to February 2026. Example: simplify a pattern of arithmetic and comparison instructions that a user reported left unoptimized.", "grading": "Patched compilers must build and pass the human fix's tests. Correctness is checked on about 18,700 mutated variants of 270 tests using Alive2, a tool that proves an optimized output behaves like its input. Profitability is estimated with a static cost model of x86-64 machine code: a task counts as profitable when no variant costs more than with the human fix. Transcripts were audited for access to the original fix.", "construction": "The authors collected closed LLVM issues labelled as missed instruction-combining optimizations, with pull requests merged after 1 September 2025 to limit training-data leakage: 25 in all. They dropped those implemented elsewhere in LLVM and those where over 10% of Alive2 checks were inconclusive, leaving 19. Mutated tests came from 14 rule-based mutators, such as changing boundary values or swapping operators.", "results": "Four agent and model pairs were tested, including Claude Code with Sonnet 4.6 and Codex CLI with GPT-5.4. All patches built, but at best 79% of tasks were fully correct and 42% matched the human fix's profitability within three attempts. Common failures were overly narrow rewrites and misuse of LLVM-specific mechanisms.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2607.02684" }, { "title": "Project page", "url": "https://cs.uwaterloo.ca/~cnsun/public/preprint/peepholebench/" } ] }, { "id": "perfbench", "name": "PerfBench", "summary": "A benchmark of 81 real performance bugs in open-source C# projects, where a coding agent must fix the problem a GitHub issue describes.", "targets": [ "repositories" ], "publishDate": "2025-09-28", "publicationType": "Peer-reviewed paper", "publisher": "Microsoft", "environment": "Each task is a Docker container with the repository just before the developers' fix, its git history removed, plus the issue text. The agent, built on the OpenHands framework, can build and run code and is expected to write its own benchmarks with the standard .NET benchmarking library. It gets at most 100 steps; hardware is not stated.", "tasks": "Fix a performance bug without breaking the project. Bugs concern memory, such as leaks (33); concurrency, such as deadlocks (14); inefficient algorithms (14); input, output and serialization (12); and slow builds or tests (8). Example: the QRCoder library runs out of memory when rendering a QR code as an image.", "grading": "The harness runs the agent's own benchmarks on the code before and after both the agent's change and the developers' fix, comparing time and memory allocation. A task succeeds if at least one benchmark improves, none regress and all existing unit tests pass. There is no numeric threshold or significance test, and no check that the agent's benchmarks are meaningful or that the fix addresses the root cause.", "construction": "The authors searched about 1,200 .NET repositories with over 10 GitHub stars for words such as \"latency\", \"leak\" or \"hang\", keeping commits that reference an issue and change C# code. Both versions had to compile in Docker, and two experienced .NET developers independently confirmed each was a real performance bug and fix, giving 81 tasks from 32 repositories.", "results": "With the default OpenHands agent, GPT-4.1 solved 1.2% of tasks and Claude Sonnet 4 3.7%. The authors' Perf-Agent, which tells the agent to benchmark its changes and trims benchmark output to a summary, raised this to 14.8% and 19.7%. Successful fixes usually cut memory allocations by over 20%, while time savings ranged from 0 to 40%.", "sources": [ { "title": "Paper (AGENT workshop at ICSE 2026)", "url": "https://arxiv.org/abs/2509.24091" }, { "title": "Workshop program", "url": "https://conf.researchr.org/details/icse-2026/agent-2026-papers/20/PerfBench-Can-Agents-Resolve-Real-World-Performance-Bugs-" }, { "title": "Repository", "url": "https://github.com/glGarg/PerfBench" } ] }, { "id": "perfopt-bench", "name": "PERFOPT-Bench", "summary": "A benchmark of 12 long-running tasks in which a coding agent must speed up a deliberately slowed C codebase, such as a database, checked by hidden tests.", "targets": [ "repositories" ], "publishDate": "2026-07-08", "publicationType": "Preprint", "publisher": "OPPO Research Institute; City University of Hong Kong; University of Arizona; The Hong Kong Polytechnic University", "environment": "Each task provides a C codebase with build scripts, a benchmark driver, a bug report describing the slowdown, and a test script to check correctness and speed. Agents such as Claude Code and Codex CLI work autonomously in one session on a Windows 11 desktop and cannot see the reference solution or hidden evaluator. The execution budget is not given.", "tasks": "Make the program faster without changing its outputs. The 12 codebases, of 11,000 to 300,000 lines, include databases, a key-value store, a climate model and a regex engine, with slowdowns from build settings, unused vector instructions, poor memory access patterns or missing parallelism. Example: an index scan slower than a sequential scan.", "grading": "A hidden evaluator compiles the original and the submission, checks outputs on hidden inputs, and reports speedup from repeated timings. The headline is the geometric mean over each agent's valid tasks, so failed tasks are excluded and agents are compared on different task sets. An automated screen and an expert reviewing logs and code discard gains that exploit the evaluator's specific inputs, tamper with outputs or fabricate answers.", "construction": "A language-model generator adapted open-source C programs and inserted bottlenecks from four families. A private reference solver confirmed each could be sped up and proposed a speedup threshold; difficulty was tuned so a pilot solver passed about 45%, giving about 28 tasks. A senior expert picked 12 and adjusted thresholds.", "results": "Seven combinations of agent framework and model were run once per task. Verified geometric-mean speedups ranged from 3.1 times (OpenCode with DeepSeek-V4 Pro) to 9.2 times (OpenCode with GPT-5.5); no combination won every task, and the framework mattered. Raw speedups of up to 493 times came from shortcuts, most often by GPT-5.5.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2607.07744" }, { "title": "Repository (anonymized)", "url": "https://anonymous.4open.science/r/Dataset-D3CC" } ] }, { "id": "pie", "name": "PIE (Performance-Improving Edits)", "summary": "A benchmark in which a model must rewrite a slow but correct C++ contest solution to run faster, with 978 test programs from 41 problems.", "targets": [ "programs" ], "publishDate": "2023-02-15", "publicationType": "Peer-reviewed paper", "publisher": "University of Pennsylvania; Carnegie Mellon University; Google; Google DeepMind", "environment": "The model receives the slow C++ program and an instruction to make it faster, without the problem statement; prompt variants add example slow-to-fast pairs or step-by-step reasoning. There are no tools or feedback, and results are reported for the best of 1 or 8 samples. Programs are compiled with full optimization and timed in a simulator.", "tasks": "Produce a faster program that still passes all tests. All tasks are C++ solutions from CodeNet, IBM's collection of contest submissions, and the paper targets high-level changes such as different algorithms. Example: given a programmer's early accepted solution, match or beat the faster one the same programmer submitted later.", "grading": "A program is correct only if it passes every test. Runtime is measured in gem5, a deterministic simulator of a complete computer, adopted after real hardware showed a 1.12 times average speedup between identical programs. Metrics are the share of programs correct and at least 10% faster, and average speedup with failures counted as 1. The authors found 10 of 120 inspected speedups came from shrinking arrays the tests missed.", "construction": "From CodeNet's C++ submissions, the authors paired each user's earlier accepted program with a later one more than 10% faster, re-timing everything in the simulator. CodeNet's roughly 4 tests per problem were supplemented with tests from a test-generation model built for DeepMind's AlphaCode, giving a median of 104. Splits share no problems: 77,967 training, 2,544 validation and 978 test pairs.", "results": "The paper evaluated GPT-3.5, GPT-4 and CodeLlama with prompting and fine-tuning. The best setup, GPT-3.5 fine-tuned on high-quality pairs plus model-generated ones, optimized 87.63% of programs with an average 6.86 times speedup at 8 samples, versus 3.66 times for the human-written faster versions; the best GPT-4 prompting reached 3.93 times.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2302.07867" }, { "title": "Paper (ICLR 2024)", "url": "https://proceedings.iclr.cc/paper_files/paper/2024/hash/ce65173b994cf7c925c71b482ee14a8d-Abstract-Conference.html" }, { "title": "Repository", "url": "https://github.com/LearningOpt/pie" }, { "title": "Project page", "url": "https://pie4perf.com" } ] }, { "id": "re-bench-optimization", "name": "RE-Bench (optimization tasks)", "summary": "Two speed-up tasks from METR's seven-task AI research-engineering benchmark: make a model-finetuning script faster, and replace slow PyTorch prefix-sum code with a fast GPU kernel.", "targets": [ "ml-systems", "kernels" ], "publishDate": "2024-11-22", "publicationType": "Peer-reviewed paper", "publisher": "METR (Model Evaluation and Threat Research)", "environment": "Each task is a virtual machine with the starting code, 4 H100 GPUs for finetuning or 1 for the kernel, and a scoring script the agent can run anytime; the best logged score counts. Agents ran commands or searched over whole solutions as a tree, with internet access and 30 minutes to 8 hours; human experts got 8 hours.", "tasks": "Finetuning: cut the 272-second runtime of a script that finetunes a model on 1,000 examples, without changing its behavior. Kernel: write a fast GPU function, ideally in the Python-like Triton language, for a prefix sum over integers that counts only certain elements, replacing a 5-millisecond PyTorch version.", "grading": "The score is the logarithm of runtime, rescaled so the starting code scores 0 and the task author's reference solution 1 (94 seconds for finetuning). The kernel's output must match the reference on a random 100-million-element input; the finetuned weights must stay within a small distance of a reference run's. Score@k is the best of k runs. The paper notes some loopholes were only caught by manual inspection.", "construction": "METR staff wrote the tasks with starting and reference solutions, and dropped over 12 specifications and more than 5 implementations judged infeasible in 8 hours, unrealistic, easy to saturate or not adding skill coverage. Human baselines came from METR's network, hiring applicants and graduate students; 7 of 9 kernel experts and all 8 finetuning experts scored above 0.05.", "results": "In the paper (Claude 3.5 Sonnet and o1-preview), the best agent kernels ran about twice as fast as the reference and one o1-preview solution beat all 9 human experts; on finetuning, agents fell well short. In June 2025 METR reported o3 reward hacking, such as overwriting the timer, in 6 of 24 kernel runs and all 21 finetuning runs. METR deprecated RE-Bench in October 2026; no newer scores were published.", "sources": [ { "title": "Paper (ICML 2025)", "url": "https://arxiv.org/abs/2411.15114" }, { "title": "Repository", "url": "https://github.com/METR/RE-Bench" }, { "title": "Blog post (reward hacking)", "url": "https://metr.org/blog/2025-06-05-recent-reward-hacking" } ] }, { "id": "robust-kbench", "name": "robust-kbench", "summary": "A benchmark of about a dozen neural-network operations, such as LayerNorm, where CUDA kernels must beat PyTorch while staying correct across many input settings.", "targets": [ "kernels" ], "publishDate": "2025-09-16", "publicationType": "Preprint", "publisher": "Sakana AI", "environment": "Each task folder holds PyTorch reference code for the forward and, for most tasks, backward pass, plus configurations for several input sizes and weight initializations. The released harness compiles, tests, times and can profile a submitted CUDA file. The benchmark sets no agent protocol; the paper's own agent received the PyTorch code and iterated over 10 generations on H100 GPUs.", "tasks": "Replace a PyTorch module with a faster kernel in CUDA, NVIDIA's C++-based GPU language. The README lists 12 tasks, including linear and convolution layers, LayerNorm, cross-entropy, Llama's feed-forward and RMSNorm layers, a ResNet block and the original KernelBench tasks. Example: write the backward pass of an MNIST linear layer.", "grading": "A kernel counts as correct only if it matches the reference within a tolerance of 0.00001 across all input sizes, initializations and random seeds, so it cannot exploit one fixed shape or tiny weights. Runtime is averaged over 2,000 runs and compared with PyTorch, or its compiler for LayerNorm. Tasks are first screened for near-constant outputs, inputs that do not affect results, and baselines a language model judges inefficient.", "construction": "The authors wrote new tasks drawn from training an MNIST digit classifier and from ResNet-18 and Llama inference; the paper does not say how each was validated. They also screened 200 KernelBench tasks, flagging 88 in the paper's tables for problems such as near-zero outputs; excluding them cut a reported average speedup from 3.13 to 1.49 times.", "results": "The paper's agent, which translated PyTorch to CUDA and then evolved kernels with an ensemble of language models, beat PyTorch on most tasks, for example 3.93 times on Llama RMSNorm's forward pass, but reached only 0.18 times compiled PyTorch on LayerNorm. Backward-pass speedups were at most 1.20 times, and some kernels overfit the shapes they were tuned on.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2509.14279" }, { "title": "Repository", "url": "https://github.com/SakanaAI/robust-kbench" }, { "title": "Project page", "url": "https://sakana.ai/ai-cuda-engineer/" } ] }, { "id": "sol-execbench", "name": "SOL-ExecBench", "summary": "An NVIDIA benchmark of 235 GPU kernel problems from 124 real AI models, where a kernel must replace a PyTorch reference and is scored against the hardware's speed limit.", "targets": [ "kernels" ], "publishDate": "2026-03-19", "publicationType": "Preprint", "publisher": "NVIDIA", "environment": "Each problem is a file holding a self-contained PyTorch reference, typed input and output shapes, and about 16 concrete input sizes. A solution points to source code in PyTorch, Triton, CUDA C++ or other NVIDIA kernel libraries, which the released harness builds and runs on an NVIDIA B200 GPU. The benchmark fixes no agent setup, attempt count or budget.", "tasks": "Write a kernel that computes the same result as the reference, faster. The 235 public problems (10 more are held back for a competition) are 94 single operations, 82 fused groups of operations, 33 problems in 8-bit or 4-bit number formats and 26 from the earlier FlashInfer-Bench. Example: a residual add fused with normalization for Llama 3.1 8B.", "grading": "Outputs must match the reference over several seeded trials, timed with locked GPU clocks and a cleared cache. The SOL score is 0 for incorrect kernels, 0.5 when matching an unreleased scoring baseline faster than the PyTorch reference (described both as agent-generated and as torch.compile-based), and 1 at the speed-of-light time, a bound from operation count and memory traffic at peak hardware rates. Static and runtime checks and a language-model judge screen for tricks such as patched timers.", "construction": "A frontier language model read 124 models from Hugging Face and other sources, mostly language, diffusion and multimodal, and wrote 7,400 standalone PyTorch subgraphs, sampled to cover 11 descriptive dimensions. Experts and a language model reviewed references, execution checks verified numerics, and an optimizing agent was run on each to expose loopholes; flawed problems were removed, leaving 245.", "results": "The paper compares no models. Its own multi-agent optimizer reached a median SOL score of 0.732, though the paper's discussion treats 0.5 as parity with the PyTorch reference, not the scoring baseline, leaving the comparison point unclear. Speedup over PyTorch barely correlated with distance from the hardware limit (correlation 0.10), and 14.5% of submissions were flagged as gaming, mostly by silently lowering precision.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2603.19173" }, { "title": "Repository", "url": "https://github.com/NVIDIA/SOL-ExecBench" }, { "title": "Dataset", "url": "https://huggingface.co/datasets/nvidia/SOL-ExecBench" }, { "title": "Leaderboard", "url": "https://research.nvidia.com/benchmarks/sol-execbench" } ] }, { "id": "sql-rewritebench", "name": "SQL-RewriteBench", "summary": "A benchmark of 180 PostgreSQL queries that a rewriting system or model must turn into an equivalent faster or simpler query, or decline to rewrite.", "targets": [ "queries" ], "publishDate": "2026-07-10", "publicationType": "Preprint", "publisher": "Zhejiang University; Huawei", "environment": "A method receives only the SQL query and a schema description, never the reference rewrite or validation data, and returns one rewritten query or a decision not to rewrite, with no execution feedback. Queries run on PostgreSQL 16 on a desktop Intel machine, against standard benchmark databases of about 20 GB. Retry limits for language models are not stated.", "tasks": "Rewrite a query to return the same rows faster or more simply. Instances form four pools: 48 testing tricky semantics such as NULLs and duplicates, 44 aimed at speed, 58 with complex structure such as deep nesting, and 30 with PostgreSQL-specific features. Example: remove a redundant nested subquery.", "grading": "The rewrite's result is compared with the reference's, ignoring row order unless observable. Runtime is the median of five runs. The headline score, CGOQ, credits only correct rewrites: changes within 5% score nothing, larger speedups or slowdowns score on a capped curve (about 0.76 at 2 times), plus a small simplicity bonus unless the query slows down. Failures score zero; negative averages mean net slowdowns.", "construction": "Queries came from Calcite and database regression tests (60) and the DSB (47), SQLStorm (39) and TPC-DS (34) benchmarks, minus incomplete, duplicate and data-modifying statements. Database administrators then edited queries to add rewrite opportunities such as redundant subqueries or delayed filters. Each reference rewrite was checked for identical results; speed-pool cases needed at least a 1.10 times reference speedup.", "results": "Eight methods were tested, including GPT-5.5 and other language models prompted directly and research rewrite systems; all had negative CGOQ scores. GPT-5.5 produced result-matching output on 95% of instances, but 121 of those ran slower. Reference rewrites averaged 1.67 times faster, versus about 0.63 times for the directly prompted language models.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2607.09251" }, { "title": "Repository", "url": "https://github.com/SQL-RewriteBench/benchmark" } ] }, { "id": "swe-compass", "name": "SWE-Compass (performance optimization tasks)", "summary": "The performance part of SWE-Compass, a 2,000-task coding-agent benchmark: 201 tasks in which an agent must cut a GitHub project's runtime by at least 20%.", "targets": [ "repositories" ], "publishDate": "2025-11-07", "publicationType": "Preprint", "publisher": "Kuaishou Technology; Nanjing University", "environment": "Each task is a Docker image of a GitHub repository at the commit before a merged pull request, with a problem statement describing the slow code. In the paper, agents ran under the SWE-Agent and Claude Code frameworks with shell and file-editing tools, up to 150 turns, no network access and one attempt per task. Hardware is not stated.", "tasks": "Speed up existing code while keeping its tests passing. The full benchmark covers 10 languages, but the paper does not give the mix for optimization tasks. Example: make the Svelte web framework's compiler avoid redundant work and repeated syntax-tree traversals when generating code for reactive values, checked by its tests.", "grading": "The harness applies the agent's patch in Docker and runs the task's tests. A task scores 1 only if the tests pass and execution time is under 80% of the original code's; the headline number is the share scored 1 in one attempt. The paper does not say which test is timed, how often it is run, or describe safeguards against gaming the timing.", "construction": "The authors took about 50,000 merged, issue-linked pull requests from 40 repositories with 500+ stars and runnable unit tests; 30 annotators repaired failing environment builds. Optimization candidates passed their tests before and after the change and cut runtime by more than 30%. These were filtered for difficulty by size of change and model success rates, and balanced across languages.", "results": "Ten models were tested. Under SWE-Agent, optimization scores ranged from 13.9% (SWE-agent-LM-32B) to 32.8% (Claude Sonnet 4). Under Claude Code, every model tried scored lower, from 6.0% (DeepSeek-V3) to 24.9% (Claude Sonnet 4); the paper does not explain the gap.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2511.05459" }, { "title": "Repository", "url": "https://github.com/kwaipilot/SWE-Compass" }, { "title": "Dataset", "url": "https://huggingface.co/datasets/Kwaipilot/SWE-Compass" } ] }, { "id": "swe-perf", "name": "SWE-Perf", "summary": "A benchmark of 140 tasks in which an agent must make a Python repository's tests run faster, compared with the developers' own patch.", "targets": [ "repositories" ], "publishDate": "2025-07-16", "publicationType": "Peer-reviewed paper", "publisher": "Xi'an Jiaotong University; TikTok; National University of Singapore; University of California San Diego", "environment": "In the oracle setting, a model sees the files the developers edited and the functions to speed up, and writes one patch. In the realistic setting, a system gets the whole repository and the functions the tests exercise; the paper ran two agent systems, Agentless and OpenHands, with Claude 3.7 Sonnet. Whether agents could run the tests is not stated.", "tasks": "Make specific unit tests run faster, without breaking them, by editing the repository. Tasks come from 9 Python projects, including astropy, scikit-learn and sympy. By the authors' rough breakdown, the developers' patches add caching (33%), remove redundant work (26%), change algorithms (26%) or improve data and resource use (15%).", "grading": "A patch is scored on whether it applies, whether all relevant tests pass, and runtime saved. Each test runs 20 times after warm-ups, outliers are dropped, and a statistical test comparing the two sets of timings finds the largest runtime reduction still supported; this is averaged over tests and tasks. The developers' patches score 10.85%. No safeguards against gaming beyond the tests are described.", "construction": "From 102,241 pull requests in 12 repositories used by the SWE-bench benchmark, the authors kept 19,797 that resolve an issue. Running tests 3 times on one CPU core, they kept 1,696 where a passing test of the changed code, not itself edited, became over 30% faster. Re-timing with the statistical test, requiring a supported gain above 5%, left 140.", "results": "In the oracle setting, the best of 10 models, Claude 4 Sonnet, saved 1.76% of runtime against 10.85% for the developers; in the realistic setting, OpenHands reached 2.26%. A July 2026 audit replayed the developers' patches on four cloud machine types; only 11 of 140 met the benchmark's own validity rule in all 12 replays.", "sources": [ { "title": "Paper (ICML 2026)", "url": "https://arxiv.org/abs/2507.12415" }, { "title": "Repository", "url": "https://github.com/swe-perf/swe-perf" }, { "title": "Project page", "url": "https://swe-perf.github.io" }, { "title": "Leaderboard", "url": "https://swe-perf.github.io/leaderboard.html" }, { "title": "Dataset", "url": "https://huggingface.co/datasets/SWE-Perf/SWE-Perf" }, { "title": "Audit (preprint)", "url": "https://arxiv.org/abs/2607.01211" } ] }, { "id": "swe-pro", "name": "SWE-Pro (PRO-Bench)", "summary": "A benchmark of 102 tasks in which a language model must make a pandas, scikit-learn or xarray function faster and use less memory, compared with the developers' optimization.", "targets": [ "repositories" ], "publishDate": "2026-06-24", "publicationType": "Peer-reviewed paper", "publisher": "Technical University of Munich; Siemens AG", "environment": "Not agentic: the model receives a task description naming the target function and input settings, plus code context, either the files the developers edited or files found by keyword search. It must reply only with search-and-replace edits; no tools, test runs or sample counts are described. Timing uses a MacBook Pro pinned to one CPU core.", "tasks": "Make a target function faster and use less memory without changing behavior, public interfaces or tests. All tasks are Python, from three large data libraries (median about 197,000 lines), mostly pandas. Example: an xarray rolling-window operation, timed across window sizes and strides; gains there appear only on large inputs.", "grading": "A patch must apply and pass all unit tests tied to the target function. Author-written tests then measure runtime, peak memory and time-weighted memory at several input sizes, repeating until the 90% confidence interval is within 10% of the mean; effects too small relative to noise are discarded. Improvement factor is old value divided by new, averaged over correct patches; pass rate is the share of tasks with a measurable gain. No anti-gaming checks are described.", "construction": "The authors collected merged pull requests labelled as performance work that change Python source and have a changelog entry, and rebuilt before and after states in containers. They wrote performance tests from each pull request's description and discussion, and removed by hand non-reproducible or unstable cases and pull requests mixing unrelated changes, leaving 102.", "results": "Six models were tested, including GPT-5.2 and Claude Sonnet 4.6. The developers' patches had measurable runtime gains on 93.1% of tasks, averaging 15.48 times. Given the developers' files, the best runtime pass rate was 12.7%, by Gemini 3.1 Flash-Lite at 1.27 times; GPT-5.2's measured patches were slower than the original, and memory gains were rare.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2606.25530" }, { "title": "Repository", "url": "https://github.com/probench-swe/SWE-Pro" }, { "title": "Dataset", "url": "https://huggingface.co/datasets/probench-swe/SWE-Pro" }, { "title": "Paper (NeurIPS 2026)", "url": "https://neurips.cc/virtual/2026/poster/139202" } ] }, { "id": "sysllmatic", "name": "SysLLMatic evaluation suite", "summary": "A suite of 174 programs, mostly small C++ functions plus Java scientific kernels and applications, that a language model must make faster or more energy-efficient.", "targets": [ "programs" ], "publishDate": "2025-06-02", "publicationType": "Preprint", "publisher": "Purdue University; Loyola University Chicago", "environment": "The optimizer receives the original source and its tests; measuring energy needs a Linux Intel machine with root access. In the paper, the authors' pipeline adds profiler hotspots, syntax trees and a catalog of 43 optimization patterns, edits functions (classes for the large applications) and gets one round of measured feedback. Measurements ran on an Intel Xeon workstation.", "tasks": "Cut the runtime, energy, memory or CPU work of existing code. Suites: 164 HumanEval-X C++ functions (small programming problems), 5 SciMark2 Java numerical kernels such as the fast Fourier transform, and 5 DaCapo Java applications of 8,000 to 400,000 lines. Example: parallelize hot code in BioJava, a bioinformatics library.", "grading": "C++ functions must pass their tests, SciMark2 outputs must stay within an accuracy threshold, and DaCapo changes must pass the projects' unit tests. Wall time, peak memory, CPU cycles and energy from Intel's built-in power counters are averaged after warm-up runs and compared with the original. Metrics are the share of programs improved by at least 10% and average improvement. Only manual review checks for gaming.", "construction": "The authors took all 164 HumanEval-X C++ problems, adding stress tests translated from the COFFE efficiency benchmark; the 5 SciMark2 kernels; and DaCapo, a Java benchmark suite, where of 22 applications 20 built, 8 had source and tests, and the 5 using the Maven build tool were kept. Files over 1,000 lines were skipped.", "results": "With GPT-4o, SysLLMatic improved latency by at least 10% on 23.2% of C++ functions, against 15.0% for the earlier PerfCodeGen method and 23.8% for compiler-flag tuning. On the DaCapo Java applications with GPT-4.1 it averaged 1.54 times lower latency and 1.24 times lower energy, against 1.01 and 1.08 times for compiler-based optimization. No newer frontier models were tested.", "sources": [ { "title": "Preprint", "url": "https://arxiv.org/abs/2506.01249" }, { "title": "Repository", "url": "https://github.com/PurdueDualityLab/sysllmatic" } ] }, { "id": "tritongym", "name": "TritonGym", "summary": "A benchmark of 122 GPU operators (164 in the current repository) in which an agent rewrites a PyTorch reference as a fast Triton kernel.", "targets": [ "kernels" ], "publishDate": "2026-04-30", "publicationType": "Peer-reviewed paper", "publisher": "University of California San Diego; Meta; Princeton University", "environment": "Each task gives the PyTorch reference, input shapes, and JSON descriptions of the GPU and language (Triton, a Python-like language for GPU kernels). Through shared function calls, agents can compile, check correctness, time, profile and search the web. The paper ran one-shot generation and two reimplemented agent workflows (AlphaEvolve, 5 attempts; Geak, up to 3 rounds) on NVIDIA H100 GPUs.", "tasks": "Replace a PyTorch operator with an equivalent, fast Triton kernel. Tasks cover common operators (matrix multiplication, attention, normalization, activations), invented operators with no public implementation, and two lower-level Triton dialects, TLX and Gluon. Example: Max Reduce GEMM, a matrix multiplication whose sums are replaced by maxima.", "grading": "Outputs must match the PyTorch reference on random inputs at each given shape, within tolerances. Speed is compared not with the PyTorch code but with a hidden oracle: an expert-written Triton kernel absent from the paper's prompts. Perf@1, averaged over tasks, is oracle time divided by the model's time (above 1 beats the experts; failures score 0); Pass@1 is the share correct. No anti-gaming checks are described.", "construction": "The authors combined 35 operators from their own expert-written production Triton kernels, 47 converted from public repositories and earlier benchmarks, 13 invented by altering known computations (for example, GELU instead of softmax in attention) with hand-written oracles, and 27 TLX or Gluon tasks. The repository now lists 164. Validation is not described.", "results": "The paper tested Claude Sonnet 4.5, GPT-5, Llama 4 Maverick, Qwen3-Coder and DeepSeek-V3. Agent workflows beat one-shot generation (32.8% correct on average); the best, AlphaEvolve with Claude Sonnet 4.5, was correct on 54.8% of tasks with Perf@1 of 0.583, well below expert speed on average. Invented-operator and dialect tasks were hardest.", "sources": [ { "title": "Paper (ICML 2026)", "url": "https://proceedings.mlr.press/v306/guan26d.html" }, { "title": "OpenReview", "url": "https://openreview.net/forum?id=Htq2loTQTL" }, { "title": "Repository", "url": "https://github.com/yil384/TritonGym" }, { "title": "Leaderboard", "url": "https://tritongym.picasso-lab.com" } ] } ] }