Start with real code
One target file, from a small algorithm to a 4,218-line application component.
A visual guide to RefactorBench-JS
Can an AI split up code without quietly breaking it?
Refactoring should change how code is organized—not what the software does. RefactorBench-JS checks that promise by giving coding agents large JavaScript and React files, then testing the behavior they were supposed to preserve.
one-big-file.jsx
123scored fixtures
90web / JavaScript
33React Native
14model × tool conditions
01 · The basic idea
Judge the behavior, not the makeover.
A refactor can look cleaner and still break a button, change an API response, or lose saved state. RefactorBench keeps the agent away from the holdout tests, then runs those tests after the work is finished.
Ask an agent to split it
The agent can create modules, move logic, and reconnect the pieces.
Keep the test hidden
The agent cannot optimize for the exact assertions used to grade it.
Run the behavior
Passing means the observable behavior survived the reorganization.
The benchmark’s rule of thumb “Behavioral preservation is a functional property, and functional properties are best verified by functional tests.”
02 · What gets tested
Not just toy functions.
The fixtures span pure logic, data, APIs, web interfaces, and mobile apps. The test changes with the kind of behavior that matters.
01Algorithms
Edge cases, return values, and data-structure invariants must stay intact.
02Web interfaces
Rendering, events, browser storage, and multi-step user flows are exercised.
03Mobile apps
Navigation, gestures, native APIs, modals, and mobile state flows are checked.
03 · Baseline results
A test tool helps—but not every model.
Seven model configurations attempted the same 123 fixtures, once with a unit-test runner available and once without it. Toggle the chart to compare them.
Hidden-test pass rate. Every model/tool condition used the same 123 fixtures.
Sonnet 4.6 passed 29 fixtures with the runner and 17 without it.
Two models scored slightly lower with the test runner available.
Failed in all 14 baseline conditions; large UI files dominated this group.
04 · The trust gap
“Done” is not the same as correct.
In the highest-scoring baseline condition, the agent reported success far more often than the hidden tests confirmed preserved behavior.
98/ 123
Agent said “success”
Gemini 3.0 Pro with the test runner enabled.
31/ 123
Hidden tests passed
The external behavioral check tells a different story.
68 false-confidence failures
The agent reported success, but the hidden tests still found broken behavior.
05 · Where attempts failed
Most failures were basic, not mysterious.
Across 1,436 failed outcomes, incomplete attempts, syntax errors, and broken module boundaries explained more than nine out of ten failures.
Reported non-success47.9%
Syntax / parse29.4%
Import / export15.3%
Everything else7.4%
06 · What it means
The model is only half the system.
A coding agent also depends on its tools, its stopping behavior, and the external checks around it. RefactorBench makes those differences measurable instead of trusting a clean-looking diff or a confident completion message.
Run the code. Compilation and visual cleanliness cannot prove behavior survived.
Evaluate model + tools together. The same tool can help one model and fail to help another.
Measure calibration. An agent that knows when it failed is safer than one that breaks things confidently.