CQB runs from Aug 4 onward
10 arms β No refactor baseline, command-center / Codex Luna (high), command-center / Codex Luna (medium), command-center / Sonnet 5 (medium), command-center / Opus 5 (high), command-center / Sonnet 5 (high), cursor / Opus 5 (high), cursor / Sonnet 5 (medium), ant / Opus 5 (high), ant / Sonnet 5 (medium) Β· generated 8/9/2026, 5:54:17 AM
At a glance β across all arms β
Best: 1 / 11Per premade, No refactor baseline's place among arms with an observed weighted churn (insertions + 2Β·edit-loc, tests-gated) (lower is better). Rank uses the point estimates; β best means the gap to first is inside the Β±8.0 noise band. Missing arms are excluded from that row's denominator and reported as incomplete coverage.
community-chat
| Premade | Rank | Weighted churn | Ξ best | Field | No refactor baseline tests | Flaw audit | Details | ||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| loose-view-params-scattered-policy | tests failed | β Β· raw 248 | +β vs cursor / Sonnet 5 (medium) | ΓΓΓΓΓΓ | β 3/4 | β | All arms
|
conference-review
| Premade | Rank | Weighted churn | Ξ best | Field | No refactor baseline tests | Flaw audit | Details | ||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| fused-author-conflicts | tests failed | β Β· raw 142 | +β vs command-center / Opus 5 (high) | ΓΓΓΓΓΓΓΓ | β 0/4 | β | All arms
|
full-names
| Premade | Rank | Weighted churn | Ξ best | Field | No refactor baseline tests | Flaw audit | Details | ||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| fixed-slot-render | 1 / 10 | 57 | β | β 4/4 | β | All arms
|
interactive-fiction
| Premade | Rank | Weighted churn | Ξ best | Field | No refactor baseline tests | Flaw audit | Details | ||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| inline-passage-records | 10 / 10 | 420 | +254 vs cursor / Sonnet 5 (medium) | β 4/4 | β | All arms
|
minesweeper
| Premade | Rank | Weighted churn | Ξ best | Field | No refactor baseline tests | Flaw audit | Details | ||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| boolean-cell-state-inline-views | tests failed | β Β· raw 517 | +β vs command-center / Opus 5 (high) | ΓΓΓ | β 2/4 | β | All arms
|
photo-manager
| Premade | Rank | Weighted churn | Ξ best | Field | No refactor baseline tests | Flaw audit | Details | ||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| working-view-selection-split | 3 / 9 | 190 | +26 vs cursor / Sonnet 5 (medium) | ΓΓΓ | β 4/4 | β | All arms
|
pixel-art
| Premade | Rank | Weighted churn | Ξ best | Field | No refactor baseline tests | Flaw audit | Details | ||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| palette-registry-by-name | 5 / 10 | 827 | +219 vs command-center / Opus 5 (high) | ΓΓΓΓΓ | β 4/4 | β | All arms
| ||||||||||||||||||||||||||||||||||||||||||||
| sequential-mirrored-paint | tests failed | β Β· raw 679 | +β vs command-center / Opus 5 (high) | ΓΓΓΓΓΓΓ | β 2/4 | β | All arms
|
portfolio-ledger
| Premade | Rank | Weighted churn | Ξ best | Field | No refactor baseline tests | Flaw audit | Details | ||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| scattered-reconciliation-logic | 4 / 9 | 284 | +74 vs cursor / Sonnet 5 (medium) | ΓΓΓΓΓ | β 4/4 | β | All arms
|
quilldoc-compiler
| Premade | Rank | Weighted churn | Ξ best | Field | No refactor baseline tests | Flaw audit | Details | ||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| source-renderer-separate-outline | all failed | β Β· raw 781 | β | ΓΓΓΓΓΓΓΓΓΓ | β 0/4 | β | All arms
|
todolist
| Premade | Rank | Weighted churn | Ξ best | Field | No refactor baseline tests | Flaw audit | Details | ||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| flat-lists-query-enums | tests failed | β Β· raw 234 | +β vs cursor / Sonnet 5 (medium) | ΓΓ | β 3/4 | β | All arms
|
Ξ best is the selected arm's distance from the best observed value; smaller is better and β means the selected arm is best. A filled dot is the selected arm; peer dots are gray. The shaded field is within one noise band of best; distance is shown in noise-band units and diamonds cap values beyond 5Γ. Γ marks a tests-gated failure. Expand βAll armsβ for exact values and failure provenance.
1.Infra trustworthiness
How much to trust the rest of this report β jobs that failed before scoring, and feature-agent instances that never produced a clean measurement. Means in Β§3βΒ§4 are over tested instances only; everything counted as a failure here is excluded from them.
Caveats present
| Arm | Jobs | Feature-agent instances | |||||||
|---|---|---|---|---|---|---|---|---|---|
| evaluated | refactor-failed | regressed | eval-failed | incomplete | tested | apply-failed | test-infra-failed | unknown | |
| No refactor baseline | 11 | 0 | 0 | 0 | 0 | 44 | 0 | 0 | 0 |
| command-center / Codex Luna (high) | 19 | 1 | 2 | 0 | 0 | 60 | 0 | 0 | 0 |
| command-center / Codex Luna (medium) | 8 | 0 | 3 | 0 | 0 | 16 | 0 | 0 | 0 |
| command-center / Sonnet 5 (medium) | 11 | 0 | 0 | 0 | 0 | 30 | 0 | 0 | 0 |
| command-center / Opus 5 (high) | 11 | 0 | 0 | 0 | 0 | 33 | 0 | 0 | 0 |
| command-center / Sonnet 5 (high) | 9 | 1 | 0 | 0 | 0 | 27 | 0 | 0 | 0 |
| cursor / Opus 5 (high) | 11 | 0 | 0 | 0 | 0 | 44 | 0 | 0 | 0 |
| cursor / Sonnet 5 (medium) | 11 | 0 | 0 | 0 | 0 | 44 | 0 | 0 | 0 |
| ant / Opus 5 (high) | 11 | 0 | 0 | 0 | 0 | 44 | 0 | 0 | 0 |
| ant / Sonnet 5 (medium) | 11 | 0 | 0 | 0 | 0 | 44 | 0 | 0 | 0 |
No refactor baseline β Clean: all 11 job(s) evaluated and all 44 feature-agent instance(s) tested.
command-center / Codex Luna (high) β Caveats: 1 refactor failed, 2 refactor regressed behavior.
Non-evaluated jobs:
photo-manager/working-view-selection-split/replicate-1β refactor-failed (failed)pixel-art/palette-registry-by-name/replicate-1β regressed (regressed)todolist/flat-lists-query-enums/replicate-1β regressed (regressed)
command-center / Codex Luna (medium) β Caveats: 3 refactor regressed behavior.
Non-evaluated jobs:
community-chat/loose-view-params-scattered-policy/replicate-1β regressed (regressed)conference-review/fused-author-conflicts/replicate-1β regressed (regressed)minesweeper/boolean-cell-state-inline-views/replicate-1β regressed (regressed)
command-center / Sonnet 5 (medium) β Clean: all 11 job(s) evaluated and all 30 feature-agent instance(s) tested.
command-center / Opus 5 (high) β Clean: all 11 job(s) evaluated and all 33 feature-agent instance(s) tested.
command-center / Sonnet 5 (high) β Caveats: 1 refactor failed.
Non-evaluated jobs:
portfolio-ledger/scattered-reconciliation-logic/replicate-1β refactor-failed (failed)
cursor / Opus 5 (high) β Clean: all 11 job(s) evaluated and all 44 feature-agent instance(s) tested.
cursor / Sonnet 5 (medium) β Clean: all 11 job(s) evaluated and all 44 feature-agent instance(s) tested.
ant / Opus 5 (high) β Clean: all 11 job(s) evaluated and all 44 feature-agent instance(s) tested.
ant / Sonnet 5 (medium) β Clean: all 11 job(s) evaluated and all 44 feature-agent instance(s) tested.
2.Feature-agent metrics
Per task Γ arm, the mean across tested replicates Γ feature-agent instances (Β± standard error; n tested observations in each cell). Diff size is the post-exclusion, canonicalized churn the scores are computed from on current-schema runs. Lower diff size is better; higher scores are better.
Total churn β lines changed (ins+mod+del, canonicalized) (lower is better)
Edit-loc (mod+del, canonicalized) (lower is better)
Diff size β files changed (post-exclusion) (lower is better)
| Arm | community-chat/loose-view-params-scattered-policy | conference-review/fused-author-conflicts | full-names/fixed-slot-render | interactive-fiction/inline-passage-records | minesweeper/boolean-cell-state-inline-views | photo-manager/working-view-selection-split | pixel-art/palette-registry-by-name | pixel-art/sequential-mirrored-paint | portfolio-ledger/scattered-reconciliation-logic | quilldoc-compiler/source-renderer-separate-outline | todolist/flat-lists-query-enums | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No refactor baseline | 1 Β·4 | 3 Β·4 | 1 Β·4 | 1 Β·4 | 1 Β·4 | 1 Β·4 | 2 Β·4 | 2 Β·4 | 1 Β·4 | 4 Β·4 | 1 Β·4 | 2 Β±0.1 Β·44 |
| command-center / Codex Luna (high) | 2 Β±0.4 Β·6 | 6 Β±0.6 Β·6 | 1 Β·6 | 4 Β±0.9 Β·6 | 2 Β±0.6 Β·6 | 2 Β·4 | 2 Β·4 | 2 Β·6 | 1 Β·6 | 5 Β±0.3 Β·6 | 2 Β·4 | 3 Β±0.2 Β·60 |
| command-center / Codex Luna (medium) | β | β | 1 Β·2 | 5 Β·2 | β | 2 Β·2 | 2 Β·2 | 2 Β·2 | 3 Β·2 | 4 Β·2 | 1 Β·2 | 3 Β±0.3 Β·16 |
| command-center / Sonnet 5 (medium) | 1 Β·3 | 4 Β·3 | 1 Β·3 | 2 Β·3 | 1 Β·2 | 1 Β·2 | 2 Β·3 | 2 Β·3 | 1 Β·3 | 4 Β·2 | 1 Β·3 | 2 Β±0.2 Β·30 |
| command-center / Opus 5 (high) | 4 Β·3 | 7 Β·3 | 1 Β·3 | 4 Β±0.3 Β·3 | 4 Β±0.3 Β·3 | 2 Β·3 | 4 Β·3 | 4 Β·3 | 6 Β±0.3 Β·3 | 12 Β±0.7 Β·3 | 6 Β·3 | 5 Β±0.5 Β·33 |
| command-center / Sonnet 5 (high) | 1 Β·3 | 3 Β·3 | 1 Β·3 | 3 Β·3 | 1 Β·3 | β | 2 Β·3 | 2 Β·3 | β | 4 Β·3 | 1 Β·3 | 2 Β±0.2 Β·27 |
| cursor / Opus 5 (high) | 8 Β±0.3 Β·4 | 9 Β·4 | 2 Β·4 | 8 Β·4 | 8 Β±0.5 Β·4 | 5 Β±0.7 Β·4 | 6 Β·4 | 7 Β±0.4 Β·4 | 9 Β·4 | 11 Β±0.3 Β·4 | 12 Β±0.3 Β·4 | 8 Β±0.4 Β·44 |
| cursor / Sonnet 5 (medium) | 7 Β·4 | 8 Β·4 | 1 Β·4 | 7 Β·4 | 3 Β·4 | 3 Β±0.3 Β·4 | 3 Β·4 | 5 Β±0.3 Β·4 | 6 Β·4 | 4 Β·4 | 9 Β·4 | 5 Β±0.4 Β·44 |
| ant / Opus 5 (high) | 7 Β·4 | 10 Β±0.3 Β·4 | 1 Β·4 | 6 Β·4 | 7 Β±0.3 Β·4 | 5 Β±0.5 Β·4 | 6 Β±0.3 Β·4 | 8 Β±0.6 Β·4 | 8 Β·4 | 8 Β±0.3 Β·4 | 8 Β·4 | 7 Β±0.3 Β·44 |
| ant / Sonnet 5 (medium) | 10 Β·4 | 10 Β·4 | 1 Β·4 | 7 Β·4 | 4 Β·4 | 5 Β±0.4 Β·4 | 5 Β±0.3 Β·4 | 2 Β·4 | 9 Β±0.6 Β·4 | 4 Β·4 | 7 Β·4 | 6 Β±0.4 Β·44 |
Line-based score (maximum β Ξlines) (higher is better)
| Arm | community-chat/loose-view-params-scattered-policy | conference-review/fused-author-conflicts | full-names/fixed-slot-render | interactive-fiction/inline-passage-records | minesweeper/boolean-cell-state-inline-views | photo-manager/working-view-selection-split | pixel-art/palette-registry-by-name | pixel-art/sequential-mirrored-paint | portfolio-ledger/scattered-reconciliation-logic | quilldoc-compiler/source-renderer-separate-outline | todolist/flat-lists-query-enums | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No refactor baseline | 66 Β±26.4 Β·4 | 0 Β·4 | 64 Β±0.9 Β·4 | 4 Β±3.3 Β·4 | 0 Β·4 | 199 Β±2.4 Β·4 | 141 Β±11.7 Β·4 | 123 Β±71.1 Β·4 | 225 Β±5.1 Β·4 | 0 Β·4 | 44 Β±17.0 Β·4 | 79 Β±13.5 Β·44 |
| command-center / Codex Luna (high) | 87 Β±20.2 Β·6 | 0 Β·6 | 31 Β±4.2 Β·6 | 10 Β±4.3 Β·6 | 91 Β±28.9 Β·6 | 0 Β·4 | 28 Β±20.3 Β·4 | 84 Β±22.8 Β·6 | 158 Β±32.8 Β·6 | 40 Β±39.7 Β·6 | 75 Β±4.7 Β·4 | 57 Β±8.8 Β·60 |
| command-center / Codex Luna (medium) | β | β | 36 Β·2 | 20 Β±17.5 Β·2 | β | 0 Β·2 | 329 Β±18.5 Β·2 | 131 Β±130.5 Β·2 | 0 Β·2 | 0 Β·2 | 49 Β±49.0 Β·2 | 70 Β±30.2 Β·16 |
| command-center / Sonnet 5 (medium) | 90 Β±45.0 Β·3 | 0 Β·3 | 54 Β±6.4 Β·3 | 121 Β±6.1 Β·3 | 59 Β±7.5 Β·2 | 153 Β±16.0 Β·2 | 156 Β±80.7 Β·3 | 254 Β±28.5 Β·3 | 63 Β±62.7 Β·3 | 0 Β·2 | 36 Β±1.5 Β·3 | 91 Β±16.7 Β·30 |
| command-center / Opus 5 (high) | 62 Β±31.3 Β·3 | 102 Β±2.2 Β·3 | 22 Β±10.0 Β·3 | 125 Β±1.2 Β·3 | 126 Β±11.0 Β·3 | 0 Β·3 | 297 Β±5.3 Β·3 | 308 Β±8.1 Β·3 | 0 Β·3 | 0 Β·3 | 94 Β±8.7 Β·3 | 103 Β±18.7 Β·33 |
| command-center / Sonnet 5 (high) | 119 Β±11.3 Β·3 | 37 Β±37.0 Β·3 | 57 Β±1.0 Β·3 | 13 Β±13.3 Β·3 | 82 Β±2.8 Β·3 | β | 242 Β±13.1 Β·3 | 91 Β±91.0 Β·3 | β | 0 Β·3 | 70 Β±4.4 Β·3 | 79 Β±16.2 Β·27 |
| cursor / Opus 5 (high) | 114 Β±38.1 Β·4 | 0 Β·4 | 49 Β±6.3 Β·4 | 110 Β±7.9 Β·4 | 111 Β±10.0 Β·4 | 101 Β±9.9 Β·4 | 168 Β±96.7 Β·4 | 115 Β±66.8 Β·4 | 92 Β±53.4 Β·4 | 0 Β·4 | 62 Β±21.6 Β·4 | 84 Β±13.3 Β·44 |
| cursor / Sonnet 5 (medium) | 141 Β±2.7 Β·4 | 0 Β·4 | 54 Β±2.2 Β·4 | 131 Β±7.9 Β·4 | 104 Β±12.3 Β·4 | 197 Β±7.7 Β·4 | 128 Β±74.8 Β·4 | 230 Β±77.0 Β·4 | 277 Β±5.2 Β·4 | 0 Β·4 | 108 Β±1.5 Β·4 | 124 Β±15.4 Β·44 |
| ant / Opus 5 (high) | 131 Β±3.5 Β·4 | 0 Β·4 | 56 Β±2.8 Β·4 | 101 Β±2.7 Β·4 | 99 Β±4.3 Β·4 | 138 Β±3.9 Β·4 | 118 Β±67.9 Β·4 | 84 Β±84.3 Β·4 | 217 Β±4.3 Β·4 | 0 Β·4 | 34 Β±14.1 Β·4 | 89 Β±12.8 Β·44 |
| ant / Sonnet 5 (medium) | 112 Β±38.0 Β·4 | 29 Β±28.5 Β·4 | 49 Β±5.8 Β·4 | 103 Β±9.3 Β·4 | 28 Β±28.0 Β·4 | 199 Β±10.9 Β·4 | 246 Β±12.7 Β·4 | 291 Β±9.9 Β·4 | 253 Β±11.5 Β·4 | 0 Β·4 | 88 Β±1.8 Β·4 | 127 Β±15.7 Β·44 |
Normalized churn score (test-gated) (higher is better)
Normalized churn: a feature agent that failed a functionality test scores 0, as in the line-based score
| Arm | community-chat/loose-view-params-scattered-policy | conference-review/fused-author-conflicts | full-names/fixed-slot-render | interactive-fiction/inline-passage-records | minesweeper/boolean-cell-state-inline-views | photo-manager/working-view-selection-split | pixel-art/palette-registry-by-name | pixel-art/sequential-mirrored-paint | portfolio-ledger/scattered-reconciliation-logic | quilldoc-compiler/source-renderer-separate-outline | todolist/flat-lists-query-enums | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No refactor baseline | 18.3 Β±6.2 Β·4 | 0.0 Β·4 | 43.3 Β±0.4 Β·4 | 20.7 Β±0.4 Β·4 | 8.0 Β±4.7 Β·4 | 23.1 Β±0.2 Β·4 | 5.9 Β±0.3 Β·4 | 4.1 Β±2.4 Β·4 | 21.9 Β±0.3 Β·4 | 0.0 Β·4 | 18.8 Β±6.3 Β·4 | 14.9 Β±2.1 Β·44 |
| command-center / Codex Luna (high) | 22.9 Β±4.6 Β·6 | 0.0 Β·6 | 33.7 Β±0.9 Β·6 | 25.0 Β±0.4 Β·6 | 21.1 Β±6.7 Β·6 | 0.0 Β·4 | 6.9 Β±2.4 Β·4 | 7.9 Β±1.6 Β·6 | 19.6 Β±3.9 Β·6 | 2.2 Β±2.2 Β·6 | 28.8 Β±0.4 Β·4 | 15.6 Β±1.7 Β·60 |
| command-center / Codex Luna (medium) | β | β | 34.6 Β·2 | 25.3 Β±1.0 Β·2 | β | 0.0 Β·2 | 12.1 Β±0.6 Β·2 | 5.3 Β±5.3 Β·2 | 0.0 Β·2 | 0.0 Β·2 | 15.2 Β±15.2 Β·2 | 11.6 Β±3.5 Β·16 |
| command-center / Sonnet 5 (medium) | 18.7 Β±9.4 Β·3 | 0.0 Β·3 | 40.2 Β±2.3 Β·3 | 29.7 Β±0.7 Β·3 | 24.1 Β±0.6 Β·2 | 19.8 Β±1.2 Β·2 | 5.3 Β±2.7 Β·3 | 9.4 Β±0.7 Β·3 | 7.2 Β±7.2 Β·3 | 0.0 Β·2 | 24.1 Β±0.1 Β·3 | 16.4 Β±2.5 Β·30 |
| command-center / Opus 5 (high) | 17.0 Β±8.5 Β·3 | 40.2 Β±0.3 Β·3 | 32.0 Β±2.1 Β·3 | 29.9 Β±0.1 Β·3 | 35.8 Β±1.2 Β·3 | 0.0 Β·3 | 11.4 Β±0.2 Β·3 | 11.6 Β±0.2 Β·3 | 0.0 Β·3 | 0.0 Β·3 | 30.1 Β±0.8 Β·3 | 18.9 Β±2.7 Β·33 |
| command-center / Sonnet 5 (high) | 27.3 Β±0.9 Β·3 | 12.5 Β±12.5 Β·3 | 40.9 Β±0.4 Β·3 | 23.2 Β±1.4 Β·3 | 25.4 Β±0.2 Β·3 | β | 8.6 Β±0.3 Β·3 | 3.2 Β±3.2 Β·3 | β | 0.0 Β·3 | 26.0 Β±0.4 Β·3 | 18.6 Β±2.7 Β·27 |
| cursor / Opus 5 (high) | 22.7 Β±7.6 Β·4 | 0.0 Β·4 | 38.7 Β±2.0 Β·4 | 28.4 Β±0.8 Β·4 | 28.3 Β±1.1 Β·4 | 17.3 Β±0.6 Β·4 | 5.9 Β±3.4 Β·4 | 4.6 Β±2.7 Β·4 | 10.0 Β±5.8 Β·4 | 0.0 Β·4 | 27.5 Β±1.6 Β·4 | 16.7 Β±2.1 Β·44 |
| cursor / Sonnet 5 (medium) | 28.4 Β±0.2 Β·4 | 0.0 Β·4 | 39.6 Β±0.8 Β·4 | 30.5 Β±1.0 Β·4 | 27.0 Β±1.1 Β·4 | 23.4 Β±0.8 Β·4 | 4.4 Β±2.6 Β·4 | 8.1 Β±2.7 Β·4 | 26.1 Β±0.4 Β·4 | 0.0 Β·4 | 30.3 Β±0.1 Β·4 | 19.8 Β±2.1 Β·44 |
| ant / Opus 5 (high) | 28.2 Β±0.3 Β·4 | 0.0 Β·4 | 40.4 Β±1.0 Β·4 | 30.6 Β±0.3 Β·4 | 30.4 Β±0.4 Β·4 | 20.0 Β±0.3 Β·4 | 4.9 Β±2.8 Β·4 | 3.1 Β±3.1 Β·4 | 22.8 Β±0.3 Β·4 | 0.0 Β·4 | 26.3 Β±0.9 Β·4 | 18.8 Β±2.1 Β·44 |
| ant / Sonnet 5 (medium) | 21.9 Β±7.3 Β·4 | 9.3 Β±9.3 Β·4 | 38.4 Β±1.8 Β·4 | 27.9 Β±0.9 Β·4 | 6.8 Β±6.8 Β·4 | 23.9 Β±1.1 Β·4 | 9.1 Β±0.3 Β·4 | 9.5 Β±0.3 Β·4 | 24.0 Β±0.8 Β·4 | 0.0 Β·4 | 28.1 Β±0.2 Β·4 | 18.1 Β±2.0 Β·44 |
File-churn score (files touched, size-normalized) (higher is better)
| Arm | community-chat/loose-view-params-scattered-policy | conference-review/fused-author-conflicts | full-names/fixed-slot-render | interactive-fiction/inline-passage-records | minesweeper/boolean-cell-state-inline-views | photo-manager/working-view-selection-split | pixel-art/palette-registry-by-name | pixel-art/sequential-mirrored-paint | portfolio-ledger/scattered-reconciliation-logic | quilldoc-compiler/source-renderer-separate-outline | todolist/flat-lists-query-enums | Overall |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No refactor baseline | 100.0 Β·4 | 66.3 Β·4 | 100.0 Β·4 | 100.0 Β·4 | 100.0 Β·4 | 100.0 Β·4 | 79.2 Β·4 | 79.2 Β·4 | 100.0 Β·4 | 57.9 Β·4 | 100.0 Β·4 | 89.3 Β±2.3 Β·44 |
| command-center / Codex Luna (high) | 78.0 Β±7.0 Β·6 | 49.8 Β±2.6 Β·6 | 100.0 Β·6 | 67.9 Β±10.2 Β·6 | 86.0 Β±8.9 Β·6 | 79.2 Β·4 | 79.2 Β·4 | 79.2 Β·6 | 100.0 Β·6 | 52.9 Β±1.8 Β·6 | 79.2 Β·4 | 77.2 Β±2.5 Β·60 |
| command-center / Codex Luna (medium) | β | β | 100.0 Β·2 | 53.1 Β·2 | β | 79.4 Β·2 | 79.2 Β·2 | 79.2 Β·2 | 67.4 Β·2 | 57.9 Β·2 | 100.0 Β·2 | 77.0 Β±4.2 Β·16 |
| command-center / Sonnet 5 (medium) | 100.0 Β·3 | 57.9 Β·3 | 100.0 Β·3 | 78.7 Β·3 | 100.0 Β·2 | 100.0 Β·2 | 79.2 Β·3 | 79.2 Β·3 | 100.0 Β·3 | 57.9 Β·2 | 100.0 Β·3 | 86.7 Β±2.9 Β·30 |
| command-center / Opus 5 (high) | 60.0 Β·3 | 44.3 Β·3 | 100.0 Β·3 | 58.2 Β±2.1 Β·3 | 60.8 Β±2.9 Β·3 | 79.2 Β·3 | 59.6 Β·3 | 59.2 Β·3 | 49.3 Β±1.4 Β·3 | 41.5 Β±1.3 Β·3 | 49.6 Β±0.0 Β·3 | 60.2 Β±2.8 Β·33 |
| command-center / Sonnet 5 (high) | 100.0 Β·3 | 66.3 Β·3 | 100.0 Β·3 | 66.3 Β·3 | 100.0 Β·3 | β | 79.2 Β·3 | 79.2 Β·3 | β | 57.9 Β·3 | 100.0 Β·3 | 83.2 Β±3.2 Β·27 |
| cursor / Opus 5 (high) | 44.9 Β±0.9 Β·4 | 40.0 Β·4 | 78.7 Β·4 | 42.8 Β·4 | 44.3 Β±1.7 Β·4 | 55.5 Β±3.7 Β·4 | 52.1 Β·4 | 46.6 Β±1.6 Β·4 | 40.4 Β·4 | 33.4 Β±0.8 Β·4 | 35.3 Β±0.6 Β·4 | 46.7 Β±1.8 Β·44 |
| cursor / Sonnet 5 (medium) | 45.7 Β·4 | 41.0 Β·4 | 100.0 Β·4 | 44.3 Β·4 | 66.7 Β·4 | 70.4 Β±3.0 Β·4 | 68.3 Β·4 | 54.8 Β±1.2 Β·4 | 50.4 Β·4 | 58.4 Β·4 | 39.2 Β·4 | 58.1 Β±2.6 Β·44 |
| ant / Opus 5 (high) | 46.1 Β·4 | 37.0 Β±0.7 Β·4 | 100.0 Β·4 | 50.0 Β·4 | 49.0 Β±1.2 Β·4 | 58.5 Β±2.8 Β·4 | 49.3 Β±1.1 Β·4 | 44.9 Β±1.9 Β·4 | 43.6 Β·4 | 40.4 Β±1.0 Β·4 | 44.7 Β·4 | 51.2 Β±2.5 Β·44 |
| ant / Sonnet 5 (medium) | 35.7 Β·4 | 36.2 Β·4 | 100.0 Β·4 | 44.3 Β·4 | 58.4 Β·4 | 54.3 Β±2.4 Β·4 | 59.1 Β±1.8 Β·4 | 79.2 Β·4 | 41.0 Β±2.1 Β·4 | 57.9 Β·4 | 46.1 Β·4 | 55.7 Β±2.8 Β·44 |
Cell format: mean Β±sem Β·n. β = no tested
observations. Β·n is the number of tested instances behind the mean.
3.Scores
One marker per tested feature-agent instance, columns by task, arms on the y-axis. Circle = all functionality tests passed; cross = a test failed; triangle = the refactor itself regressed a prerefactor test, so that cell never produced a scored instance. Behind the markers, each cell with enough instances carries a box summarizing its spread, and the two best-placed arms in each column are badged in the column's right margin β each plot's caption states its own rule. Hover for details.
Line-based score (maximum β Ξlines)
Normalized churn score (test-gated)
Total churn β lines changed (ins+mod+del, canonicalized)
Edit-loc (mod+del, canonicalized)
Diff size β files changed (post-exclusion)
File-churn score (files touched, size-normalized)
4.Feature-agent token usage
Output tokens reported by every feature-agent attempt, including attempts that failed before scoring. Hover for details.
Feature-agent output-token usage
5.Seeded-flaw audit
Each bad premade has a catalog of deliberately seeded design flaws. A churn number can look good because an agent tidied the code while leaving the flaw in place, so these verdicts are read off the refactored source and the agent's own trajectory rather than measured: fixed asks whether the flaw is gone from the code, diagnosed whether the agent named the actual mechanism. An issue counts as fixed for the right reasons only when both say yes.
Fix rate across audited flaws
Each dot is the proportion of seeded flaw slots fixed by that arm. Hover for fixed, partial, surviving, and ungraded counts.
command-center / Codex Luna (high)
81/121
command-center / Codex Luna (medium)
34/63
command-center / Sonnet 5 (medium)
27/63
command-center / Opus 5 (high)
47/63
command-center / Sonnet 5 (high)
28/54
cursor / Opus 5 (high)
37/63
cursor / Sonnet 5 (medium)
19/63
ant / Opus 5 (high)
35/63
ant / Sonnet 5 (medium)
11/63
Fix rate by project and premade
community-chat/loose-view-params-scattered-policy
command-center / Codex Luna (high)
8/10
command-center / Codex Luna (medium)
3/5
command-center / Sonnet 5 (medium)
2/5
command-center / Opus 5 (high)
4/5
command-center / Sonnet 5 (high)
2/5
cursor / Opus 5 (high)
4/5
cursor / Sonnet 5 (medium)
3/5
ant / Opus 5 (high)
4/5
ant / Sonnet 5 (medium)
2/5
conference-review/fused-author-conflicts
command-center / Codex Luna (high)
2/4
command-center / Codex Luna (medium)
1/2
command-center / Sonnet 5 (medium)
0/2
command-center / Opus 5 (high)
2/2
command-center / Sonnet 5 (high)
0/2
cursor / Opus 5 (high)
0/2
cursor / Sonnet 5 (medium)
0/2
ant / Opus 5 (high)
0/2
ant / Sonnet 5 (medium)
0/2
full-names/fixed-slot-render
command-center / Codex Luna (high)
0/4
command-center / Codex Luna (medium)
0/2
command-center / Sonnet 5 (medium)
0/2
command-center / Opus 5 (high)
1/2
command-center / Sonnet 5 (high)
0/2
cursor / Opus 5 (high)
1/2
cursor / Sonnet 5 (medium)
0/2
ant / Opus 5 (high)
0/2
ant / Sonnet 5 (medium)
0/2
interactive-fiction/inline-passage-records
command-center / Codex Luna (high)
8/8
command-center / Codex Luna (medium)
3/4
command-center / Sonnet 5 (medium)
3/4
command-center / Opus 5 (high)
3/4
command-center / Sonnet 5 (high)
4/4
cursor / Opus 5 (high)
3/4
cursor / Sonnet 5 (medium)
3/4
ant / Opus 5 (high)
2/4
ant / Sonnet 5 (medium)
1/4
minesweeper/boolean-cell-state-inline-views
command-center / Codex Luna (high)
13/14
command-center / Codex Luna (medium)
4/7
command-center / Sonnet 5 (medium)
5/7
command-center / Opus 5 (high)
6/7
command-center / Sonnet 5 (high)
5/7
cursor / Opus 5 (high)
5/7
cursor / Sonnet 5 (medium)
2/7
ant / Opus 5 (high)
5/7
ant / Sonnet 5 (medium)
2/7
photo-manager/working-view-selection-split
command-center / Codex Luna (high)
5/5
command-center / Codex Luna (medium)
3/5
command-center / Sonnet 5 (medium)
1/5
command-center / Opus 5 (high)
5/5
cursor / Opus 5 (high)
3/5
cursor / Sonnet 5 (medium)
0/5
ant / Opus 5 (high)
2/5
ant / Sonnet 5 (medium)
0/5
pixel-art/palette-registry-by-name
command-center / Codex Luna (high)
17/26
command-center / Codex Luna (medium)
6/13
command-center / Sonnet 5 (medium)
6/13
command-center / Opus 5 (high)
11/13
command-center / Sonnet 5 (high)
7/13
cursor / Opus 5 (high)
9/13
cursor / Sonnet 5 (medium)
3/13
ant / Opus 5 (high)
8/13
ant / Sonnet 5 (medium)
0/13
pixel-art/sequential-mirrored-paint
command-center / Codex Luna (high)
13/22
command-center / Codex Luna (medium)
6/11
command-center / Sonnet 5 (medium)
3/11
command-center / Opus 5 (high)
7/11
command-center / Sonnet 5 (high)
5/11
cursor / Opus 5 (high)
6/11
cursor / Sonnet 5 (medium)
4/11
ant / Opus 5 (high)
7/11
ant / Sonnet 5 (medium)
1/11
portfolio-ledger/scattered-reconciliation-logic
command-center / Codex Luna (high)
4/8
command-center / Codex Luna (medium)
2/4
command-center / Sonnet 5 (medium)
3/4
command-center / Opus 5 (high)
3/4
cursor / Opus 5 (high)
3/4
cursor / Sonnet 5 (medium)
1/4
ant / Opus 5 (high)
2/4
ant / Sonnet 5 (medium)
2/4
quilldoc-compiler/source-renderer-separate-outline
command-center / Codex Luna (high)
8/14
command-center / Codex Luna (medium)
5/7
command-center / Sonnet 5 (medium)
3/7
command-center / Opus 5 (high)
3/7
command-center / Sonnet 5 (high)
4/7
cursor / Opus 5 (high)
2/7
cursor / Sonnet 5 (medium)
3/7
ant / Opus 5 (high)
4/7
ant / Sonnet 5 (medium)
3/7
todolist/flat-lists-query-enums
command-center / Codex Luna (high)
3/6
command-center / Codex Luna (medium)
1/3
command-center / Sonnet 5 (medium)
1/3
command-center / Opus 5 (high)
2/3
command-center / Sonnet 5 (high)
1/3
cursor / Opus 5 (high)
1/3
cursor / Sonnet 5 (medium)
0/3
ant / Opus 5 (high)
1/3
ant / Sonnet 5 (medium)
0/3
These are hand-graded judgments, not measurements β expand any row for the evidence behind its verdict. No audit covers No refactor baseline, so it is absent from the table below β that is "nobody checked", not "nothing found".
Across every audited premade
| Arm | graded slots | fixed | partial | survives | fixed for the right reasons | understood, left in place | fixed without naming why |
|---|---|---|---|---|---|---|---|
| command-center / Codex Luna (high) 2 audited runs | 121 | 81 | 26 | 14 | 81 | 21 | 0 |
| command-center / Codex Luna (medium) | 63 | 34 | 18 | 11 | 33 | 15 | 1 |
| command-center / Sonnet 5 (medium) | 63 | 27 | 18 | 18 | 27 | 10 | 0 |
| command-center / Opus 5 (high) | 63 | 47 | 6 | 10 | 47 | 7 | 0 |
| command-center / Sonnet 5 (high) | 54 | 28 | 15 | 11 | 28 | 4 | 0 |
| cursor / Opus 5 (high) | 63 | 37 | 9 | 17 | 34 | 2 | 3 |
| cursor / Sonnet 5 (medium) | 63 | 19 | 14 | 30 | 17 | 1 | 2 |
| ant / Opus 5 (high) | 63 | 35 | 11 | 17 | 30 | 0 | 5 |
| ant / Sonnet 5 (medium) | 63 | 11 | 10 | 42 | 9 | 2 | 2 |
command-center / Codex Luna (high) 81/121
seeded-flaw verdicts fixed
command-center / Codex Luna (medium) 34/63
seeded flaws fixed
command-center / Sonnet 5 (medium) 27/63
seeded flaws fixed
command-center / Opus 5 (high) 47/63
seeded flaws fixed
command-center / Sonnet 5 (high) 28/54
seeded flaws fixed
cursor / Opus 5 (high) 37/63
seeded flaws fixed
cursor / Sonnet 5 (medium) 19/63
seeded flaws fixed
ant / Opus 5 (high) 35/63
seeded flaws fixed
ant / Sonnet 5 (medium) 11/63
seeded flaws fixed
Per-premade tally
| Premade | command-center / Codex Luna (high) | command-center / Codex Luna (medium) | command-center / Sonnet 5 (medium) | command-center / Opus 5 (high) | command-center / Sonnet 5 (high) | cursor / Opus 5 (high) | cursor / Sonnet 5 (medium) | ant / Opus 5 (high) | ant / Sonnet 5 (medium) |
|---|---|---|---|---|---|---|---|---|---|
| community-chat/loose-view-params-scattered-policy Γ2 audited runs | 8/10 +1 partial | 3/5 +1 partial | 2/5 +2 partial | 4/5 +1 partial | 2/5 +3 partial | 4/5 | 3/5 | 4/5 | 2/5 +1 partial |
| conference-review/fused-author-conflicts Γ2 audited runs | 2/4 +2 partial | 1/2 | 0/2 +1 partial | 2/2 | 0/2 +1 partial | 0/2 | 0/2 | 0/2 | 0/2 |
| full-names/fixed-slot-render Γ2 audited runs | 0/4 +3 partial | 0/2 +1 partial | 0/2 +2 partial | 1/2 | 0/2 +2 partial | 1/2 | 0/2 +1 partial | 0/2 +1 partial | 0/2 +1 partial |
| interactive-fiction/inline-passage-records Γ2 audited runs | 8/8 | 3/4 | 3/4 | 3/4 | 4/4 | 3/4 | 3/4 | 2/4 +1 partial | 1/4 +1 partial |
| minesweeper/boolean-cell-state-inline-views Γ2 audited runs | 13/14 +1 partial | 4/7 +3 partial | 5/7 +1 partial | 6/7 | 5/7 | 5/7 | 2/7 | 5/7 | 2/7 |
| photo-manager/working-view-selection-split | 5/5 | 3/5 +1 partial | 1/5 +2 partial | 5/5 | β | 3/5 +1 partial | 0/5 +2 partial | 2/5 +2 partial | 0/5 +3 partial |
| pixel-art/palette-registry-by-name Γ2 audited runs | 17/26 +7 partial | 6/13 +4 partial | 6/13 +3 partial | 11/13 | 7/13 +3 partial | 9/13 +2 partial | 3/13 +4 partial | 8/13 +3 partial | 0/13 +2 partial |
| pixel-art/sequential-mirrored-paint Γ2 audited runs | 13/22 +5 partial | 6/11 +4 partial | 3/11 +3 partial | 7/11 +3 partial | 5/11 +3 partial | 6/11 +3 partial | 4/11 +3 partial | 7/11 +2 partial | 1/11 |
| portfolio-ledger/scattered-reconciliation-logic Γ2 audited runs | 4/8 +3 partial | 2/4 +1 partial | 3/4 | 3/4 +1 partial | β | 3/4 | 1/4 +1 partial | 2/4 | 2/4 |
| quilldoc-compiler/source-renderer-separate-outline Γ2 audited runs | 8/14 +1 partial | 5/7 +1 partial | 3/7 +2 partial | 3/7 | 4/7 +1 partial | 2/7 +1 partial | 3/7 +1 partial | 4/7 +1 partial | 3/7 +1 partial |
| todolist/flat-lists-query-enums Γ2 audited runs | 3/6 +3 partial | 1/3 +2 partial | 1/3 +2 partial | 2/3 +1 partial | 1/3 +2 partial | 1/3 +2 partial | 0/3 +2 partial | 1/3 +1 partial | 0/3 +1 partial |
Cells are fixed / graded slots β one slot per catalog issue per audited run, so an arm with three audited runs answers the catalog three times over. Partial fixes are counted separately rather than folded in, since a partial fix leaves the flaw in the code. A premade marked no catalog has no manifest to be denominated in, so its fraction is only over the issues the audit chose to grade. Expand a premade below for its issue-by-issue verdicts and evidence.
community-chat/loose-view-params-scattered-policy command-center / Codex Luna (high) 8/10 Β· command-center / Codex Luna (medium) 3/5 Β· command-center / Sonnet 5 (medium) 2/5 Β· command-center / Opus 5 (high) 4/5 Β· command-center / Sonnet 5 (high) 2/5 Β· cursor / Opus 5 (high) 4/5 Β· cursor / Sonnet 5 (medium) 3/5 Β· ant / Opus 5 (high) 4/5 Β· ant / Sonnet 5 (medium) 2/5
| # | Seeded flaw | command-center / Codex Luna (high) | command-center / Codex Luna (medium) | command-center / Sonnet 5 (medium) | command-center / Opus 5 (high) | command-center / Sonnet 5 (high) | cursor / Opus 5 (high) | cursor / Sonnet 5 (medium) | ant / Opus 5 (high) | ant / Sonnet 5 (medium) |
|---|---|---|---|---|---|---|---|---|---|---|
| 1run 1 | fixed diagnosed | partial diagnosed | partial diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | |
| 1run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 2run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | partial diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | |
| 2run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 3run 1 | partial diagnosed | survives diagnosed | survives part-diagnosed | partial diagnosed | partial part-diagnosed | survives part-diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 3run 2 | survives part-diagnosed | β | β | β | β | β | β | β | β | |
| 4run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | fixed diagnosed | survives not diagnosed | |
| 4run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 5run 1 | fixed diagnosed | fixed part-diagnosed | partial diagnosed | fixed diagnosed | partial part-diagnosed | fixed diagnosed | fixed part-diagnosed | fixed part-diagnosed | partial part-diagnosed | |
| 5run 2 | fixed diagnosed | β | β | β | β | β | β | β | β |
conference-review/fused-author-conflicts command-center / Codex Luna (high) 2/4 Β· command-center / Codex Luna (medium) 1/2 Β· command-center / Sonnet 5 (medium) 0/2 Β· command-center / Opus 5 (high) 2/2 Β· command-center / Sonnet 5 (high) 0/2 Β· cursor / Opus 5 (high) 0/2 Β· cursor / Sonnet 5 (medium) 0/2 Β· ant / Opus 5 (high) 0/2 Β· ant / Sonnet 5 (medium) 0/2
| # | Seeded flaw | command-center / Codex Luna (high) | command-center / Codex Luna (medium) | command-center / Sonnet 5 (medium) | command-center / Opus 5 (high) | command-center / Sonnet 5 (high) | cursor / Opus 5 (high) | cursor / Sonnet 5 (medium) | ant / Opus 5 (high) | ant / Sonnet 5 (medium) |
|---|---|---|---|---|---|---|---|---|---|---|
| 1run 1 | partial diagnosed | survives part-diagnosed | survives part-diagnosed | fixed diagnosed | survives diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 1run 2 | partial diagnosed | β | β | β | β | β | β | β | β | |
| 2run 1 | fixed diagnosed | fixed diagnosed | partial diagnosed | fixed diagnosed | partial part-diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 2run 2 | fixed diagnosed | β | β | β | β | β | β | β | β |
full-names/fixed-slot-render command-center / Codex Luna (high) 0/4 Β· command-center / Codex Luna (medium) 0/2 Β· command-center / Sonnet 5 (medium) 0/2 Β· command-center / Opus 5 (high) 1/2 Β· command-center / Sonnet 5 (high) 0/2 Β· cursor / Opus 5 (high) 1/2 Β· cursor / Sonnet 5 (medium) 0/2 Β· ant / Opus 5 (high) 0/2 Β· ant / Sonnet 5 (medium) 0/2
| # | Seeded flaw | command-center / Codex Luna (high) | command-center / Codex Luna (medium) | command-center / Sonnet 5 (medium) | command-center / Opus 5 (high) | command-center / Sonnet 5 (high) | cursor / Opus 5 (high) | cursor / Sonnet 5 (medium) | ant / Opus 5 (high) | ant / Sonnet 5 (medium) |
|---|---|---|---|---|---|---|---|---|---|---|
| 1run 1 | survives part-diagnosed | survives part-diagnosed | partial part-diagnosed | survives not diagnosed | partial part-diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 1run 2 | partial diagnosed | β | β | β | β | β | β | β | β | |
| 2run 1 | partial diagnosed | partial diagnosed | partial diagnosed | fixed diagnosed | partial diagnosed | fixed diagnosed | partial part-diagnosed | partial part-diagnosed | partial part-diagnosed | |
| 2run 2 | partial diagnosed | β | β | β | β | β | β | β | β |
interactive-fiction/inline-passage-records command-center / Codex Luna (high) 8/8 Β· command-center / Codex Luna (medium) 3/4 Β· command-center / Sonnet 5 (medium) 3/4 Β· command-center / Opus 5 (high) 3/4 Β· command-center / Sonnet 5 (high) 4/4 Β· cursor / Opus 5 (high) 3/4 Β· cursor / Sonnet 5 (medium) 3/4 Β· ant / Opus 5 (high) 2/4 Β· ant / Sonnet 5 (medium) 1/4
| # | Seeded flaw | command-center / Codex Luna (high) | command-center / Codex Luna (medium) | command-center / Sonnet 5 (medium) | command-center / Opus 5 (high) | command-center / Sonnet 5 (high) | cursor / Opus 5 (high) | cursor / Sonnet 5 (medium) | ant / Opus 5 (high) | ant / Sonnet 5 (medium) |
|---|---|---|---|---|---|---|---|---|---|---|
| 1run 1 | fixed diagnosed | survives not diagnosed | fixed diagnosed | survives part-diagnosed | fixed diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 1run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 2run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | |
| 2run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 3run 1 | fixed diagnosed | fixed diagnosed | survives not diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | partial not diagnosed | survives not diagnosed | |
| 3run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 4run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed part-diagnosed | fixed diagnosed | fixed diagnosed | partial part-diagnosed | |
| 4run 2 | fixed diagnosed | β | β | β | β | β | β | β | β |
minesweeper/boolean-cell-state-inline-views command-center / Codex Luna (high) 13/14 Β· command-center / Codex Luna (medium) 4/7 Β· command-center / Sonnet 5 (medium) 5/7 Β· command-center / Opus 5 (high) 6/7 Β· command-center / Sonnet 5 (high) 5/7 Β· cursor / Opus 5 (high) 5/7 Β· cursor / Sonnet 5 (medium) 2/7 Β· ant / Opus 5 (high) 5/7 Β· ant / Sonnet 5 (medium) 2/7
| # | Seeded flaw | command-center / Codex Luna (high) | command-center / Codex Luna (medium) | command-center / Sonnet 5 (medium) | command-center / Opus 5 (high) | command-center / Sonnet 5 (high) | cursor / Opus 5 (high) | cursor / Sonnet 5 (medium) | ant / Opus 5 (high) | ant / Sonnet 5 (medium) |
|---|---|---|---|---|---|---|---|---|---|---|
| 1run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 1run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 2run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | fixed not diagnosed | survives not diagnosed | |
| 2run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 3run 1 | fixed diagnosed | partial diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | |
| 3run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 4run 1 | fixed diagnosed | fixed diagnosed | survives diagnosed | fixed diagnosed | survives part-diagnosed | fixed not diagnosed | fixed not diagnosed | fixed not diagnosed | fixed not diagnosed | |
| 4run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 5run 1 | fixed diagnosed | partial diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | fixed diagnosed | survives not diagnosed | |
| 5run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 6run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | fixed diagnosed | survives not diagnosed | |
| 6run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 7run 1 | partial part-diagnosed | partial diagnosed | partial part-diagnosed | survives part-diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 7run 2 | fixed diagnosed | β | β | β | β | β | β | β | β |
photo-manager/working-view-selection-split command-center / Codex Luna (high) 5/5 Β· command-center / Codex Luna (medium) 3/5 Β· command-center / Sonnet 5 (medium) 1/5 Β· command-center / Opus 5 (high) 5/5 Β· cursor / Opus 5 (high) 3/5 Β· cursor / Sonnet 5 (medium) 0/5 Β· ant / Opus 5 (high) 2/5 Β· ant / Sonnet 5 (medium) 0/5
| # | Seeded flaw | command-center / Codex Luna (high) | command-center / Codex Luna (medium) | command-center / Sonnet 5 (medium) | command-center / Opus 5 (high) | command-center / Sonnet 5 (high) | cursor / Opus 5 (high) | cursor / Sonnet 5 (medium) | ant / Opus 5 (high) | ant / Sonnet 5 (medium) |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | fixed diagnosed | survives diagnosed | survives part-diagnosed | fixed diagnosed | β | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 2 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | β | fixed diagnosed | partial part-diagnosed | partial part-diagnosed | partial part-diagnosed | |
| 3 | fixed diagnosed | fixed diagnosed | partial diagnosed | fixed diagnosed | β | fixed diagnosed | survives not diagnosed | fixed not diagnosed | survives not diagnosed | |
| 4 | fixed diagnosed | partial diagnosed | partial part-diagnosed | fixed diagnosed | β | fixed diagnosed | partial diagnosed | fixed part-diagnosed | partial part-diagnosed | |
| 5 | fixed diagnosed | fixed diagnosed | survives not diagnosed | fixed diagnosed | β | partial not diagnosed | survives not diagnosed | partial not diagnosed | partial not diagnosed |
pixel-art/palette-registry-by-name command-center / Codex Luna (high) 17/26 Β· command-center / Codex Luna (medium) 6/13 Β· command-center / Sonnet 5 (medium) 6/13 Β· command-center / Opus 5 (high) 11/13 Β· command-center / Sonnet 5 (high) 7/13 Β· cursor / Opus 5 (high) 9/13 Β· cursor / Sonnet 5 (medium) 3/13 Β· ant / Opus 5 (high) 8/13 Β· ant / Sonnet 5 (medium) 0/13
| # | Seeded flaw | command-center / Codex Luna (high) | command-center / Codex Luna (medium) | command-center / Sonnet 5 (medium) | command-center / Opus 5 (high) | command-center / Sonnet 5 (high) | cursor / Opus 5 (high) | cursor / Sonnet 5 (medium) | ant / Opus 5 (high) | ant / Sonnet 5 (medium) |
|---|---|---|---|---|---|---|---|---|---|---|
| 1run 1 | partial diagnosed | partial diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 1run 2 | partial part-diagnosed | β | β | β | β | β | β | β | β | |
| 2run 1 | fixed diagnosed | survives part-diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | fixed diagnosed | survives not diagnosed | |
| 2run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 3run 1 | survives part-diagnosed | survives diagnosed | survives part-diagnosed | survives diagnosed | survives part-diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 3run 2 | survives part-diagnosed | β | β | β | β | β | β | β | β | |
| 4run 1 | partial diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | fixed diagnosed | survives not diagnosed | |
| 4run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 5run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | |
| 5run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 6run 1 | partial diagnosed | partial part-diagnosed | partial part-diagnosed | fixed diagnosed | partial part-diagnosed | fixed diagnosed | partial part-diagnosed | fixed diagnosed | partial not diagnosed | |
| 6run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 7run 1 | fixed diagnosed | fixed diagnosed | survives not diagnosed | fixed diagnosed | survives not diagnosed | fixed part-diagnosed | survives not diagnosed | fixed diagnosed | survives not diagnosed | |
| 7run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 8run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | |
| 8run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 9run 1 | partial part-diagnosed | partial part-diagnosed | partial part-diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | partial not diagnosed | partial not diagnosed | |
| 9run 2 | partial part-diagnosed | β | β | β | β | β | β | β | β | |
| 10run 1 | fixed diagnosed | partial diagnosed | partial diagnosed | fixed diagnosed | partial part-diagnosed | partial part-diagnosed | partial part-diagnosed | partial part-diagnosed | survives not diagnosed | |
| 10run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 11run 1 | fixed diagnosed | survives not diagnosed | survives not diagnosed | fixed diagnosed | partial part-diagnosed | partial part-diagnosed | partial part-diagnosed | partial not diagnosed | survives not diagnosed | |
| 11run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 12run 1 | partial diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | fixed diagnosed | survives not diagnosed | |
| 12run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 13run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | partial part-diagnosed | fixed diagnosed | survives diagnosed | |
| 13run 2 | fixed diagnosed | β | β | β | β | β | β | β | β |
pixel-art/sequential-mirrored-paint command-center / Codex Luna (high) 13/22 Β· command-center / Codex Luna (medium) 6/11 Β· command-center / Sonnet 5 (medium) 3/11 Β· command-center / Opus 5 (high) 7/11 Β· command-center / Sonnet 5 (high) 5/11 Β· cursor / Opus 5 (high) 6/11 Β· cursor / Sonnet 5 (medium) 4/11 Β· ant / Opus 5 (high) 7/11 Β· ant / Sonnet 5 (medium) 1/11
| # | Seeded flaw | command-center / Codex Luna (high) | command-center / Codex Luna (medium) | command-center / Sonnet 5 (medium) | command-center / Opus 5 (high) | command-center / Sonnet 5 (high) | cursor / Opus 5 (high) | cursor / Sonnet 5 (medium) | ant / Opus 5 (high) | ant / Sonnet 5 (medium) |
|---|---|---|---|---|---|---|---|---|---|---|
| 1run 1 | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives part-diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 1run 2 | survives not diagnosed | β | β | β | β | β | β | β | β | |
| 2run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | |
| 2run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 3run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | |
| 3run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 4run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | |
| 4run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 5run 1 | partial part-diagnosed | partial part-diagnosed | partial part-diagnosed | fixed diagnosed | partial part-diagnosed | fixed diagnosed | partial part-diagnosed | fixed diagnosed | fixed not diagnosed | |
| 5run 2 | survives part-diagnosed | β | β | β | β | β | β | β | β | |
| 6run 1 | fixed diagnosed | fixed diagnosed | partial diagnosed | fixed diagnosed | fixed diagnosed | partial part-diagnosed | partial part-diagnosed | fixed diagnosed | survives diagnosed | |
| 6run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 7run 1 | fixed diagnosed | partial diagnosed | partial diagnosed | partial diagnosed | partial part-diagnosed | partial part-diagnosed | partial part-diagnosed | partial part-diagnosed | survives not diagnosed | |
| 7run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 8run 1 | fixed diagnosed | fixed diagnosed | survives not diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | |
| 8run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 9run 1 | fixed diagnosed | partial part-diagnosed | survives part-diagnosed | fixed diagnosed | survives part-diagnosed | fixed diagnosed | survives not diagnosed | fixed diagnosed | survives not diagnosed | |
| 9run 2 | survives not diagnosed | β | β | β | β | β | β | β | β | |
| 10run 1 | partial part-diagnosed | fixed diagnosed | survives not diagnosed | partial part-diagnosed | partial part-diagnosed | partial part-diagnosed | survives not diagnosed | partial part-diagnosed | survives not diagnosed | |
| 10run 2 | partial diagnosed | β | β | β | β | β | β | β | β | |
| 11run 1 | partial diagnosed | partial diagnosed | survives diagnosed | partial not diagnosed | survives part-diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 11run 2 | partial diagnosed | β | β | β | β | β | β | β | β |
portfolio-ledger/scattered-reconciliation-logic command-center / Codex Luna (high) 4/8 Β· command-center / Codex Luna (medium) 2/4 Β· command-center / Sonnet 5 (medium) 3/4 Β· command-center / Opus 5 (high) 3/4 Β· cursor / Opus 5 (high) 3/4 Β· cursor / Sonnet 5 (medium) 1/4 Β· ant / Opus 5 (high) 2/4 Β· ant / Sonnet 5 (medium) 2/4
| # | Seeded flaw | command-center / Codex Luna (high) | command-center / Codex Luna (medium) | command-center / Sonnet 5 (medium) | command-center / Opus 5 (high) | command-center / Sonnet 5 (high) | cursor / Opus 5 (high) | cursor / Sonnet 5 (medium) | ant / Opus 5 (high) | ant / Sonnet 5 (medium) |
|---|---|---|---|---|---|---|---|---|---|---|
| 1run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | β | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | |
| 1run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 2run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | β | fixed diagnosed | partial part-diagnosed | fixed diagnosed | fixed diagnosed | |
| 2run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 3run 1 | survives part-diagnosed | survives part-diagnosed | survives not diagnosed | partial part-diagnosed | β | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 3run 2 | partial diagnosed | β | β | β | β | β | β | β | β | |
| 4run 1 | partial diagnosed | partial part-diagnosed | fixed diagnosed | fixed diagnosed | β | fixed diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 4run 2 | partial diagnosed | β | β | β | β | β | β | β | β |
quilldoc-compiler/source-renderer-separate-outline command-center / Codex Luna (high) 8/14 Β· command-center / Codex Luna (medium) 5/7 Β· command-center / Sonnet 5 (medium) 3/7 Β· command-center / Opus 5 (high) 3/7 Β· command-center / Sonnet 5 (high) 4/7 Β· cursor / Opus 5 (high) 2/7 Β· cursor / Sonnet 5 (medium) 3/7 Β· ant / Opus 5 (high) 4/7 Β· ant / Sonnet 5 (medium) 3/7
| # | Seeded flaw | command-center / Codex Luna (high) | command-center / Codex Luna (medium) | command-center / Sonnet 5 (medium) | command-center / Opus 5 (high) | command-center / Sonnet 5 (high) | cursor / Opus 5 (high) | cursor / Sonnet 5 (medium) | ant / Opus 5 (high) | ant / Sonnet 5 (medium) |
|---|---|---|---|---|---|---|---|---|---|---|
| 1run 1 | survives diagnosed | fixed diagnosed | fixed diagnosed | survives diagnosed | fixed diagnosed | survives diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | |
| 1run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 2run 1 | partial diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | partial diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | |
| 2run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 3run 1 | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | fixed diagnosed | fixed diagnosed | |
| 3run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 4run 1 | survives diagnosed | fixed diagnosed | survives not diagnosed | survives part-diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 4run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 5run 1 | survives part-diagnosed | survives not diagnosed | survives not diagnosed | survives diagnosed | survives part-diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | survives not diagnosed | |
| 5run 2 | survives not diagnosed | β | β | β | β | β | β | β | β | |
| 6run 1 | fixed diagnosed | fixed diagnosed | partial part-diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | |
| 6run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 7run 1 | survives diagnosed | partial diagnosed | partial part-diagnosed | survives diagnosed | partial part-diagnosed | survives not diagnosed | partial not diagnosed | partial not diagnosed | partial not diagnosed | |
| 7run 2 | fixed diagnosed | β | β | β | β | β | β | β | β |
todolist/flat-lists-query-enums command-center / Codex Luna (high) 3/6 Β· command-center / Codex Luna (medium) 1/3 Β· command-center / Sonnet 5 (medium) 1/3 Β· command-center / Opus 5 (high) 2/3 Β· command-center / Sonnet 5 (high) 1/3 Β· cursor / Opus 5 (high) 1/3 Β· cursor / Sonnet 5 (medium) 0/3 Β· ant / Opus 5 (high) 1/3 Β· ant / Sonnet 5 (medium) 0/3
| # | Seeded flaw | command-center / Codex Luna (high) | command-center / Codex Luna (medium) | command-center / Sonnet 5 (medium) | command-center / Opus 5 (high) | command-center / Sonnet 5 (high) | cursor / Opus 5 (high) | cursor / Sonnet 5 (medium) | ant / Opus 5 (high) | ant / Sonnet 5 (medium) |
|---|---|---|---|---|---|---|---|---|---|---|
| 1run 1 | fixed diagnosed | fixed diagnosed | partial part-diagnosed | fixed diagnosed | partial diagnosed | partial part-diagnosed | partial part-diagnosed | partial part-diagnosed | partial part-diagnosed | |
| 1run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 2run 1 | partial diagnosed | partial diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | fixed diagnosed | survives not diagnosed | fixed diagnosed | survives not diagnosed | |
| 2run 2 | fixed diagnosed | β | β | β | β | β | β | β | β | |
| 3run 1 | partial part-diagnosed | partial part-diagnosed | partial part-diagnosed | partial diagnosed | partial part-diagnosed | partial part-diagnosed | partial not diagnosed | survives not diagnosed | survives not diagnosed | |
| 3run 2 | partial part-diagnosed | β | β | β | β | β | β | β | β |
6.Refactoring agent time/tokens
Time spent by the refactoring agent itself, before the feature update. The
table summarizes successful final attempts; the plot also shows failed final
attempts at lower opacity. A retried job overwrites its prior result.json, so these figures exclude earlier attempts, retry
backoff, regression checks, and feature-agent evaluation.
| Arm | Model | Effort | Mean Β± SEM (min) | Range (min) | Succeeded | Failed |
|---|---|---|---|---|---|---|
| No refactor baseline | claude-sonnet-4-6 | low | 0.0 Β± 0.0 | 0.0β0.0 | 11 | 0 |
| command-center / Codex Luna (high) | gpt-5.6-luna | high | 40.6 Β± 1.7 | 21.8β51.3 | 21 | 1 |
| command-center / Codex Luna (medium) | gpt-5.6-luna | medium | 20.2 Β± 1.1 | 12.4β25.6 | 11 | 0 |
| command-center / Sonnet 5 (medium) | claude-sonnet-5 | medium | 29.3 Β± 6.3 | 9.3β89.5 | 11 | 0 |
| command-center / Opus 5 (high) | claude-opus-5 | high | 92.5 Β± 7.0 | 54.9β134.7 | 11 | 0 |
| command-center / Sonnet 5 (high) | claude-sonnet-5 | high | 63.5 Β± 10.9 | 14.9β123.5 | 9 | 1 |
| cursor / Opus 5 (high) | claude-opus-5 | high | 25.6 Β± 4.0 | 9.1β53.4 | 11 | 0 |
| cursor / Sonnet 5 (medium) | claude-sonnet-5 | medium | 6.2 Β± 0.7 | 1.9β9.2 | 11 | 0 |
| ant / Opus 5 (high) | claude-opus-5 | high | 16.4 Β± 2.0 | 4.3β27.6 | 11 | 0 |
| ant / Sonnet 5 (medium) | claude-sonnet-5 | medium | 5.8 Β± 0.6 | 1.0β8.4 | 11 | 0 |
7.How the metrics were computed
Scores are taken verbatim from the harness (this report displays them, it does not recompute them). For current-schema runs, the report uses the harness's primary canonicalized measurement: supported JavaScript and TypeScript files are parsed and both revisions are reprinted into a stable layout before they are diffed, so formatting-only edits do not count as churn. The pre-update line-count baseline is canonicalized the same way. Files that cannot be canonicalized use the textual diff instead. Legacy rows retain their historical post-exclusion measurements, whose measurement method was not recorded. All churn figures are post-exclusion (see Β§7).
Total churn β lines changed (ins+mod+del, canonicalized)
Ξlines = total lines changed by the feature agent (insertions + modifications + deletions), from the post-exclusion canonicalized diff on current-schema runs. Supported JS/TS files are reprinted before diffing; files that cannot be canonicalized fall back to textual counting. Legacy runs did not record the measurement method.
Edit-loc (mod+del, canonicalized)
editLoc = Ξ£ per-file (modifications + deletions) over the canonicalized diff β churn in pre-existing lines, ignoring pure insertions. Null on old-schema runs (no per-file churn kinds).
Diff size β files changed (post-exclusion)
Ξfiles = files touched by the feature agent, after exclusions.
Line-based score (maximum β Ξlines)
lineBased = testsPass ? max(0, benchmark maxChangeSize β Ξlines) : 0 β where Ξlines is post-exclusion canonicalized churn on current-schema runs (legacy runs did not record their measurement method or maximum).
Normalized churn score (test-gated)
normalizedChurn = 0 Γ (Ξlines β€ 0 ? 100 : 100Β·(1 β ln(Ξlines) / ln(preUpdateLines + 500)) β on current-schema runs, both churn and the pre-update line-count baseline use the same canonicalized representation.) when tests fail; otherwise (Ξlines β€ 0 ? 100 : 100Β·(1 β ln(Ξlines) / ln(preUpdateLines + 500)) β on current-schema runs, both churn and the pre-update line-count baseline use the same canonicalized representation.).
File-churn score (files touched, size-normalized)
fileChurn = Ξfiles β€ 0 ? 100 : 100Β·(1 β ln(Ξfiles) / ln(preUpdateFiles + 20)) β Ξfiles is post-exclusion, and the baseline is the pre-update file count of the same tree.
testsPass means every functionality test passed (passed === total); a failure forces the line-based score to 0, and the normalized churn
score too while the report is set to test-gated. A refactor that regressed a
prerefactor test produced no scored instance at all, so it is not a 0 here β
Β§3 plots it as its own marker. Cell means aggregate tested instances only β apply-failed and test-infra-failed instances
do not contribute measurements. Any harness-zeroed scores on legacy failure rows
are artifacts and are excluded from the means (but counted in Β§1).
8.Diff-stat exclusions
The displayed churn and churn-based scores count production source churn only. The harness's exclusion recipe drops:
- test files (
*.test.*,*.spec.*, anything under a test directory) - Markdown docs (
*.md) - Dafny specs (
*.dfy) - agent scratch dirs (anything under
cc-notes/or.cc-scratch/) - comment & docstring lines, and blank lines (metrics are SLOC-based; JS/TS comment stripping)
- vendored dependencies & lockfiles β always dropped (
node_modules/,package-lock.json,yarn.lock,pnpm-lock.yaml) - whitespace-only churn (reindentation) does not count (
git diff --ignore-space-change)
This is the harness's default exclusion recipe, applied identically to the diff and to the pre-update baseline that normalizes the churn score, so the numerator and denominator stay commensurable. Canonical printing also removes comments and formatting differences from supported JavaScript and TypeScript files; unsupported or unparsable files fall back to textual counting after the applicable exclusions.