Infrastructure configuration affects agentic coding evals as much as model capability¶
Insight: Infrastructure configuration can shift agentic coding benchmark scores by 6 percentage points or more due to resource constraints affecting runtime evaluation environments. Resource allocation with zero headroom causes transient memory spikes leading to unnecessary failures independent of model capability. Meaningful benchmark comparisons require published infrastructure specifications; differences below 3 percentage points deserve skepticism without configuration documentation.
Detail: The problem stems from agentic coding's runtime nature: agents write, test, and iterate code against container resource limits. Unlike static benchmarks, infrastructure becomes part of the evaluation. Treating allocation floor and ceiling separately (with 3x headroom) stabilizes results. Beyond 3x multipliers, agents deploy memory-intensive strategies solving previously impossible problems.