Bansal et al. — best prediction of ML experiment outcomes is R²=0.49; all models underestimate effect sizes
| Date | 2026-10-19 |
|---|---|
| Category | Benchmarks |
| Judgement | Refutes |
| Gauge weight | 2 / 3 |
| Source | Bansal et al., Can LLMs predict the outcomes of machine learning experiments? (arXiv 2610.22599, announced 2026-10-19) |
Summary
In a study of 288 LLM-proposed ideas across 8 MLE-bench tasks, the best ex-ante effect prediction among five frontier models was R²=0.49 by Opus 5.5. All models systematically underestimated effect sizes, with no self-preference found.
A/B Impact Analysis
By quantifying a bottleneck of judgment rather than execution, this is Scenario B refutation evidence. If reading experiment outcomes in advance is at this level, high-risk high-return ideas are systematically underfunded, weakening the fast recursive improvement premise. The Scenario A reply of partial offset through parallel search is also recorded, but the debate grouped it with InnovationEval as a broken research loop. R² of 0.75 was later adopted as the reference line.