Classical Hyperparameter Optimization Methods Outperform LLM Agents in Empirical Test

A study published on arXiv on 9 June 2026 compared LLM agents with direct code-editing access against established hyperparameter optimizers. Within a fixed search space, CMA-ES and TPE consistently outperformed the language model approach. The finding calibrates expectations for LLM utility in ML automation and suggests hybrid architectures may offer the practical path forward.