Traditional Optimization Tools Still Beat LLM Agents at Tuning Machine Learning

A June 2026 study compared large language models given code access against conventional hyperparameter optimizers—algorithms designed to fine-tune ML systems. Established methods like CMA-ES and TPE outperformed the LLM approach within a fixed search space. The finding suggests hybrid systems combining both approaches may be more practical than relying on LLMs alone for automation.