Back to Search View Original Cite This Article

Abstract

<title>Abstract</title> <p> Large Language Models (LLMs) generate individual functions well, but turning a natural-language goal into a working multi-file project — with resolved dependencies and an entry point that actually runs — remains unreliable: single-pass output frequently crashes on missing packages, deprecated APIs, and unobserved runtime errors. We present SAGE, an open-source multi-agent framework that treats project synthesis as a closed-loop, self-adaptive control problem, a MAPE-K control loop: a Planner decomposes the goal, a Coding Agent generates every file, a Validator executes the project inside an isolated virtual environment, and a Feedback Agent converts the resulting tracebacks into typed repairs, iterating until the project runs. We evaluate SAGE on <italic>SAGE-Bench-v2</italic> , a benchmark we designed to exercise the failure modes that actually break LLM-generated code — API version-drift, type and schema errors, and silent incompleteness — using an <italic>independent</italic> re-execution oracle and the unbiased Pass@k estimator, with significance assessed by a paired Wilcoxon signed-rank test and Vargha–Delaney effect sizes. On gpt-4.1-mini, execution-guided repair raises Pass@1 from 69.3% to 92.0% (+ 22.7 points; \(\:p=0.042\), medium effect), rescuing tasks that open-loop generation never solves — for example, code that emits a pandas frequency alias removed in a recent release. We further introduce and test a <italic>correctness-aware oracle</italic> that repairs toward planner-specified success criteria; it does not significantly beat the exit-code oracle (+ 2.7 points, negligible effect), because the extra repair churn regresses roughly as many tasks as it rescues. SAGE offers a practical framework for self-healing synthesis and evidence that <italic>specifying</italic> success helps more than aggressively <italic>enforcing</italic> it. </p>

Show More

Keywords

project sage oracle effect goal

Related Articles

PORE

About

Connect