Build-Recipe Inference Agent

Software almost never ships with a reliable specification of how to build it. This project treats that missing specification as the artifact worth recovering, and treats the question of where automated recovery works as a measurement problem rather than a scoreboard.

Type

Independent research and engineering project

Started

May 2026 – ongoing

Status

Harness in progress; comparison design complete

Build systems

Autotools, Make, CMake, Maven, Gradle, Ant

Ground truth

~370 hand-verified build recipes

Corpora

Three, cross-compared

• Python 3.12
• Docker
• Anthropic API
• Linux / WSL2
• Bash

The problem

The dependency list, the required compiler or JDK version, the exact phase commands and flags — all of it usually lives in the head of whoever last got it working. Engineers lose days getting an unfamiliar codebase to build. Companies cannot reproduce old builds, and legacy projects quietly become unbuildable.

The insight is that the recipe is recoverable, just not by reading. An agent attempts a build in a clean container, reads the failure, adjusts the environment, and retries; the trace of what finally worked is the recipe.

The concrete case that explains why: a legacy Apache Java project fails to compile because it only works under an older JDK, and nothing in the source tree says so. The only path to that answer is to build it, read the error, and try a different JDK. That loop is the problem, and it is also why filename-based heuristics cannot solve it.

The comparison design

This is the part worth leading with, because it is what separates the project from a single-tool demo. Every existing system in this space reports results on the program set its own authors chose. Running three tools over all three corpora exposes how much of each reported success rate is capability and how much is corpus fit.

That is a measurement question, and answering it is the contribution. The deliverable is a map rather than a score — a grid of where builds succeed and fail, broken down by tool, build ecosystem, corpus, recipe field, and failure category.

The hypothesis the map tests

Inference accuracy should track how declared a field is. A build system is declared by the presence of pom.xml or configure.ac, so it should be inferred nearly perfectly. A required JDK version is often declared nowhere in the source tree and only surfaces as a compile error, so it should be inferred poorly by anything that does not actually attempt the build and read the error.

Evidence that the hypothesis is worth testing, drawn from hand-labeled build data across roughly 1,000 candidate programs:

• 92 of 161 projects with a recorded Java version need JDK 8, 11, or 17 rather than the default JDK 21, and nothing in the file tree says so.
• 103 programs where the default build commands fail outright.
• 25 projects where a plain mvn verify does not succeed and module exclusions or flags are required.
• 19 programs that will not compile under the default compiler flags.
• Configure invocations that depend on non-obvious compatibility flags such as CFLAGS+=-fcommon or FORCE_UNSAFE_CONFIGURE=1.

None of that is recoverable from filenames. All of it is recoverable by building, failing, and reading the error.

Scope and limits

Stating the limit before being asked is the point, so here it is up front.

• Deliberately scoped to programs whose dependencies are installable through the system package manager. That excludes projects needing pip, npm, cargo, specific hardware, or unpackaged legacy versions — roughly 141 of the 1,000 candidate programs.
• Two of the six build systems have very few examples in the corpus, so those results will be suggestive rather than statistically solid.
• The cross-tool comparison depends on the other systems being runnable. One is confirmed public; the other's artifacts are not yet confirmed available. Until both are verified this remains a comparison design, not a completed result.
• The ground-truth corpus is one I contributed to, not one I built alone. It came out of lab work with multiple contributors.

How the pass criterion is defined

The criterion was fixed before anything ran: the inferred recipe, executed in a clean container, produces a successful build. Every attempt runs in a fresh container with state reset. Results are scored against recipes a human already recorded by hand for the same programs.

Two design choices follow from treating this as evaluation work rather than tool-building:

• Constrained action space. The agent gets domain-specific tool wrappers rather than raw shell access, with an ablation designed to measure the effect on success rate and on invalid actions.
• Append-only event logging for deterministic replay, plus per-run cost and token accounting, so a result can be re-examined rather than only re-run.

Status

Environment and harness in progress; comparison design complete, results pending. Splitting what is designed from what is running is deliberate — it is the same honesty that makes the evaluation argument credible in the first place.

Done and defensible now

• Environment and repository setup.
• Six build systems worked with directly, hands-on.
• Build-failure diagnosis and failure-category construction.
• Study design, metric definition, and the cross-tool comparison design.
• Literature synthesis across the existing systems in this space.

In progress

• The containerized executor and build-system detection.
• The LLM inference call and failure-recovery loop with attempt budgets.
• Per-ecosystem results, ablation results, and the finished map.

What I would do differently

• Verify the availability of the external systems' artifacts before designing the comparison around them. The design is sound, but one leg of it is still contingent on something outside my control.
• Sample the corpus for build-system balance earlier. Two ecosystems ended up thin enough that their cells in the map will carry a caveat rather than a finding.

More work on the projects page, or back to Selected Work.