Agents, evaluation and governance
From copilots to AI software factories: what must change
A fictional reservation change illustrates the decisions, evidence and responsibilities needed before delegating more work.
Joan ArandaAn AI software factory looks attractive when we see agents completing tasks. Bringing that idea into an organization requires decisions about how work is prepared, who accepts it and what happens when the evidence is insufficient.
Before expanding autonomy, I would examine a small change from beginning to end. If its acceptance conditions remain ambiguous, adding execution capacity will probably make that ambiguity harder to manage.
A fictional change
We will use a hypothetical reservation application. The request is to allow cancellation when at least 24 hours remain before the reservation starts. It does not describe a real product or migration.
To make the example assessable, we establish these rules: the server compares UTC instants; cancellation is allowed exactly at the 24-hour boundary; only a confirmed reservation permits the initial cancellation; repeating an accepted cancellation does not duplicate its effects. Refund policies and the user interface are outside the scope of this change.
These are not universal decisions. Another product may require another policy. Their value is making the requested behavior visible.
Conditions for moving forward
| Point | Required evidence | Responsibility | If evidence is missing |
|---|---|---|---|
| Understand the system | Relevant existing states and effects, with uncertainties identified | Engineering and functional owner | Narrow the change or investigate |
| Agree on behavior | Boundary rule, states, repetition and scope | Functional owner with engineering | Resolve disagreements before implementation |
| Prepare execution | Affected modules, constraints and a proportional plan | Engineering | Reduce scope or complete context |
| Review the proposal | Changes consistent with rules and contracts | Technical reviewer | Correct or reject the proposal |
| Validate the result | Functional cases and preservation of relevant behavior | QA/engineering and functional acceptance | Iterate; do not declare the change complete |
| Deliver | Recorded acceptance and an appropriate recovery mechanism | Delivery owner | Do not activate the change |
A useful validation gate is defined by a decision, its evidence and a responsibility that can stop the workflow. Its name is secondary. A few people may perform these functions; the table does not require hiring a role for every row.
What validation would demonstrate
In this example, we would check a confirmed reservation exactly 24 hours before its start, one just outside the permitted cancellation period and one just inside. We would also check non-cancellable states, repetition of an accepted request and absence of duplicate effects. Test instants would be fixed so results do not depend on the current clock.
Testing the boundary does not establish equivalence across the entire system. We need to delimit which behavior is being preserved and examine relevant integration. An evaluator that repeats the implementation’s mistaken assumptions can falsely approve it.
Anthropic recommends well-specified tasks, stable environments and appropriate tests for assessing coding agents. That guidance is useful, while acceptance of a specific delivery still requires the team’s context and responsibility.
The specification can change too
If review uncovers an unexpected constraint, we must decide whether it changes the agreement. Preventing every change to preserve the document would create false stability. Changing it without recording why would make it an unreliable reference.
The response is to maintain small cycles: revise intent or context, update the checks and repeat affected execution. The workflow need not become a cascade of lengthy documents.
What should improve when capacity expands
Only after understanding a task would I consider parallel execution or multiple agents. I would want to know which work can be separated, where conflicts emerge and what integration and review cost.
Expansion makes sense when it improves accepted outcomes within understood limits. More open proposals can consume review capacity without increasing deliveries. The operating model needs to make that trade-off observable.
An AI software factory becomes an assessable proposal when we can explain its conditions for progress and its failures. Execution is a capability; the ability to accept or stop the result belongs to the system that makes it usable.
Reference
- Anthropic Engineering, Demystifying evals for AI agents: evaluation guidance, not empirical validation of this example.
The table, rules and case are original hypothetical illustrations prepared for publication. No tests of this fictional product have been executed, and no production results are reported.