What AI Cannot Replace in Software Development, Part 1
A perspective on the structural limits of automated test generation, and what they reveal about the nature of work in software engineering.

The Problem Nobody Names

A few years ago, I reviewed a software development contract between an organization and a vendor. The terms were standard: a fixed delivery date, a penalty clause for late delivery, and an obligation to maintain 80% code coverage. The intent was twofold. It would protect the organization from negligence, and guard against the kind of deliberate opportunism that governance frameworks exist precisely to prevent.

When the project fell behind schedule, the vendor found a way to satisfy the coverage requirement without slowing down: they had the AI generate unit tests after the developers had written the code. The coverage metric turned green, the delivery date was met, and the penalty was avoided.

Then came UAT, and the acceptance phase surfaced problem after problem. The tests had protected nothing, because they had been written to describe what the code did rather than what the system was supposed to do.

The contract had specified a metric, and the metric had been satisfied. The organization had paid, contractually and technically, for a test suite that institutionalized its own blind spots.

I am often asked whether it is a good idea to let AI generate unit tests. That experience is part of my answer. The complete answer is more interesting, because it touches on the nature of testing, the source of truth in a software system, and what AI actually changes in the work of a developer.

The Test Oracle Problem: What Research Already Knows

In the software engineering literature, the problem has a precise name: the test oracle problem. It designates the central challenge of any automated test generation. How do you automatically distinguish correct behavior from incorrect behavior?

Automated generation tools, including those based on large language models, produce what are called regression oracles: tests that capture what the code does today, not what it should do.

When you ask AI to generate tests from source code, you invert the causality of testing. The test becomes a mirror of the implementation, and if the implementation contains a bug, the test will capture it in order to protect it rather than flag it. Beyond introducing a bug into the system, you put in place an infrastructure that resists its correction.

Alan Turing demonstrated formally that one can never prove a program is free of all bugs. This is a consequence of undecidability: no general procedure can determine whether a program behaves correctly in all cases. In practice, this means there will always be incorrect behaviors the code produces without anyone having intended them. When AI takes the code as its source of truth, it freezes these invisible behaviors at industrial scale.

The test oracle problem is not new; what AI changes is its scale. A developer writing tests manually is constrained by the time it takes. An AI generating tests from code can institutionalize incorrect behavior at industrial speed, producing in seconds a test suite that would have taken days to write and that is wrong in exactly the same way throughout.

What Kent Beck Understood, and What Automated Generation Ignores

Kent Beck, in his Test-Driven Development philosophy, establishes a distinction that remains underappreciated: a test must speak to behavior, not implementation. Behavior is the very reason the code needs to exist. Unlike code, it is stable: it survives refactorings, technology changes, and complete rewrites.

A behavior-based test is a living document of the system’s intent. Future developers will refer to it to understand what the system must do, as opposed to what it happens to do today.

A test generated from code lies about that intent, documenting the accident rather than the design.

This has an organizational consequence the technical literature does not make explicit. An untested bug can be discovered and fixed freely. A bug protected by a test becomes a norm. Any attempt to correct it will cause the test to fail, creating institutional pressure not to fix it. At that point the problem is one of governance as much as of technique.

A test generated from code does not protect against bugs. It protects against change, including the change that would fix the bug.

The Limits of Specifications as a Substitute

One might think the problem is solved by asking AI to generate tests from specifications rather than from code. This is a direction academic research is actively exploring, enriching prompts with documentation, developer comments, and semantic context. It is better, but it is not enough.

A specification is always incomplete. Consider a typical example: a spec describes how a search feature should return results ranked by relevance. It says nothing about what happens when two users submit the same query simultaneously, or how the system should behave when the underlying index is being updated, or what “relevance” means when the user’s implicit intent differs from their explicit query. No negligent author forgot these edge cases. They belong to the class of conditions that only become visible when the system meets real users in a real environment.

Human judgment detects these gaps naturally, by confronting the spec with lived experience of how systems fail, how users actually behave, and what the production environment will impose. AI generates tests that are consistent with what is written, without the ability to signal what is missing.

A better model will not remove this limitation, because it is structural rather than temporary: understanding the real world, with its complexity, ambiguity, and unspoken assumptions, remains a capability current systems do not possess.

The Real Asset of a Software Organization

In a world where AI generates code, the real asset of an organization becomes its specifications and its tests.

The code itself will be rewritten, migrated, replaced. What endures is the knowledge of expected behavior, captured by humans who understand the problem in its full complexity.

This conclusion does not appear in the technical literature or in analyst reports, which measure the value of AI in terms of volume produced. It is a thesis about the nature of a software organization’s intellectual asset, and it radically changes what is worth investing in.

Organizations that have understood this invest in the quality of their specifications and in the human rigor of their tests. Those that delegate both to AI, working from existing code, leave their technical debt in place while making it harder to see and harder to correct.

The question of what the developer must retain, and what can legitimately be delegated, extends beyond the problem of tests alone. It touches on the cognitive division between human and machine, which is the subject of the next article in this series.

If the real asset of a software organization is not its code but its understanding of expected behavior, who in your organization is responsible for building and protecting that understanding? And what happens to it when the pressure to deliver is high enough?

Gabriel Mailhot
Software Architect and Software Engineering Quality Advisor
Series: What AI Cannot Replace in Software Development