What AI Cannot Replace in Software Development, Part 3

In a previous article in this series, I proposed a redefinition of legacy: legacy is not a property of the code itself, but a state of the relationship between the code and those expected to maintain it. A system becomes legacy the moment no one in the organization mentally inhabits what they are nonetheless expected to evolve. The problem lies less in the age of the technology than in the developer’s lack of deep cognitive engagement with their own code.

This definition raises an urgent question: if generative AI produces code in place of developers, and if those developers only supervise the output without having genuinely thought through the underlying reasoning, are we creating the conditions for legacy at scale from the very first generated line?

The purpose of this article is to examine this question and its organizational consequences. Partial integration of AI into the development process tends to create a structural instability, one that leads logically to what I call a tipping point: a discontinuity in how an organization must configure itself. Getting through this transition also requires a precise understanding of which organizational assets are truly durable, and the answer is less obvious than it appears.

Cognitive Engagement as a Condition of Maintainability

When a developer writes code, they do more than produce text: they build a mental representation of the problem, its constraints, edge cases, and trade-offs. This representation lives in procedural memory and enables effective maintenance: the ability to anticipate where things will break, to understand why a decision was made, and to assess the impact of a change without simulating the entire system.

When a developer merely supervises AI-generated code, this mental representation does not fully form. They can read the code, validate its apparent logic, and approve a pull request, but they have not traversed the reasoning that produced it.

Supervising AI-generated code without engagement in the reasoning that produces it is functionally similar to maintaining a legacy system: one understands what the code does, but not why it does it that way.

I am not criticizing generative AI as such. I am describing a structural property of the situation: intellectual ownership of code is a consequence of the creation process, not of reading the result, and review cannot supply it after the fact.

The Tipping Point: A Structural Discontinuity

Organizations that integrate generative AI without changing the structure of responsibility tend to drift into an unstable intermediate state, which typically becomes visible only during the first serious incidents.

Developers believe they are supervising, but have progressively lost the capacity to intervene autonomously. The system keeps growing, technically modern and cognitively orphaned. In several AI-assisted transitions I have observed, the pattern is consistent: productivity is high for months, velocity metrics look excellent, confidence is strong. Then comes the first non-trivial evolution (a scaling constraint, an architectural shift, or a subtle behavioral regression), and no one can confidently explain why the system behaves as it does.

The problem with gradual transition is precisely that it produces this situation without anyone having made the deliberate choice to accept it. The organization bears the costs of both models without benefiting from the advantages of either.

The relationship between human intellectual ownership and AI integration is not linear. Below a certain threshold of engagement, maintainability tends to collapse in a way that is difficult to reverse once the accumulated gap reaches critical mass. This threshold is the tipping point. It separates two internally stable configurations from a fragile intermediate zone.

The Two Stable Configurations

The Human-Author Model

Humans remain the cognitive authors of the code. Generative AI serves as an accelerator for boilerplate, refactoring, or exploration, but the developer still traverses the reasoning. They can explain every significant decision, debug unexpected behaviors, and anticipate the consequences of changes. This model preserves long-term maintainability and integrates naturally with existing quality practices. Its main constraint is speed: AI amplifies productivity but does not replace the human traversal of reasoning.

The Agent-Author Model

Agents become responsible for producing the code. Humans step out of the traditional technical loop. In exchange, the organization reconfigures itself deeply: code loses its status as a durable strategic asset and becomes a temporary instantiation of behavior defined elsewhere, rewritten on demand from specifications and behavioral tests, which now form the primary reference. Classical technical debt tends to be largely replaced by a different form of accumulated cost centered on specification quality and test completeness. Architecture becomes declarative: focused on what the system must do, not how.

Which of these assets, then, are truly durable over time? Current agent-oriented frameworks largely leave this question unanswered, and it deserves careful examination before committing to this model.

The Hierarchy of Durable Assets

Specification Drift and the Limits of Natural Language Stability

Current agent-oriented frameworks tend to treat specifications as a stable abstraction layer above the model. The assumption is that you write the spec once and can reliably regenerate the system across model versions and technology changes.

This assumption deserves scrutiny. Practitioners already observe prompt drift, LLM drift, and context drift: the gradual degradation in how a specification is interpreted as models evolve. A specification that worked reliably with one model version can produce subtly different behaviors in its successor, not because the spec changed, but because the model’s implicit understanding of its constructs has shifted. The long-term stability of a specification is coupled, implicitly, to the stability of the model that interprets it.

In systems meant to last years or decades, this specification drift becomes more than a prompt engineering inconvenience; it threatens the long-term reproducibility of the system itself. Subtle divergences can accumulate in the spaces between explicit requirements, even when tests continue to pass on covered behaviors. Current frameworks do not yet formally manage this coupling.

The Asset Hierarchy

This leads to a clear hierarchy of durability:

• Code is ephemeral by design, the least durable asset. It is regenerated, not evolved.
• Specifications are semi-stable. More durable than code, but subject to semantic drift relative to evolving models. They require active, periodic re-validation.
• Behavioral tests constitute the most stable anchor. A well-designed test yields a binary, objective verdict (it passes or it fails), independent of any model’s interpretation of intent.

However, the durability of tests ultimately rests on the quality of the human oracle that defines them. A poorly conceived test can protect incorrect behavior as reliably as a good one protects the right one. This brings us back directly to the oracle problem examined in the first article of this series: tests are only as strong as the human reasoning behind them.

In the agent-author model, the true organizational anchor is the behavioral test suite, but only insofar as it crystallizes genuine human understanding of expected behavior. Tests are the formal record of the oracle’s judgment, not a substitute for it.

Quality Assurance as a Foundational Discipline

This inversion is structural. In traditional development, QA is positioned as downstream validation. In the agent-author model, defining, maintaining, and evolving behavioral tests becomes the first step in the process, and the most consequential one.

The quality engineer who designs solid behavioral test suites is no longer in a support role. They steward the behavioral knowledge that makes reliable system regeneration possible across time, model versions, and technology generations. QA shifts from a verification function to a core strategic competency.

The Human Role in the Agent-Author Model

Agents operate within a bounded context, a necessary reduction of reality that always implies information loss. The real world is structurally larger than any context an agent can receive: regulatory constraints, internal political dynamics, implicit contractual commitments, strategic directions, stakeholder relationships. None of this is fully visible to a system operating within a defined context.

A concrete example: agents might correctly choose a technically superior tool within their context, yet overlook a token cost ten times higher, an unevaluated vendor dependency, or a strategic direction not yet formally communicated. These elements simply were not in their context.

The human role is therefore not to supervise code, but to maintain alignment between the space of agents and the real world that will never be entirely captured in a context. It is a translation role: transposing the constraints, objectives, and values of the real world into clear intentions for systems that operate within bounded spaces. Humans retain responsibility for the what and the why. Agents manage the how.

In this model, IT professionals do not disappear; they migrate. Their expertise moves from writing code to mastering specification quality, behavioral verification, and domain knowledge. The longstanding comprehension gap between business and IT dissolves here through a deeper reconfiguration of the IT role itself.

A Historical Precedent: The Transition to Deep Learning

The shift I describe has a direct parallel in AI’s own history. For several decades, symbolic expert systems emphasized full interpretability: every rule was explicit and traceable. The transition to deep learning abandoned this property. Neural networks produce impressive results through mechanisms their creators cannot fully explain. The debate was settled by results rather than philosophy: deep learning produced capabilities that symbolic systems could not reach.

AI became what it is today because the scientific community collectively accepted abandoning internal interpretability as a necessary condition of a system’s validity. Observable performance replaced transparency of mechanism as the primary evaluation criterion.

The parallel with the tipping point is direct. In the agent-author model, one abandons code comprehensibility as a condition of maintainability. It is replaced by the quality of specifications and, above all, the completeness of behavioral tests. Observable performance replaces internal understanding.

This precedent does not prove the tipping point is inevitable everywhere, but it shows that abandoning one form of comprehensibility for another form of validation can produce significant gains, and that resisting the shift can carry a competitive cost.

The Organizational Question: The Greenfield Model

For large organizations with decades of technological history, transforming the existing base is rarely feasible in the short term. Legacy systems (those for which intellectual ownership has already been lost) do not lend themselves to migration toward the agent-author model without massive prior investment in specification.

The coherent pattern is the greenfield model: creating a new entity, such as a division, a subsidiary, or a distinct organization, configured according to the agent-author model from day one. This entity develops the competencies in specification, declarative architecture, and behavioral testing while the existing organization continues under the human-author model.

This pattern is not without precedent. Large financial institutions have used it for digital transformation initiatives. Cooperatives have created subsidiaries to explore new business models without risking their operational core. When transformation requires a reconfiguration too deep to be achieved from within, one creates something new alongside.

The resulting coexistence, with units in human-author mode managing the existing systems and units in agent-author mode building the future, is stable when the boundary between the two models is explicit and deliberate. The danger lies in blurring the line between them, not in having two models.

Implications for Practitioners and Decision-Makers

Three practical observations emerge:

First, diagnose honestly. The relevant question is not “what percentage of our code is AI-generated?” but “can our developers intervene autonomously and confidently in the code that exists?” An uncertain answer signals that the organization is likely already in the unstable intermediate zone, without having made the deliberate choice to be there.

Second, choose deliberately. Maintaining the human-author model requires frameworks that preserve developers’ traversal of reasoning: not excluding AI, but ensuring they remain genuine authors of what they maintain. Moving to the agent-author model demands significant prior investment in specification quality and behavioral tests before the transition, not after.

Third, protect the right asset. Resist the temptation to treat specifications as the primary investment. The behavioral test suite, grounded in rigorous human judgment, is the more stable and strategically critical anchor. Specifications communicate intent to agents; tests verify that the intent was correctly understood. Only the latter survives model evolution without active re-validation.

The intermediate zone, where humans supervise code they have not genuinely inhabited without strong specifications and tests, is a trap rather than a transitional state. It is precisely where tomorrow’s legacy forms, at the accelerated speed of generative AI.

In the previous article in this series, I argued that the capacity to judge the correctness of a result is an irreplaceably human competency. Machines can evaluate outputs; what they cannot do is define what constitutes a correct output, because that requires an understanding of the real world that exceeds any formalizable context.

The tipping point does not contradict this thesis. It extends it upward. In the agent-author model, humans no longer judge the correctness of code. They define the behavioral tests that encode what correctness means, and they translate the real-world constraints that no bounded context will ever fully capture. The oracle does not disappear: it moves upstream and becomes the organization’s most critical competency.

AI is already transforming software development.
The question is whether your organization will deliberately choose its tipping point, or simply drift into one without realizing what it has surrendered.

Gabriel Mailhot
Organizational Engineering for the AI Era
Researcher (2005–2010): Image Processing & Creative AI
Business Expert / Domain Expert (2010-2026): Software Architecture, Software Development and Quality Practices

Series: What AI Cannot Replace in Software Development

An Operational Distinction

I have spent a significant part of my career working with organizations whose human foundation is unstable: high turnover, frequent reassignments, temporary teams assembled for a project and dissolved at its close. In these organizations, something specific happens to software systems over time. The knowledge of why the code is the way it is gradually disappears.

It is most visible with legacy systems. Take VB6 applications, COBOL modules, early Java codebases: the developers who built them have moved on, retired, or simply forgotten. What remains is the code itself, and a thin layer of documentation that rarely captures the decisions and constraints that shaped it. New developers inherit a system they can run but not fully understand.

I had assumed this was a problem of time, in which knowledge erodes because the people who held it leave. Then I started observing something different in teams using AI coding assistants intensively. The developers were still there, yet the knowledge was still eroding. The mechanism was disengagement rather than departure.

AI creates distance through engagement rather than through time, and that distinction changes how we should think about what developers must retain.

The Structural Advantages of AI: Memory and Computational Speed

AI has two advantages that do not depend on the quality of the model or its version: near-unlimited memory and processing speed that vastly exceeds human cognition.

Memory first. A large language model has been trained on billions of lines of code, millions of technical discussions, years of documentation. It can instantly retrieve the exact syntax of a rarely used API, the correct pattern for an obscure edge case, the exhaustive list of parameters for a given function. No developer, however experienced, can compete on this terrain.

Speed next. AI accomplishes in seconds what would take hours: generating boilerplate, converting an algorithm from one language to another, reformatting a data structure, producing variants. This is a real, measurable advantage, and it is not going away.

These two advantages are what they are: memory and performance, not judgment.

What the Developer Must Retain: Causal Understanding

The boundary between what AI does well and what it cannot do is more precise than a split between memory and reasoning in the abstract sense. It runs between pattern recognition and causal understanding.

An experienced developer who identifies a race condition in multithreaded code does more than recognize a familiar pattern. They understand why that pattern is problematic in this specific context: which timing constraint creates the risk, which property of the system makes the bug intermittent, which fix will resolve the problem without creating another.

AI can recognize the pattern. It does not possess the causal understanding that allows one to evaluate whether that recognition is relevant in this context. This understanding of the system, the domain, and the real constraints is what the developer must retain.

The concern has empirical support. A randomized controlled trial examining how AI coding assistance affects skill development found that developers using AI assistance scored 17 percentage points lower on comprehension and debugging tests than those coding manually (50% versus 67% on average). The contrast was even sharper by usage pattern: developers who delegated code generation to AI scored below 40%, while those who used AI for conceptual inquiry while coding independently scored 65% or higher. The engagement pattern mattered more than the tool itself.

This includes, as discussed in the first article of this series, the definition of expected behaviors and the design of tests. These activities call for judgment about what the system must do in the real world, not memorization.

Knowledge Erosion: An Underestimated Organizational Risk

When a developer delegates to AI not only memory and speed, but also understanding, a silent phenomenon takes hold: knowledge erosion.

The traditional form of this erosion is familiar. Developers leave, and with them goes the understanding of why the system is structured the way it is (the decisions, constraints, and trade-offs that shaped it). Call it a distance of time: the gap between the people who understood the system and those who now maintain it.

What AI introduces is structurally different. The developers are still there, but their engagement with the code has diminished. The cognitive effort of reading it, holding its structure in mind, and reasoning about its consequences no longer happens. They generate, accept, and move on, and the knowledge never forms.

This is a distance of engagement, and it is more insidious than the distance of time because it is invisible. The team is stable, the velocity metrics are green, the system runs. Yet the understanding that would allow someone to reason confidently about changing it has quietly failed to accumulate: anticipating the consequences of a modification, recognizing a subtle regression, formulating a test without looking at the code.

Teams using coding assistants intensively report that developers increasingly struggle to explain code they have recently written themselves. Understanding atrophies from lack of exercise, and a developer who does not fully understand their own system can no longer evaluate its quality. This closes the loop with the testing problem from the first article. Such developers can no longer verify that the system does what it should. They may not have lost sight of the intent, but they no longer understand the code well enough to reason about whether it delivers it.

The Optimal Division of Labor

The right question is therefore not “should we use AI?” but “for what should we use it?”

AI as a memory extension: syntax, APIs, known patterns, mechanical conversions, boilerplate. The developer formulates the request, evaluates the result, decides whether to integrate it. AI accelerates access to what they already know how to look for.

The developer as guardian of understanding: architecture, expected behaviors, edge cases, domain constraints, tests, decisions that commit the future of the system. These activities are not delegated. AI may well produce something that resembles them, but their value lies precisely in the human judgment that produces them.

This division is a stable structure rather than a temporary limitation, grounded in what causal understanding requires: lived experience of the system, knowledge of the domain, and accountability for the outcome.

Implications for Organizations and Training

This analysis has direct consequences for how organizations train their developers and evaluate the value of AI.

On training: the goal should not be to teach developers to use AI, but to ensure that AI use does not replace the development of understanding. Practices that require reading code, manual debugging, and peer review, precisely those that delivery pressure tends to eliminate, become more important.

On value assessment: the usual metrics (lines of code produced, delivery speed, test coverage rates) do not capture knowledge erosion. An organization can see its metrics improve while its capacity to master its own system degrades. The indicators that matter are harder to measure: a developer’s ability to explain why an architectural decision was made, to recognize a subtle regression without running the tests, or to reason confidently about the consequences of a change they have not yet made.

AI does not save time; it changes the nature of work by shifting value toward what the machine cannot do. Organizations that understand this invest in human understanding. Those that do not let it atrophy without even noticing.

Legacy systems are already showing us what a system looks like when its knowledge has disappeared. The question is whether we are building the legacy systems of 2035 right now. If the specifications and tests that protect expected behavior are themselves produced by developers who no longer fully understand the system, the circle closes on itself, and the erosion spreads beyond the code to the knowledge that was supposed to protect it.

The distance of engagement is a choice, until it isn’t.

Gabriel Mailhot
Software Architect and Software Engineering Quality Advisor
Series: What AI Cannot Replace in Software Development

In the organization where I work, there is a mainframe COBOL system that has been running at the core of operations for decades. By every conventional definition, it should be legacy: old language, old architecture, a technology that most developers under forty have never touched.

It is not legacy. The team responsible for it is among the most mature and capable in the organization. They understand every decision embedded in that system: the constraints that shaped it, the business rules it encodes, the edge cases it handles. They can explain why a specific field is structured a certain way, what regulatory requirement drove a particular branching condition, and exactly what would happen downstream if you changed it. There is no question about the system they cannot answer.

Contrast that with a microservices architecture I reviewed at another organization. It had been built two years prior on a modern stack, with 85% test coverage and a full CI/CD pipeline. The team that built it had since been reorganized. No one could explain why the service boundaries had been drawn that way. Modifying anything felt like defusing a bomb. That system was legacy from the inside out, despite being less than two years old.

Michael Feathers defined legacy code as code without tests. The definition is pragmatic and useful, since tests reduce the uncertainty of change, but it is incomplete.

Tests are not the cause of human engagement with code. They are a partial substitute for its erosion.

When a team genuinely understands a system (its decisions, its constraints, its intended behaviors), tests are almost redundant as a knowledge preservation mechanism. When that understanding has eroded, tests become the last line of defense against the unknown. They matter enormously in that role. But their presence does not mean the understanding is there.

Legacy is not a property of the code. It is the state of the relationship between the code and the people responsible for it.

A system becomes legacy at the moment when no one can confidently answer: why does it work this way? Not how, since any developer can read code, but why: the reasoning, the trade-offs, the domain constraints that shaped the implementation. When that knowledge is gone, whether through team turnover, disengagement, or the gradual accumulation of changes nobody fully understood, the system has crossed the threshold.

This distinction matters because it changes what the problem actually is. If legacy is a technical property, the solution is technical: rewrite, refactor, add tests, modernize the stack. If legacy is a relational property (a broken link between a system and its human maintainers), then the solution requires restoring that link. Sometimes that means tests. More often it means something harder: rebuilding the shared understanding of what the system is supposed to do, and why.

Tests, documentation, and architecture diagrams are mechanisms for transmitting intent across time and team boundaries. They matter, but they are proxies for something more fundamental: the capacity of a human being to understand a system well enough to change it with confidence.

As long as that capacity exists, the software is alive, regardless of its age, its language, or its test coverage.

The moment it disappears, the system becomes legacy. And no amount of refactoring will fix that without first addressing the human dimension.

This is the foundation on which the rest of this series is built. The next article takes up the question that follows: what happens to that human capacity when AI generates the code?

Gabriel Mailhot
Software Architect and Software Engineering Quality Advisor
Series: What AI Cannot Replace in Software Development