What AI Cannot Replace in Software Development, Part 3

In a previous article in this series, I proposed a redefinition of legacy: legacy is not a property of the code itself, but a state of the relationship between the code and those expected to maintain it. A system becomes legacy the moment no one in the organization mentally inhabits what they are nonetheless expected to evolve. The problem lies less in the age of the technology than in the developer’s lack of deep cognitive engagement with their own code.

This definition raises an urgent question: if generative AI produces code in place of developers, and if those developers only supervise the output without having genuinely thought through the underlying reasoning, are we creating the conditions for legacy at scale from the very first generated line?

The purpose of this article is to examine this question and its organizational consequences. Partial integration of AI into the development process tends to create a structural instability, one that leads logically to what I call a tipping point: a discontinuity in how an organization must configure itself. Getting through this transition also requires a precise understanding of which organizational assets are truly durable, and the answer is less obvious than it appears.

Cognitive Engagement as a Condition of Maintainability

When a developer writes code, they do more than produce text: they build a mental representation of the problem, its constraints, edge cases, and trade-offs. This representation lives in procedural memory and enables effective maintenance: the ability to anticipate where things will break, to understand why a decision was made, and to assess the impact of a change without simulating the entire system.

When a developer merely supervises AI-generated code, this mental representation does not fully form. They can read the code, validate its apparent logic, and approve a pull request, but they have not traversed the reasoning that produced it.

Supervising AI-generated code without engagement in the reasoning that produces it is functionally similar to maintaining a legacy system: one understands what the code does, but not why it does it that way.

I am not criticizing generative AI as such. I am describing a structural property of the situation: intellectual ownership of code is a consequence of the creation process, not of reading the result, and review cannot supply it after the fact.

The Tipping Point: A Structural Discontinuity

Organizations that integrate generative AI without changing the structure of responsibility tend to drift into an unstable intermediate state, which typically becomes visible only during the first serious incidents.

Developers believe they are supervising, but have progressively lost the capacity to intervene autonomously. The system keeps growing, technically modern and cognitively orphaned. In several AI-assisted transitions I have observed, the pattern is consistent: productivity is high for months, velocity metrics look excellent, confidence is strong. Then comes the first non-trivial evolution (a scaling constraint, an architectural shift, or a subtle behavioral regression), and no one can confidently explain why the system behaves as it does.

The problem with gradual transition is precisely that it produces this situation without anyone having made the deliberate choice to accept it. The organization bears the costs of both models without benefiting from the advantages of either.

The relationship between human intellectual ownership and AI integration is not linear. Below a certain threshold of engagement, maintainability tends to collapse in a way that is difficult to reverse once the accumulated gap reaches critical mass. This threshold is the tipping point. It separates two internally stable configurations from a fragile intermediate zone.

The Two Stable Configurations

The Human-Author Model

Humans remain the cognitive authors of the code. Generative AI serves as an accelerator for boilerplate, refactoring, or exploration, but the developer still traverses the reasoning. They can explain every significant decision, debug unexpected behaviors, and anticipate the consequences of changes. This model preserves long-term maintainability and integrates naturally with existing quality practices. Its main constraint is speed: AI amplifies productivity but does not replace the human traversal of reasoning.

The Agent-Author Model

Agents become responsible for producing the code. Humans step out of the traditional technical loop. In exchange, the organization reconfigures itself deeply: code loses its status as a durable strategic asset and becomes a temporary instantiation of behavior defined elsewhere, rewritten on demand from specifications and behavioral tests, which now form the primary reference. Classical technical debt tends to be largely replaced by a different form of accumulated cost centered on specification quality and test completeness. Architecture becomes declarative: focused on what the system must do, not how.

Which of these assets, then, are truly durable over time? Current agent-oriented frameworks largely leave this question unanswered, and it deserves careful examination before committing to this model.

The Hierarchy of Durable Assets

Specification Drift and the Limits of Natural Language Stability

Current agent-oriented frameworks tend to treat specifications as a stable abstraction layer above the model. The assumption is that you write the spec once and can reliably regenerate the system across model versions and technology changes.

This assumption deserves scrutiny. Practitioners already observe prompt drift, LLM drift, and context drift: the gradual degradation in how a specification is interpreted as models evolve. A specification that worked reliably with one model version can produce subtly different behaviors in its successor, not because the spec changed, but because the model’s implicit understanding of its constructs has shifted. The long-term stability of a specification is coupled, implicitly, to the stability of the model that interprets it.

In systems meant to last years or decades, this specification drift becomes more than a prompt engineering inconvenience; it threatens the long-term reproducibility of the system itself. Subtle divergences can accumulate in the spaces between explicit requirements, even when tests continue to pass on covered behaviors. Current frameworks do not yet formally manage this coupling.

The Asset Hierarchy

This leads to a clear hierarchy of durability:

• Code is ephemeral by design, the least durable asset. It is regenerated, not evolved.
• Specifications are semi-stable. More durable than code, but subject to semantic drift relative to evolving models. They require active, periodic re-validation.
• Behavioral tests constitute the most stable anchor. A well-designed test yields a binary, objective verdict (it passes or it fails), independent of any model’s interpretation of intent.

However, the durability of tests ultimately rests on the quality of the human oracle that defines them. A poorly conceived test can protect incorrect behavior as reliably as a good one protects the right one. This brings us back directly to the oracle problem examined in the first article of this series: tests are only as strong as the human reasoning behind them.

In the agent-author model, the true organizational anchor is the behavioral test suite, but only insofar as it crystallizes genuine human understanding of expected behavior. Tests are the formal record of the oracle’s judgment, not a substitute for it.

Quality Assurance as a Foundational Discipline

This inversion is structural. In traditional development, QA is positioned as downstream validation. In the agent-author model, defining, maintaining, and evolving behavioral tests becomes the first step in the process, and the most consequential one.

The quality engineer who designs solid behavioral test suites is no longer in a support role. They steward the behavioral knowledge that makes reliable system regeneration possible across time, model versions, and technology generations. QA shifts from a verification function to a core strategic competency.

The Human Role in the Agent-Author Model

Agents operate within a bounded context, a necessary reduction of reality that always implies information loss. The real world is structurally larger than any context an agent can receive: regulatory constraints, internal political dynamics, implicit contractual commitments, strategic directions, stakeholder relationships. None of this is fully visible to a system operating within a defined context.

A concrete example: agents might correctly choose a technically superior tool within their context, yet overlook a token cost ten times higher, an unevaluated vendor dependency, or a strategic direction not yet formally communicated. These elements simply were not in their context.

The human role is therefore not to supervise code, but to maintain alignment between the space of agents and the real world that will never be entirely captured in a context. It is a translation role: transposing the constraints, objectives, and values of the real world into clear intentions for systems that operate within bounded spaces. Humans retain responsibility for the what and the why. Agents manage the how.

In this model, IT professionals do not disappear; they migrate. Their expertise moves from writing code to mastering specification quality, behavioral verification, and domain knowledge. The longstanding comprehension gap between business and IT dissolves here through a deeper reconfiguration of the IT role itself.

A Historical Precedent: The Transition to Deep Learning

The shift I describe has a direct parallel in AI’s own history. For several decades, symbolic expert systems emphasized full interpretability: every rule was explicit and traceable. The transition to deep learning abandoned this property. Neural networks produce impressive results through mechanisms their creators cannot fully explain. The debate was settled by results rather than philosophy: deep learning produced capabilities that symbolic systems could not reach.

AI became what it is today because the scientific community collectively accepted abandoning internal interpretability as a necessary condition of a system’s validity. Observable performance replaced transparency of mechanism as the primary evaluation criterion.

The parallel with the tipping point is direct. In the agent-author model, one abandons code comprehensibility as a condition of maintainability. It is replaced by the quality of specifications and, above all, the completeness of behavioral tests. Observable performance replaces internal understanding.

This precedent does not prove the tipping point is inevitable everywhere, but it shows that abandoning one form of comprehensibility for another form of validation can produce significant gains, and that resisting the shift can carry a competitive cost.

The Organizational Question: The Greenfield Model

For large organizations with decades of technological history, transforming the existing base is rarely feasible in the short term. Legacy systems (those for which intellectual ownership has already been lost) do not lend themselves to migration toward the agent-author model without massive prior investment in specification.

The coherent pattern is the greenfield model: creating a new entity, such as a division, a subsidiary, or a distinct organization, configured according to the agent-author model from day one. This entity develops the competencies in specification, declarative architecture, and behavioral testing while the existing organization continues under the human-author model.

This pattern is not without precedent. Large financial institutions have used it for digital transformation initiatives. Cooperatives have created subsidiaries to explore new business models without risking their operational core. When transformation requires a reconfiguration too deep to be achieved from within, one creates something new alongside.

The resulting coexistence, with units in human-author mode managing the existing systems and units in agent-author mode building the future, is stable when the boundary between the two models is explicit and deliberate. The danger lies in blurring the line between them, not in having two models.

Implications for Practitioners and Decision-Makers

Three practical observations emerge:

First, diagnose honestly. The relevant question is not “what percentage of our code is AI-generated?” but “can our developers intervene autonomously and confidently in the code that exists?” An uncertain answer signals that the organization is likely already in the unstable intermediate zone, without having made the deliberate choice to be there.

Second, choose deliberately. Maintaining the human-author model requires frameworks that preserve developers’ traversal of reasoning: not excluding AI, but ensuring they remain genuine authors of what they maintain. Moving to the agent-author model demands significant prior investment in specification quality and behavioral tests before the transition, not after.

Third, protect the right asset. Resist the temptation to treat specifications as the primary investment. The behavioral test suite, grounded in rigorous human judgment, is the more stable and strategically critical anchor. Specifications communicate intent to agents; tests verify that the intent was correctly understood. Only the latter survives model evolution without active re-validation.

The intermediate zone, where humans supervise code they have not genuinely inhabited without strong specifications and tests, is a trap rather than a transitional state. It is precisely where tomorrow’s legacy forms, at the accelerated speed of generative AI.

In the previous article in this series, I argued that the capacity to judge the correctness of a result is an irreplaceably human competency. Machines can evaluate outputs; what they cannot do is define what constitutes a correct output, because that requires an understanding of the real world that exceeds any formalizable context.

The tipping point does not contradict this thesis. It extends it upward. In the agent-author model, humans no longer judge the correctness of code. They define the behavioral tests that encode what correctness means, and they translate the real-world constraints that no bounded context will ever fully capture. The oracle does not disappear: it moves upstream and becomes the organization’s most critical competency.

AI is already transforming software development.
The question is whether your organization will deliberately choose its tipping point, or simply drift into one without realizing what it has surrendered.

Gabriel Mailhot
Organizational Engineering for the AI Era
Researcher (2005–2010): Image Processing & Creative AI
Business Expert / Domain Expert (2010-2026): Software Architecture, Software Development and Quality Practices

Series: What AI Cannot Replace in Software Development

The first three articles in this series examined specific symptoms: the instability of the People-Process-Tools model under high human variability, the strategic liability of tool positioning in a fast-moving AI market, and the asymmetric risk that regulated organizations are structurally failing to measure. This fourth article steps back to name the underlying condition, the diagnosis from which those symptoms follow.

AI is not just a technological disruption. It is an organizational architecture disruption.

A Mismatch That Went Unnoticed Until Now

For most of the industrial and information age, there was a comfortable relationship between the pace of technological change and the pace of organizational adaptation. Technology evolved, but slowly enough that organizations could absorb each shift, stabilize their processes around it, and build institutional knowledge before the next wave arrived. The lag between technological possibility and organizational practice was an inconvenience, not a structural threat.

That relationship has inverted. For the first time in the history of organized work, the speed of technological evolution is structurally outpacing the speed of organizational adaptation. Rather than a temporary disruption caused by one particularly fast technology cycle, this is a change of regime: a permanent shift in the relative velocities of technological change and organizational response.

The evidence is visible in every domain. Organizations complete a tool evaluation cycle and find the market has already moved. They design a training program for a capability that AI has partially automated by the time the program launches. They build processes around a set of constraints that are redefined by regulatory guidance six months later. The gap is widening rather than closing, and the organizational models built for a slower world are showing the strain.

Why Processes Were the Right Answer, and What They Assumed

Consider what processes actually are, and why they represented such a powerful organizational technology for so long. Beyond being a list of steps, a process is an encoded solution to a recurring problem, a crystallization of organizational learning into a repeatable, transferable, auditable form. When an organization documents a process, it is converting tacit knowledge into institutional memory, individual judgment into scalable behavior.

This is genuinely valuable, and it rests on three assumptions that held for most of the twentieth century and well into the twenty-first: that the problem being solved is sufficiently stable to justify encoding, that the conditions under which the process operates will remain predictable enough for the steps to stay valid, and that repeatability is itself a source of value (doing the same thing reliably, at scale, is better than improvising every time).

AI attacks all three assumptions simultaneously. It renders stable problems unstable by continuously shifting what is technically possible and economically rational. It introduces permanent uncertainty into operating conditions. And it transforms repeatability from an asset into a fragility: the ability to execute the same steps reliably becomes a liability when those steps are precisely what needs to change.

Processes are not so much wrong as misaligned, optimized for a world that is receding faster than most organizations have recognized.

Processes Are Becoming Legacy Systems

In technology, we use the term ’legacy system’ to describe something specific: not a system that is broken, but a system that was built for a set of constraints that no longer match the environment in which it now operates. A legacy system can be technically functional, even well-maintained. What makes it legacy is the growing gap between its underlying assumptions and current reality.

This is precisely the condition of process-driven organizational architectures in the face of AI-driven technological turbulence. The processes themselves may be executing correctly. The people following them may be doing so diligently. The problem is architectural. The way the organization governs work (through predefined steps, stable roles, documented procedures, sequential approvals) was designed for a world of slower change, and that world is no longer the operating environment.

The recognition is uncomfortable because process-driven governance is deeply embedded in how organizations understand accountability, compliance, and quality. Audits verify process adherence. Performance frameworks reward process execution. Regulatory frameworks often mandate specific processes. Saying that processes are becoming legacy systems does not mean that accountability, compliance, and quality no longer matter. It means that the mechanism through which they have been achieved is reaching the limits of its fitness for purpose.

Principle-Driven Systems as the Architectural Response

If processes assume stability, principles assume change. This is the essential distinction, and it points toward a different architecture for organizational governance.

A principle defines what must be true: the quality threshold that cannot be compromised, the constraint that is non-negotiable, the outcome that must be achieved. It does not prescribe how. It governs by results rather than by steps. Teams operating under a principle-driven system follow guardrails and pursue defined objectives, but they adapt their methods continuously as the environment evolves, while the principle itself stays stable.

This is not a novel idea in theory. Agile movements, outcome-based management, and mission-command doctrine in military organizations have all pointed in this direction. What is new is the urgency and the mechanism. The urgency comes from the acceleration of technological change. The mechanism comes from agentic AI architectures: systems that are natively coupled to objectives and constraints rather than to specific tools or procedures, and that can adapt to changing operating conditions without requiring the governance layer to be redesigned every time the tooling shifts.

Agentic architectures are not simply a more efficient way to execute processes. They are an architectural alternative to them, embedding adaptability at the level of the system itself rather than relying on human judgment to compensate for procedural rigidity.

Consider a concrete illustration. A logistics team operating under a process-driven model has a documented approval workflow for any change to its supply chain routing tools, a cycle that takes weeks and requires sign-off from architecture, security, and procurement. When a new AI-powered routing tool becomes available, the process triggers in full, regardless of how minor the change. Under a principle-driven model, the governing constraint is different: “all routing decisions must maintain delivery reliability above X%, cost variance below Y%, and full auditability.” An agent operating under those principles can evaluate and adopt a new routing tool autonomously, because what is being governed is the outcome rather than the mechanism. The principle holds, and the approval cycle dissolves.

The shift from process-driven to principle-driven governance is therefore primarily an architectural decision rather than a technology one. It concerns where stability should live in an organization. In a world of permanent technological turbulence, the answer is that stability must live in principles and outcomes, not in procedures and tools.

Organizations Must Be Engineered for Technological Turbulence

None of this argues for improvisation, or for abandoning the rigor that processes have historically provided. The argument is that rigor itself must be rearchitected. The question to settle is what to govern and at what level of abstraction, not whether to govern.

The organizations that handle the coming decade well will not be the ones that execute their current processes better, but the ones that have made a deliberate architectural choice: to invest in stable, well-defined principles rather than brittle, step-by-step procedures; to build agentic systems capable of working with continuously shifting tooling; and to develop adaptive governance frameworks that measure outcomes rather than adherence.

Engineering an organization for technological turbulence means building a particular kind of resilience. Bouncing back from disruption is one form of it; being structurally designed to absorb continuous change without losing coherence is another. The distinction matters, because the first is a recovery capability and the second is an architectural property.

The organizations that possess this property will not be immune to the challenges of an AI-accelerated environment. But they will face those challenges with a structural advantage: the ability to adapt their methods while holding their principles constant, and to absorb technological change as a normal operating condition rather than as an exceptional disruption.

Gabriel Mailhot
Software Architect and Software Engineering Quality Advisor
Series: Organizational Engineering for the AI Era

In a previous article, I argued that the People-Process-Tools model needs to be rebalanced toward platform engineering in contexts of high resource variability. This article extends that argument one step further: in a world where AI is redefining the available tools every few months, even the ‘Tools’ layer of that platform deserves to be rethought.

The Paradox of Tool Positioning

In large organizations, the process of selecting, evaluating, approving, and deploying a new tool is rarely a lightweight exercise. Between security reviews, architecture assessments, procurement cycles, integration planning, and training programs, the full lifecycle from initial evaluation to production readiness commonly spans six to eighteen months. For highly regulated environments, that timeline can extend further.

This investment was once rational. When technology moved slowly, the cost of a rigorous positioning process was amortized over years of standardized usage. The approved tool became part of the organizational fabric, embedded in workflows, documentation, onboarding paths, and institutional memory.

That calculus has changed. The current cycle of AI-driven tool evolution is measured not in years but in months, sometimes weeks. By the time an organization completes its positioning process for a given tool, the competitive situation may have shifted, the tool itself may have been substantially transformed, and a new generation of alternatives may have emerged. The organization ends up optimized for a world that no longer exists.

Why This Was Rational, and Why It No Longer Is

It would be unfair to dismiss tool positioning processes as bureaucratic excess. They exist for legitimate reasons: security and compliance requirements, interoperability constraints, support and vendor stability considerations, and the very real cost of fragmented tooling across teams. These concerns have not disappeared.

What has changed is the nature of the coupling between an organization and its tools. In a slower-moving environment, investing deeply in a specific tool (building expertise, integrations, and institutional knowledge around it) generated durable returns. That depth was an asset.

In today’s environment, that same depth increasingly functions as a liability. Deep coupling to a specific tool creates switching costs that slow adaptation. Institutional knowledge built around a particular interface or workflow becomes obsolete as AI transforms the tools themselves. The more thoroughly an organization has embedded a tool into its operating fabric, the more painful the inevitable transition becomes.

The issue lies less in the tools themselves than in the strategic weight organizations assign to tool selection. When tool positioning is treated as a long-horizon strategic commitment, the organization assumes a stability that the environment no longer provides.

The Pivot: From Tools to Principles

If tools are no longer a stable strategic variable, what is? I would argue for principles: the articulation of expected behaviors, quality standards, operational constraints, and governance guardrails that define how work should be conducted, independently of which tool happens to be executing it.

This shift goes deeper than it may initially appear. Principle-driven governance asks organizations to invest in defining what good looks like (what outcomes are expected, what boundaries must be respected, what quality thresholds must be met) rather than prescribing the specific mechanism through which those outcomes are achieved. The principle is stable; the tooling that enacts it is fluid.

Consider the difference between these two governance postures:

Tool-centric: “All code reviews must be conducted using platform X, following workflow Y.”

Principle-centric: “All code changes must demonstrate coverage above threshold T, pass static analysis at severity level S, and include a human review sign-off before merging.”

The second posture is tool-agnostic. It can be enacted by a human reviewer, a CI/CD pipeline, an AI-assisted review agent, or any combination thereof. As tooling evolves, the principle remains intact. Only the mechanism of enactment changes, and that change becomes an optimization rather than a disruption.

Agentic Architectures as a Tooling Abstraction Layer

This is precisely where agentic AI architectures become strategically relevant, as a structural response to tooling volatility and not merely as a productivity lever.

A well-designed agent is coupled to an objective, a set of constraints, and a definition of acceptable outputs, rather than to a specific tool. Given those parameters, the agent selects, orchestrates, and adapts its tooling dynamically. The tool becomes an implementation detail, visible to the agent but invisible to the governance layer.

In a multi-agent architecture, this abstraction scales. Individual agents can specialize (one handling code analysis, another managing test generation, another monitoring technical debt), each operating against the same set of organizational principles while independently managing their tooling layer. As new, more capable tools emerge, the agents can incorporate them without requiring a governance review of the tool itself. What gets reviewed is whether the agent’s outputs continue to satisfy the principles.

This represents a meaningful inversion of the traditional governance model. Rather than controlling quality by controlling tools, the organization controls quality by controlling outcomes. The locus of governance shifts from inputs to outputs, from “what tool are you using” to “are the results meeting the standard.”

The implications for large organizations are significant. Tool evaluation cycles shorten dramatically. Shadow tooling, a persistent governance challenge in organizations with slow approval processes, largely dissolves as a concern: if the agent is producing compliant outputs, the underlying tooling becomes an operational detail rather than a governance risk.

Limits and Conditions for Success

This architecture relocates governance complexity rather than eliminating it. Several challenges deserve honest acknowledgment.

Defining good principles is harder than it appears. Principle-driven governance requires organizations to articulate, with precision, what they actually care about. Vague principles produce inconsistent agent behavior. Overly rigid principles reproduce the brittleness of tool-centric governance in a different form. Getting the level of abstraction right requires significant organizational investment and ongoing calibration.

Output evaluation at scale is a partially unsolved problem. For well-defined, measurable outcomes such as test coverage, static analysis scores, or deployment frequency, this is tractable. For more qualitative dimensions like architectural coherence, domain appropriateness, and long-term maintainability, it remains genuinely difficult.

Agent maturity remains a limiting factor. Current agentic systems are capable and improving rapidly, but they are not infallible. Human oversight, particularly for high-stakes decisions, cannot be fully removed from the loop, at least not yet.

Regulated environments require special treatment. In domains where regulatory compliance is tied to specific tools, processes, or audit trails, the fluidity of agentic tooling introduces compliance risks that cannot be resolved through principle-definition alone.

A Governance Model Built for the Speed of Change

The organizations best equipped for the current period of technological acceleration are not necessarily those with the best tools. They are those with the clearest principles, along with the architectural flexibility to let those principles be enacted by whatever mechanisms become available.

This is a direct extension of the argument made in the first article of this series. If the People-Process-Tools model needs to be rebalanced toward platform and process in a context of human variability, then the tooling layer of that platform needs to be rebalanced toward principles and agents in a context of technological volatility.

The strategic question for technology leaders has shifted from “which tools should we standardize on” to “which principles are non-negotiable, and how do we build systems, human and agentic, that reliably enact them, regardless of what the tooling landscape looks like six months from now.”

Gabriel Mailhot
Software Architect and Software Engineering Quality Advisor
Series: Organizational Engineering for the AI Era