An Operational Distinction

I have spent a significant part of my career working with organizations whose human foundation is unstable: high turnover, frequent reassignments, temporary teams assembled for a project and dissolved at its close. In these organizations, something specific happens to software systems over time. The knowledge of why the code is the way it is gradually disappears.

It is most visible with legacy systems. Take VB6 applications, COBOL modules, early Java codebases: the developers who built them have moved on, retired, or simply forgotten. What remains is the code itself, and a thin layer of documentation that rarely captures the decisions and constraints that shaped it. New developers inherit a system they can run but not fully understand.

I had assumed this was a problem of time, in which knowledge erodes because the people who held it leave. Then I started observing something different in teams using AI coding assistants intensively. The developers were still there, yet the knowledge was still eroding. The mechanism was disengagement rather than departure.

AI creates distance through engagement rather than through time, and that distinction changes how we should think about what developers must retain.

The Structural Advantages of AI: Memory and Computational Speed

AI has two advantages that do not depend on the quality of the model or its version: near-unlimited memory and processing speed that vastly exceeds human cognition.

Memory first. A large language model has been trained on billions of lines of code, millions of technical discussions, years of documentation. It can instantly retrieve the exact syntax of a rarely used API, the correct pattern for an obscure edge case, the exhaustive list of parameters for a given function. No developer, however experienced, can compete on this terrain.

Speed next. AI accomplishes in seconds what would take hours: generating boilerplate, converting an algorithm from one language to another, reformatting a data structure, producing variants. This is a real, measurable advantage, and it is not going away.

These two advantages are what they are: memory and performance, not judgment.

What the Developer Must Retain: Causal Understanding

The boundary between what AI does well and what it cannot do is more precise than a split between memory and reasoning in the abstract sense. It runs between pattern recognition and causal understanding.

An experienced developer who identifies a race condition in multithreaded code does more than recognize a familiar pattern. They understand why that pattern is problematic in this specific context: which timing constraint creates the risk, which property of the system makes the bug intermittent, which fix will resolve the problem without creating another.

AI can recognize the pattern. It does not possess the causal understanding that allows one to evaluate whether that recognition is relevant in this context. This understanding of the system, the domain, and the real constraints is what the developer must retain.

The concern has empirical support. A randomized controlled trial examining how AI coding assistance affects skill development found that developers using AI assistance scored 17 percentage points lower on comprehension and debugging tests than those coding manually (50% versus 67% on average). The contrast was even sharper by usage pattern: developers who delegated code generation to AI scored below 40%, while those who used AI for conceptual inquiry while coding independently scored 65% or higher. The engagement pattern mattered more than the tool itself.

This includes, as discussed in the first article of this series, the definition of expected behaviors and the design of tests. These activities call for judgment about what the system must do in the real world, not memorization.

Knowledge Erosion: An Underestimated Organizational Risk

When a developer delegates to AI not only memory and speed, but also understanding, a silent phenomenon takes hold: knowledge erosion.

The traditional form of this erosion is familiar. Developers leave, and with them goes the understanding of why the system is structured the way it is (the decisions, constraints, and trade-offs that shaped it). Call it a distance of time: the gap between the people who understood the system and those who now maintain it.

What AI introduces is structurally different. The developers are still there, but their engagement with the code has diminished. The cognitive effort of reading it, holding its structure in mind, and reasoning about its consequences no longer happens. They generate, accept, and move on, and the knowledge never forms.

This is a distance of engagement, and it is more insidious than the distance of time because it is invisible. The team is stable, the velocity metrics are green, the system runs. Yet the understanding that would allow someone to reason confidently about changing it has quietly failed to accumulate: anticipating the consequences of a modification, recognizing a subtle regression, formulating a test without looking at the code.

Teams using coding assistants intensively report that developers increasingly struggle to explain code they have recently written themselves. Understanding atrophies from lack of exercise, and a developer who does not fully understand their own system can no longer evaluate its quality. This closes the loop with the testing problem from the first article. Such developers can no longer verify that the system does what it should. They may not have lost sight of the intent, but they no longer understand the code well enough to reason about whether it delivers it.

The Optimal Division of Labor

The right question is therefore not “should we use AI?” but “for what should we use it?”

AI as a memory extension: syntax, APIs, known patterns, mechanical conversions, boilerplate. The developer formulates the request, evaluates the result, decides whether to integrate it. AI accelerates access to what they already know how to look for.

The developer as guardian of understanding: architecture, expected behaviors, edge cases, domain constraints, tests, decisions that commit the future of the system. These activities are not delegated. AI may well produce something that resembles them, but their value lies precisely in the human judgment that produces them.

This division is a stable structure rather than a temporary limitation, grounded in what causal understanding requires: lived experience of the system, knowledge of the domain, and accountability for the outcome.

Implications for Organizations and Training

This analysis has direct consequences for how organizations train their developers and evaluate the value of AI.

On training: the goal should not be to teach developers to use AI, but to ensure that AI use does not replace the development of understanding. Practices that require reading code, manual debugging, and peer review, precisely those that delivery pressure tends to eliminate, become more important.

On value assessment: the usual metrics (lines of code produced, delivery speed, test coverage rates) do not capture knowledge erosion. An organization can see its metrics improve while its capacity to master its own system degrades. The indicators that matter are harder to measure: a developer’s ability to explain why an architectural decision was made, to recognize a subtle regression without running the tests, or to reason confidently about the consequences of a change they have not yet made.

AI does not save time; it changes the nature of work by shifting value toward what the machine cannot do. Organizations that understand this invest in human understanding. Those that do not let it atrophy without even noticing.

Legacy systems are already showing us what a system looks like when its knowledge has disappeared. The question is whether we are building the legacy systems of 2035 right now. If the specifications and tests that protect expected behavior are themselves produced by developers who no longer fully understand the system, the circle closes on itself, and the erosion spreads beyond the code to the knowledge that was supposed to protect it.

The distance of engagement is a choice, until it isn’t.

Gabriel Mailhot
Software Architect and Software Engineering Quality Advisor
Series: What AI Cannot Replace in Software Development