预测曾是测试的替代品

Prediction Was a Substitute for Testing

§1 — The Deletion#

On 2026-05-04, commit f25d2d5f, I deleted 34,523 lines of code from a single repository.1

One hundred and twenty-two files. A legacy surface built around twelve tools, collapsed down to four. Not because the twelve were broken — they worked. Not because the project was failing — it had 2,727 commits spanning four and a half months of active development. The deletion happened because the twelve-tool surface was always a probe, and the probe had returned its reading.

The four tools that survived are the ones that accumulated real use. The eight that didn't aren't failures. They were questions, phrased in running code, and the answers came back: not needed. In a pre-AI build economy, shipping twelve tools to learn that four were worth keeping would have been prohibitively expensive — weeks of team-time to discover what a single commit now discards and replaces. The economy has changed.

What kind of practice looks like this? What cost model makes a 34,523-line deletion look like good work rather than recklessness?

The deletion didn't happen in chaos. The same repository holds .claude/hooks/gha-tdd-guard.mjs — a hook that blocks production code from being committed without a sibling test file. Its AGENTS.md carries the rule: "Plans >500 lines require Codex audit before Phase 1 commits." The 34,523 lines were removed inside structure. The structure is the point, and the story begins decades before any of this code existed.

§2 — Why the Old Habit Was Rational#

Frederick Winslow Taylor did not invent careful planning because he thought workers were stupid. He invented it because workers faced a genuine physical constraint: "it would be physically impossible for him to work at his machine and at a desk at the same time."2

His solution was architectural. Planning specialists worked in the planning room; execution workers worked on the factory floor. "The work of every workman is fully planned out by the management at least one day in advance, and each man receives in most cases complete written instructions, describing in detail the task which he is to accomplish."2 Taylor's rationale was cognitive specialization and physical impossibility — but the structure he built encoded a specific cost asymmetry. Planning time was the cheaper variable: it required only thought, paper, and specialist attention, none of which wore out machinery or consumed materials. Execution time was the expensive bottleneck: it required time, money, and coordination on the factory floor.

The cost-asymmetry framing is the article's inference from Taylor's structure, not Taylor's stated claim. Taylor was describing what he observed about efficient organization. But the structure he observed was responding to the economics of its moment: thought was cheap relative to action.

That asymmetry shaped a century of professional culture. The careful-planning ethos is not a personality trait or a virtue in disguise — it is a cost-optimized heuristic, calibrated to an environment where execution was the scarce and expensive resource. When building anything required time, money, and coordination, thought had to do the testing's job. The mental simulation that said "this design will not work" was saving the weeks of physical work that would have demonstrated the same thing. The old habit was rational. The question is whether the costs that justified it still apply.

§3 — The Method That Waited for Its Economics#

Thomas Edison did not invent the empirical method. He industrialized it at a moment when no adequate theory existed to replace it.

The Menlo Park laboratory ran thousands of filament tests for the incandescent bulb — materials tested systematically, results recorded, failures used to constrain the next guess. Tesla's contemporaneous critique was technically accurate: "just a little theory and calculation would have saved him 90 percent of the labour."3 What Tesla missed was the total-cost calculation when no adequate theory existed to perform the calculation from. Wikipedia's characterization of Edison's method is precise: he resorted to trial-and-error specifically "in the absence of, or lack of awareness of, adequate theories."3 When theory was available — Joule's and Ohm's laws — Edison used it. Enumeration was the residual strategy when prediction was theoretically optimal but practically unavailable.

The point is not that Edison was brilliant or that Tesla was wrong. The point is that empirical enumeration was always a valid epistemology. What gated it was cost per trial: each test required building a physical component, running a physical experiment, recording results by hand. The method was ancient. The constraint was economic.

What changed the calculus for software was not a new idea. It was the arrival of economics that the empirical method had been waiting a century for.

§4 — The Cost Collapse, in Numbers#

Three codebases, approximately five months, roughly 4,929 commits and 1,177,000 lines of code across a plugin linter, a Tauri editor, and a Tauri desktop control panel — built by one person, all greenfield, with no live users.4 At least three explicit disposal episodes: a 34,523-line deletion, a 568 MB archive discarded after two days, a 70% production LOC reduction in a single refactor.

These are the author's own repositories. That matters, and the limits of what they prove will be addressed directly in §9.

The first-pass build cost has collapsed. Not total lifecycle cost — operational costs, maintenance debt, audit expense all persist and compound across maintenance cycles. But the cost of constructing a first pass, of getting something running to learn whether the shape is right, has dropped to where it is no longer the binding constraint in the decision of whether to test or predict.

nlpm (Natural-Language Programming Manager): 1,109 commits in 50 days, 28,706 lines of code.4 Peak velocity: 92 commits in a single day. On day one, seven commits in nine minutes scaffolded the core. On day two, a flag introduced at 20:11 was renamed by 20:20 — a nine-minute decision cycle from recognition to execution, with zero coordination overhead, zero branch negotiation, zero context-switch tax. The repository has 13 GitHub Actions workflows and zero external Python dependencies. The speed sits inside structure.

claudepot​: 568 MB monolith — a marketing site, ingest pipeline, and content directories packaged together — archived on 2026-04-14 after two days of work.4 Restarted from scratch as a focused Tauri 2 application. Within 31 days, 1,093 commits and 223,303 lines of code; 1,700+ Rust tests and 400+ JavaScript tests; one commit reads "audit fix: address 39 of 40 Codex audit findings"; a subsequent refactor reduced production LOC from 817 to 253 — a 70% reduction in one pass. The 568 MB archive is not wreckage. It answered a question: this shape does not fit. That answer was the deliverable from two days of work.

The three case studies together demonstrate the logic: build, measure, keep or delete. The judgment machinery — rule files, TDD guards, cross-model audit loops — is what makes the speed directed rather than chaotic. A developer with 10+ rule files in .claude/rules/, a CLAUDE.md that names a domain model of four nouns, and a commit log containing codex audit-fix cycles is not vibe coding in the pejorative sense. The apparatus is present throughout.

§5 — The Mechanism Named#

When the cost of building a first pass falls below the cost of predicting what that pass will discover, the optimal strategy inverts.

This is not a preference or an aesthetic. It is arithmetic. If a careful analysis of whether a twelve-tool MCP surface will see real use costs three days of focused attention — architectural review, user research, prioritization sessions — and building the twelve-tool surface and measuring actual usage costs two days plus one commit that takes what the measurement showed, then prediction has become the expensive path. Reality is the cheaper classifier.

The 34,523 deleted lines are not wreckage — they are signal. The carcass was the measurement. The instrument was spent, not wasted; the thermometer does not become less useful because it has taken the temperature.

The 568 MB claudepot monolith answered "this shape does not fit." That answer is worth exactly two days of first-pass build cost to obtain — the same answer that three weeks of architectural planning might or might not have produced, with far higher prediction overhead and no guarantee of accuracy. The probe is not the product. The probe is what produces the data the product decision requires.

This is where the argument pivots. The prior sections established that the old careful-planning habit was rational under old costs. The case studies establish that costs have shifted in scope. The mechanism — the point where strategy inverts — is the cost crossover: the moment when testing becomes cheaper than predicting, and reality becomes a more reliable classifier than foresight. Chess and pharma both hit this crossover before software did.

§6 — Two Domains, Same Outcome#

In 1973, the Northwestern University Chess 4.x team abandoned heuristic-based selective search and switched to full-width brute-force search. Their discovery was cost-comparative: the time required to simply enumerate all available moves was substantially less than the time required to apply knowledge-intensive heuristics to select a few promising ones. The enumerator beat the heuristic player not by being smarter but by making search cheaper than prediction. Chess 4.x won eight of ten ACM computer chess championships from 1970 to 1979 and, as one chess historian noted, set the paradigm that modern chess programs still follow.5 When enumeration became cheap, expert heuristic selection lost its competitive advantage.

The pharmaceutical industry followed a structurally identical path in the 1990s. High-throughput screening — automated experimental testing of large compound libraries — became the confirmation layer for drug discovery as robotics and miniaturization reduced the cost per experimental assay. Computational prediction did not disappear; it evolved into a prefilter. As one peer-reviewed survey confirmed, "computer-assisted drug design, such as generation of virtual libraries... now becomes the standard procedure" — defining subsets of chemical space likely to yield hits before the more expensive physical screening confirms them.6 Virtual screening became the first pass; HTS became the verification. Prediction did not die. It changed function.

The pharma case prefigures what follows in §7: when experimental testing becomes cheap enough, prediction does not disappear from the pipeline. It relocates. It becomes the cheap initial pass that narrows the space before the more reliable classifier — the actual experiment — renders its judgment. Software planning is undergoing the same structural adjustment. The four-function decomposition is the map of that adjustment.

§7 — The Four Functions; The One That Loses Its Job#

Before asking which parts of planning survive, it helps to ask what planning was actually doing.

Building on Suchman's 1987 argument that plans function as resources for action rather than control structures, and on Mintzberg's 1994 distinction between planning-as-prediction and planning-as-coordination-device, planning-as-control-device, and planning-as-communication-device, this article identifies four separable functions:7

Filtering — eliminating paths that are impossible or very unlikely to work before spending resources on them. This is the function that answers: do we need to build version B if version A is going to fail for reasons we can anticipate?

Coordination — aligning multiple actors around a shared model before they begin acting. Teams, stakeholders, and dependencies need a common frame. Plans provide it.

Legibility — making the intended work visible to people who will evaluate, fund, or inherit it. Future-self, funders, collaborators, and managers all need to understand what is happening.

Practice by imagining — the act of planning develops domain understanding. Thinking through how a feature will work teaches the builder things about the domain that running the code would eventually teach anyway, but faster and at lower cost when the build is expensive.

When first-pass build cost collapses, only the first function — filtering — loses its job to a cheaper competitor. Reality is a better filter than foresight: it catches edge cases that planning misses, it is not subject to the planner's blind spots, and it never costs more than building the pass that reveals the answer.

Coordination, legibility, and practice by imagining all survive intact. A team still needs alignment. A project still needs to be visible to its stakeholders. A builder still benefits from working through what they are about to do. Nothing in the cost collapse of execution touches any of these three functions. The apparatus does not replace them — it replaces only the filtering function, while those three continue to operate at the rule layer rather than the outcome layer.

The article's original contribution is the cost-mechanism filter on this decomposition. Prior work (Suchman, Mintzberg) described the functions. The question of which function is specifically obsoleted when build cost falls — and which survive — is the article's analytical move. The answer is: filtering is made redundant; the rest are not.

There is a quiet corollary for the audience's concern about deleted code and lost knowledge. The knowledge is in the probe's result — what was learned by running the code — not in the code itself. The deleted nlpm flag, the archived claudepot monolith: the instrument was spent, not the knowledge. Knowledge lives in the measurement, not in the thermometer.

§8 — The Counter-Corpus#

The steelman deserves its strongest form, not its weakest.

The 2026 counter-corpus is not CodeRabbit's December 2025 report alone — though that study, analyzing 470 pull requests, found AI-co-authored code produced approximately 1.7 times more issues per PR than human-only code (10.83 vs. 6.45 per PR), with cross-site scripting vulnerabilities at 2.74 times the rate specifically for XSS.8 That study used signal-based authorship classification rather than confirmed attribution and was conducted by a vendor with commercial interest in surfacing AI code problems — both legitimate caveats, but neither explains the finding away.

The stronger 2026 entry is a large-scale empirical study by Liu et al. — 302,600 verified AI-authored commits across 6,299 GitHub repositories, tracked across five AI coding assistants. The headline finding: over 15% of commits from every tool introduce at least one issue. More consequentially, 22.7% of tracked AI-introduced issues survive to the latest version of the repository.9 This is not a vendor analysis of 470 pull requests; it is a systematic examination of 302,600 commits, and the persistence figure — nearly one in four AI-introduced issues still present at latest version — is a direct measurement of maintenance debt. The study is an arXiv preprint and has not yet been peer-reviewed, which is a genuine limitation, but the methodology (explicit Git metadata for authorship classification, rather than signal inference) is substantially more reliable than CodeRabbit's approach.

METR's July 2025 randomized controlled trial found that 16 experienced open-source developers completed tasks 19% more slowly when AI tools were permitted — on their own mature codebases with an average of five years of prior contributor experience.10 METR's own February 2026 follow-up complicates this: the same cohort returned and, in a late-2025 measurement, showed an 18% speedup — but METR explicitly disclaimed the reliability of that figure due to severe selection bias. Thirty to fifty percent of developers avoided submitting tasks they didn't want to complete without AI, making the measurement unreliable in the opposite direction.11 The original paper was not retracted; the follow-up data is not evidence of a productivity gain — METR described it as "only very weak evidence for the size of this increase." The honest read is that AI productivity is too cohort-dependent to measure cleanly at this moment, and the original 19% slowdown finding should not be treated as settled science.

This counter-corpus is not the refutation. It is the prediction.

The article's contention is that the apparatus — rule files, TDD guards, audit loops, cross-model review — is the necessary alternative locus for judgment when first-pass build cost collapses. The 2026 data is direct evidence for what happens at population scale when the apparatus is absent. The CodeRabbit population, the 6,299 repositories in Liu et al., METR's measured developers — these are samples of AI-assisted development without the systematic judgment machinery the case studies describe. The counter-corpus shows, at scale, what the apparatus exists to prevent.

The article's honest position on this: the apparatus is necessary. Whether any apparatus is sufficient to close the population-scale gap — to drive that 22.7% persistence rate to something comparable to careful human development — is an empirical question the article does not resolve. The article has one disciplined practitioner's data. That is not a controlled experiment.

§9 — The n=1 Problem and the Failure That Fits#

The survivor-bias objection deserves to be named by name, not softened into a scope disclaimer.

The three case studies share three properties that limit what they prove: they are all built by one author; that author is the writer of this article; and the judgment apparatus the author used was assembled over years before any of these case studies began. The .claude/ infrastructure, the rule files, the cross-model audit workflows — none of these materialized when nlpm started. They were in place. The case studies measure outcomes given the apparatus, not outcomes when a practitioner first encounters this approach.

Name the meta-investment: the apparatus represents years of deliberate investment in codified judgment. A developer reading this article and attempting to replicate the practice will face a bootstrap problem — the apparatus takes time to build, and building the apparatus is itself an exercise in outcome prediction (which rules will hold, which constraints will generalize). The case studies are not benchmarks for what a median developer with AI tools will produce. They are mechanism demonstrations — evidence that the cost collapse is real and that directed empiricism inside judgment machinery is structurally feasible.

Name the failure: the 568 MB claudepot monolith is presented in §4 as the abandonment-option exhibit — evidence that discarding a misfit costs two days rather than two months. That is accurate. It is also evidence that the apparatus did not prevent the failure. Two days of work went into a shape that did not fit, and the apparatus caught nothing about the shape mismatch before the two days were spent. The apparatus contained the cost; it did not eliminate the error. There is a difference. A second instance: the abandoned com.claudepot.app fork, 95 commits, abandoned April 14, 2026. The apparatus was present. The direction was wrong. The cost was low; the failure was real.

The apparatus did not prevent the failure. It contained it.

Name the falsifier: what evidence would refute the mechanism claim? A controlled comparison — a practitioner with the apparatus against the same practitioner without it, on equivalent greenfield projects — would refute the mechanism claim if the with-apparatus version produced the population-scale failure modes the counter-corpus documents. No such study exists. Until it does, the claim is a hypothesis about which conditions invert the cost calculus, not a demonstration that the apparatus is the explanatory variable.

The article advances the hypothesis. It cannot confirm it.

§10 — Planning Relocates; the Question Remains Open#

The prescription is not "plan less." It is "plan differently."

In the old economy, planning predicted outcomes: what would be built, which features would succeed, how users would respond, what edge cases would arise. Those predictions were cost-justified because the alternative — building to discover — was expensive. In the new economy, outcome prediction at that level is cheaper to bypass than to perform, in domains where the artifact can be built, machine-checked, and discarded cheaply. Surgery is explicitly not such a domain. Neither are live production systems with real users, hardware with physical manufacturing constraints, or civil infrastructure. The scope is real.

What survives is rule prediction — not prediction about what will happen in this project, but prediction about what constraints will govern all projects. Rule files, TDD guards, cross-model audit mandates: these are predictions too, but at a different cost grain. A rule is designed once and applied across many builds; an outcome prediction is made once and tested once. The cost calculus has inverted for outcome prediction; it has not inverted for rule prediction. Planning has a new object, not a new verdict.

Karpathy named the structural question in April 2026, writing that in agentic engineering "you are in charge of taste, engineering, design, and whether the system makes sense."12 That framing centers craft and oversight, which is directionally right. The article's contribution is the mechanism that explains why this structure is now economically optimal: outcome prediction has become the expensive path, and the apparatus is where careful deliberation now belongs. Karpathy reached the destination by a different route. The cost mechanism is what explains why the destination is stable.

The judgment apparatus is the new load-bearing investment. Not the sprint, not the shipping velocity, not the commit count. The apparatus is what separates directed empiricism from undirected thrashing, and what separates the case studies in this article from the population measured by Liu et al. Whether it is sufficient to close the gap at population scale is the next empirical question. The article identifies it; it does not answer it.

The builder who designed the apparatus inside which fast movement is directed — not the builder who moved fastest — is the hero of the new economy. The constraints, the apparatus, the rule: these are where prediction now lives. Not gone. Relocated.

References#

1. Author's own vmark repository: https://github.com/xiaolai/vmark. Commit f25d2d5f, 2026-05-04. Full commit message: "Hard cut of the legacy MCP surface — 122 files, ~34,500 lines deleted." Numbers extracted 2026-05-13; reproducible via git show f25d2d5f --stat. ↩

2. Taylor, F.W. The Principles of Scientific Management. Harper & Brothers, 1911. Chapter 2. Project Gutenberg EBook #6435. marxists.org/reference/subject/economics/taylor/principles/ch02.htm ↩↩2

3. "Edisonian Approach." Wikipedia. Fetched 2026-05-13. Tesla quote: Nikola Tesla, as cited in Wikipedia, confirmed verbatim. Also: "The Edisonian Method: Trial and Error," Springer Nature, link.springer.com/chapter/10.1007/978-3-030-29940-8_10. ↩↩2

4. Author's own repositories: claudepot-app (https://github.com/xiaolai/claudepot-app, 1,093 commits, first commit 2026-04-12); nlpm (https://github.com/xiaolai/nlpm-for-claude, 1,109 commits, first commit 2026-03-25); vmark (https://github.com/xiaolai/vmark, 2,727 commits). Numbers extracted 2026-05-13 and reproducible via git log. ↩↩2↩3

5. "Chess (Northwestern University)." Wikipedia. Fetched 2026-05-13; ACM championship record confirmed; brute-force vs. selective-search paradigm shift confirmed. Also: chessprogramming.org/ACM_1974. ↩

6. Baig, M.H. et al. "Combinatorial chemistry and drug discovery." PMC5645069. Journal of Biosciences, 2017. pmc.ncbi.nlm.nih.gov/articles/PMC5645069/ ↩

7. The four-function decomposition is the article's analytical contribution, refining prior work: Suchman, L. Plans and Situated Actions. Cambridge University Press, 1987 (plans as resources for action, not control structures); Mintzberg, H. The Rise and Fall of Strategic Planning. Free Press, 1994 (planning as prediction, coordination, control, and communication). ↩

8. CodeRabbit. "State of AI vs Human Code Generation Report." coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report, December 17, 2025. Independent coverage: The Register, December 17, 2025. theregister.com/2025/12/17/ai_code_bugs/ ↩

9. Liu, Y., Widyasari, R., Zhao, Y., Irsan, I.C., Chen, J., & Lo, D. "Debt Behind the AI Boom: A Large-Scale Empirical Study of AI-Generated Code in the Wild." arXiv:2603.28592. Submitted March 30, 2026; revised April 26, 2026. arxiv.org/abs/2603.28592 (arXiv preprint; not yet peer-reviewed as of 2026-05-13.) ↩

10. METR. "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity." metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/, July 10, 2025. arXiv:2507.09089 (preprint; not yet peer-reviewed as of 2026-05-13.) ↩

11. METR. "We are Changing our Developer Productivity Experiment Design." metr.org/blog/2026-02-24-uplift-update/, February 24, 2026. ↩

12. Karpathy, A. "Sequoia Ascent 2026 summary." karpathy.bearblog.dev/sequoia-ascent-2026/, April 30, 2026. Quote confirmed verbatim. ↩

Further Reading#

• Suchman, L. Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge University Press, 1987. — The foundational argument that plans function as resources for action rather than as control structures. The four-function decomposition in §7 builds on this directly.

• Mintzberg, H. The Rise and Fall of Strategic Planning. Free Press, 1994. — Distinguishes planning-as-prediction from planning-as-coordination, -control, and -communication. The article's four-function decomposition refines this earlier taxonomy and extends it with a cost-mechanism filter.

• Taylor, F.W. The Principles of Scientific Management. Harper & Brothers, 1911. Project Gutenberg EBook #6435. — Read in full to understand the planning-execution separation in its original form. The cost-asymmetry framing in §2 is the article's inference from Taylor's structure; the primary text confirms what Taylor built and why.

• Karpathy, A. "Vibe Coding." x.com/karpathy/status/1886192184808149383, February 2, 2025. — The coinage that named the practice this article analyzes. The article's argument does not depend on Karpathy's framing but does enter the current conversation through the vocabulary he established.

• Ries, E. The Lean Startup. Crown Business, 2011. — The build-measure-learn loop that partially anticipated the article's argument, but within a cost structure where "build" still meant weeks. Useful background for understanding what the article is updating and why the update is nontrivial.