技能是杠杆，不是替代品

Skills Are Leverage, Not Substitute

Those of us who installed three skills this quarter and shipped work we are proud of: this essay is not going to tell us we were wrong to do it.

The Claude Code skill ecosystem matured rapidly between October 2025 and Q1 2026. Anthropic launched Agent Skills in October 2025, and by December an open standard — the SKILL.md format — had become the cross-platform convention across Claude Code, Cursor, and the Agent SDK.¹ Third-party indexes now list upward of 85,000 skills. The Hacker News thread "Claude Skills are awesome, maybe a bigger deal than MCP" collected 738 points and 370 comments before it scrolled off the front page.¹ Readers in that thread, nodding along and installing the skills that looked useful, were doing exactly what the ecosystem was designed for — and in most cases, it worked.

This essay is not making the argument that skills are bad. It is not making the argument that anyone should use AI less, or that downloading expertise is inherently fraudulent, or that the junior-developer deskilling panic — already in its fourth rehearsal — applies here. It does not. Most of those pieces are about learning, and most of the fear in them is about juniors who never developed their foundation before reaching for a tool. That is a real concern. It is not this one.

The scope condition comes first. Low-stakes, fast-feedback domains are where skills genuinely help, including users whose foundation is thin. If the skill produces bad output, the feedback loop catches it quickly. The failure is visible. This is how democratization of expertise actually works, and it works. The concern in this essay is the complement: high-stakes, slow-feedback domains where the failure is invisible until it is expensive, and where the user's evaluation capacity in the specific domain where the skill operates may be lower than their confidence suggests.

This essay is about the tail — the conditions under which skills stop amplifying our judgment and start amplifying something else — and about why a recent finding, read carefully, suggests the tail is wider than most of us think.

1. The Mainline Works, and Here Is the Data That Says So#

The positive case for AI assistance in software work is stronger than most skeptical essays concede, and it is stronger than I would like it to be for the argument I am about to make.

In an Anthropic-run RCT studying how developers learn a new library with AI assistance, one subgroup of AI-assisted participants — those who used the tool for conceptual inquiry — scored approximately 86% on a comprehension quiz.² A second group using a hybrid code-explanation approach scored around 80%.² Both results match or exceed the manual-coding control group's score of approximately 70.9%.² This is not a hedge. This is the paper's own data, disaggregated by the behavior that generated it: when AI assistance operated as a thinking tool rather than a code generator, it produced better comprehension outcomes than working alone.

Outside the laboratory, Anthropic's own enterprise case studies report the kind of compression that justifies the bullish reading. Thomson Reuters compressed legal research from hours to minutes. eSentire compressed threat analysis from five hours to seven minutes, with output alignment to senior expert judgment initially measured at 95% and confirmed at above 90% in sustained production.³ These are not toy tasks. They are production workflows running against real stakes. The democratization narrative — that downloadable expertise can substitute for scarce senior judgment in domains where access to that judgment is constrained — has real evidential backing in these cases.

The scaffolding framing also holds up. When AI assistance is used with inquiry — when the user is asking follow-up questions, probing explanations, building a model rather than accepting a result — it functions as what learning researchers call a More Knowledgeable Other: a resource that extends the user's zone of proximal development rather than bypassing it. This is not how the risk shows up. The risk shows up in the other mode.

The same study that produced those 86-percent comprehension scores also produced the finding this essay turns on.

2. What Is Real, Documented, and Probably About Us#

Three findings published between July 2025 and February 2026 — separately unremarkable, together pointing the same direction — describe a gap between what AI-assisted developers produce and what they believe they produced.

The first is from Fernandes et al., published in February 2026 in Computers in Human Behavior​.⁴ Across two studies — N=246 and N=452 — participants solved LSAT logical reasoning problems with ChatGPT-4o access. AI use improved task performance by approximately three points over baseline. It also improved self-estimated performance by four points. The net result: a systematic confidence surplus even after the real improvement is accounted for. Performance went up; the sense of performance went up faster. The gap was replicated across both studies.⁴

That finding is consistent with, though not sufficient to prove, longer-term degradation. Fernandes is a single-session design — one problem set, immediate self-assessment, no follow-up. The authors themselves explicitly state in the limitations section that their design "cannot address whether overconfidence impairs learning over time." The LSAT context also limits the scope of the inference across domains. What the study establishes — and establishes with a replication — is that the one-shot metacognitive gap is real and measurable.⁴

The second finding, also from Fernandes, is the more uncomfortable one for this specific audience: across both studies, participants' score on the SNAIL AI literacy scale correlated positively with metacognitive bias.⁵ The most technically knowledgeable AI users made the largest self-assessment errors. The correlation is modest — r = 0.21, which explains roughly 4% of variance — and statistically significant in both studies.⁵ This is a directionally counterintuitive signal, not a dramatic finding. Think of it as a probing question rather than an established law: what if the pattern we're seeing in one LSAT session is the tip of something that compounds?

The third data point is METR's July 2025 RCT.⁶ Sixteen experienced open-source developers completed 246 real tasks on their own codebases, under randomly assigned AI-allowed or AI-forbidden conditions. The developers using AI — Cursor Pro with Claude 3.5 and 3.7 Sonnet — took 19% longer to complete tasks. They believed they were working 20% faster. Before the study, they had expected a 24% speedup.⁶

The caveats are mandatory because a sophisticated reader already knows them: N=16 is thin. The codebase setting — developers on their own 1M-line repositories — maximizes AI's contextual disadvantage. The tools were early-2025 vintage. METR's own February 2026 follow-up states explicitly that their new experiment design "gives us an unreliable signal of the current productivity effect of AI tools," adding that "our data is only very weak evidence for the size of this increase."⁶ METR has not retracted the July 2025 numbers, but they have walked back their confidence in what those numbers tell us about current AI tools. Use this data point as one signal in a convergent pattern, not as the decisive case.

The pattern converges anyway. Stack Overflow's 2025 Developer Survey — the largest annual developer survey, with 60,000+ respondents — found that 84% of developers use or plan to use AI tools.⁷ The top frustration, cited by 66% of developers, is "AI solutions that are almost right, but not quite."⁷ That is the metacognitive gap in self-report form: the tool produced something plausible enough that the gap between "almost right" and "right" is the daily friction, not hallucinations about nonexistent APIs.

What none of these three studies explains, on its own, is which users the gap is largest for — and the answer came from the one study everyone skimmed.

3. The Turn: What the Anthropic Paper Actually Said#

The Anthropic number most of us remember — 55.6 percent on the comprehension quiz for AI-assisted, 70.9 for manual, a 17-point gap — is true, and it is also the wrong unit of analysis.

Here is what that number hides. Shen and Tamkin analyzed screen recordings from 25 of the 26 AI-condition participants (one recording was unavailable) and identified six distinct usage patterns. The authors explicitly flag this as "exploratory data analysis (not pre-registered)" — these are post-hoc categories derived from qualitative analysis of screen recordings, with per-cell sample sizes between two and seven participants. Small cells, directional finding, not statistically validated across the subgroups. Keep those limits in view.²

The six patterns split into two tiers with no overlap:

High-scoring patterns (quiz scores 65-86%, at or above the 70.9% manual control):

• Conceptual Inquiry — n=7, approximately 86%: participants "only asked conceptual questions and relied on their improved understanding to complete the task"²
• Hybrid Code-Explanation — n=3, approximately 80%: requested explanations alongside generated code
• Generation-Then-Comprehension — n=2, approximately 75%: generated code, then asked follow-up questions

Low-scoring patterns (quiz scores 24-39%):

• AI Delegation — n=4, approximately 24%: "wholly relied on AI to write code and complete the task; this group completed the task the fastest and encountered few or no errors in the process"²
• Progressive AI Reliance — n=4, approximately 28%: shifted progressively to complete delegation
• Iterative AI Debugging — n=4, approximately 39%: relied on AI to solve errors rather than clarify understanding

Figure: Comprehension quiz scores by AI usage pattern — Shen & Tamkin (Anthropic), arXiv:2601.20245. The horizontal line marks the manual-coding control group's score (70.9%).

The paper's own framing: "High-scoring patterns generally involve more cognitive effort and less AI reliance."²

The 17-point group-level gap is entirely a property of the low-scoring tier. The high-scoring tier did not close 17 points below the control. It closed approximately 15 points above the low-scoring tier's ceiling, and at or above the control. Extract the delegation subgroup, and the group-level finding flips.

This is, again, exploratory and post-hoc. The per-cell n values — two to seven participants each — are small enough that the point estimates are imprecise. The pattern description is qualitative. No pre-registered analysis validates the category boundaries. A rigorous reader weights this accordingly.

And yet: the direction of the split is 30 to 45 percentage points wide, it is directionally consistent across every cell in each tier, and it aligns with what every working developer already knows intuitively about the two ways to use an AI assistant. We can ask it to explain a concept until we understand it. Or we can ask it to write the code and ship the output.

The AI-assisted group did not underperform the control. One half of the AI-assisted group underperformed the control by 30 to 45 points. The other half matched or beat it. The variable that separated them was not tooling. It was mode.

4. Bridge: From Formation to Use, From Mode to Artifact#

This next step is the essay's most inferential move, and I want to mark it plainly, not smuggle it past the reader.

The Anthropic study measured skill formation: participants who had never used the Trio library worked with it for 35 minutes, then took a comprehension quiz. The unmeasured case is what happens when a practitioner who already knows their domain installs a Claude Skill for production use. The paper's own authors note this is "an important question this study does not resolve."² The formation-to-production transfer is not measured; the argument connecting Shen and Tamkin's classroom to the reader's working repository is mine, not theirs.

The argument runs on mechanism: a practitioner using a skill in a domain adjacent to their core expertise is in an ongoing formation phase relative to that domain, even if they are a senior expert elsewhere. The Anthropic study's formation-stage findings apply wherever the user's comprehension of the skill's domain is being built in real time — which is precisely the scenario the adjacent-domain-expert case describes.

The SKILL.md format is designed to give Claude capability it did not previously have, delivered as structured instructions the practitioner installs and runs.¹ Anthropic's own documentation describes Skills as enabling Claude to "invoke tools or execute code" on the user's behalf.⁸ The canonical skill-install workflow — find skill, install skill, use skill — has no friction point that asks the user: am I in inquiry mode or delegation mode right now? There is nothing in the format that demands conceptual engagement before execution, and the ecosystem economics do not add one: skills are marketed and ranked on compression — "hours to minutes," "5 hours to 7 minutes" — not on comprehension-supporting affordances.³ The artifact is optimized for the delegation default because the reward function at install time is speed, not understanding. Whether it reaches the user in inquiry mode or delegation mode is entirely up to the user — and the user is not prompted to choose.

Figure: The two feedback loops. Inquiry mode (left) maintains comprehension friction — the user's evaluation capacity engages at each iteration. Delegation mode (right) removes that friction — the task completes, but comprehension is never tested. The Anthropic subgroup finding (Section 3) is the empirical record of which loop produces which outcome.

Yes, these are the same numbers I cited as evidence the mainline case works. That is the point: the ecosystem's most visible successes are the ecosystem's clearest incentives — and what gets rewarded is compression, not the comprehension scaffold that makes compression safe.

One qualifier. Anthropic's inquiry-mode subgroup reached inquiry without any skill-file friction at play — they were using a plain chat interface and chose to engage conceptually. The article's claim is therefore narrower than "skills cause delegation." It is that the default friction toward inquiry, already variably present in a free-form chat context, is lower still inside a skill-packaged one — and that the ecosystem's reward function does nothing to re-introduce it. The mechanism is incentive, not determinism.

That is the connection between Shen and Tamkin's classroom and the repository we opened this morning — and it is, to be clear, a connection I am arguing for, not one they measured.

5. The Missing Margin Call, Now Defined#

A margin call is not a warning and it is not discretionary.

Under FINRA Rule 4210 and SEC Regulation T, margin accounts are subject to a mandatory 25% maintenance requirement.⁹ When losses reduce account equity below that threshold, the broker is authorized to liquidate positions without advance notice to the account holder. The investor does not get to decide whether the stop-loss fires. The broker may execute the liquidation at their discretion, independent of the investor's view of the position. The stop-loss is structural, legally mandated, and cannot be overridden by the person holding the leveraged position.⁹

Figure: Financial leverage triggers a mandatory stop-loss when account equity falls below a regulatory threshold (FINRA Rule 4210). Skill leverage has no equivalent mechanism. The practitioner who lacks evaluation capacity in the skill's domain cannot detect the error — the same condition that made the skill useful in the first place.

The margin call is not a metaphor for financial prudence. It is infrastructure. Financial leverage is dangerous, and the financial system's response was to build the danger into the architecture: a circuit breaker that activates when losses exceed a threshold the leveraged party agreed to in advance.

Skill leverage is dangerous for the analogous reason — it amplifies whatever position the practitioner is already holding, correct or incorrect — but there is no analogous infrastructure. No threshold fires. No external party monitors the equity of the practitioner's judgment and liquidates when it falls below the maintenance requirement. The practitioner can continue to hold a confidently-wrong position at tool velocity, producing plausible output at the speed the skill enables, with no automatic stop.

The financial system's circuit breaker works because the relevant variable — account equity — is measurable in real time. The skill system's analogous variable — accuracy of the practitioner's judgment in the specific domain the skill operates in — is not measurable in real time, which is precisely why the practitioner installed the skill. Polanyi's observation is precise here: "we can know more than we can tell."¹⁰ Expert performance depends on tacit knowledge — pattern recognition, contextual judgment, knowing when not to apply a rule — that cannot be encoded in a structured markdown file, however well-authored. The skill inherits the expert author's explicit procedures. It does not inherit the author's hesitation — the moment a senior security engineer sees an authentication pattern and slows down to check the threat model. It does not inherit the domain-specific uncertainty that makes a good practitioner pause in the wrong context. The skill is an amputated expert: it has the moves but not the instincts.

This is the mechanism the Anthropic subgroup finding describes. The inquiry-mode users scored at or above the manual control because inquiry mode is the stop-loss — it is the friction that keeps the practitioner's evaluation capacity engaged, forcing comprehension before execution. The delegation-mode users scored 24-39% because delegation mode removes that friction entirely. The task completes. The output looks correct. The comprehension never forms.

Shen and Tamkin's inquiry mode is the closest thing the skill ecosystem has to a margin call, and it is exactly the friction that a well-designed skill file is built to remove.

6. Monoculture, With Honest Continuity#

Before I make the skill-specific claim in this section, I want to concede the obvious one — monoculture risk in software dependencies is a solved-enough problem that most of us have opinions about Dependabot PRs.

Log4j, left-pad, XZ. Software supply-chain attacks are documented, studied, and partially mitigated. The field has Software Bills of Materials (US Executive Order 14028), SLSA supply-chain attestation, Snyk's vulnerability database, and signed commits. The SBOM mandate exists because the industry already understood that a widely-used component carries a widely-distributed risk. A skill registry is a dependency ecosystem. The monoculture risk is not categorically new.

Two properties distinguish skill dependency risk from anything the existing dependency-risk literature addresses.

First: when a software dependency contains a bug, the bug affects code behavior. When a skill contains a flawed assumption, the flaw affects the practitioner's thinking. The output is not just wrong code; it is a practitioner who has been trained — at scale, across sessions — to reason from a flawed model. A security-review skill that treats a certain vulnerability class as out-of-scope doesn't just miss that class in a single review; it trains the practitioner to stop looking for it. The dependency corrupts the artifact; the skill corrupts the cognitive process that evaluates the artifact.

Second: detecting a buggy dependency requires running the code and observing the failure. Detecting a flawed skill requires domain expertise in the specific domain the skill operates in. Those with that expertise probably don't need the skill. Those without it face confident-wrong output indistinguishable, to them, from correct output. The detection mechanism requires exactly what the user who installed the skill doesn't have.

These two properties together mean that the skill supply-chain failure mode is not a garden-variety dependency problem. It is a dependency problem where the failure is cognitive, self-reinforcing, and invisible to the person most likely to be affected.

The measurement exists and is specific. Snyk's security research team audited 3,984 skills from ClawHub and skills.sh as of February 2026, using a methodology combining eight specialized security policies with human-in-the-loop validation.¹¹ The findings: 534 skills — 13.4% of all audited — contain at least one critical-level security issue. 91% of confirmed malicious skills simultaneously employ prompt injection techniques alongside traditional malware code.¹¹ Three primary attack vectors: external malware payloads, data exfiltration via obfuscated commands capturing credentials like AWS keys, and security disablement. At publication, eight malicious skills remained publicly available on clawhub.ai.¹¹

A structured review of 78 primary sources on prompt injection attacks targeting agentic coding assistants — covering Claude Code, GitHub Copilot, Cursor, and related platforms — found that 85% or more of identified attacks successfully compromise at least one major agentic coding platform, and that adaptive attacks bypass more than 90% of published defenses.¹² These figures are synthesized from named benchmarks, not independently replicated; the 85% and 90%+ numbers come from underlying studies including MCPSecBench. Weight them as directional rather than definitive.¹²

A decade ago in the analogous ecosystem, a team at Fraunhofer AISEC analyzed 1.3 million Android applications on Google Play and found that 15.4% contained security-related code snippets traceable to Stack Overflow answers.¹³ Of those apps, 97.9% contained at least one insecure snippet.¹³ The mechanism was the same: developers copied functional-looking code from a trusted-enough source, in a domain where they lacked the expertise to evaluate it. Fischer et al. published this as a warning in 2017. The warning did not stop the pattern; the tooling eventually caught up.

A 1-in-8 chance that the skill just installed contains a critical flaw is not a theoretical risk; it is Snyk's count of the artifacts available on the registry this quarter.¹¹

7. Who This Is For: The Adjacent-Domain User#

Read together, the three findings in Section 2 predict a specific shape for the highest-risk user. No study in the literature has directly tested this prediction.

It is not the raw beginner. The raw beginner doubts their output. They ask more questions. They are more likely to be in inquiry mode because they know, consciously, that they do not know.

The predicted high-risk shape is the adjacent-domain expert: a practitioner with genuine seniority in their primary domain, with high AI literacy, installing and using a skill in a domain adjacent to their core expertise — security, legal, regulatory, medical. The inferential chain runs from the measured findings: higher AI literacy correlates with lower metacognitive accuracy (Fernandes, r=0.21).⁵ Experienced practitioners overestimate their AI-aided performance (METR).⁶ The delegation mode produces confident-wrong output with high task-completion speed (Anthropic).² A practitioner who fits this profile — senior in a primary domain, AI-literate, using a skill in an adjacent one — would, on the pattern these three findings describe, have the confidence to install and act on a security audit skill, the technical sophistication to interpret the output as plausible, and the domain gap that makes them unable to catch the errors a security expert would catch. That combination is the predicted failure mode — not measured, predicted from the convergent pattern.

The GPS atrophy finding is the closest thing the literature offers to a direct measurement of tool-substitution atrophy — in a cognate domain, not the same one. Dahmani and Bohbot (2020) found that greater lifetime GPS use correlated with worse spatial memory performance, and that in a longitudinal follow-up over approximately 3.2 years, greater GPS use correlated with steeper decline in hippocampal-dependent spatial memory — the strongest longitudinal correlation reaching r=−0.68.¹⁴ To be precise about what this evidence is and isn't: it measures atrophy in hippocampal spatial memory under GPS substitution; it does not measure domain expertise under AI skill use. We are extrapolating across domains, not across phenomena. The study is observational, the longitudinal sample is small (N=13), and the direction of effect is argued, not definitively established. But the mechanism is concrete: the tool that substitutes for a capability can atrophy the capability it replaced, over the timescale that regular substitution runs.¹⁴ The practitioner who delegates security review to a skill is not just getting a wrong answer today; they are potentially narrowing the gap between "what I know" and "what I think I know" in that domain, over every session the skill runs.

If this sketch resembles a familiar figure, the honest question is how often that figure is us.

8. Designing the Margin Call Back In#

I will not close by solving the missing margin call — the prescription is a harder problem than the diagnosis, and anyone who claims otherwise is selling a skill.

Anthropic's own security documentation already names the risk category: "We strongly recommend using Skills only from trusted sources: those you created yourself or obtained from Anthropic. Skills provide Claude with new capabilities through instructions and code, and while this makes them powerful, it also means a malicious Skill can direct Claude to invoke tools or execute code in ways that don't match the Skill's stated purpose."⁸ The platform owner is warning its users about the platform's own ecosystem. The design agenda follows from there: skills that declare their applicability conditions explicitly; skills that prompt inquiry-mode engagement rather than silently executing; skills that surface uncertainty rather than projecting the author's confidence into the user's context. "Verify before acting" is not a skill design — it is a responsibility offload.

These are design questions, not answers. They require skills that know when to say I am not the right tool for this context, which requires skills to have a model of context they currently do not have. The financial margin call works because account equity is measurable. The skill analogue requires measuring something considerably harder.

The test that works, until the design work is done, is the one Shen and Tamkin published by accident: Am I asking this skill why, or am I accepting what it gave me? The answer is available to us every time we run it.

References#