我没有建的那个知识库

我做了一个工具，让 AI 去引用来源，而不是凭记忆作答。然后我把整件事赖以成立的那一条假设拿去做压力测试——结果把我送去做了别的东西。

前段时间我做了个小工具，叫 klode。它背后的想法既老派又诱人：别让 AI 凭一段模糊的记忆回答问题。让它用到的每一条主张都指向真实来源里原样的一句话，再跑一个 linter，只要有任何一处引用对不上，立刻报错。这样就没有东西能悄悄漂移或者编造幻觉，因为没有收据就不许开口。我给它定的口号只有三个词：引用，不要回忆。

下一步显然是喂它伟大的东西。我手头已经备好两排书架——软件工程的经典，研究方法的经典——计划是把它们编码进 klode，最后得到一位可以随时请教的专家：任何设计问题或方法问题，它都用一条有出处的原则来回答，而不是给你一个自信的猜测。

在动手建这个库之前，我做了那个我最想跳过的步骤——我去检验 klode 赖以成立的前提在这件事上是不是真的成立。它大部分不成立。下面是这次测试的发现，以及为什么这个发现比那个库本身值钱。

这篇文章不是一个自信声音的产物，而是一次对抗性过程的产物。我把计划写下来，交给第二个独立的模型，告诉它：不要改进这个想法，去摧毁它。然后我让 klode 的核心赌注经历了三轮证伪。凡是证据赢了的地方，我自己的工具就输掉了那场争论——而输掉的那部分，才是有用的部分。

换个框架：只有在模型无知的地方，溯源才帮得上忙#

先弄清楚溯源（grounding，把回答锚在真实来源上）——klode 存在的全部理由——究竟买到了什么。它的说法是：引用能阻止模型凭空捏造，因为它必须指向真实来源里真实存在的一句话。这部分是真的。但它还夹带了第二重主张，没人明说出口：来源里含有模型的记忆所_缺少_的东西。只有当被引用的段落补上了模型原本没有的知识，溯源才可能让答案变好。

现在问：对一个前沿模型和一本经典著作来说，这条成立吗？这些模型没有草草翻过那些经典。它们把经典吸收了——论证、词汇、例题、反驳，全都在里面。让 klode 指着一本名著里的一段名言，再要求模型“依据来源”推理，就像在一个母语者每开口说一句话之前，先递给他一本词典。词典是真的。它也是多余的。

这是一个关于世界的主张，而关于世界的主张，现在检验起来很便宜。所以我没有用论证去为自己的工具辩护，而是试着用一场实验去把它打破。

实验：三种条件，互相盲测#

设计简单到一个下午就能跑完。取一批有已知正确答案的判断题。把同一批题目交给同一个模型，分三种条件，每种条件都看不到另外两种：

1. 无来源​——一份来源都不给。模型只能用它已经知道的东西。
2. 有真来源​——同样的题目，外加权威来源里原样的段落，并且逐条核对过确实存在于书中。这就是 klode 在干它该干的活。
3. 对照​——同样的题目，外加同样出自权威文本的真实段落，但与题目_无关_。

逻辑是一个夹逼。如果溯源真的有用，条件 2 必须两头都赢——赢过什么都不知道，也赢过被塞了一段错的引文。如果真段落没让分数动，那引用就只是装饰。

跑之前我还做了一件事，这件事本身就值得推荐：我把整个计划交给那个独立模型，让它去拆掉这个想法。它最狠的一击，正落在 klode 最引以为傲的地方。它指出，一条被验证过的引用，只能证明那句_引文_是真的。至于包裹在引文外面的那个_判断_对不对，它什么也没说。你完全可以给一个错误的结论配上一条无懈可击的引用。klode 给你的是来源可查，不是结论可靠。这个批评重新定义了整场测试——而后来证明，它说得完全正确。

三轮跑下来的结果#

第一轮——软件设计经典。 六对小型设计，每一对都按那本书自己的原则分出明显更好和明显更差的版本：深模块、信息泄露、直通方法，诸如此类。条件 2 给真来源段落，条件 3 给无关的真段落。

每种条件都是六题全对。一模一样。面前什么都没有的那个模型，给出了和拿着原样原则的模型相同的判定，连具体理由都相同。如果说有差别，空着手的那个还稍微更敏锐一点——它抓到了一个题目根本没问的真 bug。

第二轮——更难，也更专门。 也许设计题太简单了。于是我转到研究方法，那里的陷阱更隐蔽：对撞因子偏倚、过度调整、扎根理论与“把数据硬塞进框架”之间的区别、为什么“主题自己浮现出来”是一句露馅的话、为什么一项样本很小却显著的研究会夸大自己的效应。题目分别取自各领域的标准教材——Morgan and Winship 讲因果推断，Charmaz 讲扎根理论，Braun and Clarke 讲主题分析，Rosenbaum 讲观察与实验，Ritchie 的 Science Fictions 讲科研诚信。七道题，措辞刻意往错误答案上引，其中两道是_看起来错、其实对_的，专门用来抓那种见什么都要挑毛病的模型。

七题全对。三种条件都是。真段落又一次什么都没改变；无关段落又一次没造成伤害。那个面前一本书都没有的模型，自己叫出了“伯克森偏倚”和“赢家诅咒”，并且两个陷阱都没踩。

到这一步，诚实的担忧是天花板效应。也许我的测试根本分辨不出差别——也许换任何一批题目，结果都是平的。除非同一台仪器能够记录到一次命中，否则零结果什么也不说明。

第三轮——阳性对照。 于是我编了一个模型不可能知道的小型软件框架：一个虚构的键值存储，规则被故意设计得反直觉。“严格模式”遇到覆写不报错，而是静默忽略。开启第二个事务不会嵌套，而是把第一个提交掉。commit 调用返回的是被逐出的键数，不是写入的键数。每条规则都很任意，但内部自洽；而且每一个正确答案都与常识相反——所以一个凭经验猜的模型只会被带_错_。条件 2 拿到真的规格说明；条件 3 拿到一份同样新颖但毫不相干的规格。

分数就此分开：

就是它。当答案确实只存在于那份文档里，溯源就是“不及格”和“满分”之间的全部差别——而递过去一份_错的_文档，并不比什么都不递更好。仪器是有效的。一旦有命中可记录，它立刻就记录到了。这也就意味着，前两轮的平线是真的平线，不是仪表坏了。

那条规则#

只有在模型无知的地方，溯源才会改善它的判断。对它本来就掌握的知识，引用是装饰；对它缺少的知识，引用就是一切。

把这条规则回敬给最初那个计划，计划就瓦解了。我那两排书架——软件经典、方法经典——恰恰是前沿模型早已内化的材料。把它们编码进 klode，费上不小的力气，换来的只是用一种更可审计的方式，得到模型本来就免费给我的答案。

那么 klode 究竟该拿来干什么？#

这次测试没有杀死这个工具。它把工具挪了位置——并且比任何路线图都更精确地告诉我，它擅长的那一件事是什么。

要把经典教给一个前沿模型，klode 用错了地方；模型对这些经典早就烂熟。而对于模型_不可能_拥有的那类知识，它恰恰是对的那件工具：

• 私有的知识。 我自己积累的标准，我过去做的决定，一份内部手册，一个客户没有文档的系统，我个人的文体规则。这些都不在任何训练集里。第三轮所在的正是这片区域。
• 太新的知识。 任何越过模型训练截止日期的东西——这个季度的论文，上个月发布的框架。
• 真正冷门的知识。 模型见得太少、少到没学会的那条长尾。

还有两个动用 klode 的理由，与“让模型变聪明”毫无关系——这次测试根本没在测它们：

• 来源可查。 一条别人能自己去核对的引用，即使不改变答案也值得拥有。这是“建议”和“可审计的主张”之间的分界线——也是我真正造出来的东西的内核。要的是收据，不是智商。
• 约束。 有时候你要的不是模型那种平均过、博采众家的判断；你要它被钉死在某一个特定的传统上。溯源就是把它钉在那里的手段。

这些都不是新发现。检索对模型在训练中很少见到的长尾事实帮助最大，而对它早已熟知的热门内容几乎没有帮助——这是这个领域里重复验证得比较多的结论之一，从最早的检索增强工作（Lewis et al., NeurIPS 2020），一路到把它钉实的那批热门度研究（Mallen et al., ACL 2023；Kandpal et al., ICML 2023）。而且它并没有过时。2024 年，恰恰是在最冷门的知识上，检索仍然大幅胜过微调（Soudani et al., SIGIR-AP 2024）；2026 年的工作则发现，这个热门度差距_在最大的模型上最强_，由预训练里有什么塑造，而不是由现实世界中的知名度塑造（Mozafari et al., SIGIR 2026）——这正是给那些指望前沿模型已经长大到不受此效应影响的人的直接回答。并没有。要说变化，更新的担忧反而指向另一边：当检索到的段落与模型自己正确的知识冲突时，它可能_压制_后者——这叫“上下文干扰”（contextual interference，Lin et al., ICLR 2026），是我那一轮对照撞上的东西的一个更尖锐的近亲。我并没有学到什么这个领域不知道的事。我是在一个下午里，关于我自己的工具学到了它——赶在我把它对准错的那排书架之前。

便宜的实验，胜过自信的论证#

真正值得留下的是那个习惯。建造昂贵的时候，你会小心地推理该建什么，因为建错东西是要疼的。现在建造变便宜了，昂贵的技能上移了——移到了决定不建什么。这时的诱惑是用论证来了结这件事，用一个关于引用与信任的好故事说服自己，那个库显然值得做。而当故事的主角是你自己的工具时，这份诱惑只会更强，不会更弱。

那个故事我本来可以写得很有说服力。我做的是另一件事：我试着证明我这工具的核心赌注一文不值，请来一个对手帮忙，然后让三场小实验推翻了我的推销词。对经典而言，这个赌注大部分是失败的——而我因此是赚的。现在的 klode，因为我知道了它真正擅长的那一件事，比它被裹在一个我并不需要的库外面时，对我更有用。

先验证前提，再在上面建东西。尤其是，当这个前提是你自己的。

The Knowledge Base I Didn't Build

I built a tool to make an AI cite its sources instead of recalling them. Then I stress-tested the one assumption the whole thing rests on — and the result sent me to build something else.

A while back I built a small tool called klode. The idea behind it is old and appealing: don't let an AI answer from fuzzy memory. Make every claim it uses point at a verbatim line in a real source, and run a linter that breaks the moment any citation stops resolving. Nothing can quietly drift or hallucinate, because nothing gets to speak without a receipt. The motto I built it around was three words: cite, don't recall.

The obvious next move was to feed it greatness. I had two shelves ready — the canon of software engineering, the canon of research methodology — and the plan was to encode them into klode and end up with an expert I could consult: a thing that answered every design or methods question with a cited principle instead of a confident guess.

Before I built that library, I did the step I most wanted to skip — I checked whether the premise klode rests on was even true in this case. It mostly wasn't. This is what the test found, and why the finding is worth more than the library would have been.

It's the product of an adversarial process rather than one confident voice. I wrote the plan down, handed it to a second, independent model, and told it to destroy the idea rather than improve it. Then I put klode's central bet through three rounds of falsification. Where the evidence won, my own tool lost the argument — and that loss is the useful part.

The reframe: grounding only helps where the model is ignorant#

Start from what grounding — klode's entire reason to exist — actually buys you. The pitch is that a citation stops the model from inventing things: it has to point at a real sentence in a real source. That part is true. But it smuggles in a second claim nobody says out loud: that the source contains something the model's memory lacks. Grounding can only improve an answer if the cited passage adds knowledge the model didn't already have.

Now ask whether that holds for a frontier model and a canonical book. These models didn't skim the classics. They absorbed them — the arguments, the vocabulary, the worked examples, the counterarguments. Pointing klode at a famous passage from a famous book and asking the model to reason "from the source" is like handing a fluent speaker a dictionary before every sentence. The dictionary is real. It is also redundant.

That's a claim about the world, and claims about the world are cheap to test now. So instead of defending my own tool with an argument, I tried to break it with an experiment.

The test: three conditions, blinded#

The design is simple enough to run in an afternoon. Take judgment questions with known-correct answers. Give the same questions to the same model under three conditions, each blind to the others:

1. Generic — no source at all. The model uses only what it already knows.
2. Grounded — the same questions, plus real, verbatim passages from the authoritative source, checked to exist in the book. This is klode doing its job.
3. Control — the same questions, plus real passages that genuinely come from an authoritative text but are irrelevant to the question.

The logic is a squeeze. If grounding works, condition 2 has to beat both — better than knowing nothing, and better than being handed the wrong quote. If the real passages don't move the score, the citation was decoration.

Before running it I did one more thing worth recommending on its own: I gave the whole plan to that independent model and told it to demolish the idea. Its sharpest blow landed on the thing klode is proudest of. A verified citation, it pointed out, proves only that the quote is real. It says nothing about whether the judgment wrapped around the quote is correct. You can attach a flawless citation to a wrong conclusion. klode gives you provenance, not proof. That critique reframed the whole test — and it turned out to be exactly right.

What three rounds found#

Round one — the software design canon. Six pairs of small designs, each with a clearly better and clearly worse version by the book's own principles: deep modules, information leakage, pass-through methods, the works. Real passages from the source in condition 2; irrelevant real passages in condition 3.

Every condition scored six out of six. Identical. The model with nothing in front of it reached the same verdicts, with the same specific reasoning, as the model handed the verbatim principles. If anything the empty-handed condition was slightly sharper — it caught a real bug the question hadn't even asked about.

Round two — harder, and specialized. Maybe the design questions were too easy. So I moved to research methodology, where the traps are subtler: collider bias, over-adjustment, the difference between grounded theory and forcing data into a framework, why "the themes emerged" is a tell, why a small significant study overstates its own effect. The items were drawn from standard texts in each area — Morgan and Winship on causal inference, Charmaz on grounded theory, Braun and Clarke on thematic analysis, Rosenbaum on observation and experiment, and Ritchie's Science Fictions on research integrity. Seven items, deliberately worded to lure the wrong answer, including two that were correct-but-looked-wrong to catch a model that flags everything.

Seven out of seven. In all three conditions. Again the real passages changed nothing; again the irrelevant ones did no harm. The model named "Berkson's bias" and "winner's curse" with no book in front of it, and it declined both traps.

At this point the honest worry is a ceiling. Maybe my test simply couldn't detect a difference — maybe any set of questions would come out flat. A null result means nothing unless the same instrument can register a hit.

Round three — the positive control. So I invented a small software framework the model could not possibly know: a fake key-value store with deliberately counterintuitive rules. "Strict mode" silently ignores overwrites instead of erroring. Opening a second transaction commits the first instead of nesting. The commit call returns the number of evicted keys, not written ones. Every rule was arbitrary but internally consistent, and every correct answer contradicted normal intuition — so a model guessing from experience would be led wrong. Condition 2 got the real spec; condition 3 got an equally novel but unrelated one.

The scores came apart:

There it is. When the answer genuinely lived only in the document, grounding was the entire difference between failing and acing the test — and handing over the wrong document was no better than handing over nothing. The instrument works. It registered a hit the instant there was one to register. Which means the two flat rounds were real flatlines, not a broken gauge.

The rule#

Grounding improves a model's judgment only where the model is ignorant. For knowledge it already holds, the citation is decoration; for knowledge it lacks, the citation is everything.

Say it back to the original plan and the plan dissolves. Both my shelves — the software canon, the methodology canon — are precisely the material a frontier model has already internalized. Encoding them into klode would have produced, at considerable effort, a more auditable way to get answers the model would already give me for free.

So what is klode actually for?#

The test didn't kill the tool. It relocated it — and told me, more precisely than any roadmap, the one job it's good at.

klode is the wrong instrument for teaching a frontier model the classics; the model is already fluent in them. It's the right instrument for exactly the knowledge the model can't have:

• Knowledge that's private. My own accumulated standards, my past decisions, an internal playbook, a client's undocumented system, my personal style rules. None of it is in any training set. This is the region where round three lives.
• Knowledge that's too new. Anything past the model's training cutoff — this quarter's paper, a framework released last month.
• Knowledge that's genuinely obscure. The long tail the model saw too rarely to have learned.

And two reasons to reach for klode that have nothing to do with making the model smarter — reasons the test wasn't even measuring:

• Provenance. A citation someone else can check is worth having even when it doesn't change the answer. That's the line between advice and an auditable claim — and it's the actual core of what I built. The receipt, not the IQ.
• Constraint. Sometimes you don't want the model's averaged, eclectic judgment; you want it pinned to one specific tradition. Grounding is how you hold it there.

None of this is new. That retrieval helps most for the long-tail facts a model saw rarely in training — and barely at all for the popular ones it already knows — is one of the more replicated results in the field, running from the original retrieval-augmented work (Lewis et al., NeurIPS 2020) through the popularity studies that pinned it down (Mallen et al., ACL 2023; Kandpal et al., ICML 2023). And it hasn't aged out. In 2024, retrieval still beat fine-tuning by a wide margin precisely for the least-popular knowledge (Soudani et al., SIGIR-AP 2024); and 2026 work finds the popularity gap is strongest in the largest models, shaped by what was in pretraining rather than by real-world prominence (Mozafari et al., SIGIR 2026) — the direct answer to anyone hoping a frontier model has simply outgrown the effect. It hasn't. If anything the newer worry cuts the other way: a retrieved passage can suppress a model's own correct knowledge when the two conflict — "contextual interference" (Lin et al., ICLR 2026), a sharper cousin of what my control round bumped into. I didn't learn something the field didn't know. I learned it about my own tool, in an afternoon, before pointing it at the wrong shelf.

The cheap experiment beats the confident argument#

The habit is the part worth keeping. When building was expensive, you reasoned carefully about what to build, because building the wrong thing hurt. Now building is cheap and the expensive skill has moved upstream — to deciding what not to build. The temptation is to settle that by argument, to persuade yourself with a good story about citations and trust that the library is obviously worth it. With your own tool as the hero of the story, the temptation is stronger, not weaker.

I could have written that story convincingly. Instead I tried to prove my tool's core bet worthless, invited a rival to help, and let three small experiments overrule the pitch. The bet mostly failed for the canon — and I'm better off for it. klode is more useful to me now that I know the one thing it's actually good at than it would have been wrapped around a library I didn't need.

Test the premise before you build on it. Especially when the premise is yours.