I asked an agent a simple question last week: which AI benchmarking tools are independently run. The first page of results was nine sites. Seven were content farms that reprint numbers the labs published about themselves. Two were real.
An agent that summarizes those nine pages produces a fluent, well-organized, confidently worded answer in which the seven fakes outvote the two real sources. Nothing hallucinated. Every sentence traceable to a page that exists. The answer is still wrong, and it is wrong in the way that is hardest to catch, because it reads exactly like a right answer.
Last verified against the linked guides on this site. Version-specific details drift — check the vendor docs before relying on an exact flag or limit.
Why reading the summary and moving on hides the weak source
The default research loop is one prompt, one summary, one decision. It fails at the step nobody watches: the moment the agent collapses ten sources of very different quality into one paragraph of uniform confidence.
Fluency is generated. It is a property of the writing, not of the evidence. A claim sourced from a peer-reviewed paper and a claim sourced from an SEO page written to rank for "best AI model 2026" come out of the summarizer sounding identical, because the summarizer's job is to sound consistent.
So the rule is not "trust agents less". It is: never accept a claim that has been separated from its source. Keep provenance attached all the way to the decision.
By the end you will have
- A four-tier source ranking you can apply before reading anything
- The mechanism by which a weak source becomes a confident claim, stated precisely
- The Claim-Source Ledger
- A test that catches invented and laundered citations
- A rule for when a single search beats a research agent
Tier the sources before you read them
Do this first, on the URL list, before any content is fetched. It costs thirty seconds and it changes what the agent is allowed to conclude.
| Tier | What qualifies | How to treat it |
|---|---|---|
| 1. Primary | The thing itself: the repository, the spec, the filing, the dataset, the leaderboard that ran the test | Quote directly, cite exactly |
| 2. Direct account | The people who did the work: lab post, maintainer changelog, first-party docs | Usable, but interested. Note the incentive |
| 3. Independent secondary | Someone who verified or reproduced, with method shown | Good corroboration, never the only source |
| 4. Aggregation | Reprints, roundups, listicles, most of page one | Discovery only. Never cite. Follow through to tier 1 |
Most of what a general web search returns for a commercially interesting query is tier 4. That is not a defect in the search tool, it is what the incentives of the open web produce, and any agent using that tool inherits the bias.
Mechanism dive: how a weak source becomes a confident claim
Three steps, each individually reasonable, and the failure is in the composition.
Step one, retrieval inherits the ranker's bias. The agent gets whatever search returns. Search ranks for engagement and optimization, not for whether the page ran an experiment. A query with commercial intent returns a result set that is mostly optimized content, so the agent's evidence pool is skewed before it reads a word.
Step two, summarization strips provenance. The model reads ten pages and writes one answer. Inside that answer, the claim from the primary source and the claim from the roundup are rendered in the same register. There is no mechanism in the generation step that carries evidential weight forward, because the output is one voice by construction.
Step three, laundering through a chain. Aggregator A cites vendor blog B. Aggregator C cites aggregator A. Your agent finds A and C, reports two sources in agreement, and the agreement is an artifact: both trace to B, which is the interested party. Independent confirmation and repetition look identical at the summary layer.
The consequence is specific and worth stating plainly. The confidence of an AI research answer correlates with how much was written about a topic, not with how well it is known. Topics with heavy SEO pressure produce the most confident and least reliable answers, and those are exactly the topics people research before spending money.
Make claims carry their source to the decision
The fix is structural, not a better prompt. Require the pipeline to keep claim and source bound together, and to state the tier.
Instruct the agent in these terms:
- Every factual claim ships with its URL and its tier, inline, in the working notes
- A tier 4 source may be used to find a tier 1 source, and may never appear as the citation
- When two sources agree, check whether they trace to the same origin before counting it as corroboration
- When a number cannot be traced to tier 1 or tier 2, report it as unverified or drop it
- Contradictions get surfaced, not averaged
The last one matters more than it looks. Averaging is the default behavior and it destroys the most valuable signal in a research task. When sources disagree, the disagreement is usually the finding.
The worked example: the benchmark question
Back to the question I opened with. Applying the ledger changed three things.
First, tiering the URL list before reading demoted seven of nine results to discovery. That inverted the answer: the majority position on page one was held entirely by sources that never ran a test.
Second, following tier 4 pages through to their origins produced the real list. The sites that execute evaluations describe their harness, publish run artifacts, and occasionally disagree with the vendors. Epoch AI publishes its Inspect run logs, which is a tier 1 artifact. That is a different kind of object from a table of numbers with a date stamp on it.
Third, and this is the part that only a ledger catches: I had a specific figure for a coding benchmark that I wanted to use. It appeared in several places and looked well corroborated. Every one of those places was tier 4, and none linked to the official leaderboard for it. So the number went into the notes as unverified and stayed out of the final answer. It might well be correct. It was not established, and the whole point of the discipline is that those are different states.
Magnet: Claim-Source Ledger
One row per claim. Keep it in the working notes, not just in the agent's head, so the audit is possible after the fact.
| # | Claim | Source URL | Tier | Origin (if not primary) | Status |
|---|-------|-----------|------|-------------------------|--------|
| 1 | Index composites 9 evals | artificialanalysis.ai | 1 | - | verified |
| 2 | Runs evals in-house, publishes logs | epoch.ai/benchmarks | 1 | - | verified |
| 3 | Model X scores 96 on bench Y | somelisticle.com | 4 | untraced | UNVERIFIED - do not use |
| 4 | Domain moved to new host | 301 observed on fetch | 1 | - | verified |Run this as the last step before writing anything:
For every row in the ledger:
1. Does the URL resolve, and does the claim appear on that page? (catches invented citations)
2. Is the tier honest, or is a roundup labelled as primary?
3. If two rows agree, do they trace to different origins? (catches laundering)
4. Anything still tier 3 or 4 with no tier 1 or 2 behind it: mark UNVERIFIED.
Report the count of rows dropped.You should see: a non-zero drop count. A research pass that verifies every claim it gathered did not search hard enough, or the checker is rubber-stamping. On a contested commercial topic, expect a large share of the initial claim set to be untraceable to a primary source. Those rows are the ones that would have shipped as confident sentences.
Failure modes
| Smell | Result | Repair |
|---|---|---|
| Summary read without the source list | Tier 4 consensus presented as fact | Ledger before prose, always |
| "Multiple sources confirm" | Often one origin repeated | Trace to origin before counting agreement |
| Citations never opened | Real-looking URLs that do not contain the claim | Resolve and grep every cited page |
| Contradictions averaged into a middle number | The most useful signal is deleted | Surface disagreement as a finding |
| One broad query, one pass | Retrieval bias goes unnoticed | Re-query from a different angle, compare pools |
| Research reused six months later | Facts decayed, ledger did not | Date every row, re-verify time-sensitive claims |
| Agent asked to be comprehensive | Volume of citations rises, quality falls | Ask for the strongest three sources, not all of them |
When not to use a research agent
- You know the primary source. If the answer is in a repository, a spec, or a file you already have, open it. An agent adds a summarization step and its associated loss for no gain.
- The question is contested and commercially valuable. Model rankings, tool comparisons, supplement claims, anything with affiliate pressure. Search results here are adversarial by construction. Go to primaries directly.
- The answer must be exact. Version numbers, API parameters, legal thresholds, dosages. Read the document.
- The topic is very recent. Under a few days old, the corpus is thin and mostly reposts of a single announcement. One good primary beats twenty derivatives.
- You cannot evaluate the output. If you have no way to tell a right answer from a plausible one, more research does not help. Find someone who can, or narrow the question until you can check it.
path
Make research a system, not a prompt
Agent OS Setup installs source policy, verification gates, and memory so findings stay traceable across sessions instead of dying in a chat log.
An AI research answer is only as good as its weakest uncited source, and the summary is written specifically to hide which one that is.
Your next action: take the last research answer you acted on, rebuild it as a Claim-Source Ledger, and open every URL. Count the rows that turn out to be tier 4 with nothing behind them. That count is the error rate you have been shipping.
Related: AI Model Benchmarks That Matter applies this method to model selection, and Context Engineering Is the Only Engineering covers what happens to findings after you verify them.
