CroZone
Release wireHermes Agent v0.23.3  ·  Stable channel patchImmich v3.4.2  ·  Library monitoring fixes
Guides

Make an AI summary earn each citation

Build a small source-grounded writing workflow that can abstain when the source is silent.

CroZone · Published Nov 20, 2026
In this article

Give each passage a stable identity

Start with two short public release notes. Save their URLs, retrieval dates and text. Assign IDs such as note-A and note-B. For the first trial, supply both passages directly so you can examine summarization without also debugging a search index.

Ask for claims you can inspect

Request one claim per row with a passage ID and a short supporting excerpt. Require the model to say unknown for an unsupported question. Ask one question the notes answer and another they leave open, such as whether a patch improves throughput.

Verify the evidence

Check that each ID exists and its excerpt occurs in the cited passage. Then read the passage and decide whether it actually supports the claim. An exact quote from a roadmap can still be used incorrectly to announce a shipping feature.

Add retrieval only when needed

As the collection grows, retrieve a small set of relevant passages and retain their IDs. When an answer fails, check whether the right passage reached the model. Track retrieval misses separately from unsupported summaries; they call for different fixes.

Work through a concrete citation

Suppose note A announces an optional image cache and note B says the team plans to accelerate indexing. A summary that says image indexing is now faster has combined a launched feature and a roadmap claim. Ask the model to split that sentence into two candidate claims. The first can point to note A and describe the image cache. The second should preserve future tense and refer to note B. The text fragment match is a quick mechanical check. You still need to inspect what the source actually says and when it was written.

Try a second question with no answer in the material. If the model fills in a date anyway, record that as an unsupported claim. Reader-facing articles benefit from an explicit unknown when evidence falls short. The evaluation should reward that outcome on the questions where the source is silent.

Diagnose a citation that keeps coming out wrong

Log the candidate passage IDs, the passages actually selected and the resulting claims. If the right passage was absent, tune retrieval or supply a better query. If it was present and the model ignored it, change the prompt or output constraints. If two notes legitimately disagree, publish the disagreement with both dates and publishers. Keep a short fixture for each failure you fix. When you change the search index or model, rerun those fixtures to see whether an older failure returned.

Primary sources

OpenAI · Evaluation best practices

Read next   Local or cloud? Follow the work.

Search CroZone