Why RAG retrieval quality — not the model — is why your bot hallucinates your refund policy

A support bot confidently tells a customer they can return an item after 90 days. Your policy is 30. The instinct is to blame the model for “hallucinating.” Nine times out of ten, the model did its job — it was handed the wrong context. The failure is in retrieval.

How RAG actually fails

• Bad chunking. Your policy got split across two chunks, and the agent retrieved the half that mentions “90 days” (a different product line) without the qualifier.
• Weak embeddings / retrieval. The most relevant passage never made it into the top results, so the model reasoned over near-misses.
• Stale content. The knowledge base still contains last year’s policy, and nothing tells the agent which version wins.
• No grounding constraint. The agent is allowed to answer even when retrieval returned nothing useful, so it fills the gap with plausible fiction.
• Conflicting sources. Two documents disagree, and the agent has no rule for which is authoritative.

Fixing each one

• Chunk on meaning, not length. Keep a policy and its conditions together; preserve headings and context.
• Evaluate retrieval separately from generation. Measure whether the right passage is being retrieved at all, before you judge the answer.
• Add freshness and authority metadata. Let the system prefer the current, canonical version.
• Constrain the agent to grounded answers. If retrieval is empty or low-confidence, the agent should say so and escalate — not improvise.
• Resolve conflicts explicitly. Give the system a source-of-truth hierarchy.

Diagnosing which of these is hurting your agent — and by how much — is a core part of the Deflection Audit. Retrieval problems are among the most common and the most fixable causes of a leaking support bot.