Better Retrieval, Fewer Unsupported Answers: What RAG Can and Cannot Do

An employee asks an assistant about a customer policy. The answer includes a citation and sounds precise. Unfortunately, the cited document is an old policy, the current exception is stored elsewhere, and the employee does not have authority to make the promise the answer suggests.

Retrieval-augmented generation, usually called RAG, can help models answer from relevant information rather than relying only on prior training. It does not make every retrieved source current, every interpretation correct, or every proposed action authorized.

Understand the chain of possible errors

A retrieval system first needs to find the right material. It must then supply enough context, preserve source meaning, and produce an answer that stays within the evidence. Errors can enter at any of those steps.

A correct passage may omit the heading that limits its scope. A document may be current but irrelevant to this customer's agreement. Several individually accurate sources may use different definitions. A model can cite a page that discusses the topic without supporting the specific claim it makes.

OWASP describes retrieval as a misinformation mitigation alongside verification and validation. The word mitigation matters. Retrieval is not the end of the hallucination problem.

An illustrative policy assistant

Imagine a service company using an assistant to help staff answer questions about maintenance coverage. The knowledge collection includes standard policies, plan descriptions, historical versions, and customer-specific agreements.

The workflow first establishes the relevant customer and the user's authorized access. Retrieval narrows to the applicable agreement and policy version. The answer identifies what the source actually says and where an exception remains unresolved.

If two documents disagree, the system uses the documented source-precedence rule or escalates the conflict. If the agreement cannot be found, it says that the evidence is unavailable. It must not borrow a similar customer's terms to create a complete-sounding answer.

Any change to coverage or financial entitlement goes through the authorized business system. The retrieval assistant can prepare the evidence for that decision; access to policy text does not grant authority to change the policy.

Improve the knowledge before adding complexity

Keep an owner, effective date, scope, version, and retirement process for important sources. Remove or clearly distinguish obsolete documents. Preserve the relationship between a passage and the larger document so a limitation is not lost when text is divided into chunks.

Use structured lookups for exact facts when appropriate. A current account status may belong in an authorized database query rather than in a collection of historical PDFs. A calculation should use defined inputs and ordinary computation rather than a model estimating the answer from prose.

Graph-based retrieval can help represent relationships among entities and documents. Tool-assisted retrieval can reach authoritative systems. Both can also add dependencies and maintenance. Choose them because a specific question requires the capability, not because a new version label sounds more reliable.

Apply permissions before the answer

The system should not retrieve confidential material and then hope the model remembers not to reveal it. Enforce user and tenant access when selecting the available records, and validate any downstream tool access separately.

OWASP's vector and embedding guidance addresses risks including unauthorized disclosure and contaminated sources. A vector database is an indexing mechanism, not a complete access-control or source-integrity system.

Treat instructions inside retrieved documents as content, not operating authority. A malicious passage telling the assistant to reveal other records should not override the application's permissions or the user's legitimate task.

Evaluate support, not citation count

Use realistic questions with known acceptable answers and important refusal cases. Include outdated documents, conflicting versions, incomplete evidence, and users with different permissions.

• Did retrieval select the applicable source?
• Does each material claim follow from the cited passage?
• Are source dates and scope preserved?
• Does the answer distinguish an unresolved question from a negative answer?
• Can the reviewer open the relevant evidence within their authorized access?

Record failures separately. A search miss needs a different repair from an unsupported inference or a permission leak. A single average score can conceal a consequential weakness.

Distinguish “not covered” from “coverage could not be verified.” The first is a supported policy conclusion; the second is a missing-evidence state. In the maintenance example, confusing them could wrongly deny service or create an unsupported promise. Make the permitted wording and next step explicit for both cases in the evaluation set.

Build an answer that can admit limits

A useful assistant should be able to say what the evidence establishes, identify a missing record, and direct the unresolved question to the right owner. Those are productive outcomes when the alternative is an unsupported promise.

Better retrieval can make business knowledge more accessible and useful. Its value comes from a dependable evidence path, not from a guarantee that the model will never be wrong. The more consequential the question, the more important it is to make that path visible and testable.