AI Ecosystem CourseLearn with Q
Data: From Business Records To Model Inputs · Lesson 13

How Retrieval Supplies Information To AI

Retrieval supplies selected evidence at runtime. A useful answer still depends on applicability, access, completeness and correct interpretation.

Study And Save Progress

Watch The Lesson

6 min 48 sec

Read the video transcript

A customer asks our fictional assistant whether a particular order can be returned. The model's general language ability cannot establish this order's current facts or the policy that applies to it. Follow the request as it obtains authorized evidence, then uses that evidence to explain an answer. We will separate finding the right information from interpreting it correctly.

Retrieval selects relevant information from a specified collection. Retrieval-augmented generation, usually called R A G, supplies retrieved material in the context used by a generative model. Indexing a policy makes evidence available through that retrieval path; it does not by itself update the model's learned parameters. Training and request-time evidence follow different processes.

Use a supplied hypothetical rule. Version A applies to purchases before June 1 and allows standard unused products within 30 elapsed days. Version B applies from June 1 and uses 14 days. Custom-made products are excluded in both versions. Order O1301 is a standard unused product bought in May and returned 21 days later. An authorized order lookup supplies those facts.

The purchase-date rule selects A; its 30-day provision and product exception are both needed for the explanation. We are describing an expected reasoning path, with no running service result. Preparing searchable evidence involves several decisions. Extraction obtains usable policy content from its source format. Chunking divides it into retrieval units. An index organizes representations to support search.

Metadata should preserve identity, version, effective dates, product scope and access conditions. The paragraph and its exception must remain connected strongly enough for the actual question. Tiny fragments can omit qualifications; oversized units can add irrelevant material and consume the model's limited context. Suppose a retrieved fragment contains only returns within 30 days.

For a custom-made product, that fragment is insufficient because the exclusion is absent. A fluent answer can then appear well sourced while missing the controlling qualification. Test the unit design and whether the required exception is retrieved. Citation presence alone does not establish that the answer is supported.

A question about the policy in force now searches the approved current set. A question about a particular order may need an appropriately controlled historical version. Our May order uses A under the supplied rule even after B becomes current. Keep archived versions available for the permitted date-qualified path where required.

Do not blend conflicting versions without their scope or assume newest always wins. Access is another independent condition. Authenticate the user and enforce the relevant authorization before exposing order facts or policy content. A tenant identifier in a prompt is not an access-control mechanism. Microsoft's Azure AI Search documentation describes

retrieval-and-generation patterns and a security-filter example. That filter uses identity strings to trim results; it does not itself authenticate the user. The surrounding application must supply and enforce the appropriate authorization correctly on every relevant path. Updates and revocations must reach runtime use. A changed source with an old index can return stale evidence; a revoked permission with an

unchanged retrieval path can disclose content improperly. Trace source, extraction, chunk, index and request identities, including refresh and deletion behavior. No particular service or filter configuration is certified by this lesson. Morgan Stanley's June 26, 2024 release describes an earlier Assistant rollout

in September 2023 that gave financial advisors quick access to the firm's intellectual capital. The same release introduces Debrief for meeting notes; those are separate tools. The Assistant is a documented example of making an organization's information accessible through an application. The release alone does not establish whether each answer retrieved the right

authorized passage or used it correctly. Our evaluation has to inspect those 2 stages directly. Evaluate 2 stages separately. First ask whether retrieval obtained the required, applicable, authorized evidence. Then ask whether generation used it correctly, preserved qualifications and responded appropriately when evidence was missing. A correct-looking answer might come from unsupported guessing,

while a good retrieved passage might still be misinterpreted. Keep source references and expected uncertainty behavior with each test case. In the course's hypothetical 10-question pilot, 8 questions retrieve the required authorized policy evidence. 7 of those 8 receive correct evidence-supported explanations. Across all 10 questions, correctness is 7 divided by 10, or 70%.

Among successful retrievals, generation correctness is 7 divided by 8, or 87.5%. Reporting only that conditional figure hides 2 retrieval failures. Retrieval coverage is 8 out of 10, or 80%. Keep any unauthorized disclosure as a separate high-consequence failure requiring its own response, even when other answers are correct.

These are teaching counts, not measured service results. Here is the checkpoint. Why does adding retrieval fail to guarantee freedom from hallucination? Think about both the selected evidence and what happens after it reaches the model. Retrieval can return missing, stale, irrelevant or unauthorized material. The model can misinterpret evidence or invent beyond it.

The system therefore needs evidence checks, access enforcement and a defined response when the necessary support is unavailable. Naming the architecture R A G does not establish those conditions. For practice, inspect a fragment missing an exception, a historical purchase and an unauthorized request. State which failure belongs to selection or access,

and which belongs to interpretation. The worksheet's answer key keeps these distinctions explicit. Our order question becomes answerable through current facts and the applicable evidence, with actions such as issuing a refund still controlled by a separate authorized workflow. Next we will examine similarity search and why a nearby result can still be the wrong one.

Main Point

Retrieval supplies selected evidence at runtime. A useful answer still depends on applicability, access, completeness and correct interpretation.

Key Points

1. Follow O1301

The fictional rule selects A for purchases before 1 June (30 elapsed days, standard unused products) and B from 1 June (14 days). Custom-made products are excluded in both. O1301 is a standard unused May purchase returned after 21 days. Its authorized facts and A's provision plus exception support the expected explanation. These are invented rules, not real consumer rights or an executed answer test.

A current-policy question and a dated order question can require different evidence. Keep historical versions available under controlled applicability rules.

2. Prepare and select complete evidence

Extraction, chunking, indexing and metadata affect what reaches the model. A fragment saying “within 30 days” without the custom-made exclusion cannot support the corresponding product decision. Test whether the retrieval unit preserves or recovers that qualification. Bigger chunks can add noise and consume context; no universal chunk size is prescribed.

Azure AI Search's RAG overview describes retrieval supplying relevant context and identifies security and evidence challenges. Its product features and performance descriptions do not establish our system's behavior. [AZ-RAG]

3. Enforce access and verify use

Authenticate and authorize the request, enforce access across lookup/retrieval, and test revocation and refresh. Azure's security-filter pattern filters by identity strings; those strings do not authenticate the requester. A filter field or a prompt instruction alone is not proof of authorization. [AZ-FILTER]

Keep separate test records: required evidence and whether it was retrieved; expected explanation and whether it is supported; access decisions; missing-evidence response. A citation alone can point to an incomplete fragment. A correct answer unsupported by retrieved evidence can conceal guessing. The worksheet probes these paths without claiming a running implementation passes.

4. Measure the full path

Evaluate two stages separately. First ask whether retrieval obtained the required, applicable, authorized evidence. Then ask whether generation used it correctly, preserved qualifications and responded appropriately when evidence was missing. A correct-looking answer might come from unsupported guessing, while a good retrieved passage might still be misinterpreted. Keep source references and expected uncertainty behavior with each test case. In the course's hypothetical ten-question pilot, eight questions retrieve the required authorized policy evidence. Seven of those eight receive correct evidence-supported explanations. Across all ten questions, correctness is seven divided by ten, or seventy percent. Among successful retrievals, generation correctness is seven divided by eight, or eighty-seven point five percent. Reporting only that conditional figure hides two retrieval failures. Retrieval coverage is eight out of ten, or eighty percent. Keep any unauthorized disclosure as a separate high-consequence failure requiring its own response, even when other answers are correct. These are teaching counts, not measured service results.

5. Distinct transfer

Twenty questions yield required authorized evidence for 16; 12 of those 16 receive correct supported explanations. Calculate end-to-end, conditional generation and retrieval coverage. Record any unauthorized disclosure separately. Use Q02 on the worksheet; the separate key gives reasoning.

6. A dated example

Morgan Stanley's June 26, 2024 release describes an earlier Assistant rollout in September 2023 that gave financial advisors quick access to the firm's intellectual capital. The same release introduces Debrief for meeting notes; those are separate tools. The Assistant is a documented example of making an organization's information accessible through an application. The release alone does not establish whether each answer retrieved the right authorized passage or used it correctly. Our evaluation has to inspect those two stages directly.

7. Bringing the lesson together

The opening order question needs authorized facts and complete applicable policy evidence before a supported explanation. You can now separate retrieval failure, interpretation failure and an access incident, with the full denominator visible.

Learning Checks

Attempt each answer before expanding the reasoning.

M13-L13-Q01

Why does adding retrieval fail to guarantee freedom from hallucination? Think about both the selected evidence and what happens after it reaches the model.

Show reasoning

Retrieval can return missing, stale, irrelevant or unauthorized material. The model can misinterpret evidence or invent beyond it. The system therefore needs evidence checks, access enforcement and a defined response when the necessary support is unavailable. Naming the architecture R A G does not establish those conditions.

M13-L13-Q02

In a hypothetical 20-question test, 16 obtain required authorized evidence and 12 of those 16 receive correct supported explanations. Report end-to-end, conditional generation and retrieval coverage. How should unauthorized disclosure be treated?

Show reasoning

12/20=60% end-to-end; 12/16=75% conditional generation; 16/20=80% retrieval coverage. Report unauthorized disclosure as a separate high-consequence failure.

Sources

Learner Materials

Download Lesson 13 Learner Materials

Use the supplied fictional cases and templates for practice.

Practice

Write your answer, compare it with the model answer and decide what to revisit. These written checkpoints are self-assessed.

Here is the checkpoint. Why does adding retrieval fail to guarantee freedom from hallucination? Think about both the selected evidence and what happens after it reaches the model.

Show The Model Answer

Retrieval can return missing, stale, irrelevant or unauthorized material. The model can misinterpret evidence or invent beyond it. The system therefore needs evidence checks, access enforcement and a defined response when the necessary support is unavailable. Naming the architecture R A G does not establish those conditions.

Compare your reasoning with the model answer. These written checks are self-assessed.

In a hypothetical 20-question test, 16 obtain required authorized evidence and 12 of those 16 receive correct supported explanations. Report end-to-end, conditional generation and retrieval coverage. How should unauthorized disclosure be treated?

Show The Model Answer

12/20=60% end-to-end; 12/16=75% conditional generation; 16/20=80% retrieval coverage. Report unauthorized disclosure as a separate high-consequence failure.

Compare your reasoning with the model answer. These written checks are self-assessed.

Open Practice And Save Your Work