Course Project And Templates

It tests later orders from customer groups absent from training; time and customer novelty vary together, so the effects are not isolated.

Use approved current documents for current-policy questions; use the documented date/product applicability rule for a dated order, including controlled historical versions when required.

7/10=70% end-to-end;7/8=87.5% generation conditional on required evidence retrieval. Report access failures separately; these hypothetical counts do not establish real performance.

M13-PROJECT-Q01

The 68/17 partition is intended to test at-checkout return risk within 30 days for later orders from customer groups absent from training. It does not isolate the effect of time from the effect of unseen customers. The group counts and dates define a synthetic split; they do not prove label quality, performance or leakage absence.

M13-PROJECT-Q02

For a current-policy question, retrieve from the approved current-policy set. For a dated order question, use the documented product and effective-date rule to select the applicable version, including a controlled historical version when needed. Newest-version-wins is not a general rule.

Model solution

For the orders assistant, define one order row as one order, one line row as one order line, and one return row as one return event. Preserve source-qualified identifiers. The operational database supplies current authorized order facts; policy documents are versioned and retrieved by product and applicable date; a scoped preference is application state; a reviewed final test set remains independent.

The example pipeline receives 100 records, excludes five duplicate copies under a defined key, quarantines ten unresolved currency/customer links, and reconciles 85 approved records. The hypothetical group-and-time split is specified in worked_case_expansion.md. Its claim is later orders for unseen customers, and it does not isolate the effects of customer novelty and time.

Tests check exact order IDs, applicability dates, missing facts, stale policy, exceptions, unauthorized tenants, delayed corrections, conflicting remembered preferences, and deletion across indexes/caches. Report retrieval and answer correctness separately. If required evidence is absent, the assistant says what is unknown and asks for review instead of inventing delivery or eligibility.

A strong response distinguishes current-policy search from a date-qualified historical policy lookup and inventories copies, indexes, caches and logs for correction/deletion. Accept equivalent architecture choices when sources, rights, authority, lifecycle and evaluation are explicit.

Worked measurement: keep the denominator

In the hypothetical ten-question course pilot, eight questions retrieve the required authorized evidence and seven of those eight receive correct evidence-supported explanations. End-to-end correctness is 7/10=70%; conditional generation correctness is 7/8=87.5%; retrieval coverage is 8/10=80%. Report all three with their denominators. Track unauthorized disclosure separately; a high average does not cancel an access failure. These numbers are teaching assumptions, not test results from a deployed service.

Complete model packet

See model_ten_case_packet.md for all ten expected-evidence/access/uncertainty rows. Results remain “Not run.” Keep this separate from the blank learner packet.

M13-PROJECT-Q03

Seven correct evidence-supported explanations across all ten questions give7/10=70% end-to-end. Seven correct among eight successful retrievals give7/8=87.5% conditional generation. Retrieval coverage is8/10=80%. Report unauthorized disclosure as a separate high-consequence failure, with investigation and access response; the percentage cannot cancel it. These are hypothetical counts.

Task, people, and outcome

• User and authorized purpose:
• Decision or response supported:
• What the system must not decide or do automatically:

Four data roles

Do not put final evaluation answers into training or tuning. If a source appears in more than one role, describe the separate process and authorization.

Pipeline and lifecycle

Draw each source, ingestion step, validation rule, transformation, store/index, application read, update path, deletion path, and monitoring check. Label data grain and authority. Show one late event, duplicate, correction, and invalid record.

Failure and response

• One missing or stale source:
• One bad join or incorrect relationship:
• One unauthorized request:
• Evidence required before the system answers:
• Safe uncertainty response:
• How a human corrects the source and verifies downstream refresh:

• Name and version:
• Owner and contact:
• Purpose and intended users:
• Row/document/event grain:
• Sources and collection process:
• Date range and time-zone conventions:
• Identifiers and namespaces:
• Units and schema:
• Population and selection process:
• Inclusion/exclusion and quarantine rules:
• Labels, definitions, annotators and adjudication:
• Training/development/final evaluation split rule:
• Known duplicates, gaps, bias and limitations:
• Rights, permitted uses and restrictions:
• Access controls and retention:
• Update/correction/deletion process:
• Downstream indexes, caches, logs and derived representations:
• Quality measures, denominator and threshold:
• Version history and next review trigger:

Required operating rules

• Define which source wins when a remembered statement conflicts with an authoritative current source.
• State what a user sees when evidence is missing, stale, conflicting, or outside their permission.
• Inventory source tables, copies, document versions, chunks, embeddings, caches, logs, evaluation traces, and any other relevant derived representations.
• Record what a deletion or correction request covers, who acts, and how completion or limits are evidenced.
• Give every quality metric a denominator, date range, owner, and response threshold.

All identities, dates and rules below are synthetic. This is a model design, not execution evidence. Orders belong to tenant A unless stated. O101 belongs to authorized customer C7; O102 belongs to C8. V1 applies to purchases through May 31, 2026 and allows unused returns within 30 elapsed days. V2 applies from June 1 and allows returns within 14 elapsed days. Product P9 has a supplied excluded-sale exception. Date-qualified historical lookup remains available under controlled access. No actual retrieval or generation has run.

The learner packet remains blank. These are expected behaviors under supplied assumptions; a real task needs its own authorized source evidence, rules and validated expected results.

Score each area from 0–5. Give points for evidence and reasoning, not diagram decoration.

1. Source meaning (0–5): identifies owner, grain, keys/namespaces, units, time meaning, rights, updates, and gaps. Full credit requires an authoritative source and reconciliation check for each critical fact.
2. Pipeline integrity (0–5): shows ingestion through runtime, update/deletion paths, lineage, monitoring, and handling of late, duplicate, corrected, and invalid records.
3. Role separation (0–5): distinguishes training, development validation/final evaluation, retrieval, and state; explains rights and why evaluation stays independent.
4. Meaningful tests (0–5): supplies ten task-relevant cases including stale, exact ID, missing, and unauthorized inputs; reports retrieval and generation separately and states uncertainty behavior.
5. Rights and governance (0–5): assigns accountable roles, access and lifecycle controls, quality thresholds, correction path, deletion inventory, and a conflict rule.

Interpretation

• 22–25: actionable design with clear evidence and operating accountability.
• 17–21: mostly coherent; resolve named gaps before implementation.
• 12–16: useful outline but major source, test, or governance details remain.
• 0–11: insufficient to support the intended decision.

A diagram of platforms without row meaning, authoritative evidence, access rules, and operating checks cannot earn full credit.

Partial credit by area

Apply the following 0–5 scale separately to each of the five areas above; assess the area's listed deliverables, not diagram appearance.

For source meaning, inspect each critical fact's grain,key,unit/time,owner,rights and reconciliation. For pipeline integrity, inspect late/duplicate/correction/invalid/update/deletion paths. For role separation, inspect the actual influence on training,development,final testing,retrieval and state. For tests, require all ten expected-evidence/access/uncertainty cases plus full and conditional denominators and separate permission failures. For governance, inspect accountable decisions,release/hold criteria,lifecycle actions and unresolved constraints. Any unauthorized disclosure design blocks operational readiness regardless of the numerical total; the score still records the demonstrated work.

Inventory questions

• What is the authoritative source for each fact or rule?
• Which source can correct or delete an earlier value?
• Which source is a copy, cache, index, or derived representation?
• What would make this source unsuitable for training, evaluation, or runtime use?

Use the orders-and-returns scenario or your chosen task. For each question, write: expected authorized evidence; source/version; access condition; expected answer or uncertainty behavior; retrieval result; generation result; and whether the question tests a high-consequence permission boundary.

A test packet should not disclose a real person’s information. Synthetic examples must be labeled as synthetic and independently checked against the intended rule.

This hypothetical worked case defines a reproducible group-and-time split. Its counts, groups and dates are teaching inputs; no business performance is measured.

Reconciliation counts

• 100 source order records received.
• Five duplicate copies under the defined source-qualified order key are excluded, leaving 95 unique records.
• Ten unique records have unresolved currency or customer relationships and are quarantined, leaving 85 approved records.
• The 85 approved records are assigned to a synthetic prediction task: at checkout, estimate whether an order from a not-yet-seen customer will be returned within 30 days of purchase. Use only features available by the checkout decision. The label is the occurrence of a return within the 30-day window, not a future delivery status.

Explicit group-and-time rule

Use customer identity as the group and purchase month as event time. Training contains all approved January–March records for customer groups C001–C016: C001–C012 have four orders each (48); C013–C016 have five each (20); total 68. Final evaluation contains April records for entirely different groups C017–C019 with four orders each (12) and C020 with five (5); total 17. No customer group crosses the partition, and the evaluation period is later than the training period. The test therefore measures the joint claim “at-checkout return risk within 30 days for later orders from unseen customers.” It does not isolate the effect of time from customer novelty. These counts and dates define a reproducible synthetic partition; they do not establish label quality, data validity, model performance or leakage absence. Inspect the feature-availability cutoff and label maturity before use. If the target or deployment claim differs, redesign the evaluation.

Do not use the 17 final-evaluation outcomes to revise labels, features, thresholds, prompts or candidate selection and then report the same 17 as an untouched final test. Use a separate development-validation process for iteration and record it. Score the final set only after the 30-day outcomes mature. If the actual deployment claim changes, redesign the split.

Policy retrieval rule

The twelve policy documents include superseded versions retained in a controlled archive. A question about the policy in force now searches the approved current set. A question about a specific order uses its purchase date, product and applicable documented rule to locate the version relevant to that order, including a historical version when required. “Archived” must not mean “impossible to inspect for a date-qualified question.” Preserve effective dates and access conditions on every chunk. Do not assume newest-version-wins.

Four data roles and three runtime sources

The four data roles are training data, evaluation data, request-time retrieval, and application state. Training examples support learning and are not an answer-time source. Reviewed final test cases measure the frozen candidate and are not a runtime answer source.

At runtime, the application uses three source paths: current order facts from an authorized structured lookup; applicable policy text from date- and product-filtered retrieval; and a scoped preference or pending task from application state. The separate final-evaluation repository is never the default production answer source.

Worked measurement: keep the denominator

In the hypothetical ten-question course pilot, eight questions retrieve the required authorized evidence and seven of those eight receive correct evidence-supported explanations. End-to-end correctness is 7/10=70%; conditional generation correctness is 7/8=87.5%; retrieval coverage is 8/10=80%. Report all three with their denominators. Track unauthorized disclosure separately; a high average does not cancel an access failure. These numbers are teaching assumptions, not test results from a deployed service.

Learner source

The supplied scenarios, counts and dates are hypothetical teaching examples. Use the public methodological references in the accompanying reading.