01 Example02 Attempt03 Explain04 Feedback05 Revise06 Fresh challengeOne fictional mission:Build an internal support assistant.
Drafts are temporary. Download before leaving or refreshing.
Step 1 of 6 · Example
An answer with a citation can still be wrong. In this worked example, a support assistant finds an access policy that was retired yesterday. The answer sounds confident and includes the old citation.
01 Notice A citation proves where an answer came from. It does not establish that the source is current or allowed.
02 Specify If the only source is retired, withhold the answer and hand the request to the support owner with the source ID and reason.
03 Test Given retired source policy-17 and an access question, expect no factual answer, an escalation result, and policy-17 in the diagnostic record.
Make the failing input and the expected behavior precise enough for someone else to run the test.
Try it yourself → Use this as a written practice worksheet The step-by-step editor needs JavaScript. You can still use this complete exercise: write an attempt and explanation, compare it with the rubric, revise, then answer the fresh challenge in your own document.
Two sources disagree. What should the assistant do? The fictional assistant retrieves two current documents with conflicting access rules. A proposed patch chooses whichever result ranked first.
Both sources are current and the requester can read them. Ranking measures retrieval relevance; it does not resolve policy conflicts. The support owner can resolve the conflict. The assistant cannot grant access. Given… When… Then… Include the inputs, expected result, and a failure this test must catch.
Why does your test catch a failure that a happy-path demonstration would miss?
Use the highest-ranked source: Answer immediately and show its citation.Scripted guidance for this choice Ranking does not establish authority. Your test should expose the disagreement instead of accepting a confident answer with a citation.
Explain the conflict and escalate: Withhold the disputed claim and send both source IDs to the support owner.Scripted guidance for this choice This preserves the unresolved conflict. Now make it testable: name the exact fixture, the withheld claim, both source IDs, and the escalation result.
Combine the two rules: Ask the model to generate a compromise.Scripted guidance for this choice A generated compromise creates a new rule nobody authorized. Include an assertion that no blended policy or access change is produced.
Review criteria Reproducible input: Could another developer reconstruct both conflicting sources and the user request?Source A permits team access; Source B requires an owner approval. Both have current dates and distinct IDs.
Observable behavior: Does the test assert what the assistant withholds, returns, and records?No policy answer or access change; escalate with both IDs and a conflict reason.
A test that can fail: Would selecting the first result or inventing a compromise fail your assertions?Assert that reversing retrieval order leaves the conflict outcome unchanged.
Revise your attempt Make your test specific enough for another developer to implement. Preserve the original attempt, then show the stronger version below.
Now one source is hidden from the requester. The same request retrieves a current document the user cannot access. The other document is public to the team. How does your boundary change?
The restricted source must not be quoted or revealed to this requester. The accessible source may be used only if it independently supports the answer. What changes in the fixture, expected response, and leakage assertions?
Explain what stays the same and what changes.
Enforce permissions before answering: Exclude restricted content and only answer from independently sufficient permitted evidence.Scripted guidance for this choice Permission is a separate control from source conflict. Add a test proving the hidden source cannot leak through text, citations, or diagnostic details returned to the requester.
Show the hidden citation as a warning: Explain that another source may disagree.Scripted guidance for this choice Even revealing the restricted source or its claim can leak information. The new test needs to cover authorization, not just answer correctness.
Tell the model to be discreet: Supply both documents and request a careful answer.Scripted guidance for this choice A wording instruction is not an authorization boundary. Exclude restricted content before it can influence the user-visible result.
Ask a teammate to implement the test from your description. Compare what they built with what you intended.
