Free learning / Measurement workshop
AI evaluation & decision science — working copy
Calculate classification and retrieval results, protect test data and price accepted outcomes with synthetic local evidence.
No account needed. Use your own copy for writing and keep the original as a reference.
Engineering lab library / Run locally
Choose your next agent lab.
Start with the durable workflow reference, then choose the boundary you want to test. Open its walkthrough for setup, commands and expected observations.
Synthetic records. No provider key or model inference. Runs locally with the standard library.
Start here / Python 3.11+ · Standard library
Evaluation arithmetic lab
Calculate actual fixture metrics and explain their limits.
Python 3.11+ · Standard library · Synthetic cases · No model or network
Original LockedIn Labs practice. All supplied cases and cost units are synthetic. No model, customer system or paid API is invoked. Python 3.11+; standard library only.
Curriculum: https://training.lockedinlabs.ai/training/curriculum?stage=3#evaluation
Download the companion Python lab, inspect it, then run it locally:
python3 --version
python3 fde-evaluation-science.pyThe script prints calculated observations. It does not score your competence or authorize a release. null means the denominator is absent, not that the score is perfect.
1. Define useful work
Choose a draft-only administrative assistant. Write:
- User task and unit of work:
- Positive class (what does selection mean?):
- False-positive consequence:
- False-negative consequence:
- Critical failure that cannot be averaged away:
- Expected manual fallback and owner:
The supplied fixture has TP=8, FP=1, FN=1, TN=0. Verify precision=8/9, recall=8/9 and accuracy=8/10. Change one true positive to a false negative. Predict the counts before running. Then change a negative prediction into an unsafe positive and explain the release consequence separately from the score. Call classification([], []) and explain why its rates are undefined.
To run a changed case without editing the original file, save this as evaluation_experiment.py beside it, then run python3 evaluation_experiment.py:
import runpy
lab = runpy.run_path("fde-evaluation-science.py")
expected = [True, True, False, False]
print(lab["classification"](expected, [True, True, False, False]))
print(lab["classification"](expected, [True, False, True, False]))Predict the second report: one correct selection, one unsafe selection, one missed useful draft and one correct rejection. Explain why precision and recall both change even though the number selected stays the same.
Illustrative learning artifact
Inspect the result behind the score
8/10 correct still contains an unsafe acceptance.
2. Protect the test
Create a manifest with case ID, authoritative source ID, permission, source version, split, task category and expected result. Keep related chunks, conversations or paraphrases together. Use source_overlap to detect a source reused between development and test. Remove the overlap and rerun.
This check cannot tell whether two differently named sources contain the same material, whether labels are correct or whether cases represent the intended population. Record how you would review those limitations. Have two reviewers label a small permitted sample; preserve their disagreements and clarify the rubric before expansion.
3. Locate a retrieval failure
The lab ranks D3, D1, D4. Relevant documents are D1 and D2; D3 is forbidden. At k=3, recall is 1/2 and reciprocal rank is 1/2. The forbidden return is a separate boundary failure.
Call retrieval_metrics with D1 ranked first, then with no relevant result. Explain which value moves and which security condition remains. Define how authorization filtering and freshness should be enforced before material reaches model context. Add a missing-source case that should abstain instead of inventing a citation.
4. Calibrate the judge
Prepare five synthetic results with explicit expected outcomes. Include one correct answer produced through an unauthorized tool action. Define deterministic checks for schema and side effects, and a rubric for the parts requiring domain judgment. Compare two human judgments before adding a model judge. Record disagreement, judge version, prompt, ordering and repeated-trial behavior. A fluent explanation is not independent evidence of correctness.
5. Count the full outcome cost
The synthetic baseline spends 120 minutes and 24 cost units for eight accepted outcomes: 15 minutes and three cost units each. The assisted process spends 30 drafting + 110 review minutes and 35 cost units for seven accepted outcomes: 20 minutes and five units each.
Change economics inputs to include failed attempts and rework. Do not remove their cost from the numerator just because they are absent from accepted outcomes. Try zero accepted outcomes. List fixed operating costs separately from per-task costs. These are arithmetic exercises, not vendor prices or measured savings.
6. Make a bounded decision
Record candidate revision, data manifest, grader version, baseline, metrics by slice, critical failures, total effort, cost and remaining uncertainty. State the allowed population, fallback, stop conditions and decision owner. A new language or changed permission is a new condition to evaluate, not an automatic extension of the old result.
Receiving review
Give a peer the packet and let them change one input without telling you in advance. Predict the affected measure, run the calculation and explain the consequence. Record what they observed, what remains untested and the next evidence needed. Reading completion and this fixture do not establish production model quality.
References: Google classification metrics, Google dataset separation, Anthropic agent evaluations, FinOps unit economics. Checked 2026-10-23.