# AI evaluation & decision science — working copy

Original LockedIn Labs practice. All supplied cases and cost units are synthetic. No model, customer system or paid API is invoked. Python 3.11+; standard library only.

Curriculum: https://training.lockedinlabs.ai/training/curriculum?stage=3#evaluation

Download the companion [Python lab](https://training.lockedinlabs.ai/templates/fde-evaluation-science.py), inspect it, then run it locally:

```sh
python3 --version
python3 fde-evaluation-science.py
```

The script prints calculated observations. It does not score your competence or authorize a release. `null` means the denominator is absent, not that the score is perfect.

## 1. Define useful work

Choose a draft-only administrative assistant. Write:

- User task and unit of work:
- Positive class (what does selection mean?):
- False-positive consequence:
- False-negative consequence:
- Critical failure that cannot be averaged away:
- Expected manual fallback and owner:

The supplied fixture has TP=8, FP=1, FN=1, TN=0. Verify precision=8/9, recall=8/9 and accuracy=8/10. Change one true positive to a false negative. Predict the counts before running. Then change a negative prediction into an unsafe positive and explain the release consequence separately from the score. Call `classification([], [])` and explain why its rates are undefined.

To run a changed case without editing the original file, save this as `evaluation_experiment.py` beside it, then run `python3 evaluation_experiment.py`:

```python
import runpy
lab = runpy.run_path("fde-evaluation-science.py")
expected = [True, True, False, False]
print(lab["classification"](expected, [True, True, False, False]))
print(lab["classification"](expected, [True, False, True, False]))
```

Predict the second report: one correct selection, one unsafe selection, one missed useful draft and one correct rejection. Explain why precision and recall both change even though the number selected stays the same.

## 2. Protect the test

Create a manifest with case ID, authoritative source ID, permission, source version, split, task category and expected result. Keep related chunks, conversations or paraphrases together. Use `source_overlap` to detect a source reused between development and test. Remove the overlap and rerun.

This check cannot tell whether two differently named sources contain the same material, whether labels are correct or whether cases represent the intended population. Record how you would review those limitations. Have two reviewers label a small permitted sample; preserve their disagreements and clarify the rubric before expansion.

## 3. Locate a retrieval failure

The lab ranks D3, D1, D4. Relevant documents are D1 and D2; D3 is forbidden. At k=3, recall is 1/2 and reciprocal rank is 1/2. The forbidden return is a separate boundary failure.

Call `retrieval_metrics` with D1 ranked first, then with no relevant result. Explain which value moves and which security condition remains. Define how authorization filtering and freshness should be enforced before material reaches model context. Add a missing-source case that should abstain instead of inventing a citation.

## 4. Calibrate the judge

Prepare five synthetic results with explicit expected outcomes. Include one correct answer produced through an unauthorized tool action. Define deterministic checks for schema and side effects, and a rubric for the parts requiring domain judgment. Compare two human judgments before adding a model judge. Record disagreement, judge version, prompt, ordering and repeated-trial behavior. A fluent explanation is not independent evidence of correctness.

## 5. Count the full outcome cost

The synthetic baseline spends 120 minutes and 24 cost units for eight accepted outcomes: 15 minutes and three cost units each. The assisted process spends 30 drafting + 110 review minutes and 35 cost units for seven accepted outcomes: 20 minutes and five units each.

Change `economics` inputs to include failed attempts and rework. Do not remove their cost from the numerator just because they are absent from accepted outcomes. Try zero accepted outcomes. List fixed operating costs separately from per-task costs. These are arithmetic exercises, not vendor prices or measured savings.

## 6. Make a bounded decision

Record candidate revision, data manifest, grader version, baseline, metrics by slice, critical failures, total effort, cost and remaining uncertainty. State the allowed population, fallback, stop conditions and decision owner. A new language or changed permission is a new condition to evaluate, not an automatic extension of the old result.

## Receiving review

Give a peer the packet and let them change one input without telling you in advance. Predict the affected measure, run the calculation and explain the consequence. Record what they observed, what remains untested and the next evidence needed. Reading completion and this fixture do not establish production model quality.

References: [Google classification metrics](https://developers.google.com/machine-learning/crash-course/classification/accuracy-precision-recall), [Google dataset separation](https://developers.google.com/machine-learning/crash-course/overfitting/dividing-datasets), [Anthropic agent evaluations](https://www.anthropic.com/research/demystifying-evals-for-ai-agents), [FinOps unit economics](https://www.finops.org/framework/capabilities/unit-economics/). Checked 2026-10-23.