# Your first evaluation: compare what you wanted with what happened

Original LockedIn Labs practice. About 45 minutes. All support requests are
invented; this is a local learning exercise, not a customer service.

An evaluation is a repeatable way to compare a system's behavior with an
expectation. You do not need a paid evaluation platform or a model key to learn
the method. In this lab, you run three small Python functions against the same
six requests and inspect the mistakes behind their scores.

## 1. Agree what the system should do

Our toy support router has two choices:

- **guide**: offer instructions when someone asks how to do something;
- **handoff**: pass a request to a person when human help is requested.

It only returns a label. It does not answer, send, transfer, change an account
or use a model. An expected route is the reference; the function's actual
route is the observation. The evaluator compares them.

Read the six cases in `CASES` before running anything. E4 says a person is not
needed. E5 requests help without the words `human` or `person`. These are useful
distinctions, not just more copies of the happy path.

Keep an answer key separate from a student's question paper. In the same way,
the function receives only the request text; its evaluator receives the expected
route. Giving that reference to the candidate would leak the answer.

## 2. Run the comparison

Download `fde-first-evaluation.py` into a disposable folder. Use Python 3.11 or
newer. No package installation, credentials, GPU or internet connection is
required for execution.

```shell
python3 fde-first-evaluation.py test
python3 fde-first-evaluation.py compare
```

On Windows, replace `python3` with `py -3`.

The seven checks verify scoring, separate references, the known-case/probe
distinction, matrix totals, missing-case accounting and invalid inputs.
They check this teaching code, not an AI model.

The JSON output contains dataset `support-routing-six/v1`, the six inputs and
references, each candidate's actual decisions and its summary. Keep the result
with the Python file you ran. Do not reuse the displayed counts after changing
the candidates or dataset.

## 3. Find the errors behind the same score

The **keyword baseline** hands off when it sees `human` or `person`.
**Always hand off** chooses a person for every request.
**Expanded rules** adds one negation rule and two phrases before applying
the keyword baseline.

All six cases run for each candidate.

| Candidate | Correct / planned | Missed handoffs | Unnecessary handoffs |
| --- | --- | --- | --- |
| Keyword baseline | 3 / 6 | 2: E5, E6 | 1: E4 |
| Always hand off | 3 / 6 | 0 | 3: E1, E3, E4 |
| Expanded rules | 6 / 6 | 0 | 0 |

The first two totals match, but their failures do not. One may leave people
stuck in automation; the other adds unnecessary work for a receiving team.
Agree which behavior is acceptable for the actual service before preferring
one score.

A **confusion matrix** organizes these observations. Put expected routes on
rows and predicted routes on columns. With handoff as the positive class:

| | Predicted handoff | Predicted guide |
| --- | --- | --- |
| Expected handoff | Correct handoff: true positive | Missed handoff: false negative |
| Expected guide | Unnecessary handoff: false positive | Correct guide: true negative |

The four cells sum to the number of executed cases. The keyword baseline's
cells are 1, 2, 1, 2 in the order above.

**Accuracy** counts correct decisions across all evaluated cases.
**Handoff recall** asks how many needed handoffs were caught: the baseline
gets 1/3. **Handoff precision** asks how many predicted handoffs were warranted:
the baseline gets 1/2. Always-handoff gets recall 3/3 and precision 3/6.
If a denominator is zero, the corresponding ratio is undefined.

This sample is deliberately balanced: three guide and three handoff references.
Those counts are not an estimate of real customer demand. Keep planned and
executed counts too. One unrun case cannot disappear from a six-case plan.

## 4. Challenge six out of six

Read the separate `PROBE`: “I've already tried twice. Can a colleague take over?”
The expected route is handoff. The expanded rules still return guide because
none of their recognized phrases appears. The probe is not included in the
six-case score.

A good score on examples used to develop rules does not establish performance
on new language. This published probe is also no longer hidden once you have
seen it. Retain it as a regression example, then ask someone who did not tune
the rules for fresh cases.

Write an expected route and rationale **before** executing each new request:

```shell
python3 fde-first-evaluation.py try \
  --text "Can a colleague take over?" --expected handoff
python3 fde-first-evaluation.py try \
  --text "Can I change my personal settings?" --expected guide
```

The `try` command reports predictions separately and does not rewrite the
six-case dataset or its scores. Use only invented text; do not paste private
customer requests into a terminal or retained report.

Now invent your own request for a person without `human` or `person`, and
one request that mentions a person but declines a handoff. Try an ambiguous
request too. When people disagree about its reference, resolve the policy or
introduce a clarify/escalate behavior in a separate design; do not conceal
disagreement in a forced label.

## 5. Improve one rule, then inspect what changed

Make a disposable copy of the Python file. Change only `predict`, retain the
original cases and run `compare` again. Did the change fix one mistake while
creating another? Keep both before and after case results.

The supplied checks intentionally assert the original teaching candidates'
observations. If you change those rules, some checks will change; explain the
new expected behavior rather than treating a changed score as a broken Python
installation. Preserve the original file for a reproducible baseline.

A held-out set stays out of tuning. Once you adapt to its failures, preserve
them as regressions and obtain new evaluation examples. A six-case tutorial
cannot establish generalization or authorize a production release.

## 6. Apply the method to an AI application

The same separation works when the candidate is a model-backed function:
input to the application, references and rubric to the evaluator, observations
retained for inspection. Record model, prompt, retrieval, tools, dataset and
evaluator versions. Repeat runs where model variability affects the decision.

Exact matching is suitable for these two labels. Free-form answers usually
need criteria such as supported facts, required detail and appropriate next
action. Have people inspect examples and resolve disagreements. A model used
as a judge can help, but its judgments need validation against the task.

For an agent, evaluate both the result and the actions that produced it.
Correct-looking prose does not establish authorized tool use or current sources.
Continue to the advanced workflow evaluation lab for those boundaries.

Keep a short comparison note:

1. The task, dataset version and acceptance criteria.
2. Planned, executed, correct and unrun counts.
3. Case IDs behind each failure category.
4. What the fresh examples changed.
5. The remaining evidence needed for a receiving-team decision.

## Sources and next practice

Primary references checked 20 October 2026:

- [LangSmith evaluation concepts](https://docs.langchain.com/langsmith/evaluation-concepts)
- [Claude evaluation guidance](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)
- [scikit-learn classification metrics](https://scikit-learn.org/stable/modules/model_evaluation.html#confusion-matrix)

The cases, Python functions and teaching sequence above are original LockedIn
Labs materials. These references identify technical sources, not affiliations.

Continue with [actual workflow boundary and recovery evaluation](/training/guides/agent-engineering#2c-evaluate-useful-work-stale-approval-and-missing-evidence).