Free learning / 45-minute evaluation lab
Your first evaluation: compare what you wanted with what happened
Start with what a good result means, compare three simple candidates on the same six requests, and inspect the mistakes behind a score.
No account needed. Use your own copy for writing and keep the original as a reference.
Engineering lab library / Run locally
Make the score explainable.
Run three small Python functions against the same six synthetic requests. Compare their actual mistakes, then challenge a perfect sample score with new wording.
Synthetic records. No provider key or model inference. Runs locally with the standard library.
Start here / Python 3.11+ · Standard library
First evaluation comparison
Reproduce three candidate reports and explain how equal scores hide different failures.
Python 3.11+ standard library · No account or installation
Original LockedIn Labs practice. About 45 minutes. All support requests are invented; this is a local learning exercise, not a customer service.
An evaluation is a repeatable way to compare a system's behavior with an expectation. You do not need a paid evaluation platform or a model key to learn the method. In this lab, you run three small Python functions against the same six requests and inspect the mistakes behind their scores.
1. Agree what the system should do
Our toy support router has two choices:
- guide: offer instructions when someone asks how to do something;
- handoff: pass a request to a person when human help is requested.
It only returns a label. It does not answer, send, transfer, change an account or use a model. An expected route is the reference; the function's actual route is the observation. The evaluator compares them.
Read the six cases in CASES before running anything. E4 says a person is not needed. E5 requests help without the words human or person. These are useful distinctions, not just more copies of the happy path.
Keep an answer key separate from a student's question paper. In the same way, the function receives only the request text; its evaluator receives the expected route. Giving that reference to the candidate would leak the answer.
2. Run the comparison
Download fde-first-evaluation.py into a disposable folder. Use Python 3.11 or newer. No package installation, credentials, GPU or internet connection is required for execution.
python3 fde-first-evaluation.py test
python3 fde-first-evaluation.py compareOn Windows, replace python3 with py -3.
The seven checks verify scoring, separate references, the known-case/probe distinction, matrix totals, missing-case accounting and invalid inputs. They check this teaching code, not an AI model.
The JSON output contains dataset support-routing-six/v1, the six inputs and references, each candidate's actual decisions and its summary. Keep the result with the Python file you ran. Do not reuse the displayed counts after changing the candidates or dataset.
3. Find the errors behind the same score
The keyword baseline hands off when it sees human or person. Always hand off chooses a person for every request. Expanded rules adds one negation rule and two phrases before applying the keyword baseline.
All six cases run for each candidate.
| Candidate | Correct / planned | Missed handoffs | Unnecessary handoffs |
|---|---|---|---|
| Keyword baseline | 3 / 6 | 2: E5, E6 | 1: E4 |
| Always hand off | 3 / 6 | 0 | 3: E1, E3, E4 |
| Expanded rules | 6 / 6 | 0 | 0 |
The first two totals match, but their failures do not. One may leave people stuck in automation; the other adds unnecessary work for a receiving team. Agree which behavior is acceptable for the actual service before preferring one score.
A confusion matrix organizes these observations. Put expected routes on rows and predicted routes on columns. With handoff as the positive class:
| | Predicted handoff | Predicted guide | | --- | --- | --- | | Expected handoff | Correct handoff: true positive | Missed handoff: false negative | | Expected guide | Unnecessary handoff: false positive | Correct guide: true negative |
The four cells sum to the number of executed cases. The keyword baseline's cells are 1, 2, 1, 2 in the order above.
Accuracy counts correct decisions across all evaluated cases. Handoff recall asks how many needed handoffs were caught: the baseline gets 1/3. Handoff precision asks how many predicted handoffs were warranted: the baseline gets 1/2. Always-handoff gets recall 3/3 and precision 3/6. If a denominator is zero, the corresponding ratio is undefined.
This sample is deliberately balanced: three guide and three handoff references. Those counts are not an estimate of real customer demand. Keep planned and executed counts too. One unrun case cannot disappear from a six-case plan.
4. Challenge six out of six
Read the separate PROBE: “I've already tried twice. Can a colleague take over?” The expected route is handoff. The expanded rules still return guide because none of their recognized phrases appears. The probe is not included in the six-case score.
A good score on examples used to develop rules does not establish performance on new language. This published probe is also no longer hidden once you have seen it. Retain it as a regression example, then ask someone who did not tune the rules for fresh cases.
Write an expected route and rationale before executing each new request:
python3 fde-first-evaluation.py try \
--text "Can a colleague take over?" --expected handoff
python3 fde-first-evaluation.py try \
--text "Can I change my personal settings?" --expected guideThe try command reports predictions separately and does not rewrite the six-case dataset or its scores. Use only invented text; do not paste private customer requests into a terminal or retained report.
Now invent your own request for a person without human or person, and one request that mentions a person but declines a handoff. Try an ambiguous request too. When people disagree about its reference, resolve the policy or introduce a clarify/escalate behavior in a separate design; do not conceal disagreement in a forced label.
5. Improve one rule, then inspect what changed
Make a disposable copy of the Python file. Change only predict, retain the original cases and run compare again. Did the change fix one mistake while creating another? Keep both before and after case results.
The supplied checks intentionally assert the original teaching candidates' observations. If you change those rules, some checks will change; explain the new expected behavior rather than treating a changed score as a broken Python installation. Preserve the original file for a reproducible baseline.
A held-out set stays out of tuning. Once you adapt to its failures, preserve them as regressions and obtain new evaluation examples. A six-case tutorial cannot establish generalization or authorize a production release.
6. Apply the method to an AI application
The same separation works when the candidate is a model-backed function: input to the application, references and rubric to the evaluator, observations retained for inspection. Record model, prompt, retrieval, tools, dataset and evaluator versions. Repeat runs where model variability affects the decision.
Exact matching is suitable for these two labels. Free-form answers usually need criteria such as supported facts, required detail and appropriate next action. Have people inspect examples and resolve disagreements. A model used as a judge can help, but its judgments need validation against the task.
For an agent, evaluate both the result and the actions that produced it. Correct-looking prose does not establish authorized tool use or current sources. Continue to the advanced workflow evaluation lab for those boundaries.
Keep a short comparison note:
- The task, dataset version and acceptance criteria.
- Planned, executed, correct and unrun counts.
- Case IDs behind each failure category.
- What the fresh examples changed.
- The remaining evidence needed for a receiving-team decision.
Sources and next practice
Primary references checked 20 October 2026:
The cases, Python functions and teaching sequence above are original LockedIn Labs materials. These references identify technical sources, not affiliations.
Continue with actual workflow boundary and recovery evaluation.