Receive an AI candidate comparison

Read sections 6b and 6c of Evaluation science. Download the Python lab and one blank individual receiving record per person. Use Python 3.11 or later in a permitted local folder. The program uses the standard library, needs no account or network, and calculates comparisons from fixed authored routing predictions. It does not call or test an AI model.

The twenty planned case IDs are the comparison roster. A reference labels each synthetic task guide or handoff. A match counts agreement with that reference; it does not establish that a real task was completed safely. Unique IDs also do not make repeated authored examples independent observations.

Agree synthetic participant aliases. A begins as builder and B as receiver; swap for the second receiving round. Retain the exact file copy, commands, reports, first predictions, prior answer exposure, corrections and assistance separately. Use permitted synthetic material throughout.

If a learner cannot run Python, use the recorded case map for a reading task. Record observed only and retain the unrun execution goal. Do not borrow an account, request a policy exception or count a facilitator's run as the learner's execution. Arrange a later permitted attempt with a named owner.

Reproduce the six commands below before teaching. Locate the planned and matched denominators, the absent predictions, and the integer bounds in the actual JSON. Explain why changing the workload can move an aggregate and why a zero endpoint permits a tie. Keep that explanation separate from claims about real AI quality.

Use the existing practitioner instructor kit to rehearse with an observer. Ask the learner to predict and run a changed omission while you coach without typing for them or supplying the answer. The observer records actual actions, help, unanswered questions and the specific repair needed; the observation workspace can retain that account. Preserve independent accounts before comparing them. This preparation creates no trainer approval or delivery authority. Topic, audience, support and engagement decisions remain with the accountable owner.

Before running anything, each person writes what a higher aggregate could mean, which roster makes a paired comparison possible, and what one missing prediction would leave unknown. Declare any answers already seen. Locate the script and receiving record together, then agree where each person's separate output goes. This is a baseline explanation to revisit, not a qualification test.

Builder A runs the same-case and changing-workload reports. Receiver B locates the actual numerator and denominator before accepting an aggregate claim.

python3 fde-paired-evaluation.py same-cases > clinic-paired.json
python3 fde-paired-evaluation.py workload-mix > clinic-mix.json

For the paired report, locate the ten baseline matches and eight candidate matches across the same twenty planned cases. Find R09's improvement and the three regressions. For the workload mix, locate the routine and complex counts: the candidate's aggregate rises from 26/100 to 64/100 while both slice rates fall. Those workloads are different, unrandomized rosters. Do not attribute that aggregate movement to the candidate.

Each person records their own explanation and any help. The receiver asks for comparable cases and an agreed reference before making a real change claim.

Receiver B predicts what removing the three regression results will change, then runs the missing-results report without builder takeover.

python3 fde-paired-evaluation.py missing-cases > clinic-missing.json

Locate the seventeen matched pairs, baseline 7/17, candidate 8/17, and three absent candidate predictions. The full-plan candidate has eight known matches and up to three additional matches; the baseline has ten. The possible candidate-minus-baseline difference is minus two through plus one matches across twenty planned cases. The observed-subset gain cannot settle its direction.

Ask for the omitted results and why they are absent. Keep the report's unknowns as unknowns. Record the exact claim the available evidence supports, rather than silently substituting seventeen observed cases for the original plan.

Swap roles: B builds and A receives. A predicts both ranges before running these two omission choices, then explains the result while B locates the supporting case rows without taking over.

python3 fde-paired-evaluation.py missing-cases --missing-id R09 > clinic-r09.json
python3 fde-paired-evaluation.py missing-cases --missing-id R07 --missing-id R08 > clinic-tie.json

With only R09 absent, the full-plan difference is minus three through minus two matches. The candidate has fewer reference matches in every completion, although the exact difference remains unknown. With R07 and R08 absent, the difference is minus two through zero: a tie remains possible, so strictly fewer is too strong. Neither range is a confidence interval, a probability or a production decision.

Use the remaining time for a changed condition. Predict the effect of omitting two known candidate matches before opening a new report:

python3 fde-paired-evaluation.py missing-cases --missing-id R01 --missing-id R03 > clinic-changed.json

Derive the lower and upper match counts from the actual known and missing rows. Keep the original prediction beside the correction. If time runs out, mark this attempt unrun and name who will coordinate it; do not fill in an observation from someone else's explanation.

Each receiver explains one claim using their own saved report and names one real assumption the fixture cannot test. The builder locates the supporting evidence and records any disagreement. Preserve both accounts and help received. A pod lead gathers the unresolved questions without converting group activity into individual independence.

• Evaluation owner: preserve the comparison plan, candidate/reference versions, missing-result explanation and next controlled run. Completion bounds do not replace representative sampling or an appropriate uncertainty analysis.
• Domain owner: review whether guide/handoff and the reference are appropriate for the real workflow, especially consequential errors hidden by a match count.
• Service owner: define operating limits, failure monitoring and stop/recovery conditions that this comparison has not exercised.
• Receiving-team owner: locate the permitted evidence, explain what their team can operate, and retain the access, support or unresolved-work gap.
• Pod lead: connect each request to an actual accountable person, next test and review point. A practice role grants no real client or release authority.

Retain each person's first explanation, actual commands and outputs, independent action, help, correction and unrun work in their individual record. Choose one changed-input follow-up after review. Attendance, a correct calculation and this clinic's completion do not establish client placement, field independence or qualification.