Tests 101: understand what a green result means

An automated test is a program that exercises behavior and checks an expectation. Imagine a colleague asks you to check a spreadsheet formula with a few worked examples. You choose inputs, calculate the result and compare it with an independently agreed answer. A software test makes that comparison repeatable. The analogy has a limit: repeating the check cannot tell you whether the agreed answer was sensible in the first place.

A fixture is prepared input or supporting state for a test. An assertion is a check written as code. The test runner selects and executes tests, records failures and prints a report. A test suite is a collection of tests. A regression is previously expected behavior that a change breaks; a regression test preserves an example of that behavior.

This lab contains unit tests: small checks around one ordinary function. An integration test would exercise collaborating components, and an end-to-end test would follow a larger user flow. These terms describe scope, not a guarantee of quality. Unit tests cannot establish that cloud credentials, deployed networking or a real customer workflow work.

For an FDE using a coding agent, a green result is useful evidence only when you can explain the question that was actually checked. The agent can write a test or run a command. You still need to understand its expectations, inputs and scope.

Our fictional rule is simple: North billing tickets may become a draft; foreign tickets and unsupported categories must be blocked. A draft is only an output label in this exercise. It sends nothing and grants no permission.

The baseline function checks category but forgets tenant. Here is one test written out without the lab's reporting machinery:

import test from 'node:test';
import assert from 'node:assert/strict';

function route(ticket) {
  return ticket.category === 'billing' ? 'draft' : 'blocked';
}

test('foreign billing stays blocked', () => {
  const ticket = { tenant: 'south', category: 'billing' }; // Arrange
  const actual = route(ticket);                          // Act
  assert.equal(actual, 'blocked');                       // Assert
});

Read it as a sentence: “Given this South ticket, call the function, then require the returned value to equal blocked.” The assertion receives actual first and expected second. It fails because the baseline returns draft. A meaningful failure points to a disagreement; it does not by itself prove whether the requirement, fixture or implementation is wrong.

The examples below keep the rule fixed so you can isolate the implementation, command and assertion changes. Select each event and compare the value produced with the comparison performed.

Download fde-test-evidence-lab.mjs into a new working folder. Open it in an editor before running it. Use Node.js 24; the file needs no npm install, model account, API key or network. It creates test files in a new system temporary directory and removes only that directory after collecting the results. It does not load or edit your existing project.

node --version
node fde-test-evidence-lab.mjs > test-evidence.json

The outer command produces one JSON report. Its exit code of 0 means the experiment finished collecting results, not that all six child suites passed. Read each case's command, exitCode, executedCases, observations, runner and original stdout. The generated test source is included so you can inspect exactly what ran. Temporary paths in diagnostic output are replaced with <temporary-lab> for readability.

Write your predictions before running:

The runner can report one failed test for the setup-error file even though no ticket test body ran. In the filtered example it can report zero skipped tests while omitting unselected tests. Compare declaredCases with executedCases and notObserved; do not infer execution from the total or skip count alone.

stdout contains the real test report and OBSERVATION lines written immediately before each assertion. These lines are learning instrumentation, not a security boundary. Their absence in setup-error is expected. The report does not execute in this webpage; run the downloaded file to produce your own evidence.

Make two separate copies of the original .mjs file so the effects stay independent.

Experiment A — repair behavior. Change REPAIR_BASELINE = false to REPAIR_BASELINE = true. Run that copy. The baseline should now pass all three contract checks, with foreign actual blocked. The independent setup-error and overcorrected cases remain failures. Explain why the previously filtered green result could not have established this repair.

Experiment B — change selection. In a fresh copy, change INCLUDE_FOREIGN_IN_FILTER = false to INCLUDE_FOREIGN_IN_FILTER = true. Run it. The filtered command now selects permitted and foreign tests. It should expose one pass and one failure without changing the function. The unsupported-category case remains unobserved by that command. State the new scope precisely.

Keep the original report and each edited report under separate filenames. For each, record the changed line, prediction, actual selected names, exit result and observed values. Neither experiment needs a provider, package installation or modifications to your real repository.

Ask for the exact command, project version and output, then read what the command includes. A new file does not necessarily join an existing npm test script. In the protected Claude Code starter, the script names test/triage.test.mjs; a separate file needs an explicit run or a reviewed script change. Check that your new test name actually appears.

Use the failure phase to choose the next action. A missing import calls for setup investigation. A failed comparison calls for checking the requirement, input and implementation. A filtered pass calls for examining the missing cases. A pass after weakening expectations calls for reviewing the test change before claiming a repair.

Changing an assertion can be appropriate when the requirement changes, but the decision needs an explanation. The weak-assertion case did not change the rule. It replaced “foreign is blocked” with “the output has a string type.” The original test name remained, which makes the green result especially misleading.

Keep useful allowed examples beside refusal examples. The always-blocked function demonstrates how an overcorrection can make a denial pass while breaking the task. Extend tests around the changed behavior rather than endlessly rerunning unrelated checks. Selectivity is useful when its scope is explicit.

Before returning to the coding project, explain all three statements in your own words:

• A process can exit successfully while the behavior you care about was never tested.
• A test can fail before reaching any assertion about application behavior.
• Every selected test can pass after a weaker assertion while a real defect remains.

Your handoff should contain the requirement, reviewed code version, command, executed cases, actual results, both changed-copy reports and remaining uncertainty. A local unit-test pass does not demonstrate authentication, model quality, a deployed integration or authorization to release.

Continue to Use failures to debug the decision in Claude Code training with your existing Academy access. Apply this reasoning to its real starter, confidence boundaries and reviewable regression test.

Primary reference checked 22 October 2026: Node.js 24 test runner, particularly test functions, name filtering and command-line execution. All instruction, examples and fixtures here are original LockedIn Labs material.