# Stop a service without abandoning accepted work

40-minute engineering lab · Node.js 24 · macOS, Linux or WSL · Original synthetic case

A release replaces a process while someone is waiting for its answer. Stopping new traffic is only half the job: the old process may already owe a response. In this lab, you will hold a real HTTP request open, signal its server, and compare a completed drain with two interrupted requests.

Download the [graceful-shutdown companion](/templates/fde-graceful-shutdown.mjs). Read it before execution. It starts only its own local child processes on `127.0.0.1`, uses operating-system-assigned ports and cleans them up. No packages, provider key, model, cloud account or production service is involved. Use Node.js 24 on macOS, Linux or WSL; native Windows signal behavior does not support this experiment and is rejected. HTTP exchanges, signals and process exits are real; the work and its release condition are synthetic.

## 1. Separate admission from completion

**Admission** means the service accepted a request. **Completion** means it produced the required result; the caller must still receive and interpret it. **Draining** means closing admission to new work while giving accepted work a bounded opportunity to finish. A **grace period** is that allowed interval, not proof that every request finished.

Picture a reception desk closing for the day. A closed sign turns away new arrivals, but a visitor already at the desk still needs their document. In software, readiness may withdraw an instance from normal traffic, but it does not itself complete the requests already inside it.

| Boundary | What the companion observes |
| --- | --- |
| Admission | The child reports that its `/work` handler accepted a real request, then deliberately holds the response. |
| Readiness | `/ready` answers whether this instance is admitting new work under this lab's policy. |
| Liveness | `/live` answers while the process is responsive, including during a graceful drain. |
| Completion | The client actually receives an HTTP response from the originally accepted request. |
| Termination | The parent records the child process's actual exit and then probes the old endpoint. |

The parent and child use inter-process communication (IPC) to coordinate the teaching steps. A child's accepted event lets the parent know the request reached the handler before it sends a signal. That acknowledgement is not an HTTP response, a durable write or a client receipt.

## 2. Follow one accepted request through a stop

`SIGTERM` requests termination. On supported systems the application can install a handler and perform a bounded shutdown. `SIGKILL` forcibly stops the process; its application cannot run a cleanup handler for that signal. Installing a SIGTERM handler transfers responsibility for finishing shutdown to the application.

This lab deliberately keeps its listener open during the graceful phase. The handler first marks the instance draining. `/ready` now returns 503, `/live` still returns 200, and a newly arriving `/work` request receives 503. The original request remains held. When the parent releases it through IPC, the child produces its response and closes the listener. The parent records both the original client’s HTTP 200 response and normal child exit; their observation order can vary across processes.

This is one application-level admission strategy. Calling Node's `server.close()` immediately would stop accepting new connections; a new caller could see a connection failure instead of this lab's HTTP 503. Do not claim those are the same observation. Readiness, the application's admission guard and listener closure are separate mechanisms.

The diagram is an authored explanation of the three executable cases, not live telemetry. Follow the originally accepted request separately from any probe or new request.

## 3. Predict and run the three shutdown cases

Use a new learning folder and retain the output:

```sh
node --version
node fde-graceful-shutdown.mjs --all > shutdown-results.json
```

Before running, predict the original request's result, any observed probe statuses and the final process exit. Do not assign an HTTP status to a connection that never receives one.

| Case | Action after the request is accepted | Observation to investigate |
| --- | --- | --- |
| `graceful` | SIGTERM; inspect draining probes; release the held request | New work is refused; the original caller receives 200 and normal process exit is observed. |
| `abrupt` | SIGKILL while the request is held | The connection fails without an HTTP response; the process exits by signal. |
| `deadline` | SIGTERM with a short real grace timer; never release the request | The grace expires, sockets are destroyed and the process exits with code 2. The caller does not receive a completed response. |

Inspect a case independently:

```sh
node fde-graceful-shutdown.mjs graceful
node fde-graceful-shutdown.mjs abrupt
node fde-graceful-shutdown.mjs deadline
```

Read `observations` in parent-observed order. `heldRequest` is the actual original caller's result. `drainingProbes` contains the graceful case's readiness, liveness and new-work observations; the other cases do not claim those probes ran. `exit` reports the operating-system exit result, and `afterExitProbe` records a fresh attempt to reach the old endpoint.

A result with `kind: "transport-failure"` has `status: null`. Keep its actual error code rather than rewriting it as HTTP 500 or 503. The operating system and client implementation determine the precise failure code. A child event saying it finished writing would also not be enough on its own: inspect the client's received response.

The configured grace is not a latency benchmark. The experiment uses coordination messages to choose when synthetic work may finish, and a real timer to bound the deadline case. Scheduling and machine load affect elapsed time. It does not measure user workloads, routing propagation or an orchestrator's stop budget.

## 4. Challenge the shutdown claim

Compare `graceful` with `deadline`, keeping the original request the same. In one run the receiver allows the work to finish before shutdown. In the other, an accepted request remains unfinished when its budget ends. Explain why “we installed a SIGTERM handler” is insufficient evidence of graceful completion.

In a separately named copy, change `DEADLINE_GRACE_MS` from `300` to `1000`, then run `node YOUR_CHANGED_COPY.mjs deadline`. Preserve the original output and label the modified candidate. The configured budget should change, but work that is never released still expires without a completed response. Compare the event sequence; do not turn the elapsed time into a benchmark.

Now inspect the admission check in the downloaded file. Explain the behavior a request arriving after draining starts should receive, and what evidence would expose accidental continued admission. Keep the late request distinct from the already accepted one. Do not change the expected result merely because a process exits with code 0.

For a real workflow that can write to a database or call a tool, a lost response leaves another question: did the effect occur before the connection failed? This companion performs no such business effect. Its transport failure cannot establish whether a real payment, message or update happened. The receiving design needs an operation identity, reconciliation and an appropriate retry policy before retrying uncertain work.

## 5. Hand the shutdown decision to the receiving team

Use a three-person pod: service owner, caller owner and receiving operator. Each writes a prediction before discussion. The receiver runs a case, finds the original request's result and explains the difference from the late request. Record hints or author takeover. Solo practice remains self-review.

```text
Candidate file / version or digest:
Case and configured grace:
My prediction before execution:
Accepted-request event and original client result:
New-request / probe observations (or not run):
Process exit and post-exit observation:
My contribution and assistance received:
Unfinished or uncertain work:
Receiving decision for this stated purpose:
Next owner / action / evidence needed:
```

The service owner defines admission, in-flight tracking, shutdown budget and cleanup. The caller owner defines how to recognize a completed response and reconcile an uncertain result. The receiving operator must reproduce the runbook with permitted access. The platform owner must verify routing, lifecycle hooks, signal delivery and actual stop budgets in the chosen environment. A pod lead coordinates these decisions; they do not gain release authority from a successful classroom command.

In Kubernetes, Pod termination and endpoint updates are coordinated by platform components; a terminating endpoint is not simply removed instantly. Shutdown behavior depends on the configured grace period, hooks, container lifecycle and consumers of endpoint state. This local companion runs no Kubernetes controller or load balancer. Use its observations to ask better questions before a separately authorized environment rehearsal.

## 6. Explain, debug and extend

**Explain:** Why can liveness remain successful while readiness refuses new work? Which request does each result describe?

**Debug:** The old caller reports a connection reset and the process is gone. What did you observe, and which business-effect question remains unanswered?

**Design:** A workload takes longer than the permitted stop budget. Propose how to bound, cancel, checkpoint or reconcile its work. Identify the business contract and owners needed before choosing a technique. Increasing a timeout alone does not prove useful completion.

Continue with [reliable services and operational ownership](/training/guides/service-operations#6b-run-the-receiver-rehearsal) to combine target-specific probes with this lifecycle evidence. The [operational handover lesson](/app/academy?course=service-operations&lesson=operational-handover&view=focus) connects the observations to an accountable receiving runbook under existing Academy access.

## Sources and scope

Original LockedIn Labs case, code and teaching sequence. Primary references checked 26 October 2026:

- [Node.js 24 HTTP server lifecycle](https://nodejs.org/docs/latest-v24.x/api/http.html#serverclosecallback): listener closure, active connections and forceful connection closure.
- [Node.js 24 signal events](https://nodejs.org/docs/latest-v24.x/api/process.html#signal-events): signal handling and platform differences.
- [Kubernetes Pod termination](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination): orchestration, grace periods and terminating endpoints.

The local run supplies HTTP and process observations. It does not demonstrate production recovery, workload durability, organizational staffing or independent field qualification.