# Reliable services & operational ownership — working copy

Original LockedIn Labs practice. Python 3.11+, no dependencies or network access. This lab calculates synthetic observations and simulates a queue. It is not a load test, monitoring integration, deployment or release authorization.

Curriculum: https://training.lockedinlabs.ai/training/curriculum?stage=5#operations

Download and inspect the [Python lab](https://training.lockedinlabs.ai/templates/fde-service-operations.py), then run:

```sh
python3 --version
python3 fde-service-operations.py
```

## 1. Follow one request

Map DNS, TLS, routing, identity, runtime, database, queue and external model API for your project. Mark persisted state and each independent owner. Explain what a VM, container and service each provide. Define readiness and liveness separately. Choose a user-outcome signal, a metric, a log event and a trace boundary; exclude secrets and sensitive payloads.

## 2. Define reliability

SLI: eligible requests that succeed within 2,000 ms / all eligible requests. Record measurement location, time window and exclusions. Semantic correctness and authorization are separate requirements.

The fixture contains 988 good requests and twelve failed requests. At a 99% target across 1,000 requests, the bad-event allowance is ten; the remaining budget is -2. Change one successful request to take 2,001 ms and explain the new result. Run an empty window and explain its undefined rate.

Write an action policy for budget exhaustion. Do not select an objective merely because it is common. Explain what actual business and workload evidence would justify it.

## 3. Model capacity, then state what is unmeasured

The queue admits arrivals before serving each discrete interval. For arrivals [20, 8, 0], capacity 10 and limit 15, it rejects five, completes 23 and ends empty. Inspect each interval; account for every arrival.

Double arrivals and then capacity. Change the queue limit. Verify `arrived = completed + rejected + remaining`. Identify the missing real-world variables: processing distribution, tail latency, database contention, memory, tool-call fanout and dependency quotas. Describe an authorized representative load test you would need before sizing infrastructure. Do not present this arithmetic as a benchmark.

Save this as `capacity_experiment.py` beside the lab and run `python3 capacity_experiment.py`:

```python
import runpy
lab = runpy.run_path("fde-service-operations.py")
for capacity in [0, 5, 10]:
    result = lab["queue_model"]([7, 9], capacity, 5)
    print(capacity, result["completed"], result["rejected"], result["remaining"])
```

The columns are capacity, completed, rejected and remaining. Explain why raising capacity above the queue limit does not admit more of a burst under this model's admission-before-service assumption. A different ordering model would produce different results.

## 4. Inspect a release decision

Call `release_decision` with a failed critical check, an empty reliability window and unproven recovery compatibility. Each must hold. Even the eligible result requires an accountable review; this function neither deploys nor grants approval.

In a real project's documented release path, identify the reviewed commit, dependency lock, SBOM, artifact digest, trusted provenance/signature, relevant checks, installation identity and live acceptance. Explain the difference between a checksum and trusted provenance. Do not place credentials in the packet.

## 5. Rehearse an incident

Scenario: a worker returns to service after a timeout period, but some requests may already have produced effects. Assign response and communication owners. State user impact, containment, reconciliation and safe resumption criteria. Preserve operation IDs; do not resubmit everything with new identities.

Write two updates: one when processing resumes and one when uncertain operations are reconciled. Make the distinction clear. Record an evidence-based timeline and a corrective action with an owner and verification.

## 6. Transfer ownership

Have a receiver find the signal, identify the affected operation, choose a mitigation and verify recovery with their actual permitted role. Record missing permissions or undocumented assumptions. Connect this review to the production-systems recovery lab and your chosen cloud environment. A classroom simulation is preparation for an authorized environment, not evidence that production has been recovered.

References: [Google SRE SLOs](https://sre.google/workbook/implementing-slos/), [OpenTelemetry signals](https://opentelemetry.io/docs/concepts/signals/), [SLSA 1.2](https://slsa.dev/spec/v1.2/), [NIST SSDF](https://csrc.nist.gov/pubs/sp/800/218/final), [DORA capabilities](https://dora.dev/capabilities/). Checked 2026-10-23.