Free learning / Operations workshop
Reliable services & operational ownership — working copy
Calculate an SLO, model bounded queues and rehearse incident recovery with explicit operating responsibility.
No account needed. Use your own copy for writing and keep the original as a reference.
Engineering lab library / Run locally
Choose your next agent lab.
Start with the durable workflow reference, then choose the boundary you want to test. Open its walkthrough for setup, commands and expected observations.
Synthetic records. No provider key or model inference. Runs locally with the standard library.
Start here / Python 3.11+ · Standard library
Reliability and queue lab
Calculate a missed objective and account for bounded work.
Python 3.11+ · Synthetic arithmetic and simulation · No load test or deployment
Original LockedIn Labs practice. Python 3.11+, no dependencies or network access. This lab calculates synthetic observations and simulates a queue. It is not a load test, monitoring integration, deployment or release authorization.
Curriculum: https://training.lockedinlabs.ai/training/curriculum?stage=5#operations
Download and inspect the Python lab, then run:
python3 --version
python3 fde-service-operations.py1. Follow one request
Map DNS, TLS, routing, identity, runtime, database, queue and external model API for your project. Mark persisted state and each independent owner. Explain what a VM, container and service each provide. Define readiness and liveness separately. Choose a user-outcome signal, a metric, a log event and a trace boundary; exclude secrets and sensitive payloads.
Illustrative learning artifact
Reliability belongs to the user task
Two beyond the allowance. Investigate the failed work.
2. Define reliability
SLI: eligible requests that succeed within 2,000 ms / all eligible requests. Record measurement location, time window and exclusions. Semantic correctness and authorization are separate requirements.
The fixture contains 988 good requests and twelve failed requests. At a 99% target across 1,000 requests, the bad-event allowance is ten; the remaining budget is -2. Change one successful request to take 2,001 ms and explain the new result. Run an empty window and explain its undefined rate.
Write an action policy for budget exhaustion. Do not select an objective merely because it is common. Explain what actual business and workload evidence would justify it.
3. Model capacity, then state what is unmeasured
The queue admits arrivals before serving each discrete interval. For arrivals [20, 8, 0], capacity 10 and limit 15, it rejects five, completes 23 and ends empty. Inspect each interval; account for every arrival.
Double arrivals and then capacity. Change the queue limit. Verify arrived = completed + rejected + remaining. Identify the missing real-world variables: processing distribution, tail latency, database contention, memory, tool-call fanout and dependency quotas. Describe an authorized representative load test you would need before sizing infrastructure. Do not present this arithmetic as a benchmark.
Save this as capacity_experiment.py beside the lab and run python3 capacity_experiment.py:
import runpy
lab = runpy.run_path("fde-service-operations.py")
for capacity in [0, 5, 10]:
result = lab["queue_model"]([7, 9], capacity, 5)
print(capacity, result["completed"], result["rejected"], result["remaining"])The columns are capacity, completed, rejected and remaining. Explain why raising capacity above the queue limit does not admit more of a burst under this model's admission-before-service assumption. A different ordering model would produce different results.
4. Inspect a release decision
Call release_decision with a failed critical check, an empty reliability window and unproven recovery compatibility. Each must hold. Even the eligible result requires an accountable review; this function neither deploys nor grants approval.
In a real project's documented release path, identify the reviewed commit, dependency lock, SBOM, artifact digest, trusted provenance/signature, relevant checks, installation identity and live acceptance. Explain the difference between a checksum and trusted provenance. Do not place credentials in the packet.
5. Rehearse an incident
Scenario: a worker returns to service after a timeout period, but some requests may already have produced effects. Assign response and communication owners. State user impact, containment, reconciliation and safe resumption criteria. Preserve operation IDs; do not resubmit everything with new identities.
Write two updates: one when processing resumes and one when uncertain operations are reconciled. Make the distinction clear. Record an evidence-based timeline and a corrective action with an owner and verification.
6. Transfer ownership
Have a receiver find the signal, identify the affected operation, choose a mitigation and verify recovery with their actual permitted role. Record missing permissions or undocumented assumptions. Connect this review to the production-systems recovery lab and your chosen cloud environment. A classroom simulation is preparation for an authorized environment, not evidence that production has been recovered.
References: Google SRE SLOs, OpenTelemetry signals, SLSA 1.2, NIST SSDF, DORA capabilities. Checked 2026-10-23.