# Concurrency isolation: reserve room for useful work

Original LockedIn Labs proof workshop · 50 planned minutes, excluding preparation · Python 3.11+ standard library. Real concurrent GETs reach one owned loopback HTTP server. An explicit event gate holds admitted slow work while another request is observed. No external dependency, queue, business write, replica, throughput or latency benchmark is involved.

## 1. Name the resource being limited

Concurrency is how much work can occupy a resource at once. A **work permit** lets a request enter the section protected by that limit. A semaphore can enforce the number of permits. A bounded semaphore also detects releasing more permits than its configured capacity; it does not choose a correct capacity or workload policy for you.

If slow work occupies every shared permit, unrelated fast work may be refused by the same gate. **Bulkhead isolation** separates resources into groups so one class cannot consume the whole shared allocation. That reservation has a cost: one group can be full while another group's capacity is idle.

This lab compares two policies with the same two total work permits. Shared mode has one pool of two. Partitioned mode gives one to slow work and reserves one for fast work. These are work classes selected by fixture routes, not tenants, identities or permissions. Neither mode establishes a global resource limit across replicas.

## 2. Follow admission before assuming completion

The runner starts real HTTP callers for `S1` and `S2` at `/slow`. The handler tries to acquire a permit without waiting. An admitted request enters the protected work section and waits at an event gate. A refused request gets HTTP 503 with `work_capacity_full`; it never enters that section and is not queued for later execution.

A control observer inside the lab records admission and signals the coordinator. That is not an HTTP 202 or a successful response received by the slow caller. Admitted slow callers still wait for their response. Once those observations are established, the coordinator sends `F1` to `/fast` before releasing the slow gate.

After observing F1's actual response, the coordinator releases admitted slow work, joins its client threads, and explicitly sends new `S-after` and `F-after` requests. They are new probes after release, not automatic replay of rejected work. Permits leave the protected work section before response bytes are sent. All callers and the server are closed before the report is emitted.

The browser mechanism is an authored explanation. The downloadable runner supplies the actual admission events, concurrent pending callers and HTTP responses.

## 3. Predict and run the two policies

Download [the standalone runner](/templates/fde-concurrency-isolation.py), inspect it and run in a disposable folder:

```sh
python3 --version
python3 fde-concurrency-isolation.py > isolation-original.json
python3 fde-concurrency-isolation.py --mode shared --slots 3 > isolation-more-slots.json
python3 fde-concurrency-isolation.py --held 1 > isolation-less-held.json
```

Use `py -3` on Windows if that is your launcher. No package, key or cloud account is needed. The runner accepts only a named mode, total work slots from 2 through 4 and held slow requests from 1 through 4. It accepts no user-selected host, URL or port, and ignores inherited proxies.

Predict the default report before opening it:

| Policy | Admitted slow callers held before F1 | Second slow request | F1 before release | After explicit release |
| --- | --- | --- | --- | --- |
| Shared: two work permits | S1 and S2 | Admitted; HTTP response still pending | 503: both shared permits occupied | Both admitted callers finish 200; new slow/fast probes return 200 |
| Partitioned: one slow + one fast | S1 only | 503: slow pool full, even though fast permit is idle | 200: reserved fast work can enter | S1 finishes 200; new slow/fast probes return 200 |

This is an admission comparison, not proof that a dependency failed or that fast work meets a response-time objective. The held interval is controlled by an event, not a measured external service delay. Safety timeouts fail a stalled exercise; they are not the workload's finish-by contract.

## 4. Locate the actual evidence and the idle reservation

Read `pool_capacities`, `admissions`, `observer_before_fast`, `slow_callers_pending_before_release`, `fast_response_before_release`, `observer_after_fast_before_release`, `responses` and `work_events`.

In shared mode, the control observer sees two active slow work sections and two callers without completed responses. F1's refusal leaves that active work unchanged. In partitioned mode, it sees one active slow section and an actual refused S2 response before F1 is sent. One allocated permit is idle at that point, but it belongs to the fast class. S2 cannot borrow it under this policy. The original requests and their completion observations remain distinct.

Inspect the F1 admission/finish events, not just its final 200. Fast work briefly occupies its permit, releases it and returns while the slow gate is still closed. `observer_after_fast_before_release` still contains the held slow work. The eventual slow responses are recorded only after explicit release. A refusal did not cancel admitted work, and it did not create a queued job.

Finally inspect `final_state`, `client_threads_stopped` and `server`: active work ends at zero, the listener is closed and owned threads have stopped. Peak active work is an observed count inside this protected section. It is not the number of TCP connections, HTTP handler threads, CPU cores, memory units or jobs per second.

## 5. Change capacity separately from the held workload

Hold two slow requests but change total shared slots to three. S1 and S2 can still be held, and F1 can now enter the third permit. The success follows an increased allocation, rather than isolation at the original allocation. A real system needs evidence that its dependencies and resources support that extra work.

Then restore two slots and change only held requests to one. Both policies allow F1. Partitioning is not universally better under every workload. With four held requests and two slots, shared admits two slow requests and refuses two; partitioned admits one and refuses three, while retaining the fast permit. Explain the useful work and refused work together.

The semaphore limits only the declared work section. Python's threaded HTTP server can still create handler threads and accept connections before that check. This program therefore proves no bound on all server resources and provides no general denial-of-service protection. Actual ingress, connection/backlog limits, thread or event-loop behavior, memory, shared dependencies, fairness, deadlines, cancellation and durable waiting need their own design and tests. This is not a production server template.

For supplied waiting times, expiry and completion arithmetic, use the [queue timeline model](/training/guides/service-operations#3a-run-and-change-the-job-timeline). That model includes a queue and logical time. This HTTP lab has no waiting queue and measures no throughput or latency; the two sets of evidence do not substitute for each other.

## 6. Hand the receiving question to another engineer

Allow ten minutes for a receiving review. One person retains the default report; another predicts and runs shared mode with three slots, then locates actual F1 admission while S1/S2 remain held. Swap roles and change only held requests to one at the original two slots. Each keeps their own prediction, command, evidence pointer, assistance and correction. Solo practice is self-review.

The application owner identifies the work classes and which refusal is useful to a caller. The service/platform owner shows the exact protected resource and how many independent instances have their own limits. The dependency owner supplies real downstream capacity and cancellation behavior. The pod lead records an authorized measurement plan and the unresolved receiving question; a successful classroom request establishes no client-placement or production-capacity decision.

Use a short individual record:

```text
Synthetic participant / exact file and Python version:
Prediction / mode / slots / held requests:
Actual admission event / pending caller / response evidence:
Fast request before release / held work still present:
One changed input / new observation / original retained:
My own action / assistance / first explanation and correction:
Protected work resource / resource limits not tested:
Application / service-platform / dependency owner question:
Next bounded test / receiving role / evidence still needed:
```

## 7. Defend the tradeoff in an interview

- **Why was a fast request refused in shared mode?** Both work permits were occupied. Its 503 describes this admission gate, not a proven downstream outage.
- **Why refuse S2 when the partitioned system still has an idle permit?** That permit is reserved for another class. Isolation trades flexible use of idle capacity for a declared boundary.
- **Did the observer's admission event mean the slow caller already got 200?** No. It was an internal control observation; the real response remained pending until work was released.
- **Does this bound HTTP handler threads or memory?** No. It bounds one work section after request arrival. Other resource and ingress limits remain separate.
- **Can two replicas each use this semaphore to enforce one global two-slot limit?** Not by themselves. These are process-local permits; distributed admission and recovery require another contract and evidence.

## Primary references

[Microsoft's bulkhead pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/bulkhead) describes resource partitioning and its tradeoffs. [Python bounded semaphores](https://docs.python.org/3.11/library/threading.html#threading.BoundedSemaphore) define the local synchronization primitive. [Google SRE overload handling](https://sre.google/sre-book/handling-overload/) discusses resource restrictions and the limits of request-rate proxies. Checked 29 October 2026. The code, cases and receiving exercise are original LockedIn Labs materials, not a vendor implementation or performance claim.