# Retry amplification: count the calls that actually arrive

Original LockedIn Labs proof workshop · 50 planned minutes, excluding setup · Python 3.11+ standard library. A caller contacts one local service, which contacts a second local provider fixture. Actual HTTP GETs, query validation and handler observations execute on loopback. There is no SDK, external provider, model, business write or load benchmark.

## 1. Separate one request from its attempts

A retry sends another attempt after a result the client considers recoverable. An attempt limit includes the initial call: three total attempts allow two retries. When a caller permits three service attempts and that service permits three provider attempts per request, one failed lookup can become nine provider requests.

The number is a bound under the stated behavior, not a universal prediction. A success or a non-retryable result can stop a loop early. Another hidden SDK, proxy or queue retry layer can change the bound. Start by identifying who owns each loop and which event consumes an attempt.

This workshop compares a deliberately broken policy with a bounded single-owner candidate. The broken policy retries every non-200 response in both layers. The candidate lets the caller alone retry a received 503, and each service request makes one provider call. This is a small read-only teaching contract, not a production retry policy to paste into a client.

## 2. Follow the two actual HTTP hops

The runner binds two owned servers to `127.0.0.1` on temporary ports. A normal request carries `GET /lookup?item=synthetic`. The provider validates that query. Invalid cases omit `item`, so its actual query check returns 400. Always-unavailable cases return a supplied 503. Transient cases return 503 for the configured number of provider requests, then 200.

Caller-attempt and service-attempt headers correlate the local observations; they are not credentials or a distributed budget. The service forwards the unchanged query and actual status. Each provider handler records the request it received. The servers and their threads are shut down and closed after each independently reset case.

The browser mechanism is an authored explanation. Your runner report supplies the actual request evidence. No request is sent by opening this page.

## 3. Predict and run six comparisons

Download [the standalone Python runner](/templates/fde-http-retries.py), inspect it and run in a disposable folder:

```sh
python3 --version
python3 fde-http-retries.py > retries-original.json
python3 fde-http-retries.py --case nested-unavailable --attempts 2 > retries-nested-two.json
python3 fde-http-retries.py --case single-owner-transient --attempts 2 --failures 1 > retries-recovered.json
```

Use `py -3` on Windows if that is your launcher. No packages or keys are needed. The runner accepts only named cases, attempts from 1 through 4 and transient failures from 0 through 8. It accepts no user-selected host, URL or port and does not use an inherited HTTP proxy. These deliberately immediate retries stay inside the two owned local servers.

Predict both hop counts before opening the default report:

| Case | Caller → service requests | Service → provider requests | Final caller status |
| --- | --- | --- | --- |
| `nested-unavailable` | 3 | 9 | 503 |
| `single-owner-unavailable` | 3 | 3 | 503 |
| `nested-invalid` | 3 | 9 | 400; unchanged missing item repeated |
| `single-owner-invalid` | 1 | 1 | 400; no second attempt |
| `nested-transient` | 1 | 3 | 200 after two provider failures |
| `single-owner-transient` | 3 | 3 | 200 after two provider failures |

The final two cases have the same provider count, with different retry ownership. A successful final status does not reveal how much work occurred or where it repeated. Reducing attempts bounds extra work; it does not repair an unavailable dependency.

## 4. Inspect the requests rather than just the final status

Read `caller_attempts`, `service_requests`, `provider_requests`, `counts` and `final_http_status`. In nested-unavailable, group the provider records by `caller_attempt`: each outer attempt contains service attempts 1, 2 and 3. These are actual received GETs, not nine entries generated from multiplication alone.

In nested-invalid, inspect `path` and status. Every forwarded query still lacks `item`; repeating it cannot satisfy the provider's unchanged validation rule. The candidate stops on that received 400 because it classifies the status separately from choosing a retry owner. A single owner that still retries every error could continue repeating the same invalid input. This is not a malformed HTTP parser test or a claim that every 4xx status has the same recovery contract.

In transient cases, the provider's availability condition is supplied fixture behavior: after two failed valid requests, its next valid request returns found. Compare the nested service's one incoming request with the single-owner service's three. Do not infer a throughput or latency improvement from those counts; the experiment measures neither production performance nor model cost.

Check `owned_servers`: both should bind to loopback, have different ports and end with `stopped: true`. The report is emitted after cleanup. Unexpected transport or handler errors fail the run rather than being silently converted into a retryable 503. The selected cases exercise received statuses; lost responses and in-flight cancellation require separate evidence.

## 5. Change the limit and the recovery condition separately

With `--attempts 2`, always-unavailable nested retry produces four provider requests; the single-owner case produces two. With the default two transient failures, two attempts at the caller-owned layer stop at 503 before the third provider call can succeed. The nested two-by-two policy can reach the third provider request during its second outer attempt and return 200. That later success does not erase its different work budget.

Then hold attempts at two and change only `--failures 1`: the caller-owned transient case returns 200 after two provider requests. With failures zero it succeeds immediately. Keep original and changed reports separate, including the prediction you made before each run.

For logical deadline and shared-budget reasoning, return to the [retry-budget model](/training/guides/integration-engineering#4a-run-and-change-the-retry-budget). Its delays and deadlines are supplied logical values. This new HTTP runner does not implement a shared cross-service budget, end-to-end deadline, backoff wait, jitter, Retry-After handling, cancellation or a fleet load test. The two exercises answer different bounded questions.

## 6. Give another engineer the receiving task

Allow ten minutes for receiving review. One person retains the default report. Another predicts and runs nested-unavailable with two attempts, then locates four actual provider requests. Swap roles for single-owner-invalid and the changed transient condition. Each person keeps their own command, prediction, received evidence, assistance and correction. Solo practice is self-review.

The application owner identifies its own loop and the useful request contract. The service/SDK owner inventories retries already performed by adapters or libraries. The dependency owner supplies retryable failure meaning, safe operation semantics, overload behavior and any waiting instructions. The pod lead retains the bounded work limit and the next authorized environment test. None inherits another owner's permissions or infers client readiness from this exercise.

Use a short receiving note:

```text
Synthetic participant / exact runner and Python version:
Prediction / command / original or changed condition:
Actual caller-to-service count / provider count / final status:
Provider request records supporting that count:
My own action / assistance / first explanation and correction:
Actual retry owner / limit / error classification still to verify:
Application owner / service or SDK owner / dependency owner:
Next bounded test / receiving role / evidence still missing:
```

## 7. Defend the policy in an interview

- **Why can two three-attempt loops send nine provider requests?** Each outer attempt starts a fresh inner loop. Both continue under the deliberately broad failure rule in that case.
- **Does a single retry owner guarantee success?** No. It can bound extra work while still returning 503, or stop before a later recovery. Its placement and limits need workload evidence.
- **Why not retry the missing-item response?** The same invalid query cannot pass the unchanged validation rule. Correct the request under its actual contract; do not generalize all error statuses from this one case.
- **Does backoff remove retry amplification?** Waiting can change arrival timing, but it does not erase requests already sent. Backoff, jitter, budgets and ownership need separate design and tests.
- **Can this be used for an uncertain payment or tool action?** This exercise performs only synthetic reads. For side effects, use the operation's authorization, idempotency and reconciliation contract before another attempt; start with the [HTTP outcome workshop](/training/guides/http-outcomes).

## Primary references

[Amazon's retry/backoff guidance](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/) discusses layered amplification and retry placement. [Google SRE overload handling](https://sre.google/sre-book/handling-overload/) distinguishes request and client retry budgets. [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-503-service-unavailable) defines the HTTP status semantics; the fixture supplies the availability condition. Checked 29 October 2026. The code, cases and receiving exercise are original LockedIn Labs materials, not a vendor SDK or performance claim.