Free learning / 55-minute gateway lab
AI gateways: control a model request before it leaves
An AI gateway sits between your application and model services. Learn what it controls, then test identity, token budgets and approved routing before mapping them to Azure API Management or Kong.
No account needed. Use your own copy for writing and keep the original as a reference.
Engineering lab library / Run locally
Follow a model request through its controls.
Run the local policy exercise, compare an unsafe budget with full reservations, and explain each refusal before mapping it to a gateway product.
Synthetic records. No provider key or model inference. Runs locally with the standard library.
Start here / Python 3.11+ · Standard library
AI gateway admission and token budgets
Compare overlapping reservations, reject forged budget hints and refuse an unapproved fallback.
Python 3.11+ standard library · Offline synthetic requests
An AI gateway is a service that sits between an application and AI backends. It receives requests, applies traffic policies and forwards allowed work to a chosen backend. Think of a reverse proxy with policies for model traffic. It is not a model, an agent framework, or permission to read every customer record.
This original LockedIn FDE lab uses one fictional maintenance assistant to make that boundary concrete. Start with Azure API Management explained or Kong AI Gateway explained for a visual introduction. Allow about 55 minutes for the explanation, local experiment and decision packet.
The Python exercise runs offline with the standard library. It models one sequential, in-memory budget window using supplied identities and token counts. It does not authenticate callers, run a model, implement either vendor's policy, or provide production middleware. No cloud account, credentials or purchase is needed.
1. Put the gateway in the right place
The fictional North maintenance team uses an assistant that drafts a response. Another team, South, uses the same model service. North has a 1,500-token exercise budget; South has 5,000. Those are synthetic limits, not provider prices or quotas.
Follow the request through five responsibilities:
| Step | Responsibility | What crosses the boundary |
|---|---|---|
| 1. Application | Verify the caller and authorize the source records | A trusted identity and permitted context |
| 2. Gateway | Select an approved route and check its budget | A bounded model request |
| 3. Model service | Generate within the selected request limits | Output, completion state and usage when available |
| 4. Application | Validate the result and check citations | A draft with explicit evidence or an error |
| 5. Operator | Accept, revise or reject the proposed business action | A separate human decision |
The gateway's decision is about a particular request. It does not replace document-level authorization, a model-quality evaluation, or approval to send a message. Draw a line around each responsibility. Explain where a model-suggested tenant name would be rejected before following any commands.
2. Run the admission experiment
Download fde-ai-gateway-lab.py to an empty working folder. Use Python 3.11 or later. Check the version first; if your computer uses a versioned command such as python3.11, use that command throughout.
python3 --version
python3 fde-ai-gateway-lab.py test
python3 fde-ai-gateway-lab.py compare
python3 fde-ai-gateway-lab.py compare --candidate prompt-onlyBoth candidates see two overlapping requests before either response arrives. Each request has 400 input tokens and may generate up to 600 output tokens. Predict the decisions before reading the JSON results.
| Candidate | First request | Second request | Maximum admitted use |
|---|---|---|---|
| reserve-output | Reserve 1,000; 500 remain | Refuse: needs another 1,000 | 1,000 |
| prompt-only, intentionally broken | Reserve 400 | Reserve another 400 | 2,000 against a 1,500 budget |
The broken candidate reports 700 tokens remaining while it has admitted work that could exceed the budget. This is the distinction between accounted reservation and possible usage. A successful HTTP response or a green test of one request would not reveal the overlapping-work defect.
Change North's synthetic budget to 2,000 in your copy, rerun both candidates, then restore it. Both can admit two requests, but only one has reserved their full envelope. A passing scenario is not a general proof of the implementation. Keep the original result when recording your changed result.
3. Keep identity, context and quota separate
python3 fde-ai-gateway-lab.py demoRead first, then forged_tenant_hint. The second request asks to use South's identity in its body. The exercise still charges North because verified_tenant comes from the supplied trusted fixture. South retains all 5,000 tokens. In a real service, that fixture must be replaced by authenticated server context; accepting a caller's arbitrary tenant argument would defeat the entire boundary.
Now distinguish three controls:
| Control | The question it answers | What it cannot establish |
|---|---|---|
| Context limit | Can this single request fit the selected model's input/output contract? | Whether the team has enough total budget |
| Request rate | How many calls may arrive in a time window? | The cost or token size of those calls |
| Token budget | How much usage may this scope consume? | Whether the requested source record belongs to this caller |
The lab checks context and a single-window token budget. It does not implement a clock or a request-rate limiter. Input counts are supplied assumptions; a real integration needs the target model's counting method and its documented limits. A token is not a universal word count, a dollar amount or a byte of GPU memory.
4. Do not turn missing usage into zero
The demonstration reserves 1,000 tokens. missing_usage has no provider receipt, so 500 remain and another 1,000-token request is refused. A timeout does not prove that no inference occurred. The conservative reservation stays until reconciled.
Next, a supplied receipt reports 550 actual tokens. confirmed_receipt replaces the reservation with that usage, leaving 950. A smaller request for 400 input plus 500 output tokens fits and leaves 50. Repeating the same receipt does not charge or release it twice; a conflicting receipt is rejected. Usage beyond the reserved envelope or a conflicting settled receipt freezes new admissions for that team. This exercise has no automatic release operation; the inconsistency requires investigation rather than a convenient lower number.
These receipts are synthetic observations, not actual provider measurements. Each new admission is a new request in this exercise; it does not deduplicate client retries or provide provider-side idempotency. Write how your real service would reconcile a lost response without starting a duplicate request. Include who investigates an impossible receipt and what stops more work while it is unresolved.
5. Refuse a convenient but unapproved fallback
In unapproved_fallback, the alternate model route is known, but its processing location is outside North's allowed set. The result is location_not_allowed. A healthy backup endpoint is not automatically an approved destination.
An approved location in this fixture is a label in a teaching policy, not proof of a provider's processing, logging or retention behavior. For a real workload, verify the exact offering and agreement. The backend still checks resource access.
Add a second approved route to your own copy. Test both legitimate fallback and the refused route. Do not remove the refusal merely to make every test green. Keep request metadata in the comparison; do not add real prompts or credentials.
6. Translate the reasoning into APIM or Kong
This exercise's reservation algorithm is original teaching code. It is not a claim that a gateway product enforces the same algorithm or a perfectly global cap. Use current documentation for the actual deployment, version and traffic format.
Azure API Management. The documented llm-token-limit policy supports per-key token rates and quotas. Prompt estimation, streaming and concurrent requests affect counting; concurrent work may exceed a configured limit before responses update it. The documentation also distinguishes accounting across gateway instances. Choose the key from trusted context and verify the applicable tier and backend schema. See Microsoft's policy reference.
Kong. The current AI Rate Limiting Advanced policy documents local, cluster and Redis counter strategies, with topology and synchronization tradeoffs. The AI Gateway 2.0 policy and the older Gateway plugin are different versioned surfaces; verify the exact offering rather than copying an unrelated example. See the current policy and Gateway plugin documentation. These references were checked on 20 October 2026.
Create a small mapping table with: requirement, product/version, documented mechanism, deployment scope, evidence still needed and test case. Include caller identity, allowed backends, processing location, token accounting, counter sharing, streaming, missing usage and overload. Leave unsupported or untested rows unknown. Neither reading a policy page nor running this offline lab proves a cloud configuration.
7. Produce a reviewable gateway decision
Keep the original comparison, your one changed scenario, and the provider mapping. Add a diagram showing where trusted identity enters and where resource access remains. Write one useful request that must pass, one forged identity hint that must fail, one overlapping-budget case and one forbidden fallback. State what observations would make you change the policy rather than weakening an expected result.
Ask a peer to explain the difference between a model, an agent and a gateway using your diagram. Then ask them to predict the second admission before running it. If the explanation does not support that prediction, repair the explanation first.
Continue with the existing governed model policy lesson using your training access. The next engineering question is whether the budget survives multiple workers, process loss and window boundaries. This local lab deliberately leaves those as a transfer design; it is not a distributed gateway.