# AI gateways: control a model request before it leaves

An **AI gateway** is a service that sits between an application and AI backends.
It receives requests, applies traffic policies and forwards allowed work to a
chosen backend. Think of a reverse proxy with policies for model traffic. It is
not a model, an agent framework, or permission to read every customer record.

This original LockedIn FDE lab uses one fictional maintenance assistant to make
that boundary concrete. Start with [Azure API Management explained](/training/fde-101?topic=apim#concept)
or [Kong AI Gateway explained](/training/fde-101?topic=kong#concept) for a visual introduction.
Allow about 55 minutes for the explanation, local experiment and decision packet.

The Python exercise runs offline with the standard library. It models one
sequential, in-memory budget window using supplied identities and token counts.
It does not authenticate callers, run a model, implement either vendor's policy,
or provide production middleware. No cloud account, credentials or purchase is needed.

## 1. Put the gateway in the right place

The fictional North maintenance team uses an assistant that drafts a response.
Another team, South, uses the same model service. North has a 1,500-token exercise
budget; South has 5,000. Those are synthetic limits, not provider prices or quotas.

Follow the request through five responsibilities:

| Step | Responsibility | What crosses the boundary |
| --- | --- | --- |
| 1. Application | Verify the caller and authorize the source records | A trusted identity and permitted context |
| 2. Gateway | Select an approved route and check its budget | A bounded model request |
| 3. Model service | Generate within the selected request limits | Output, completion state and usage when available |
| 4. Application | Validate the result and check citations | A draft with explicit evidence or an error |
| 5. Operator | Accept, revise or reject the proposed business action | A separate human decision |

The gateway's decision is about a particular request. It does not replace
document-level authorization, a model-quality evaluation, or approval to send a
message. Draw a line around each responsibility. Explain where a model-suggested
tenant name would be rejected before following any commands.

## 2. Run the admission experiment

Download [fde-ai-gateway-lab.py](/templates/fde-ai-gateway-lab.py) to an empty working folder.
Use Python 3.11 or later. Check the version first; if your computer uses a
versioned command such as `python3.11`, use that command throughout.

```shell
python3 --version
python3 fde-ai-gateway-lab.py test
python3 fde-ai-gateway-lab.py compare
python3 fde-ai-gateway-lab.py compare --candidate prompt-only
```

Both candidates see **two overlapping requests** before either response arrives.
Each request has 400 input tokens and may generate up to 600 output tokens.
Predict the decisions before reading the JSON results.

| Candidate | First request | Second request | Maximum admitted use |
| --- | --- | --- | --- |
| reserve-output | Reserve 1,000; 500 remain | Refuse: needs another 1,000 | 1,000 |
| prompt-only, intentionally broken | Reserve 400 | Reserve another 400 | 2,000 against a 1,500 budget |

The broken candidate reports 700 tokens remaining while it has admitted work
that could exceed the budget. This is the distinction between **accounted
reservation** and **possible usage**. A successful HTTP response or a green test
of one request would not reveal the overlapping-work defect.

Change North's synthetic budget to 2,000 in your copy, rerun both candidates,
then restore it. Both can admit two requests, but only one has reserved their
full envelope. A passing scenario is not a general proof of the implementation.
Keep the original result when recording your changed result.

## 3. Keep identity, context and quota separate

```shell
python3 fde-ai-gateway-lab.py demo
```

Read `first`, then `forged_tenant_hint`. The second request asks to use South's
identity in its body. The exercise still charges North because `verified_tenant`
comes from the supplied trusted fixture. South retains all 5,000 tokens.
In a real service, that fixture must be replaced by authenticated server context;
accepting a caller's arbitrary tenant argument would defeat the entire boundary.

Now distinguish three controls:

| Control | The question it answers | What it cannot establish |
| --- | --- | --- |
| Context limit | Can this single request fit the selected model's input/output contract? | Whether the team has enough total budget |
| Request rate | How many calls may arrive in a time window? | The cost or token size of those calls |
| Token budget | How much usage may this scope consume? | Whether the requested source record belongs to this caller |

The lab checks context and a single-window token budget. It does not implement a
clock or a request-rate limiter. Input counts are supplied assumptions; a real
integration needs the target model's counting method and its documented limits.
A token is not a universal word count, a dollar amount or a byte of GPU memory.

## 4. Do not turn missing usage into zero

The demonstration reserves 1,000 tokens. `missing_usage` has no provider receipt,
so 500 remain and another 1,000-token request is refused. A timeout does not prove
that no inference occurred. The conservative reservation stays until reconciled.

Next, a supplied receipt reports **550 actual tokens**. `confirmed_receipt`
replaces the reservation with that usage, leaving 950. A smaller request for
400 input plus 500 output tokens fits and leaves 50. Repeating the same receipt
does not charge or release it twice; a conflicting receipt is rejected.
Usage beyond the reserved envelope or a conflicting settled receipt freezes
new admissions for that team. This exercise has no automatic release operation;
the inconsistency requires investigation rather than a convenient lower number.

These receipts are synthetic observations, not actual provider measurements.
Each new admission is a new request in this exercise; it does not deduplicate
client retries or provide provider-side idempotency. Write how your real service
would reconcile a lost response without starting a duplicate request. Include
who investigates an impossible receipt and what stops more work while it is unresolved.

## 5. Refuse a convenient but unapproved fallback

In `unapproved_fallback`, the alternate model route is known, but its processing
location is outside North's allowed set. The result is `location_not_allowed`.
A healthy backup endpoint is not automatically an approved destination.

An approved location in this fixture is a label in a teaching policy, not proof
of a provider's processing, logging or retention behavior. For a real workload,
verify the exact offering and agreement. The backend still checks resource access.

Add a second approved route to your own copy. Test both legitimate fallback and
the refused route. Do not remove the refusal merely to make every test green.
Keep request metadata in the comparison; do not add real prompts or credentials.

## 6. Translate the reasoning into APIM or Kong

This exercise's reservation algorithm is original teaching code. It is not a
claim that a gateway product enforces the same algorithm or a perfectly global cap.
Use current documentation for the actual deployment, version and traffic format.

**Azure API Management.** The documented `llm-token-limit` policy supports
per-key token rates and quotas. Prompt estimation, streaming and concurrent
requests affect counting; concurrent work may exceed a configured limit before
responses update it. The documentation also distinguishes accounting across
gateway instances. Choose the key from trusted context and verify the applicable
tier and backend schema. See [Microsoft's policy reference](https://learn.microsoft.com/en-us/azure/api-management/llm-token-limit-policy).

**Kong.** The current AI Rate Limiting Advanced policy documents local, cluster
and Redis counter strategies, with topology and synchronization tradeoffs.
The AI Gateway 2.0 policy and the older Gateway plugin are different versioned
surfaces; verify the exact offering rather than copying an unrelated example.
See the [current policy](https://developer.konghq.com/ai-gateway/policies/ai-rate-limiting-advanced/)
and [Gateway plugin documentation](https://developer.konghq.com/plugins/ai-rate-limiting-advanced/).
These references were checked on 20 October 2026.

Create a small mapping table with: requirement, product/version, documented
mechanism, deployment scope, evidence still needed and test case. Include caller
identity, allowed backends, processing location, token accounting, counter sharing,
streaming, missing usage and overload. Leave unsupported or untested rows unknown.
Neither reading a policy page nor running this offline lab proves a cloud configuration.

## 7. Produce a reviewable gateway decision

Keep the original comparison, your one changed scenario, and the provider mapping.
Add a diagram showing where trusted identity enters and where resource access remains.
Write one useful request that must pass, one forged identity hint that must fail,
one overlapping-budget case and one forbidden fallback. State what observations
would make you change the policy rather than weakening an expected result.

Ask a peer to explain the difference between a model, an agent and a gateway using
your diagram. Then ask them to predict the second admission before running it.
If the explanation does not support that prediction, repair the explanation first.

Continue with the existing [governed model policy lesson](/app/academy?course=enterprise-ai-operations&lesson=governed-control-plane&view=focus)
using your training access. The next engineering question is whether the budget
survives multiple workers, process loss and window boundaries. This local lab
deliberately leaves those as a transfer design; it is not a distributed gateway.