# See tokens before planning a model request

Original LockedIn Labs exercise · 35 minutes · Python 3.11+.

A tokenizer turns text into a sequence of IDs using an encoding's vocabulary and
rules. This practice makes that representation visible before you budget a
request. The examples are synthetic. No model, account, GPU or API key is needed.

## 1. Prepare a small local workspace

Download both files into a new folder:

- [Python exercise](/templates/fde-token-first-budget.py)
- [Pinned requirements](/templates/fde-token-first-budget-requirements.txt)

Read the file before running it. Use an approved Python installation, then:

```shell
python3 -m venv .venv
.venv/bin/python -m pip install -r fde-token-first-budget-requirements.txt
.venv/bin/python fde-token-first-budget.py prepare
.venv/bin/python fde-token-first-budget.py test
```

On Windows, create the environment with `py -3 -m venv .venv` and use
`.venv\Scripts\python.exe` instead of `.venv/bin/python`.

Installation uses the package registry. `prepare` downloads two public encoding
dictionaries and caches them in `.tokenizer-cache` inside the current folder.
Keep this folder in place. All other commands block network connections and use
the cache. If it is missing, run `prepare` again with network access before
continuing. No model is downloaded and no input text is sent to a model service.

The pinned tokenizer is tiktoken 0.14.0. Eight checks cover lossless reconstruction,
different encodings, partial UTF-8 characters and budget boundaries.

## 2. Predict the pieces, then inspect them

Before running the command, predict whether these will have the same number of
Unicode code points, bytes and tokens:

- `The pump needs inspection.`
- A code assignment with punctuation and a newline.
- `ポンプを点検してください。` — a request to inspect the pump.

```shell
.venv/bin/python fde-token-first-budget.py demo
.venv/bin/python fde-token-first-budget.py inspect --text "The pump needs inspection."
```

The English sentence has 26 code points and five tokens in both supplied
encodings. One token contains the leading space and `pump` together. An ID is an
entry in that encoding's vocabulary; its size is not a confidence score.

In the Japanese case, compare eleven tokens under `cl100k_base` with seven under
`o200k_base`. Some pieces in the first encoding contain only part of a UTF-8
character. Their `text` is null, and their exact bytes remain visible. Join all
token bytes to recover the original text. Decoding each partial piece as though
it were a complete character could lose information.

The code-point count is also not a count of visible letters in every language:
some visible characters are composed of multiple code points.

## 3. Change one thing

Inspect a new short sentence made of synthetic text. Keep the text identical and
change only the encoding:

```shell
.venv/bin/python fde-token-first-budget.py inspect --encoding cl100k_base --text "Inspect pump 104 before restart."
.venv/bin/python fde-token-first-budget.py inspect --encoding o200k_base --text "Inspect pump 104 before restart."
```

Record the input, encoding, token IDs and bytes. Counts may match while IDs or
boundaries differ. Now change one number or punctuation mark, rerun, and compare.
Do not generalize a single example into a universal words-to-tokens ratio.

## 4. Reserve room for an answer

This exercise uses a **hypothetical 128-token window**, 24 tokens of other input,
optional 80-token history and a chosen answer allowance. Only the current text
count is measured. These small numbers are teaching assumptions, not model specs.

```shell
.venv/bin/python fde-token-first-budget.py budget --history 0 --output 48
.venv/bin/python fde-token-first-budget.py budget --history 80 --output 48
.venv/bin/python fde-token-first-budget.py budget --history 80 --output 16
```

For the default five-token sentence, predict and then check:

| Text | Other input | History | Output allowance | Reserved | Result |
| --- | --- | --- | --- | --- | --- |
| 5 | 24 | 0 | 48 | 77 | 51 left |
| 5 | 24 | 80 | 48 | 157 | 29 over |
| 5 | 24 | 80 | 16 | 125 | 3 left |

Choose a repair for the oversized request. Explain whether removing irrelevant
history or shortening the answer is appropriate for the job. Preserve necessary
instructions, current source evidence and authorization boundaries.

An allowance is a maximum you plan for, not an observed output or a bill. An
agent task may make several requests, so a response cap alone cannot bound the
whole task's calls or spending.

## 5. Use the right counting method in a real service

These two encodings do not count Claude requests. They illustrate text
representation. A complete provider request can include message wrappers,
tool definitions, media and other components absent from this exercise.

Use the documented counting method for the actual provider and model, then
retain reported usage after a request. Keep unknown usage unknown. Do not infer
price, latency, quality or GPU requirements from this lab's small text counts.

- [tiktoken source and API](https://github.com/openai/tiktoken)
- [OpenAI's token-counting explanation](https://developers.openai.com/cookbook/examples/how_to_count_tokens_with_tiktoken)
- [Claude's message-token counting API](https://platform.claude.com/docs/en/build-with-claude/token-counting)

Sources checked 20 October 2026. Provider documentation is reference material;
this exercise is original instruction, not a provider certification.

## 6. Keep a small evidence packet

Keep your predictions, actual output, package version and one controlled change.
Add a budget table separating measured counts from assumed values. Finish with
two sentences: what a token is, and what is still missing before your count can
represent a complete production request.

Continue into context management in the protected Academy lesson or use the
existing NVIDIA capacity guide to see where workload reservations meet serving
constraints. A successful local exercise establishes the behavior you observed;
it does not establish provider billing or production readiness.