Free learning / 35-minute tokenizer lab

See tokens before planning a model request

Inspect actual text-to-ID conversion, compare encoding dictionaries and reserve room for an answer in a small local exercise.

Begin reading Download a working copy

No account needed. Use your own copy for writing and keep the original as a reference.

Engineering lab library / Run locally

Turn text into tokens you can inspect.

Run a real tokenizer, compare its dictionaries and plan a small request budget. Prepare the dictionaries once; the exercise then runs without network access or a model call.

Synthetic records. No provider key or model inference. Package setup needs package-registry access.

Start here / Real library, SDK or framework · Scripted inputs
First tokenization and request budget
Reproduce real token boundaries and compare fit with over-capacity before any model call.
Python 3.11+ · Pinned packages · Cache dictionaries once
Open walkthrough Download file
Keep both files together.
This lab also needs fde-token-first-budget-requirements.txt. Use the original filenames in the same folder.
Download required workflow

Original LockedIn Labs exercise · 35 minutes · Python 3.11+.

A tokenizer turns text into a sequence of IDs using an encoding's vocabulary and rules. This practice makes that representation visible before you budget a request. The examples are synthetic. No model, account, GPU or API key is needed.

1. Prepare a small local workspace

← Back to lab library Download First tokenization and request budget ↓

Download both files into a new folder:

Read the file before running it. Use an approved Python installation, then:

Code example

python3 -m venv .venv
.venv/bin/python -m pip install -r fde-token-first-budget-requirements.txt
.venv/bin/python fde-token-first-budget.py prepare
.venv/bin/python fde-token-first-budget.py test

On Windows, create the environment with py -3 -m venv .venv and use .venv\Scripts\python.exe instead of .venv/bin/python.

Installation uses the package registry. prepare downloads two public encoding dictionaries and caches them in .tokenizer-cache inside the current folder. Keep this folder in place. All other commands block network connections and use the cache. If it is missing, run prepare again with network access before continuing. No model is downloaded and no input text is sent to a model service.

The pinned tokenizer is tiktoken 0.14.0. Eight checks cover lossless reconstruction, different encodings, partial UTF-8 characters and budget boundaries.

2. Predict the pieces, then inspect them

Before running the command, predict whether these will have the same number of Unicode code points, bytes and tokens:

The pump needs inspection.
A code assignment with punctuation and a newline.
ポンプを点検してください。 — a request to inspect the pump.

Code example

.venv/bin/python fde-token-first-budget.py demo
.venv/bin/python fde-token-first-budget.py inspect --text "The pump needs inspection."

The English sentence has 26 code points and five tokens in both supplied encodings. One token contains the leading space and pump together. An ID is an entry in that encoding's vocabulary; its size is not a confidence score.

In the Japanese case, compare eleven tokens under cl100k_base with seven under o200k_base. Some pieces in the first encoding contain only part of a UTF-8 character. Their text is null, and their exact bytes remain visible. Join all token bytes to recover the original text. Decoding each partial piece as though it were a complete character could lose information.

The code-point count is also not a count of visible letters in every language: some visible characters are composed of multiple code points.

3. Change one thing

Inspect a new short sentence made of synthetic text. Keep the text identical and change only the encoding:

Code example

.venv/bin/python fde-token-first-budget.py inspect --encoding cl100k_base --text "Inspect pump 104 before restart."
.venv/bin/python fde-token-first-budget.py inspect --encoding o200k_base --text "Inspect pump 104 before restart."

Record the input, encoding, token IDs and bytes. Counts may match while IDs or boundaries differ. Now change one number or punctuation mark, rerun, and compare. Do not generalize a single example into a universal words-to-tokens ratio.

4. Reserve room for an answer

This exercise uses a hypothetical 128-token window, 24 tokens of other input, optional 80-token history and a chosen answer allowance. Only the current text count is measured. These small numbers are teaching assumptions, not model specs.

Code example

.venv/bin/python fde-token-first-budget.py budget --history 0 --output 48
.venv/bin/python fde-token-first-budget.py budget --history 80 --output 48
.venv/bin/python fde-token-first-budget.py budget --history 80 --output 16

For the default five-token sentence, predict and then check:

Text	Other input	History	Output allowance	Reserved	Result
5	24	0	48	77	51 left
5	24	80	48	157	29 over
5	24	80	16	125	3 left

Choose a repair for the oversized request. Explain whether removing irrelevant history or shortening the answer is appropriate for the job. Preserve necessary instructions, current source evidence and authorization boundaries.

An allowance is a maximum you plan for, not an observed output or a bill. An agent task may make several requests, so a response cap alone cannot bound the whole task's calls or spending.

5. Use the right counting method in a real service

These two encodings do not count Claude requests. They illustrate text representation. A complete provider request can include message wrappers, tool definitions, media and other components absent from this exercise.

Use the documented counting method for the actual provider and model, then retain reported usage after a request. Keep unknown usage unknown. Do not infer price, latency, quality or GPU requirements from this lab's small text counts.

Sources checked 20 October 2026. Provider documentation is reference material; this exercise is original instruction, not a provider certification.

6. Keep a small evidence packet

Keep your predictions, actual output, package version and one controlled change. Add a budget table separating measured counts from assumed values. Finish with two sentences: what a token is, and what is still missing before your count can represent a complete production request.

Continue into context management in the protected Academy lesson or use the existing NVIDIA capacity guide to see where workload reservations meet serving constraints. A successful local exercise establishes the behavior you observed; it does not establish provider billing or production readiness.