Model 101: from learned patterns to generated text

A large language model (LLM) is a trained neural network: software whose many numerical parameters were adjusted using data. A language-generating model uses those numbers and its current input to calculate scores for possible next tokens. A token is a unit of encoded content; it need not be a whole word. The surrounding program chooses a token, adds it to the working sequence and continues until a stopping condition. A model is neither a programming language nor an agent framework.

Think of a function with two inputs: a saved parameter set and this request's working context. Training changes the parameter set. Ordinary inference uses it to calculate an output. Giving the function a different request can change the output without rewriting the saved parameters. This analogy does not mean the model is a database lookup or that its output has been verified.

An FDE needs these distinctions to diagnose the right part of a system. A stale procedure may require better retrieval and source checks. An output limit may need a smaller task. Neither observation alone demonstrates a need to retrain a model. Provider logging, retention and any later training use are separate service policies; “inference does not update these weights” is not a privacy promise.

The lab trains a word-bigram model. “Bigram” means a pair of successive tokens. It counts which word follows another in five fictional phrases, then turns those counts into a probability distribution. It splits on spaces and uses a special <end> marker. Real LLM tokenizers use richer vocabularies.

Its entire training collection is:

the pump needs inspection
the pump needs inspection
the pump needs inspection
the pump needs repair
the valve needs repair

After needs, the learned table has three inspection occurrences and two repair occurrences. At temperature 1, these become 3/5 and 2/5. A probability distribution allocates a total of 1 across possible next tokens. It describes this model's next-token calculation, not confidence that a real pump needs work.

This is an actual small statistical model, not an LLM or a miniature Transformer. It remembers only the immediately previous word; it has no neural layers, attention, embeddings, instruction following or factual reasoning. A generative Transformer computes with learned neural weights and representations of a much larger permitted context. Causal attention lets positions use earlier context positions; it does not grant access to an external database. The toy isolates training, prediction and selection so you can explain those jobs before studying the larger architecture.

Download fde-model-first-inference.py into your practice folder. Use Python 3.11 or later; no packages, model download, GPU, key or account are required. The program runs in memory, writes no files and makes no network calls.

python3 --version
python3 fde-model-first-inference.py compare
python3 fde-model-first-inference.py sampled

Before running, predict the continuation, stop reason and whether the model's parameters changed. Then compare your prediction with trace, parameters_before, parameters_after and model_changed_from_base.

The fingerprint summarizes the learned count table to make equality visible. It is not a credential or security attestation. Each inference result should show identical before/after fingerprints, including inference on the separately retrained model. Training changes a model; generation does not mutate it here.

Read the train, distribution and generate functions in that order.

1. train counts adjacent word pairs. Repeated examples affect the table.
2. distribution turns the relevant row into probabilities. At temperature 1, these are the relative counts. The code uses logarithms and softmax, which converts scores into nonnegative values that sum to 1.
3. generate either selects the largest probability (greedy decoding) or takes a pseudo-random draw from the distribution (sampling).
4. The chosen word becomes the next context word. The loop stops on <end>, an absent transition or the explicit step limit.

For sampling, temperature rescales scores before normalization. A smaller positive temperature concentrates mass on stronger candidates; it does not verify a claim or make all other tokens impossible. The lab's temperature 0.5 turns the 3:2 counts into normalized 9:4 masses. With seed 0 the first draw is about 0.844, so it still lands on repair. Greedy is a separate mode; do not divide by zero. Production parameter names, supported settings and repeatability depend on the selected model and serving system.

The earlier-context and unseen cases reveal limitations of this specific toy. Do not conclude that a modern LLM ignores all but one word or always refuses unfamiliar input. It may instead generate a fluent but unsupported answer.

Make a copy of the file before editing. Keep predictions and actual results.

Change A — selection without training. Change SAMPLE_SEED = 0 to SAMPLE_SEED = 1 and run sampled. The continuation becomes inspection. The counts, probabilities and parameter fingerprint stay the same. Explain why two outputs do not require two models. Restore the seed before the next edit.

Change B — training without changing the prompt. Change EXTRA_REPAIR_EXAMPLES = 4 to EXTRA_REPAIR_EXAMPLES = 0 and run retrained. The learned table now equals the base model; model_changed_from_base is false and the greedy continuation returns to inspection. Explain which data changed the earlier prediction. Repetition here is a transparent teaching input, not a recommended production fine-tuning strategy.

Do not judge a model from these seven examples. This experiment verifies the specified mechanics, not language quality, reliability or workplace performance.

Your fictional assistant receives a new approved procedure: P-17 revision 3 requires a human review before a draft is sent. A generated draft omits that review. First draw where P-17 enters context, which model/version is called, where the output is checked and which application code controls sending.

Choose an observation before proposing a change:

• If P-17 never entered the request, inspect retrieval and context construction.
• If P-17 was present but the draft contradicts it, retain the case and evaluate the prompt/model combination. A low temperature is not a factuality control.
• If the draft ends early, inspect the recorded completion reason and allowance.
• If text proposes sending, the application still has to enforce permission and review. A model's words do not execute that action.

Produce a one-page explanation with the parameter set, context, decoding choice, observed result and application boundary. State what evidence would justify a training experiment and how you would compare it on held-out tasks. “We trained it” is not a substitute for evaluation.

Keep your seven predictions, actual traces, both changed-copy outputs and the one-page design. Explain the same mechanism to a colleague without running the program. They should be able to point to what changed and what stayed fixed.

Continue with real tokenization and request budgeting, then context construction and agent engineering. The protected Model 101 lesson adds knowledge checks and a transfer discussion under existing Academy access.

Primary references, checked 21 October 2026:

• Google's language-model introduction for n-grams, tokens and context.
• Google's Transformer introduction for neural parameters and attention.
• Hugging Face generation strategies for greedy decoding and sampling.

The lesson and lab are original LockedIn Labs instruction. These references provide deeper study; they do not imply a vendor partnership or certification.