# Model 101: from learned patterns to generated text An original 45-minute lesson for an engineer meeting language models for the first time. Start with the explanation, follow the visual, predict a result, then inspect a small program. The maintenance phrases below are fictional text, not advice about equipment. ## 1. Understand the model before building the agent A **large language model (LLM)** is a trained neural network: software whose many numerical parameters were adjusted using data. A language-generating model uses those numbers and its current input to calculate scores for possible next tokens. A token is a unit of encoded content; it need not be a whole word. The surrounding program chooses a token, adds it to the working sequence and continues until a stopping condition. A model is neither a programming language nor an agent framework. Think of a function with two inputs: a saved parameter set and this request's working context. Training changes the parameter set. Ordinary inference uses it to calculate an output. Giving the function a different request can change the output without rewriting the saved parameters. This analogy does not mean the model is a database lookup or that its output has been verified. | Name | What it does | What it does not establish | | --- | --- | --- | | Parameters / weights | Learned numbers used by the network's calculations | An approved, current company procedure | | Context | Instructions, messages and evidence supplied for this request | A permanent update to the model's weights | | Inference | Calculates outputs using a trained model | Truth, permission or completed business work | | Decoding | Selects output tokens from the model's scores | Whether the generated statement is supported | | Training / fine-tuning | Updates learned parameters from examples and an objective | That every future response will be correct | | Agent application | Gives the model tools and enforces the working loop | Unlimited permission to act | An FDE needs these distinctions to diagnose the right part of a system. A stale procedure may require better retrieval and source checks. An output limit may need a smaller task. Neither observation alone demonstrates a need to retrain a model. Provider logging, retention and any later training use are separate service policies; “inference does not update these weights” is not a privacy promise. ## 2. Meet a model small enough to inspect The lab trains a **word-bigram model**. “Bigram” means a pair of successive tokens. It counts which word follows another in five fictional phrases, then turns those counts into a probability distribution. It splits on spaces and uses a special `` marker. Real LLM tokenizers use richer vocabularies. Its entire training collection is: ```text the pump needs inspection the pump needs inspection the pump needs inspection the pump needs repair the valve needs repair ``` After `needs`, the learned table has three `inspection` occurrences and two `repair` occurrences. At temperature 1, these become 3/5 and 2/5. A probability distribution allocates a total of 1 across possible next tokens. It describes this model's next-token calculation, not confidence that a real pump needs work. This is an actual small statistical model, **not an LLM or a miniature Transformer**. It remembers only the immediately previous word; it has no neural layers, attention, embeddings, instruction following or factual reasoning. A generative Transformer computes with learned neural weights and representations of a much larger permitted context. Causal attention lets positions use earlier context positions; it does not grant access to an external database. The toy isolates training, prediction and selection so you can explain those jobs before studying the larger architecture. ## 3. Predict and run seven cases Download `fde-model-first-inference.py` into your practice folder. Use Python 3.11 or later; no packages, model download, GPU, key or account are required. The program runs in memory, writes no files and makes no network calls. ```sh python3 --version python3 fde-model-first-inference.py compare python3 fde-model-first-inference.py sampled ``` Before running, predict the continuation, stop reason and whether the model's parameters changed. Then compare your prediction with `trace`, `parameters_before`, `parameters_after` and `model_changed_from_base`. | Case | What changes | Expected observation | | --- | --- | --- | | `greedy` | Pick the largest next-token probability | `inspection`, then `` | | `sampled` | Draw from the same distribution, seed 0 | `repair`, then ``; weights unchanged | | `cooler` | Sampling temperature becomes 0.5, same seed | `repair` still appears; its first-step probability drops from 0.4 to about 0.308 | | `earlier-context` | Replace `pump` with `valve` in the prompt | Still `inspection`: this toy sees only the last word `needs` | | `output-limit` | Allow just one generation step | `inspection` with `token-limit`, without observing `` | | `unseen` | End the prompt with `compressor` | Empty continuation and `unseen-context` | | `retrained` | Train a separate model with four extra repair phrases | Counts become 3:6; greedy chooses `repair`; the model fingerprint differs from the base | The fingerprint summarizes the learned count table to make equality visible. It is not a credential or security attestation. Each inference result should show identical before/after fingerprints, including inference on the separately retrained model. Training changes a model; generation does not mutate it here. ## 4. Follow one prediction through the program Read the `train`, `distribution` and `generate` functions in that order. 1. `train` counts adjacent word pairs. Repeated examples affect the table. 2. `distribution` turns the relevant row into probabilities. At temperature 1, these are the relative counts. The code uses logarithms and softmax, which converts scores into nonnegative values that sum to 1. 3. `generate` either selects the largest probability (**greedy decoding**) or takes a pseudo-random draw from the distribution (**sampling**). 4. The chosen word becomes the next context word. The loop stops on ``, an absent transition or the explicit step limit. For sampling, temperature rescales scores before normalization. A smaller positive temperature concentrates mass on stronger candidates; it does not verify a claim or make all other tokens impossible. The lab's temperature 0.5 turns the 3:2 counts into normalized 9:4 masses. With seed 0 the first draw is about 0.844, so it still lands on `repair`. Greedy is a separate mode; do not divide by zero. Production parameter names, supported settings and repeatability depend on the selected model and serving system. The `earlier-context` and `unseen` cases reveal limitations of this specific toy. Do not conclude that a modern LLM ignores all but one word or always refuses unfamiliar input. It may instead generate a fluent but unsupported answer. ## 5. Change one cause at a time Make a copy of the file before editing. Keep predictions and actual results. **Change A — selection without training.** Change `SAMPLE_SEED = 0` to `SAMPLE_SEED = 1` and run `sampled`. The continuation becomes `inspection`. The counts, probabilities and parameter fingerprint stay the same. Explain why two outputs do not require two models. Restore the seed before the next edit. **Change B — training without changing the prompt.** Change `EXTRA_REPAIR_EXAMPLES = 4` to `EXTRA_REPAIR_EXAMPLES = 0` and run `retrained`. The learned table now equals the base model; `model_changed_from_base` is false and the greedy continuation returns to `inspection`. Explain which data changed the earlier prediction. Repetition here is a transparent teaching input, not a recommended production fine-tuning strategy. Do not judge a model from these seven examples. This experiment verifies the specified mechanics, not language quality, reliability or workplace performance. ## 6. Carry the distinction into an FDE design Your fictional assistant receives a new approved procedure: **P-17 revision 3 requires a human review before a draft is sent**. A generated draft omits that review. First draw where P-17 enters context, which model/version is called, where the output is checked and which application code controls sending. Choose an observation before proposing a change: - If P-17 never entered the request, inspect retrieval and context construction. - If P-17 was present but the draft contradicts it, retain the case and evaluate the prompt/model combination. A low temperature is not a factuality control. - If the draft ends early, inspect the recorded completion reason and allowance. - If text proposes sending, the application still has to enforce permission and review. A model's words do not execute that action. Produce a one-page explanation with the parameter set, context, decoding choice, observed result and application boundary. State what evidence would justify a training experiment and how you would compare it on held-out tasks. “We trained it” is not a substitute for evaluation. ## 7. Keep the work and go deeper Keep your seven predictions, actual traces, both changed-copy outputs and the one-page design. Explain the same mechanism to a colleague without running the program. They should be able to point to what changed and what stayed fixed. Continue with [real tokenization and request budgeting](/training/guides/tokens), then [context construction](/training/guides/context) and [agent engineering](/training/guides/agent-engineering). The protected Model 101 lesson adds knowledge checks and a transfer discussion under existing Academy access. Primary references, checked 21 October 2026: - [Google's language-model introduction](https://developers.google.com/machine-learning/crash-course/llm) for n-grams, tokens and context. - [Google's Transformer introduction](https://developers.google.com/machine-learning/crash-course/llm/transformers) for neural parameters and attention. - [Hugging Face generation strategies](https://huggingface.co/docs/transformers/generation_strategies) for greedy decoding and sampling. The lesson and lab are original LockedIn Labs instruction. These references provide deeper study; they do not imply a vendor partnership or certification.