# Your first trace: see what happened inside a request An assistant returns “a person should review this.” Was the guide missing, did a lookup fail, or did the application choose the wrong route? The final answer cannot tell you by itself. In this original 40-minute exercise, you will run a small Python program and inspect real OpenTelemetry spans. Start with familiar function calls, then connect their records and investigate a missing observation. ## 1. Know what you are looking at **Observability** means being able to investigate a system through the evidence it emits. **Instrumentation** is the code that produces that evidence. A **trace** connects operations in one execution. A **span** records one operation, including its start, end and relationship to other operations. If you have used a debugger, think of the question “what called what?” A trace retains selected evidence after execution. It is not a recording of every variable, and it cannot reveal work that was never instrumented. Our program contains ordinary Python functions: ```text support.request — handle a request | +-- guide.lookup — look up a synthetic guide | +-- reply.format — return the guide OR handoff.choose — choose a fallback ``` The enclosing operation is the **root**. The operations inside it are **children**. The request and children share a trace ID. Each operation has its own span ID; a child's parent ID points to its enclosing operation. OpenTelemetry is an instrumentation framework with APIs, SDKs and exporters. It is not a model or a hosted dashboard. This file uses its actual Python SDK with an in-memory exporter. There is no model inference, external service, collector connection or production telemetry. Logs describe events, metrics summarize measurements, and traces connect operations. An evaluation asks whether behavior matches an expectation. You may need several of these to explain a real issue. ## 2. Prepare the local exercise Download both files into one folder: - [First request trace · Python](/templates/fde-first-trace.py) - [Pinned OpenTelemetry requirements](/templates/fde-first-trace-requirements.txt) Use Python 3.11 or later. On macOS or Linux: ```shell python3 -m venv .venv-trace . .venv-trace/bin/activate python -m pip install -r fde-first-trace-requirements.txt python fde-first-trace.py test python fde-first-trace.py compare ``` On Windows, use Command Prompt: ```shell py -3 -m venv .venv-trace .venv-trace\Scripts\activate.bat python -m pip install -r fde-first-trace-requirements.txt python fde-first-trace.py test python fde-first-trace.py compare ``` Installation needs internet access. The exercise commands block socket connections and DNS lookup. They do not need a provider key or account. The pinned requirements include OpenTelemetry API and SDK 1.45.1 and their dependencies. Keep the package versions with your observations. Seven checks exercise parent/child relationships, interval boundaries, missing content, a handled exception, an omitted span, isolation and invalid inputs. If the SDK is disabled and no spans are recorded, the exercise reports an incomplete run instead of claiming that an export succeeded. ## 3. Connect one request Find the `success` run in the JSON from `compare`. It contains three spans: `support.request`, `guide.lookup` and `reply.format`. The application result is `guide`. This is synthetic text from a dictionary, not a generated answer. Try these observations before looking at a dashboard: 1. Find the span with `parent_id: null`. That is the root. 2. Compare all three `trace_id` values. They should match. 3. Compare the three `span_id` values. Each identifies a different operation. 4. Check that each child's `parent_id` matches the root's `span_id`. 5. Inspect `start_ms` and `duration_ms`, measured relative to the root start. The SDK creates new IDs and timestamps each run. Your numbers should not match the browser recording exactly. The names and relationships are what you are checking. Each span is opened around work and closed when its `with` block ends. The supplied program explicitly starts a new root context and uses a private provider; it does not replace the process-wide tracer provider. ## 4. Read elapsed time without counting it twice The root starts before its children and ends after them. Its duration already includes their time. If a request lasts 50 ms and a lookup inside it lasts 30 ms, the request still lasts 50 ms, not 80 ms. The exercise inserts deliberate sleeps: 30 ms in lookup, 15 ms in formatting, and 5 ms in fallback. The recorded durations come from actual SDK timing. Scheduling and instrumentation add overhead, so the values are not exact sleep requests. They are teaching measurements, not a database or model benchmark. In a disposable copy, change only the lookup sleep from `0.030` to `0.060`. Predict what should change, then run again: ```shell python fde-first-trace.py run --case success ``` The lookup interval should grow, and the enclosing request should include the extra time. Small differences in other intervals can be local scheduling noise. Do not turn a single run into a performance claim. These spans measure elapsed wall time, not CPU use. If operations run concurrently, their intervals can overlap; summing child durations may then exceed the enclosing request's duration. ## 5. Separate a missing guide from a failed lookup Run the two cases: ```shell python fde-first-trace.py run --case missing python fde-first-trace.py run --case failure ``` Both return the application result `handoff`. The distinction is in the trace: | Case | Lookup evidence | Next operation | | --- | --- | --- | | Guide missing | `training.found: false`; status `UNSET` | `handoff.choose` | | Lookup failed | status `ERROR`; event `lookup.failed` with fixed `lookup_unavailable` category | `handoff.choose` | The missing guide is an expected branch. The failed lookup raises an exception that the application catches. The parent remains `UNSET` because this example handles the failure and returns a fallback. Child error status does not automatically set parent status. `UNSET` is the SDK default. It is not a score for answer quality, completeness of instrumentation or deployment readiness. Read the instrumentation and application contract together. The result is only a handoff **decision**. No message is sent and no human acceptance is recorded. A receiving engineer should not infer those effects from an operation name. ## 6. Remove an observation without removing the work ```shell python fde-first-trace.py run --case success --omit-lookup-span ``` The result is still `guide`, but only two spans appear: the root and formatting. Read the code: `lookup_guide` still executes. Only its surrounding span is omitted. One of the checks verifies that call directly. The root still includes the time spent in the lookup. From the reduced trace alone, however, you cannot assign the unobserved interval to a specific function. An absent span is not proof that work did not run or took zero time. In a real deployment, missing instrumentation, sampling and export loss can all leave gaps. This exercise isolates missing instrumentation. It does not simulate a remote collector failure or prove cross-process propagation. ## 7. Keep useful evidence without copying the conversation Inspect the recorded attributes and events. An attribute describes an operation; an event marks a moment inside it. The exercise emits fixed names and a small set of bounded values. It excludes guide text, prompt content and raw exception messages. Automatic exception capture is explicitly disabled. This is a scoped example, not a general-purpose redactor. When adding a field, state the diagnostic question it answers and how its value is bounded. Do not serialize a whole request or exception just because it is easy. The local exporter retains spans in memory. No collector, remote storage, access policy or retention system is tested here. Across services, trace context must be propagated deliberately, while identity and permission checks remain separate. A trace ID never grants access. ## 8. Keep a small investigation note Record: - The program and package versions, and which case you ran. - One trace ID with its root and child span IDs. - The enclosing duration, without adding child time again. - The field or event distinguishing a missing guide from a failed lookup. - What disappeared when instrumentation was removed, and what still ran. - The next bounded observation you would add to a real application. Explain it to another engineer without starting with the raw JSON: “One request looked up a guide. That lookup failed, so the application chose a fallback. Here is the recorded error, and here is what we still do not know.” Continue in [the authenticated observability lesson](/app/academy?course=agent-engineering&lesson=observability-first-trace&view=focus). Then use the advanced agent-engineering lesson for event projection, evaluation and recovery boundaries. ## Sources and scope Original LockedIn Labs instruction and synthetic exercise. Primary references checked 20 October 2026: - [OpenTelemetry: traces and spans](https://opentelemetry.io/docs/concepts/signals/traces/) - [OpenTelemetry: Python instrumentation](https://opentelemetry.io/docs/languages/python/instrumentation/) - [OpenTelemetry: handling sensitive data](https://opentelemetry.io/docs/security/handling-sensitive-data/) Local SDK observations do not establish real model behavior, service latency, collector delivery, production readiness or a learner credential.