Your first trace: see what happened inside a request

Observability means being able to investigate a system through the evidence it emits. Instrumentation is the code that produces that evidence. A trace connects operations in one execution. A span records one operation, including its start, end and relationship to other operations.

If you have used a debugger, think of the question “what called what?” A trace retains selected evidence after execution. It is not a recording of every variable, and it cannot reveal work that was never instrumented.

Our program contains ordinary Python functions:

support.request — handle a request
  |
  +-- guide.lookup — look up a synthetic guide
  |
  +-- reply.format — return the guide
      OR
      handoff.choose — choose a fallback

The enclosing operation is the root. The operations inside it are children. The request and children share a trace ID. Each operation has its own span ID; a child's parent ID points to its enclosing operation.

OpenTelemetry is an instrumentation framework with APIs, SDKs and exporters. It is not a model or a hosted dashboard. This file uses its actual Python SDK with an in-memory exporter. There is no model inference, external service, collector connection or production telemetry.

Logs describe events, metrics summarize measurements, and traces connect operations. An evaluation asks whether behavior matches an expectation. You may need several of these to explain a real issue.

Download both files into one folder:

• First request trace · Python
• Pinned OpenTelemetry requirements

Use Python 3.11 or later. On macOS or Linux:

python3 -m venv .venv-trace
. .venv-trace/bin/activate
python -m pip install -r fde-first-trace-requirements.txt
python fde-first-trace.py test
python fde-first-trace.py compare

On Windows, use Command Prompt:

py -3 -m venv .venv-trace
.venv-trace\Scripts\activate.bat
python -m pip install -r fde-first-trace-requirements.txt
python fde-first-trace.py test
python fde-first-trace.py compare

Installation needs internet access. The exercise commands block socket connections and DNS lookup. They do not need a provider key or account. The pinned requirements include OpenTelemetry API and SDK 1.45.1 and their dependencies. Keep the package versions with your observations.

Seven checks exercise parent/child relationships, interval boundaries, missing content, a handled exception, an omitted span, isolation and invalid inputs. If the SDK is disabled and no spans are recorded, the exercise reports an incomplete run instead of claiming that an export succeeded.

Find the success run in the JSON from compare. It contains three spans: support.request, guide.lookup and reply.format. The application result is guide. This is synthetic text from a dictionary, not a generated answer.

Try these observations before looking at a dashboard:

1. Find the span with parent_id: null. That is the root.
2. Compare all three trace_id values. They should match.
3. Compare the three span_id values. Each identifies a different operation.
4. Check that each child's parent_id matches the root's span_id.
5. Inspect start_ms and duration_ms, measured relative to the root start.

The SDK creates new IDs and timestamps each run. Your numbers should not match the browser recording exactly. The names and relationships are what you are checking.

Each span is opened around work and closed when its with block ends. The supplied program explicitly starts a new root context and uses a private provider; it does not replace the process-wide tracer provider.

The root starts before its children and ends after them. Its duration already includes their time. If a request lasts 50 ms and a lookup inside it lasts 30 ms, the request still lasts 50 ms, not 80 ms.

The exercise inserts deliberate sleeps: 30 ms in lookup, 15 ms in formatting, and 5 ms in fallback. The recorded durations come from actual SDK timing. Scheduling and instrumentation add overhead, so the values are not exact sleep requests. They are teaching measurements, not a database or model benchmark.

In a disposable copy, change only the lookup sleep from 0.030 to 0.060. Predict what should change, then run again:

python fde-first-trace.py run --case success

The lookup interval should grow, and the enclosing request should include the extra time. Small differences in other intervals can be local scheduling noise. Do not turn a single run into a performance claim.

These spans measure elapsed wall time, not CPU use. If operations run concurrently, their intervals can overlap; summing child durations may then exceed the enclosing request's duration.

Run the two cases:

python fde-first-trace.py run --case missing
python fde-first-trace.py run --case failure

Both return the application result handoff. The distinction is in the trace:

The missing guide is an expected branch. The failed lookup raises an exception that the application catches. The parent remains UNSET because this example handles the failure and returns a fallback. Child error status does not automatically set parent status.

UNSET is the SDK default. It is not a score for answer quality, completeness of instrumentation or deployment readiness. Read the instrumentation and application contract together.

The result is only a handoff decision. No message is sent and no human acceptance is recorded. A receiving engineer should not infer those effects from an operation name.

python fde-first-trace.py run --case success --omit-lookup-span

The result is still guide, but only two spans appear: the root and formatting. Read the code: lookup_guide still executes. Only its surrounding span is omitted. One of the checks verifies that call directly.

The root still includes the time spent in the lookup. From the reduced trace alone, however, you cannot assign the unobserved interval to a specific function. An absent span is not proof that work did not run or took zero time.

In a real deployment, missing instrumentation, sampling and export loss can all leave gaps. This exercise isolates missing instrumentation. It does not simulate a remote collector failure or prove cross-process propagation.

Inspect the recorded attributes and events. An attribute describes an operation; an event marks a moment inside it. The exercise emits fixed names and a small set of bounded values. It excludes guide text, prompt content and raw exception messages. Automatic exception capture is explicitly disabled.

This is a scoped example, not a general-purpose redactor. When adding a field, state the diagnostic question it answers and how its value is bounded. Do not serialize a whole request or exception just because it is easy.

The local exporter retains spans in memory. No collector, remote storage, access policy or retention system is tested here. Across services, trace context must be propagated deliberately, while identity and permission checks remain separate. A trace ID never grants access.

Record:

• The program and package versions, and which case you ran.
• One trace ID with its root and child span IDs.
• The enclosing duration, without adding child time again.
• The field or event distinguishing a missing guide from a failed lookup.
• What disappeared when instrumentation was removed, and what still ran.
• The next bounded observation you would add to a real application.

Explain it to another engineer without starting with the raw JSON: “One request looked up a guide. That lookup failed, so the application chose a fallback. Here is the recorded error, and here is what we still do not know.”

Continue in the authenticated observability lesson. Then use the advanced agent-engineering lesson for event projection, evaluation and recovery boundaries.

Original LockedIn Labs instruction and synthetic exercise. Primary references checked 20 October 2026:

• OpenTelemetry: traces and spans
• OpenTelemetry: Python instrumentation
• OpenTelemetry: handling sensitive data

Local SDK observations do not establish real model behavior, service latency, collector delivery, production readiness or a learner credential.