# Trace context: a successful request can still have a broken trace

Original LockedIn Labs workshop · 50 minutes · Python 3.11+ and the existing pinned OpenTelemetry environment. Seven cases use two owned HTTP listeners on `127.0.0.1`, separate SDK providers and an in-memory exporter. The listeners are in one process, on fresh ephemeral ports, with separate request threads. No collector, model, TLS, external service, patient/customer data or durable business operation is involved.

## 1. Keep delivery and the trace relationship separate

A gateway can receive HTTP 200 from a worker while the worker's trace is disconnected or attached to the wrong parent. Response status answers a transport/application question; it does not certify the instrumentation. A **trace ID** groups related spans. A **span ID** names one operation. A propagated **parent ID** tells the next receiver which span it should regard as its parent.

The gateway extracts incoming context, creates its own server span, then injects the active context into the worker request. The worker extracts that actual received header and creates another server span. Our teaching code deliberately omits a separate HTTP client span. In an instrumented HTTP client, the outbound carrier would normally identify that client span instead. Always follow the actual instrumentation topology.

OpenTelemetry provides the SDK, context handling and W3C propagator. It is not a hosted monitoring product. The Python process emits the local records; this page's diagram is an authored explanation, not live telemetry.

## 2. Read the carrier before drawing the parent chain

For the version-00 fixture, `traceparent` has this shape:

```text
00-1234567890abcdef1234567890abcdef-1234567890abcdef-01
version       32 hex trace ID          16 hex parent ID flags
```

The final `01` has the sampled bit set; `00` clears it. The trace and parent IDs cannot be all zero. The SDK handles extraction rather than a home-grown validator. This workshop exercises selected version-00 cases, not every W3C version, malformed carrier or forward-compatibility rule. Consult the standard and the pinned propagator for the full contract.

The receiving server span has a new span ID, keeps a valid incoming trace ID and identifies the extracted parent as remote. When this gateway injects its active context, the outgoing parent ID becomes the gateway span ID. Copying the incoming header unchanged keeps the trace ID but names the upstream parent again: the worker becomes the gateway's apparent sibling despite the observed gateway-to-worker call.

The supplied upstream context is synthetic. The caller's span is not created or exported here. A remote parent reference does not prove that a corresponding record exists in a backend, nor that the caller is trusted.

## 3. Execute seven transport comparisons

Download [the runner](/templates/fde-http-trace.py) and [the existing pinned requirements](/templates/fde-first-trace-requirements.txt) into a disposable folder. Use a fresh environment or the unchanged environment from the first tracing lesson:

```sh
python3 -m venv .venv-trace
. .venv-trace/bin/activate
python -m pip install -r fde-first-trace-requirements.txt
python fde-http-trace.py > http-trace-original.json
python fde-http-trace.py --case correct --sampled 0 > http-trace-unsampled.json
```

On Windows Command Prompt, use `py -3 -m venv .venv-trace` and `.venv-trace\Scripts\activate.bat`; the remaining commands are unchanged. Installation needs internet access. Execution connects only to the two loopback listeners it allocates. If local socket policy blocks them, retain that setup failure; reading a report is a different practice mode.

The runner checks the exact existing pins: OpenTelemetry API/SDK 1.45.1, semantic conventions 0.66b1 and `typing_extensions` 4.16.0. It sets no global tracer provider and loads no remote exporter. Each endpoint has its own `TracerProvider`, service name and `ParentBased(ALWAYS_ON)` sampler. Only the in-memory exporter is shared for inspection.

Predict before executing:

| Case | Actual HTTP/task observation | Trace evidence to inspect |
| --- | --- | --- |
| `correct` | Both handlers run once; client receives 200 and `SYNTHETIC-OK` | Same trace; worker parent equals gateway span; two finished spans |
| `unsampled` | The same two handlers and successful result remain | Valid linked contexts; neither server span records; zero finished spans |
| `dropped` | Both requests still succeed | Gateway sends no trace header; worker starts another trace |
| `reused-parent` | Both requests still succeed | Trace ID agrees, but worker parent is the original upstream ID, not the gateway span |
| `invalid-zero` | Both requests still succeed | SDK declines the all-zero trace ID; gateway starts a new root and correctly links the worker |
| `missing` | Both requests still succeed | Gateway has no incoming context; it starts a root and forwards its own active context |
| `baggage` | Both requests still succeed | Gateway receives synthetic baggage; the trace-only propagator sends no baggage to the worker |

Use one comparison at a time:

```sh
python fde-http-trace.py --case dropped
python fde-http-trace.py --case reused-parent
python fde-http-trace.py --case baggage
```

Keep original reports before rerunning; shell redirection overwrites an existing file of the same name. The program accepts no URL, file, tenant or credential input. Its fixed GET performs no business mutation. Request success here establishes only the supplied local reply.

## 4. Reconcile wire context with exported spans

Start with `gateway.received`, `gateway.sent` and `worker.received`. These are the selected headers actually supplied to and read by the owned HTTP endpoints. Then compare each endpoint's `parent`, current `span`, `recording` flag and independently incremented `handler_counts`.

```text
For the correctly linked case:
  gateway.span.trace_id == worker.span.trace_id
  gateway.span.span_id  == worker.parent.span_id
  worker.parent.remote == true
```

Recompute those relationships instead of trusting the report's `comparison` summary. For recorded cases, reconcile them with `finished_spans`: actual SDK names, service resources, contexts and parents. SDK-generated IDs change on every run; compare relationships inside one report, not literal generated IDs across files. Span export order is not a causal-order contract.

For `reused-parent`, both spans still name the same supplied upstream parent. Two exported records and one common trace ID therefore do not establish the right causal edge. For `dropped`, the worker creates a separate root even though the gateway actually called it. Nearby timestamps and similar names cannot restore the lost parent.

The process retains each handler's direct observations separately from exported telemetry. These counters and wire records are teaching evidence, not evidence that production instrumentation always has an independent audit channel.

## 5. Change sampling without changing the task

Compare `correct` with `--sampled 1` and `--sampled 0`. This changes the incoming sampled bit for that case; the separately named `unsampled` case always uses zero. Under this lab's parent-based sampler, the unsampled remote parent yields valid non-recording contexts. Injection still passes the new active span's context onward. Both handlers execute and return the same synthetic result while the exporter receives zero finished spans.

Absence from this exporter cannot mean the task did not run. Nor can a sampled trace collection automatically provide all-request counts for a service-level indicator. Different sampler or pipeline configurations can behave differently; this is one observed SDK policy, not a universal absence rule.

Explain the boundary: the lab does not send OTLP, test collector delivery/retention, align clocks across hosts, instrument real services or prove an enterprise's sampling policy. An installed SDK and a correct local carrier are necessary evidence for their own boundaries, not a complete observability rollout.

## 6. Receive trace evidence as a delivery pod

The trace-only propagator carries trace context, not baggage. In `baggage`, the gateway sees `synthetic.department=practice`; the worker does not. That is this adapter's explicit propagation choice, not a requirement that all applications discard all baggage. Baggage is separately defined metadata and is not automatically a span attribute. Any real propagation policy needs approved keys, destinations, limits and retention evidence.

Trace IDs and metadata can correlate activity and still require privacy review. Never place secrets or private person/customer content in a carrier. A caller can supply a valid-looking trace ID or baggage value. Neither authenticates the caller, grants access, selects an authorized tenant or supplies an idempotency key. This lab implements none of those controls.

Work in three roles:

- **Builder:** retain the exact runner, package versions, original/changed command and report. Locate the actual outgoing and received carrier, not just the friendly summary.
- **Receiver:** reproduce the unchanged-header bug, identify the incorrect parent and run the sampling change. Record your own action and assistance separately from the builder's account.
- **Application/platform owners:** agree the intended service/client span topology, propagation boundary, sampling policy and next collector or receiving-service probe. Keep identity, tenant authorization and business receipts with their accountable owners.

Each participant keeps a separate record:

```text
Synthetic alias / role:
Exact runner / package versions / command:
Prediction before execution:
Gateway received / sent carrier evidence path:
Worker received carrier / actual parent evidence path:
Own observation / finished spans / handler counts:
Own contribution / assistance:
Changed sampled bit / prediction / separately retained result:
Receiving action actually attempted / assistance:
Missing evidence / next bounded probe / accountable role:
```

No practice record establishes qualification, independence, placement or permission to deploy. Continue with [operational ownership](/training/guides/service-operations) to connect trace evidence to user promises and receiving responsibilities.

## 7. Defend the evidence in an interview

Explain how `reused-parent` differs from `dropped` despite the same successful task. Sketch the intended trace chain and the actual wrong-parent chain using the IDs in one report. Then explain why an actual HTTP client span would change the expected parent.

Explain how the unsampled case has valid span IDs without finished exported spans, and which observation establishes that the handlers ran. Name the sampler configuration before generalizing. Describe why a backend with only sampled traces cannot automatically supply the complete request denominator.

Finally, describe one real receiving-service test and one collector-retention test still needed. State why a valid header or a baggage tenant label must never replace application authorization. An operational answer names the missing boundary and its owner instead of making a tracing success carry unrelated claims.

Sources checked 28 October 2026: [W3C Trace Context Recommendation, 23 November 2021](https://www.w3.org/TR/2021/REC-trace-context-1-20211123/), [OpenTelemetry Python propagation](https://opentelemetry.io/docs/languages/python/propagation/), [OpenTelemetry parent-based sampling](https://opentelemetry.io/docs/specs/otel/trace/sdk/#parentbased), and [OpenTelemetry baggage](https://opentelemetry.io/docs/concepts/signals/baggage/). These primary references explain the protocols and SDK concepts. The fixture, comparisons, diagrams and receiving exercise are original.