Distributed Tracing Explained

In a system of many services, one slow user request might pass through an API gateway, three services, a queue, two databases and a third-party API. Distributed tracing reconstructs that whole path.

Traces and spans

• A trace is the full journey of one request, identified by a trace ID.
• A span is one operation in it (an HTTP call, a query, a function), with a start time, duration, attributes and a parent span.
• Spans nest into a tree, usually shown as a waterfall chart.

Context propagation

For spans in different services to join one trace, the trace ID must travel with the request. The W3C Trace Context standard defines the traceparent header:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^  ^ trace id                       ^ parent span id ^ flags (sampled)

OpenTelemetry instrumentation adds this header to outgoing HTTP calls and reads it from incoming ones automatically. For queues, it's stored in message headers or attributes.

Reading a trace

• The longest bar - where most of the time went.
• Sequential calls that could run in parallel - a staircase of spans.
• N+1 patterns - dozens of identical short database spans.
• Errors - spans marked with error status and exception events.
• Gaps - time not covered by any child span, often uninstrumented work or waiting.

Sampling

Recording every trace at high traffic is expensive. Options:

• Head sampling - decide at the start (for example keep 10%). Simple and cheap, but may miss rare errors.
• Tail sampling - decide after the trace completes, keeping all errors and slow traces plus a sample of the rest. Usually done in an OpenTelemetry Collector.

Make traces useful

• Add business attributes: tenant.id, order.id, feature.flag.
• Write the trace ID into every log line so you can jump between them.
• Return the trace ID in an error response header; support can then look up exactly what happened.