Observability 101: Logs, Metrics and Traces

Monitoring tells you that something is wrong. Observability helps you understand why, including for problems you didn't predict. It's built from three kinds of telemetry.

Metrics: numbers over time

http_requests_total{route="/checkout", status="500"}  1243
http_request_duration_seconds{route="/checkout", quantile="0.99"}  2.4

• Cheap to store and fast to query, even over months.
• Ideal for dashboards and alerts: error rate, latency, CPU, queue depth.
• Aggregated, so they show that checkout is slow, not which request.

Logs: individual events

{"level":"error","msg":"payment declined","order_id":"ord_42","provider":"razorpay","trace_id":"4bf92f35..."}

• Rich detail about specific events.
• Expensive at high volume; structure them (JSON) so they're searchable.

Traces: the journey of one request

POST /checkout                         1.92s
├── auth.verifyToken                   12ms
├── cart.load (postgres)               48ms
├── payments.charge (HTTP)            1.71s   <- here's the problem
└── email.sendReceipt (queue)           9ms

• Show where time is spent across services, databases and external APIs.
• Each step is a span; spans share a trace ID.

Connecting them

The real power comes from correlation: an alert on a metric leads to slow traces, and a trace links to the logs written during that request (via the trace ID in each log line).

Where to start

1. Structured logs with a request or trace ID.
2. Request rate, error rate and latency metrics per endpoint.
3. Distributed tracing with OpenTelemetry once you have more than one service.

OpenTelemetry is the vendor-neutral standard for producing all three, so you can switch backends (Grafana, Datadog, Honeycomb, New Relic, cloud tools) without re-instrumenting your code.