What to Monitor: Golden Signals, RED and USE

You can measure thousands of things. These three well-known frameworks tell you which handful matter most.

The four golden signals

From Google's SRE book, for any user-facing system:

1. Latency - how long requests take. Track successful and failed requests separately; fast errors can make latency look great.
2. Traffic - how much demand: requests per second, messages per second.
3. Errors - the rate of failed requests, including "successful" responses with wrong content.
4. Saturation - how full the system is: CPU, memory, connection pools, queue depth. Often the earliest warning.

The RED method

A simpler version for every request-driven service or endpoint:

• Rate - requests per second.
• Errors - failed requests per second.
• Duration - distribution of response times (use percentiles, not averages).

A standard RED dashboard per service makes it easy for anyone on the team to read any service.

The USE method

For resources (CPUs, disks, network, connection pools, thread pools):

• Utilisation - percentage of time busy.
• Saturation - extra work queued that can't be served yet.
• Errors - error events for the resource.

Why percentiles, not averages

99 requests at 100ms + 1 request at 10s
average = 199ms   (looks fine)
p99     = 10s     (one in a hundred users waits ten seconds)

Track p50 for typical experience and p95/p99 for the tail. Users who make many requests will hit the tail often.

Putting it together

• Dashboards: RED per service at the top, USE for its key resources below.
• Alerts: on symptoms users feel (errors, latency) rather than causes (CPU), with causes on the dashboard for diagnosis.