Monitoring LLM Applications: Tokens, Latency and Quality

LLM features fail in new ways: answers get worse without any error, costs spike because a prompt grew, an agent loops through tools. Normal API monitoring doesn't catch these, so add LLM-specific telemetry.

1. Usage and cost

const response = await client.messages.create(params);

metrics.counter('llm_input_tokens_total', response.usage.input_tokens, { feature: 'support-bot', model: params.model });
metrics.counter('llm_output_tokens_total', response.usage.output_tokens, { feature: 'support-bot', model: params.model });
metrics.counter('llm_cache_read_tokens_total', response.usage.cache_read_input_tokens ?? 0, { feature: 'support-bot' });

Break costs down by feature (and by customer, in logs rather than metric labels) so you know what's driving the bill. A falling cache-read ratio is an early warning that something broke your prompt caching.

2. Latency

• Time to first token - what users feel in streaming UIs.
• Total generation time and output tokens per second.
• For agents: number of steps and tool calls per task, and total task duration.

3. Errors and stop reasons

• Rate limits (429), overloaded and server errors, timeouts.
• stop_reason distribution: a rise in max_tokens means truncated answers; refusal means declined requests.
• Structured output or tool-argument validation failures.

4. Traces

OpenTelemetry has semantic conventions for generative AI, with attributes for the model, token counts and operations. A trace per request shows retrieval, each model call and each tool call, which makes slow or looping agents easy to spot.

5. Quality

• Collect explicit feedback (thumbs up/down) and implicit signals (did the user retry or rephrase?).
• Run an offline evaluation set on every prompt or model change.
• Score a sample of production conversations with an LLM judge for correctness and policy compliance, and trend the score.

Privacy

Prompts and responses often contain personal data. Log them only with consent and a clear retention policy, redact sensitive fields, and restrict who can read them.