Two systems can report the same average throughput and behave nothing alike. One receives a steady 1,200 requests per second, spread evenly across the minute. The other receives 200 requests per second for fifty seconds, then 12,000 for ten. On a graph of hourly averages they are identical. In production, only one of them is in trouble.
Throughput is a rate, and a rate is a summary. What actually reaches your workers is a distribution: arrivals with gaps, bursts, and correlations. Capacity planning that ignores the distribution is planning for a system that never exists.
The nonlinear part
Queueing delay does not grow linearly with utilization. It stays flat while there is slack, then turns upward sharply as the server approaches saturation. A useful rule of thumb: at 70% utilization the wait is roughly twice the service time, at 85% it is four to five times, and past 90% it climbs faster than any dashboard refreshes.
This is why a service can look comfortable at 50% CPU and still time out. The resource that saturates first is often not the one on the graph. It might be a connection pool, a single writer, a lock, or the network path between two components that each look fine in isolation.
If you only alert on averages, you will learn about saturation from your users.
Queue depth is the honest signal
Latency is a lagging indicator: by the time p99 moves, the queue has already been growing. Queue depth, in-flight requests, and the age of the oldest item are closer to the truth. They move before the mean does, and they stay visible when traffic is briefly zero.
Little's law gives the relationship a useful shape:
L = λ × W
L items in the system
λ arrival rate
W time each item spends in the system
If arrivals double and service time stays constant, the number of items in flight doubles. If service time also rises because the system is contended, the queue grows quadratically. Nothing in that equation cares about the hourly average.
Measure the shape
Practical changes that cost little and pay back during the first incident:
- Keep per-second maxima alongside the average, and keep them long enough to compare weekdays.
- Track queue depth and saturation for every bounded resource, not just CPU.
- Measure latency at the client, where queueing and retries are visible together.
- Record timeouts as a separate signal. A timeout is not a slow request; it is a request whose outcome you no longer know.
None of this requires a new platform. It requires deciding that the distribution is the metric and the average is a convenience.