Scalability: The Bottleneck Is Almost Never the CPU

Your dashboard shows 28% CPU. Your p99 latency is 4 seconds. Someone suggests adding more pods.

This is the most common wrong turn in scaling work, and it comes from a reasonable assumption: that a slow system must be a busy system. It usually is not. It is a system where requests are queuing for something that is not CPU, and adding instances often makes it worse.

Here is what is actually limiting you, and the arithmetic to find it.

One formula that sizes almost everything

Little’s Law is the most useful thing in capacity planning, and it fits on one line:

concurrency = arrival rate x latency

L = λ × W

If 500 requests arrive per second and each spends 40 ms in the database, then at any instant you need:

500 req/s × 0.04 s = 20 concurrent database connections

Twenty. Not two hundred.

Now run it backwards, which is where it earns its keep. You have a pool of 20 connections and each query takes 50 ms:

20 / 0.05 s = 400 requests per second

That is your ceiling. Not a soft limit — a hard one. You can run forty pods; the pool still admits 400 requests per second and everything else waits. CPU stays low because the CPU is not doing anything. It is waiting.

Every queue in your system obeys this. Thread pools, connection pools, worker counts, in-flight request limits. If you know two of the three terms, you know the third.

Why 80% utilisation is not 80% fine

Here is the part that explains why capacity planning feels counter-intuitive.

For a simple queue, response time relates to utilisation like this:

response time = service time / (1 - utilisation)

R = S / (1 - ρ)

That denominator is the whole story. As utilisation approaches 1, response time does not rise gently — it goes to infinity.

Going from 50% to 80% utilisation does not cost you 30% of your latency budget. It costs you 150% — response time goes from twice service time to five times it.

This is why experienced teams keep what looks like wasteful headroom, and why a system that was fine yesterday at 70% falls over today at 85% after a modest traffic increase. Nothing changed except your position on a curve that turns vertical.

The honest caveat: this exact formula assumes a single server with Poisson arrivals and exponentially distributed service times — an M/M/1 queue. Real systems have multiple servers and burstier traffic, so your constants will differ. But the shape is universal: queueing delay is hyperbolic in utilisation, not linear. Every queue you own has a knee, and it is closer to the left than intuition suggests.

Practical consequence: stop setting autoscaling targets at 80% CPU. By the time you are there you are already deep into the curve, and the new instance takes time to become useful.

The ceilings, in the order they usually bite

1. Database connections

This is the most common real limit, and it has a number you can look up.

PostgreSQL defaults to max_connections = 100, of which 3 are reserved for superusers by default. So roughly 97 are actually available to your application.

Now count what you are asking for. Twelve pods with a pool of 10 each is 120 connections against a limit of 97. The twelfth pod gets connection errors, and the error looks like a database problem rather than an arithmetic one.

Two things make this sharper:

• PostgreSQL forks a process per connection, so connections cost real memory. The docs are explicit that raising max_connections raises shared memory allocation with it. This is why a pooler like PgBouncer exists.
• A bigger pool is not more throughput. Past the point where the database can genuinely execute queries in parallel, extra connections add lock contention and context switching. Throughput flattens, then falls.

MySQL uses a thread per connection and is cheaper here, which is one of the structural differences between Postgres and MySQL worth knowing before you pick.

2. File descriptors

Every socket, every open file, every connection is a file descriptor. The per-process limit is often 1024 by default on the soft limit, which sounds generous until you notice it covers inbound connections, outbound connections, log files and library internals simultaneously.

ulimit -n              # soft limit for this shell
cat /proc/<pid>/limits # what the running process actually has
ls /proc/<pid>/fd | wc -l   # how many it is using right now

The failure mode is memorable: EMFILE: too many open files, usually under exactly the load where you least want to be reading a stack trace.

3. Ephemeral ports and TIME_WAIT

This one surprises people because nothing in the application mentions it.

When your service opens an outbound connection, the kernel picks a source port from the ephemeral range. On Linux:

cat /proc/sys/net/ipv4/ip_local_port_range
# typically: 32768 60999

That is about 28,000 ports per unique destination address and port combination.

Then the closing side holds the socket in TIME_WAIT for 60 seconds — and on Linux that duration is compiled into the kernel, not a sysctl you can turn down.

Do the arithmetic. Open 500 short-lived connections per second to the same downstream service:

500 conn/s × 60 s = 30,000 sockets in TIME_WAIT

You have run out. New connections fail, and the error will be a connect timeout that looks like the downstream service is down.

The fix is not raising the range. It is not opening the connections: HTTP keep-alive, connection pooling, HTTP/2 multiplexing. Reuse beats provisioning.

4. Connection tracking

If your traffic passes through a NAT gateway, a firewall or a Kubernetes node using iptables, the kernel keeps a conntrack entry for every flow. That table is finite, and when it fills, packets are dropped silently — no error, no log line in your application, just latency and retries.

sysctl net.netfilter.nf_conntrack_max
sysctl net.netfilter.nf_conntrack_count

The gap between those two numbers is a metric worth alerting on, and almost nobody exports it.

5. Threads and event loops

A thread-per-request server is a pool by another name, and Little’s Law applies unchanged. An event-loop server has no such limit — which is its own hazard, because it will happily accept ten thousand concurrent requests and serve all of them slowly instead of refusing some quickly.

One blocking call on an event loop stalls every request on that loop. The CPU graph will look calm throughout.

Horizontal scaling and the shared denominator

Adding instances multiplies capacity for the independent part of a request and does nothing for the shared part.

Amdahl’s Law puts a ceiling on the speedup available from parallelising anything with a serial fraction. The Universal Scalability Law adds the part people miss, and it is worth writing out because the second term is the one that bites:

C(N) = N / (1 + σ(N-1) + κN(N-1))

  N  number of workers
  σ  contention   — the serial fraction, queueing for a shared resource
  κ  coherency    — the cost of keeping workers consistent with each other

With κ = 0 this reduces to Amdahl’s Law: throughput flattens but never falls. With κ > 0 something worse happens — there is a maximum, and past it throughput decreases:

peak at N* = sqrt((1 - σ) / κ)

The coherency term is why adding capacity can make a system slower. It covers cache-line bouncing between cores, lock handoffs, cluster gossip, distributed consensus, cross-region replication — anything whose cost grows with the square of the number of participants because each one must agree with every other.

Concretely, if every new pod opens more connections to the same primary database, you are not scaling. You are converting a connection limit into a queue and then into a timeout. If every new node must gossip with every other node, you have bought yourself a κ term.

Ask this before adding instances:

1. What does a request touch that is shared?
2. What is the current concurrency limit on that shared thing?
3. Does another instance raise that limit, or compete for it?

If the answer to 3 is “compete”, scaling out makes it worse.

Load balancing: what the diagrams leave out

The box labelled “LB” in your architecture diagram hides three decisions that determine whether scaling works.

Health checks that lie. A check that returns 200 whenever the process is alive will keep routing traffic into an instance whose database pool is exhausted and whose queue is thirty seconds deep. A useful check verifies the dependencies a request actually needs — and distinguishes liveness (restart me) from readiness (stop sending me traffic). Conflating them causes restart loops under load, exactly when you need stability.

L4 versus L7 under HTTP/2. L4 balances connections. With HTTP/2 or gRPC, one long-lived connection carries many multiplexed requests — so connection-level balancing can leave one backend doing most of the work while the others idle. The graph shows even connection counts and wildly uneven CPU. For HTTP/2 and gRPC you generally want request-level balancing.

Sticky sessions. Pinning a user to an instance defeats balancing on purpose. It also means a restart drops those users’ state, and a hot user becomes a hot instance you cannot spread.

Then there is the retry trap: when a system is already saturated, automatic retries multiply the load that saturated it. Retries need rate limiting and a budget, plus jitter, or they turn a brown-out into an outage.

Queues absorb bursts, not overload

A queue in front of a slow consumer buys you time across a burst. It does nothing about a sustained arrival rate above your service rate — it just moves where the failure appears.

If arrivals exceed service capacity indefinitely, the queue grows without bound, and latency grows with it. Callers time out. They retry. Arrivals increase. This is a feedback loop with the sign pointing the wrong way.

Backpressure is the answer, and it means being willing to say no:

• Bounded queues. An unbounded queue is an out-of-memory error with extra steps.
• Reject fast. A 429 with Retry-After in 5 ms is far kinder than a timeout at 30 seconds — to the caller and to you.
• Shed by value. Under pressure, drop the batch export before you drop the checkout.
• Cap in-flight work. Concurrency limits per dependency stop one slow downstream from consuming every worker you have.

The instinct is that rejecting requests is failure. The alternative is accepting requests you cannot serve, which is also failure — just later, and for everyone at once.

What multi-region does and does not give you

Multi-region buys you availability when a region fails and lower latency for users near a region.

It does not buy you throughput on anything that must stay consistent. A globally consistent write still costs at least one cross-region round trip, and the speed of light is not a configuration parameter — roughly 5 ms per 1,000 km in fibre before any equipment touches the packet. Frankfurt to Virginia is a physical floor of tens of milliseconds, per round trip.

If your write path crosses regions, you have not scaled writes. You have added latency and a partition mode to test.

Finding the actual bottleneck

Brendan Gregg’s USE method is the fastest route. For every resource, measure three things:

Most teams export utilisation and stop. But a pool at 100% utilisation with an empty queue is perfectly healthy, while a pool at 60% with a queue thirty deep is your outage. Saturation is the metric that names the bottleneck, and it is the one almost nobody graphs.

Practical version: for every pool in your system — database connections, HTTP clients, thread pools, worker queues — export the queue depth and the wait time, not just the in-use count.

Your load test is lying to you

This one deserves its own section, because it invalidates the measurement most teams trust most.

Most load generators work in a closed loop: send a request, wait for the response, send the next one. It seems reasonable. It produces latency numbers that are quietly, systematically wrong.

Picture a generator with 100 virtual users hammering a service that stalls for 2 seconds. During that stall, each user is blocked waiting. They send nothing. When the stall clears, everyone resumes and records a normal-looking latency.

The result: the worst two seconds of your test contributed almost no samples. Your p99 is computed over a set of measurements that excludes the period you actually care about.

This is coordinated omission, named by Gil Tene. The load generator has unwittingly coordinated with the system under test to only sample it when it is healthy.

Real users do not behave that way. They arrive on their own schedule. A user who clicks during your 2-second stall waits the full 2 seconds — plus however long the queue in front of them takes to drain.

How to tell if you have it: if your p99 under load is suspiciously close to your median, and your users report freezes you cannot reproduce, you almost certainly do.

The fixes:

• Measure against the intended schedule, not the actual one. If you meant to send a request at t=0 and only sent it at t=1.8s because you were blocked, that request’s latency starts at t=0.
• Use a corrected tool. wrk2 was written specifically for this. HdrHistogram provides recordValueWithExpectedInterval to backfill the missing samples.
• Prefer an open model. Generate load at a fixed arrival rate regardless of whether previous responses came back — which is what real traffic does.

Until you fix this, every latency number you have collected under saturation is an underestimate, and the worse the stall, the bigger the lie.

Finding it in practice

Enough theory. Here is the actual sequence when latency is up and CPU is not.

Start with the database, because it usually is.

-- What is everyone waiting on, right now?
SELECT state, wait_event_type, wait_event, count(*)
FROM pg_stat_activity
GROUP BY 1, 2, 3
ORDER BY 4 DESC;

A pile of rows in Client / ClientRead means the database is idle and waiting on you — the bottleneck is elsewhere. A pile in Lock or LWLock means contention inside the database. Many idle in transaction means your application is holding connections open without using them, which is the same as having a smaller pool than you think.

-- How much connection headroom is left?
SELECT count(*) AS used,
       current_setting('max_connections')::int AS max_conn
FROM pg_stat_activity;

Then the sockets.

ss -s                                   # totals, including TIME_WAIT
ss -tan state time-wait | wc -l         # how deep is the TIME_WAIT hole
ss -tan state established | wc -l       # what you actually have open

If TIME_WAIT is in the tens of thousands, you are not reusing connections. That is a client configuration problem, not a capacity problem.

Then the kernel tables.

cat /proc/sys/net/ipv4/ip_local_port_range
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max

Then the process limits.

cat /proc/<pid>/limits | grep 'open files'
ls /proc/<pid>/fd | wc -l

The whole sequence takes about three minutes and will identify the binding constraint far more often than another dashboard will.

The checklist

The bottom line

CPU is the easiest thing to graph, which is why it is the first thing people look at and the last thing that is usually wrong.

Requests queue for connections, descriptors, ports, locks and threads — and every one of those queues follows the same arithmetic. Concurrency equals arrival rate times latency. Know two terms and you know the third, which means you can calculate your ceiling instead of discovering it during an incident.

Before adding capacity, answer one question: what is the request waiting for? If you cannot name it, another instance will not fix it — it will just give the same queue more customers.