--- title: "Scalability: The Bottleneck Is Almost Never the CPU" description: "Why CPU sits at 30% while your service times out: the utilisation knee, connection pools, ephemeral ports, coordinated omission in your load tests, and the commands that find the binding constraint." author: Aleksei Aleinikov date: 2026-09-11 lang: en tags: [scalability, load-balancing, distributed-systems, connection-pool, backpressure, queueing-theory] canonical: https://www.alekseialeinikov.com/en/blog/topics/architecture/scalability-bottleneck-is-not-the-cpu-2026 source: alekseialeinikov.com --- # Scalability: The Bottleneck Is Almost Never the CPU Your dashboard shows 28% CPU. Your p99 latency is 4 seconds. Someone suggests adding more pods. This is the most common wrong turn in scaling work, and it comes from a reasonable assumption: that a slow system must be a busy system. It usually is not. It is a system where requests are queuing for something that is not CPU, and adding instances often makes it worse. Here is what is actually limiting you, and the arithmetic to find it. ![Scalability bottlenecks: why CPU is idle while requests queue elsewhere.](https://www.alekseialeinikov.com/blog/scalability-bottleneck-2026.webp) ## One formula that sizes almost everything Little's Law is the most useful thing in capacity planning, and it fits on one line: ``` concurrency = arrival rate x latency L = λ × W ``` If 500 requests arrive per second and each spends 40 ms in the database, then at any instant you need: ``` 500 req/s × 0.04 s = 20 concurrent database connections ``` Twenty. Not two hundred. Now run it backwards, which is where it earns its keep. You have a pool of **20 connections** and each query takes **50 ms**: ``` 20 / 0.05 s = 400 requests per second ``` **That is your ceiling.** Not a soft limit — a hard one. You can run forty pods; the pool still admits 400 requests per second and everything else waits. CPU stays low because the CPU is not doing anything. It is waiting. Every queue in your system obeys this. Thread pools, connection pools, worker counts, in-flight request limits. If you know two of the three terms, you know the third. ## Why 80% utilisation is not 80% fine Here is the part that explains why capacity planning feels counter-intuitive. For a simple queue, response time relates to utilisation like this: ``` response time = service time / (1 - utilisation) R = S / (1 - ρ) ``` That denominator is the whole story. As utilisation approaches 1, response time does not rise gently — it goes to infinity. | Utilisation | Response time vs. an idle system | | --- | --- | | 50% | **2×** | | 80% | **5×** | | 90% | **10×** | | 95% | **20×** | | 99% | **100×** | Going from 50% to 80% utilisation does not cost you 30% of your latency budget. It costs you **150%** — response time goes from twice service time to five times it. This is why experienced teams keep what looks like wasteful headroom, and why a system that was fine yesterday at 70% falls over today at 85% after a modest traffic increase. Nothing changed except your position on a curve that turns vertical. **The honest caveat:** this exact formula assumes a single server with Poisson arrivals and exponentially distributed service times — an M/M/1 queue. Real systems have multiple servers and burstier traffic, so your constants will differ. But the *shape* is universal: queueing delay is hyperbolic in utilisation, not linear. Every queue you own has a knee, and it is closer to the left than intuition suggests. Practical consequence: **stop setting autoscaling targets at 80% CPU**. By the time you are there you are already deep into the curve, and the new instance takes time to become useful. ## The ceilings, in the order they usually bite ### 1. Database connections This is the most common real limit, and it has a number you can look up. **PostgreSQL defaults to `max_connections = 100`**, of which **3 are reserved for superusers** by default. So roughly 97 are actually available to your application. Now count what you are asking for. Twelve pods with a pool of 10 each is 120 connections against a limit of 97. The twelfth pod gets connection errors, and the error looks like a database problem rather than an arithmetic one. Two things make this sharper: - PostgreSQL **forks a process per connection**, so connections cost real memory. The docs are explicit that raising `max_connections` raises shared memory allocation with it. This is why a pooler like PgBouncer exists. - **A bigger pool is not more throughput.** Past the point where the database can genuinely execute queries in parallel, extra connections add lock contention and context switching. Throughput flattens, then falls. MySQL uses a thread per connection and is cheaper here, which is one of the [structural differences between Postgres and MySQL](https://www.alekseialeinikov.com/en/blog/topics/data/postgres-vs-mysql-2026) worth knowing before you pick. ### 2. File descriptors Every socket, every open file, every connection is a file descriptor. The per-process limit is often **1024** by default on the soft limit, which sounds generous until you notice it covers inbound connections, outbound connections, log files and library internals simultaneously. ```bash ulimit -n # soft limit for this shell cat /proc//limits # what the running process actually has ls /proc//fd | wc -l # how many it is using right now ``` The failure mode is memorable: `EMFILE: too many open files`, usually under exactly the load where you least want to be reading a stack trace. ### 3. Ephemeral ports and TIME_WAIT This one surprises people because nothing in the application mentions it. When your service opens an outbound connection, the kernel picks a source port from the ephemeral range. On Linux: ```bash cat /proc/sys/net/ipv4/ip_local_port_range # typically: 32768 60999 ``` That is about **28,000 ports** per unique destination address and port combination. Then the closing side holds the socket in **TIME_WAIT for 60 seconds** — and on Linux that duration is compiled into the kernel, not a sysctl you can turn down. Do the arithmetic. Open 500 short-lived connections per second to the same downstream service: ``` 500 conn/s × 60 s = 30,000 sockets in TIME_WAIT ``` You have run out. New connections fail, and the error will be a connect timeout that looks like the downstream service is down. **The fix is not raising the range.** It is not opening the connections: HTTP keep-alive, connection pooling, HTTP/2 multiplexing. Reuse beats provisioning. ### 4. Connection tracking If your traffic passes through a NAT gateway, a firewall or a Kubernetes node using iptables, the kernel keeps a **conntrack** entry for every flow. That table is finite, and when it fills, packets are dropped silently — no error, no log line in your application, just latency and retries. ```bash sysctl net.netfilter.nf_conntrack_max sysctl net.netfilter.nf_conntrack_count ``` The gap between those two numbers is a metric worth alerting on, and almost nobody exports it. ### 5. Threads and event loops A thread-per-request server is a pool by another name, and Little's Law applies unchanged. An event-loop server has no such limit — which is its own hazard, because it will happily accept ten thousand concurrent requests and serve all of them slowly instead of refusing some quickly. One blocking call on an event loop stalls every request on that loop. The CPU graph will look calm throughout. ## Horizontal scaling and the shared denominator Adding instances multiplies capacity for the **independent** part of a request and does nothing for the shared part. Amdahl's Law puts a ceiling on the speedup available from parallelising anything with a serial fraction. The **Universal Scalability Law** adds the part people miss, and it is worth writing out because the second term is the one that bites: ``` C(N) = N / (1 + σ(N-1) + κN(N-1)) N number of workers σ contention — the serial fraction, queueing for a shared resource κ coherency — the cost of keeping workers consistent with each other ``` With κ = 0 this reduces to Amdahl's Law: throughput flattens but never falls. With **κ > 0** something worse happens — there is a maximum, and past it throughput **decreases**: ``` peak at N* = sqrt((1 - σ) / κ) ``` The coherency term is why adding capacity can make a system slower. It covers cache-line bouncing between cores, lock handoffs, cluster gossip, distributed consensus, cross-region replication — anything whose cost grows with the *square* of the number of participants because each one must agree with every other. Concretely, if every new pod opens more connections to the same primary database, you are not scaling. You are converting a connection limit into a queue and then into a timeout. If every new node must gossip with every other node, you have bought yourself a κ term. Ask this before adding instances: 1. What does a request touch that is **shared**? 2. What is the current concurrency limit on that shared thing? 3. Does another instance **raise** that limit, or **compete** for it? If the answer to 3 is "compete", scaling out makes it worse. ## Load balancing: what the diagrams leave out The box labelled "LB" in your architecture diagram hides three decisions that determine whether scaling works. **Health checks that lie.** A check that returns 200 whenever the process is alive will keep routing traffic into an instance whose database pool is exhausted and whose queue is thirty seconds deep. A useful check verifies the dependencies a request actually needs — and distinguishes *liveness* (restart me) from *readiness* (stop sending me traffic). Conflating them causes restart loops under load, exactly when you need stability. **L4 versus L7 under HTTP/2.** L4 balances **connections**. With HTTP/2 or gRPC, one long-lived connection carries many multiplexed requests — so connection-level balancing can leave one backend doing most of the work while the others idle. The graph shows even connection counts and wildly uneven CPU. For HTTP/2 and gRPC you generally want **request-level** balancing. **Sticky sessions.** Pinning a user to an instance defeats balancing on purpose. It also means a restart drops those users' state, and a hot user becomes a hot instance you cannot spread. Then there is the retry trap: when a system is already saturated, automatic retries multiply the load that saturated it. Retries need [rate limiting and a budget](https://www.alekseialeinikov.com/en/blog/topics/architecture/rate-limiting-in-practice-algorithms-headers-2026), plus jitter, or they turn a brown-out into an outage. ## Queues absorb bursts, not overload A queue in front of a slow consumer buys you time across a **burst**. It does nothing about a sustained arrival rate above your service rate — it just moves where the failure appears. If arrivals exceed service capacity indefinitely, the queue grows without bound, and latency grows with it. Callers time out. They retry. Arrivals increase. This is a feedback loop with the sign pointing the wrong way. **Backpressure** is the answer, and it means being willing to say no: - **Bounded queues.** An unbounded queue is an out-of-memory error with extra steps. - **Reject fast.** A 429 with `Retry-After` in 5 ms is far kinder than a timeout at 30 seconds — to the caller and to you. - **Shed by value.** Under pressure, drop the batch export before you drop the checkout. - **Cap in-flight work.** Concurrency limits per dependency stop one slow downstream from consuming every worker you have. The instinct is that rejecting requests is failure. The alternative is accepting requests you cannot serve, which is also failure — just later, and for everyone at once. ## What multi-region does and does not give you Multi-region buys you **availability** when a region fails and **lower latency** for users near a region. It does not buy you throughput on anything that must stay consistent. A globally consistent write still costs at least one cross-region round trip, and the speed of light is not a configuration parameter — roughly 5 ms per 1,000 km in fibre before any equipment touches the packet. Frankfurt to Virginia is a physical floor of tens of milliseconds, per round trip. If your write path crosses regions, you have not scaled writes. You have added latency and a partition mode to test. ## Finding the actual bottleneck Brendan Gregg's **USE method** is the fastest route. For every resource, measure three things: | | What it means | Why it matters | | --- | --- | --- | | **Utilisation** | % of time the resource is busy | Misleads on its own | | **Saturation** | How much work is **queued** | Usually the real signal | | **Errors** | Rejections, timeouts, drops | Tells you what already broke | Most teams export utilisation and stop. But a pool at 100% utilisation with an empty queue is perfectly healthy, while a pool at 60% with a queue thirty deep is your outage. **Saturation is the metric that names the bottleneck, and it is the one almost nobody graphs.** Practical version: for every pool in your system — database connections, HTTP clients, thread pools, worker queues — export the **queue depth and the wait time**, not just the in-use count. ## Your load test is lying to you This one deserves its own section, because it invalidates the measurement most teams trust most. Most load generators work in a **closed loop**: send a request, wait for the response, send the next one. It seems reasonable. It produces latency numbers that are quietly, systematically wrong. Picture a generator with 100 virtual users hammering a service that stalls for 2 seconds. During that stall, each user is blocked waiting. They send **nothing**. When the stall clears, everyone resumes and records a normal-looking latency. The result: the worst two seconds of your test contributed almost **no samples**. Your p99 is computed over a set of measurements that excludes the period you actually care about. This is **coordinated omission**, named by Gil Tene. The load generator has unwittingly coordinated with the system under test to only sample it when it is healthy. Real users do not behave that way. They arrive on their own schedule. A user who clicks during your 2-second stall waits the full 2 seconds — plus however long the queue in front of them takes to drain. **How to tell if you have it:** if your p99 under load is suspiciously close to your median, and your users report freezes you cannot reproduce, you almost certainly do. **The fixes:** - **Measure against the intended schedule, not the actual one.** If you meant to send a request at t=0 and only sent it at t=1.8s because you were blocked, that request's latency starts at t=0. - **Use a corrected tool.** `wrk2` was written specifically for this. HdrHistogram provides `recordValueWithExpectedInterval` to backfill the missing samples. - **Prefer an open model.** Generate load at a fixed arrival rate regardless of whether previous responses came back — which is what real traffic does. Until you fix this, every latency number you have collected under saturation is an underestimate, and the worse the stall, the bigger the lie. ## Finding it in practice Enough theory. Here is the actual sequence when latency is up and CPU is not. **Start with the database, because it usually is.** ```sql -- What is everyone waiting on, right now? SELECT state, wait_event_type, wait_event, count(*) FROM pg_stat_activity GROUP BY 1, 2, 3 ORDER BY 4 DESC; ``` A pile of rows in `Client / ClientRead` means the database is idle and waiting on *you* — the bottleneck is elsewhere. A pile in `Lock` or `LWLock` means contention inside the database. Many `idle in transaction` means your application is holding connections open without using them, which is the same as having a smaller pool than you think. ```sql -- How much connection headroom is left? SELECT count(*) AS used, current_setting('max_connections')::int AS max_conn FROM pg_stat_activity; ``` **Then the sockets.** ```bash ss -s # totals, including TIME_WAIT ss -tan state time-wait | wc -l # how deep is the TIME_WAIT hole ss -tan state established | wc -l # what you actually have open ``` If `TIME_WAIT` is in the tens of thousands, you are not reusing connections. That is a client configuration problem, not a capacity problem. **Then the kernel tables.** ```bash cat /proc/sys/net/ipv4/ip_local_port_range sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max ``` **Then the process limits.** ```bash cat /proc//limits | grep 'open files' ls /proc//fd | wc -l ``` The whole sequence takes about three minutes and will identify the binding constraint far more often than another dashboard will. ## The checklist | Symptom | Likely cause | First thing to check | | --- | --- | --- | | Low CPU, high latency | Queuing on a pool | Pool wait time, queue depth | | Errors at N pods but not N-1 | `max_connections` exceeded | Pods × pool size vs DB limit | | `too many open files` | FD limit | `ulimit -n`, `/proc//limits` | | Connect timeouts to one downstream | Ephemeral ports / TIME_WAIT | Connection reuse, keep-alive | | Silent packet loss under load | conntrack table full | `nf_conntrack_count` vs `max` | | Uneven backend CPU, even connections | L4 balancing HTTP/2 | Move to request-level balancing | | Latency grows, never recovers | No backpressure | Bound the queue, shed load | | Adding pods makes it slower | Shared-resource contention | What do all pods compete for? | ## The bottom line CPU is the easiest thing to graph, which is why it is the first thing people look at and the last thing that is usually wrong. Requests queue for connections, descriptors, ports, locks and threads — and every one of those queues follows the same arithmetic. Concurrency equals arrival rate times latency. Know two terms and you know the third, which means you can calculate your ceiling instead of discovering it during an incident. Before adding capacity, answer one question: **what is the request waiting for?** If you cannot name it, another instance will not fix it — it will just give the same queue more customers.