The cliff is not where the CPU is
We pushed a 2 vCPU production box until it fell over, to answer how many players fit. The number is useful; the three ways our first attempt was wrong are more useful — we measured the ramp instead of the steady state, we measured our own laptop, and we measured a machine still full of abandoned tables. Plus the counterintuitive part: when the p99 rose seventeen-fold, a core and a half was idle.
The question everyone answers with a guess
“How many people can our server handle?” is one of the few engineering questions where a confident answer is usually a made-up one. The honest version has three parts, and most teams skip two of them: how many, doing what, before which number gets bad.
We run a browser poker service on one small box: 2 vCPU, 3.6 GiB, Postgres on the same host, Caddy terminating TLS. Last night we stopped guessing and measured it — against the real production origin, not a loopback model. The number matters to us. The three ways our first attempt was wrong matter to everyone.
First: decide what “too slow” means, per surface
A single latency target is the most common mistake, and it is a mistake in both directions at once. Our own starting assumption was “keep everything under 500 ms.” That is simultaneously too loose and too strict.
The same millisecond costs a different amount depending on what the person is waiting for:
| What the user is waiting for | Budget we set | Anchored to |
|---|---|---|
| I tap CALL → my chips move | 150 ms server-side | 1 % of the 15 s action clock; the thing that decides whether the table feels alive |
| Someone else acts → I see it | 500 ms | Nobody is holding a button down waiting for it |
| A list of past hands loads | 500 ms | Our web client sets no fetch timeout at all, so a slow read hangs the UI rather than failing it |
| Login (memory-hard password hash) | 1500 ms | Paid once per session, and the hash parameters are at the security floor |
Note where each budget comes from: a constant already in the codebase, not taste. If you cannot point at the thing your budget is derived from, you do not have a budget, you have a preference — and preferences lose arguments at 2 a.m.
For us the practical upshot was blunt: 500 ms is about right for an API read and roughly 3× too loose for the action round-trip. A ceiling measured against the wrong budget is not a conservative estimate. It is a different number about a different question.
Second: the three ways we measured the wrong machine
We measured the ramp, not the steady state
The first run connected N clients and started timing immediately. That measures N users arriving, which is a real and interesting event, but it is not the same as N users being online — and it is the second one that decides how many people fit.
Once each rung warmed up for 20 s and then reset its samples, the difference was visible inside a single run. At 200 concurrent tables, the ramp window and the steady window disagreed by half:
200 tables, same run:
ramp window p99 379.9 ms
steady window p99 246.4 msIf you report the first number you will under-provision; if you report the second without saying so, you will be surprised by every marketing push. Measure both, label both.
We measured our own laptop
Halfway through the first ladder the tail started climbing while the server’s CPU sat still. The load generator was a single Node process on a developer machine — and on that machine, a stuck helper process was burning a full core.
The fix is not “be careful.” The fix is that a load generator must report its own health as a first-class output. Ours now records its event-loop delay every run:
generator_event_loop_delay_ms: { mean: 11.0, p99: 12.5, max: 15.1 }That line is what lets you say the numbers are about the server. Flat across every rung from 50 to 300 clients, well under the reported p99 — so the queueing being measured is the server’s. Without it, every tail number is an accusation you cannot support.
We measured a machine that was still full of ghosts
When a player closes their tab, our server keeps their table alive for 10 to 20 minutes before an idle reaper collects it. That is a reasonable product decision — people reconnect. It is also a measurement trap: after four ladder runs the box was holding ~800 abandoned tables, quietly costing about 7 % of both cores.
And then do the thing that turns a suspicion into a fact: we re-ran the collapsing rung once on a completely empty box — zero live tables, checked in the database seconds before the run. It collapsed identically. The ghosts were real, and they were not the cause. A control run is cheap; an unexamined confounder gets quoted for years.
Two lessons, and the second is the useful one. Obviously: check what is already running before you attribute a number to your load. Less obviously: that ghost load is real in production too. A busy hour leaves a residue of tables nobody is sitting at, and they consume the same runtime as the live ones. Any capacity limit that counts only active users will be wrong during exactly the churny hour when it matters.
Third: the cliff is not where the CPU is
Here is the ladder, solo tables (one human plus five bots — our most expensive shape per person), measured end-to-end over TLS from a client ~45 ms away:
| live tables | action p50 | action p99 | server-side p99 | app CPU |
|---|---|---|---|---|
| 50 | 43.0 ms | 115.9 ms | 70 ms | 15 % |
| 100 | 43.2 ms | 115.0 ms | 72 ms | 24 % |
| 150 | 43.9 ms | 161.4 ms | 118 ms | 34 % |
| 200 | 44.3 ms | 246.4 ms | 201 ms | 41 % |
| 300 | 85.6 ms | 4116 ms | 4071 ms | — collapse |
Read the last two rows together. Between 200 and 300 the p99 went up by a factor of seventeen, and the tables stopped making progress: the same 60-second window that produced 697 finished hands at 200 tables produced 148 at 300, and the run collected 90 latency samples where the rung below it collected 2994. All of that while roughly a core and a half sat idle.
This is not exotic. It is what a small async runtime does. Work arrives in bursts; a burst that needs both worker threads for 40 ms delays everything that arrives during those 40 ms; the delayed work makes the next burst bigger. Averaged over a minute, the box looks half asleep. The queue does not average.
Two consequences worth stealing:
- An autoscaling or alerting rule on average CPU will not fire before your users are in the collapse. Ours would have been comfortably green at the last healthy rung and still green while the tail was seconds long. Alert on the tail, or on a queue depth, or on shed count — not on how busy the box looks.
- Extrapolating from a healthy point is worthless near the cliff. 41 % CPU at 200 tables does not mean 480 tables is possible. The relationship is not linear and it does not degrade gracefully; it holds and then it falls over.
The unit you are measuring is probably not the unit you charge
We started with a mental model of “concurrent users,” because that is the number a founder gets asked about. The server does not have that concept. It has tables: one long-lived task per game, owning the state machine, the timers, the bot decisions and the fan-out.
Six friends at one six-max table are one of those. Six people each practising against bots are six. Same six “concurrent users,” six times the expensive object.
So before you pick a number to defend, find the thing that actually costs money and count that. For a chat service it is probably rooms and not sockets; for a video app it is probably transcodes and not sessions; for us it is tables. Then express the limit in that unit — otherwise the same limit is too tight for the cheap population and too loose for the expensive one.
The checklist
- Write the budget down per surface, anchored to a constant you can point at. One global latency number is always wrong for something.
- Measure the box you actually run, over the real network, through the real TLS terminator. Loopback numbers are a different machine wearing your machine’s name.
- Warm up before you time. Report arrival and steady state separately, and say which one your ceiling is.
- Instrument the load generator. If it cannot prove it was healthy, its tail numbers are unfalsifiable.
- Check what the machine was already doing. Idle-but-alive work is real work, in the test and in production.
- Walk the ladder past the knee. The interesting information is where it stops being linear, and you only get that by breaking it once, deliberately, at a time you choose.
- Alert on the tail, never on average CPU.
Then, and only then, pick your operating point below the measured ceiling — and enforce it, because a ceiling nobody enforces is a ceiling you discover from your users. That enforcement is its own subject, and its own post.