Why WebSocket delivery backed up: the 2 Mbps suspect and a measured 22.3% byte cut

Three answers up front: bottleneck, change, and proof

In one sentence: the server could compute the game, but it generated outbound JSON faster than a 2 Mbps link could drain it; this pass reduced bytes per hand, and the next production retest must locate the new knee.

A load test should leave more than a capacity number

Our previous production test produced a clear curve: 150 solo-practice tables stayed inside the latency budget, 200 missed it, and 300 drove p99 to 4.1 seconds. The collapsing rung also produced 199 outbound-backpressure evictions.

CPU was not saturated. Neither was Postgres. That made outbound delivery the leading hypothesis, not the conclusion. Two questions were still open:

• Was the limit Tokio scheduling, JSON serialization, per-connection queues, or the production box's fixed 2 Mbps public egress?
• If we changed the code, how much work did it actually remove, and did the capacity boundary move?

“300 collapsed” was therefore the beginning of the optimization pass. We needed to split vague outbound pressure into frames, bytes, and message types, then put the old and new code under the same workload.

Instrument the scarce resource first

The load generator already recorded action latency, disconnects, and completed hands, but not how much data the server sent. We added the UTF-8 byte length of every received WebSocket text frame, grouped it by message kind, and separated ramp traffic from the steady window. Four shards opened the same 20-second window at one future timestamp, so their traffic and percentiles described the same seconds.

Before changing the protocol, we ran an isolated local baseline: a release binary, two Tokio workers, four shards of 250 connections, and 1,000 simultaneously playing solo-practice tables. Each table had one simulated human and five bots.

That changed the question. The same server code did not hit a 200-table scheduling ceiling locally; 1,000 tables still held a 1.2 ms p99. But JSON payload alone required about 4.07 Mbit/s—more than twice the production instance's fixed public egress. “200 is the CPU limit” was false. Outbound bytes became the first useful optimization target.

Why we did not enlarge the queue or enable generic compression

A larger queue only stores congestion for longer. Egress stays fixed while memory use and tail latency grow. Generic WebSocket compression might save more bytes, but it can also put secret cards and user-controlled text in one compression context. A card game should not enlarge that side-channel surface for a better benchmark result.

We could not silently change the meaning of missing JSON fields either. Flutter, old H5 builds, and unknown clients still consume the full protocol. If the server simply stopped sending fields, bandwidth savings would become a compatibility incident.

We chose an explicitly negotiated outbound profile:

GET /ws?session_id=...&role=player&wire=compact-v1

• Only the current H5 opts into wire=compact-v1.
• An absent or unknown value keeps the old raw full-JSON path.
• Every socket owns its compact state. Reconnect starts with an empty cache, so the first state does not depend on the old connection.
• Per-viewer spectator and card-privacy projection runs before field omission. Optimization cannot bypass the information boundary.

Send fewer bytes without changing table state

compact-v1 does not redesign the whole protocol. It touches only data whose missing-value semantics can be written as deterministic rules:

• Omit defaults. A snapshot no longer repeats zero pot/current bet, empty board/chat, zero committed or pending chips, and folded=false. H5 restores missing values as zero, empty, or false.
• Deduplicate static fields. Later snapshots on one connection may omit an unchanged player name and cumulative buy-in. Standings may omit unchanged name, human flag, cumulative buy-in, and presence. Dynamic stacks, net results, and hand counts are still sent.
• Suppress no-op frames. A hole_cards_dealt message sent to a non-owner carried only cards:null, and H5 did nothing with it. The compact path drops that frame. A real card pair is never omitted.
• Deduplicate idempotent state.room_rules and time_bank are suppressed only when their data is identical to the prior frame on that socket. Any change is sent in full. Messages that release optimistic locks or trigger one-shot side effects are excluded.

Why not replace long names with one-letter codes?

We can. Encoding the message type hole_cards_dealt as h saves 15 bytes per occurrence; encoding the field name folded as f saves 5. But compact-v1 first took the larger wins: it drops the entire no-op hole_cards_dealt frame for non-owners and omits the entire folded=false field.

Of the optimized run's 7,912,044 B, every remaining hole_cards_dealt message together used only 61,100 B, or 0.77%. Even deleting that whole category would cap the saving at 0.77%; shortening only its type name must save less. By comparison, action_applied plus snapshot accounted for 52.07% of optimized bytes. A useful compact-v2 would therefore version a complete short-code table for frequent message types, keys, and enum values, decode it back into the existing H5 structures, and rerun the same A/B. Renaming two long strings alone will not remove “a lot” more traffic.

The governing rule is not “remove as many fields as possible.” It is: every field the server omits must have one deterministic, tested reconstruction rule in the client. Otherwise bandwidth falls while state divergence accumulates.

The result under the same 1,000-table load

We rebuilt the release binary and repeated the two-worker, 4×250-shard run over the same 20-second steady window:

Raw window bytes fell 22.3%. Because the optimized run completed 25 more hands, normalized bytes per hand fell 23.3%. Tail latency and stability did not regress, so the result did not come from doing less work.

Optimized application JSON averaged about 3.16 Mbit/s. That is lower, but still above the production instance's 2 Mbps fixed egress—and it excludes WebSocket, TCP, TLS, and reverse-proxy overhead.

This did not move production from 200 to 1,000 tables

The most attractive conclusion would also be the wrong one: production began struggling at 200 tables; the optimized code passed 1,000 locally; therefore capacity grew from 200 to 1,000.

Those numbers came from different environments. In fact, the pre-optimization code already held 1,000 tables locally. This change did not move a local limit from 200 to 1,000. It showed that local scheduling was not enough to explain the production cliff, then cut application egress for the same workload by about 22%.

Production capacity at 1,000 tables remains unproven. The optimized number itself says not to claim it: 3.16 Mbit/s of application payload is still above a 2 Mbps link. The next valid experiment is to deploy the code, repeat the original production ladder at 150, 200, 300, and higher rungs, and observe the real interface, Caddy queues, fanout_backpressure_evict, and action p99. A new knee under the same environment and workload is a capacity improvement; a local number is not.

A reusable performance-optimization loop

1. Define the workload. Connections are not enough. Record sessions per connection, bots, action frequency, and the steady window.
2. Turn the load-test result into a hypothesis. What do CPU, database, queues, and egress each support? A mechanism that has not been isolated is only a leading hypothesis.
3. Instrument the suspected resource. If egress is suspect, record frames, bytes, and message kinds. Latency alone cannot tell you which cost to remove.
4. Fence compatibility and security first. Negotiate the new profile, preserve old clients, make reconnect self-contained, and keep privacy projection ahead of optimization.
5. Compare under the same window. Track raw throughput, cost per unit of work, p99, errors, and load-generator delay together.
6. State what remains unproven. Local success is not production success, and a byte reduction is not a proportional capacity increase.

Recap

1. The production load test narrowed the problem to outbound delivery but did not prove one mechanism.
2. A local 1,000-table baseline held stable scheduling while exposing about 4.07 Mbit/s of application JSON egress.
3. An explicitly negotiated compact-v1 omitted defaults, merged repeated static state, and suppressed no-op frames without changing the legacy client protocol.
4. Under the same load, raw bytes fell 22.3%, bytes per hand fell 23.3%, and frames fell 18.2%; action p99 and early closes did not change.
5. This completes one “find the bottleneck → optimize → rerun” loop. It does not prove 1,000-table production capacity. The next loop returns to the real 2 Mbps link.