BluffKing latency: tracing delays and measuring public capacity

1) How did we locate the latency?

At 01:22 on September 8, we ran 100 nine-player tables with 900 simulated player connections, alongside 1,000 history, session and statistics queries per second. Of 60,000 HTTP requests, 4,655 returned 503: a 7.76% failure rate. Overall HTTP p99 was 422.73ms, and the statistics endpoint reached 654.19ms. The p99 from a scheduled poker action to all nine players receiving its result was 108.41ms. A p99 is the duration at or below which approximately 99% of measured samples fall.

The first run exposed a query problem, but one percentile could not identify its cause. We separated each request into time waiting to be sent by the load generator, transport, waiting for the server process to read an arrived request, application processing, and delivery to every player. We also inspected database queries, thread scheduling, connection limits and background computation.

One production trace made the distinction concrete. A request spent about 111.85ms inside the server boundary, but only 0.048ms in the application handler. About 111.78ms elapsed after arrival and before the process read it, including approximately 80.74ms waiting for CPU. Timing only the handler would miss the delay that dominated the response.

The load generator could also distort measurements. When a timer fell behind, it issued overdue requests in a burst and delayed processing responses and table messages. Diagnosis therefore had to cover both the server and the tool sending and receiving the load.

2) What did we change to reduce latency?

Reduce database waits on live paths.Current table state, live standings, lobby and room queries now use memory projections. Player records and action timelines are persisted in batches per hand. History reads gained indexes, caches invalidated by committed versions, and batched version checks, reducing repeated reads and database round trips.

Separate required persistence from deferrable analysis.Required writes use bounded, ordered background queues. Successful balance, membership and settlement changes still require a database commit. Recovery checkpoints and ownership protection during process handoff remain in place. Equity estimates moved off realtime workers; post-hand analysis uses a dedicated background CPU worker with lower priority on Linux.

Avoid repeating the same computation and construction.Preflop hands with equivalent ranks and suitedness share equity estimates. Once the river board is fixed, an estimate reuses hand ranks while preserving its seeded draws, trial counts and results. Shared hand fields and identical broadcast content reuse construction and serialization work while preserving message order and private-card visibility.

Remove additional system waits.HTTP and WebSocket connections disable Nagle buffering. The table service cannot use swap. Its file descriptor limit rose from 1,024 to 65,536, leaving room for HTTP and database connections beyond the 900 player sockets. We increased the container's CPU scheduling weight and corrected its scheduling hierarchy so that priority also applies against other host workloads.

The last optimization commit in this sequence was 111f247d at 23:20, merged as e89e574f. The load generator now yields to network I/O after every eight overdue requests, retaining the original request count and deadlines. This improves the measuring tool's scheduling; it is not itself a player-facing server speedup.

The initial run used local history queries; the final run used production realtime queries. Hardware, endpoints and the generator changed, so these runs do not establish a controlled overall speedup percentage. The initial 4,655 HTTP failures and the final zero failures also belong to different test conditions.

3) Is there still room to improve performance?

Yes. Queuing and timing variability are the next useful targets. In the final September 8 run, the realtime HTTP handler's p99 was only 0.171ms, while the complete server ingress-to-egress p99 was 7.08ms. WebSocket p99 from server ingress to writing the result for all nine recipients was 12.20ms. Waiting outside the handler, table-task scheduling and the final part of broadcast delivery still warrant investigation.

The generator is not yet free of interference. That run’s HTTP scheduling-delay p99 was 618.80ms, and end-to-end p99 including that wait was 625.09ms. We cannot attribute all of it to the server or simply remove it from acceptance. For the subsequent public-path tests, we used a separate client machine and calibrated 900 connections against an independent echo process. Across 5,000 nine-recipient group actions, all-recipient p99 was 21.16ms and scheduling p99 was 2.20ms. This validates the generator for the intended connection count and action load; it does not substitute for a public-path measurement.

Further isolation of background analysis, less repeated state transmission and distributing tables across service instances offer possible gains, but each needs measurement. Independently calculated stage p99 values cannot be added together. Server time also excludes the final screen update on a player's device. That run still missed the original goal of keeping every server p99 below 10ms.

The September 9 public-path follow-up adds evidence of tail-latency headroom: with 225 players and one action per table every 2.5 seconds, p95 was 114.55ms but p99 reached 4.03 seconds. The next diagnosis should correlate thread waiting, broadcast queues and public egress during those spikes. This run does not identify a unique bottleneck or justify scaling player capacity in proportion to average action rate.

4) How many people can play on production with p99 below 500ms?

Public-path results: 117 players is the highest repeated passing slower-play load; 45 passed rapid play, while 54 showed variability across the threshold.With a target of five action opportunities per table per second, 45 players achieved 267.41ms p99 over three minutes. Two runs with 54 players returned 506.12ms and 290.04ms, showing variability around the limit; 63 players returned 536.41ms over three minutes. Thus, 45 players is the passing point below the observed unstable range in this test. The highest single passing run was 54 players, which is not a stable service commitment.

The table includes every rapid-play run on the same production version. Rates describe target action opportunities; hand transitions or waiting for the previous acknowledgement can reduce completed actions. The table reports actual acknowledged samples. Each table has nine authenticated simulated players checking or calling over H5’s compact-v1 protocol, with sending times staggered across tables. Latency runs from the scheduled action time until all nine receive the result, including generator scheduling and public-network transport. We also report the maximum: p99 below 500ms does not mean every action finishes within 500ms.

The highest passing slower-play load is 13 tables and 117 players.With a target of one action per table every 2.5 seconds, two three-minute runs returned 215.48ms and 142.37ms p99 across 906 and 905 samples. The adjacent load of 14 tables and 126 players returned 606.36ms; 15 tables returned 638.93ms, 18 returned 1,534.61ms, and 25 returned 4,031.06ms. For this public route and workload, 117 players is the highest repeated passing load, while 126 exceeds the 500ms limit. This slower pace is a test model, not a rate derived from observed human behavior.

How did we test many connections from one public address?Production allows 900 ordinary WebSocket connections, plus 100 reserved returning-connection slots, with a limit of 12 connections per public IP. We used an expiring accounting lease for authenticated fixture accounts, bound to the source IP, run, runtime version, account list and expiry. It changes only those accounts’ per-IP accounting; global and account-level protection remains active. Connections still traverse the public network, TLS and reverse proxy at https://app.bluffking.ai. After testing, we removed the lease and verified that all fixture seats were released. This lets one source measure service capacity over the public path; it does not simulate independent regions, carriers or devices.

Every listed formal window completed its intended duration, and every table completed actual hands. There were no disconnects; all actions were acknowledged, and required persistence and chip conservation passed. The 225-player run also timed out in one table’s seat-release step and failed its cleanup criterion; a final database check confirmed no fixture seats remained active. Cleanup checks passed for the other listed runs. There was no concurrent HTTP query load. The older test also requires p99 at most 200ms and maximum latency at most 1,000ms. Those failures remain in the original reports; this article evaluates the requested p99 strictly below 500ms separately.

Why did the earlier test support 900 players?The September 8 run completed at 23:29 on a production host with 2 vCPUs and approximately 4GB RAM. The generator shared the host and used a private address. With 100 tables, 900 connections, five actions per table per second and 1,000 realtime HTTP queries per second, it completed 24,665 actions and 681 hands in about a minute. Action p99 was 214.64ms. All 60,000 HTTP requests succeeded, but HTTP end-to-end p99 was 625.09ms. That establishes private-path processing under those conditions, excluding public transport. The initial one-table public follow-up after publication recorded 355.33ms p99 across 253 actions.

These are measured boundaries for a specified version, route and workload, not a permanent maximum for every network. Three-minute windows remain short; mixed queries, longer operation, other carriers and mobile networks can reduce usable capacity. We also cannot promise imperceptible delay from server or message-receipt timing alone: that requires measuring button feedback and screen updates on real devices. A configured ordinary connection ceiling of 900 does not guarantee sub-500ms public gameplay for 900 active players.

All times are Singapore time. The optimization history covers the September 8, 2026 baseline at 01:22 through commit 111f247d at 23:20 (mainline e89e574f). The September 9 public capacity ladder ran on web-2026.09.09.3 (bb0d2c9c). Each row is calculated independently; p99 values from different windows were not averaged.