The test invoice was fake. The money was real.

The incident

On the night of July 15 we were proving the Telegram Stars subscription loop for BluffKing — invoice → payment → paid order → entitlement → refund — on Telegram's dedicated test environment: the Bot API's /test namespace, a separate universe with its own accounts, its own bots, its own Stars balances. The bot side behaved perfectly. Our 1,000 XTR test invoice was created under /test, paid from the test account, activated the subscription, and was later refunded by the bot — every Star returned.

Apple's purchase history told a different story. S$1,077.00 of real money had moved.

The first hypothesis — that a production Telegram account had somehow crossed into the test flow — was wrong. Telegram's own transaction view showed both top-ups credited straight into the test account. Nothing crossed, and no external component malfunctioned. The failure was in our own heads: our test runbook asserted the flow “could not charge real money”, and nobody had verified that sentence at a real payment sheet.

Root cause: we specified the test invoice, not where test Stars come from

The root cause was not a Telegram bug or an Apple bug. Our runbook configured the Test Server invoice but never defined a safe source for the wallet balance that would pay it. One sentence papered over that gap: “the Test Server flow cannot charge real money”. It was true for the invoice and false for the top-up — we bet the money on the second half becoming true automatically.

To see why the top-up half failed, unpack what paying a Stars invoice actually touches — three independent ledgers:

1. The bot's invoice ledger (Bot API): invoices, payments, refundStarPayment. This is what /test isolates.
2. The user's Stars wallet — an XTR balance attached to the Telegram account.
3. The platform IAP ledger that funds the wallet — Apple StoreKit (or Google Play Billing) selling Stars for cash.

The test flag lives on ledger #1. We were paying from the stable Telegram app downloaded from the App Store; signing a Test Server account into that client does not turn the client into a StoreKit sandbox. When the account lacked enough Stars, Telegram opened its normal top-up sheet. Nothing on that sheet said Sandbox, so StoreKit treated it as a production purchase against the Apple ID's real payment method. That was the S$29. Forty-eight minutes later — the test stack already torn down, but the mental model “Test Server Stars are free” still standing — a 35,000-Stars top-up meant to bankroll the next day's renewal test went through the same production sheet: S$1,048.

The hypothesis we had to rule out: “just make the local test stack open Apple's sandbox”

In the review we kept returning to one idea: the test stack ran locally, yet the cashier was production — isn't that just a misconfigured environment? Point local at Apple's sandbox and the problem is solved. That instinct is exactly right for first-party rails — our own test environment does target Stripe's test mode, and our own app's IAP does run in the StoreKit sandbox. On this flow it fails, for three independent reasons:

1. None of our environments opened the cashier. Our test stack's part in this chain is bot-side only: it creates the XTR invoice through the /test Bot API, receives its payment webhooks, and issues the Stars refund. The “Buy Stars” sheet was opened by the Telegram client itself when the balance came up short, selling Telegram's own in-app product — not one byte of that top-up touched our server. On our side it has no order, no webhook, no log line. Our local-vs-production switch isn't on the top-up path at all.
2. The sandbox switch is welded to whose build it is, not to any server. Apple's rule: an App Store build always transacts in production; TestFlight and development-signed builds always transact in the sandbox. The environment is a property of how the app binary is distributed, and sandbox access is provisioned by the developer team that owns the app. That app is Telegram's — only Telegram can produce a sandbox Stars cashier. The Bot API has no parameter, webhook, or setting by which a merchant bot can choose the payer's StoreKit environment. “Make local open the sandbox” wasn't a switch we forgot to set; it's a switch that was never ours.
3. The bigger charge happened with the test stack not even running. The S$1,048 top-up postdates the teardown, with no order, webhook, or consumption anywhere near it — a manual client-side purchase to stockpile Stars for the next day's renewal test, motivated precisely by “test-environment Stars are free anyway”. A root cause has to explain both charges: “local opened the production cashier” cannot touch the second one; “no safe source of test Stars, plus an unverified free-money sentence” explains both.

The actionable half of the instinct — a test stack must never march anyone up to a real cashier — is exactly what the fix below implements: the amount is pinned to 1 XTR and the balance must be attested up front. That is a contract, not a fuse: enter within it and the top-up sheet has no reason to appear; if it appears anyway, the run is already outside its contract and the only correct move is to cancel. The half that doesn't exist is “the sheet that appears should be a sandbox sheet” — because the sheet was never ours.

Where test Stars actually come from

Telegram's payment guide says a bot can freely test Stars payments after connecting to the dedicated test environment. It does not promise that “Buy Stars” in a stable client is free. The Test Server's @izpremiumbot previously issued free balance through test-card payments, but a maintainer in Telegram Desktop's official repository said in October 2025 that free Stars had been locked after abuse. As of a 2026-07-17 re-check, we have not verified any still-working official zero-cash top-up path.

The safe flow today is therefore not “top up in test”. It is pre-provision a non-store test balance: receive Stars inside the Test Server from another already-funded Test Server account, for example through Telegram's protocol-supported paid messages, or a Star Gift that can be converted to Stars. Check the transaction history before testing: the source must not be App Store, Google Play, Fragment or @PremiumBot, and it must not be a refunded or disputed balance awaiting clawback. If no trusted test-network balance exists, stop real-provider E2E. Keep running local simulations, but never buy Stars just to make the test finish.

Apple does provide a real StoreKit sandbox, where development-signed and TestFlight purchases do not incur charges. That is a separate switch from Telegram's Test Server. Unless both the client build and the payment sheet explicitly identify Sandbox, treat the sheet as production. The stable Telegram build in this incident did not meet that condition.

The /test namespace isolates the invoice ledger. In this incident, the wallet was funded by production StoreKit opened from the stable Telegram app.

One asymmetry made the loss feel especially helpless: the refund tool we did have, refundStarPayment, operates on ledger #2 — it returns XTR to the account. It cannot reverse a StoreKit charge on ledger #3. Different ledgers, different refund systems — the bot refunded our test payment in full, and the cash was still gone.

The reusable lesson is double. First, test mode is a property of each ledger, not of your session: a flow that crosses N ledgers needs the “this is fake money” property verified on all N — the sandbox you configured covers exactly the ledger it belongs to, nothing upstream, nothing downstream. Second, any “this is free” sentence in a runbook is a guess until someone has verified it at the actual payment sheet.

Getting S$1,077 back

Apple granted both refunds on July 16 — the same day the request was filed, well inside the 48-hour window its own support line had quoted. Four things made the case winnable:

• Freeze everything. From the moment the charges were discovered: no spending, no gifting, no further purchases, and the test account stays alive. All 36,000 Stars sat untouched, explicitly offered back for complete reversal. A refund case where the purchased goods are fully preserved is a different conversation from one where they're half-consumed.
• Correlate the ledgers, minute-exact. Telegram's transaction view showed both top-ups arriving “via App Store” into the test account; Apple's purchase history showed the matching S$29 and S$1,048. The 1,000-Stars top-up is minute-exact against the sole test payment; the 35,000-Stars top-up has no consumption at all. Two independent ledgers telling one consistent timeline is evidence, not narrative.
• One case, both purchases, a manual review. In our case the self-serve refund page showed “this purchase does not have an invoice” while the charges were still posting. The path that worked here: put both transactions in a single support case and ask for a manual exception review, with the timeline above as the story. Refund decisions belong to Apple and outcomes vary — this is what worked once, not a guarantee.
• Never misreport it as fraud. It's tempting — a fraud claim feels like a fast lane. It's also false, easily disproven (the top-ups came from your own device), and it can end with a disabled Apple ID. The truthful story — an accidental purchase during payment testing, fully preserved for reversal — was enough.

The fix: the guard moved from prose to code

“Never top up during a test” was already written down before the incident — in the instructions. Instructions are exactly where it failed. The rule now lives in the server as a fail-closed guard (enforce_telegram_test_server_money_guard, server/src/handlers/billing.rs), and the doc-comment on it is the incident in three lines:

/// Telegram's Bot API Test Server isolates the invoice, but does not by itself
/// make a store top-up free. A stable iOS/Android client can still open a live
/// App Store/Google Play purchase sheet when the test account lacks Stars.

What it enforces:

• Restricted runtime or nothing. Every non-production real-provider invoice requires the dedicated wrapper-established runtime; outside it the request is refused. The wrapper script itself accepts exactly two fixed actions — register the test webhook, or run the repository-built server — and rejects arbitrary child commands, so there is no side door to the raw Bot API token.
• Amount pinned to 1 XTR — Telegram's minimum. Any other amount is refused. The blast radius of the worst repeat incident is one Star of already-owned balance.
• Attested existing balance. The operator must pass --existing-test-stars=N, attesting to Stars already visible in the test account. If the balance is short, the run stops before any invoice exists.
• Attested provenance. The wrapper also requires --test-stars-provenance=non-store-test-balance: the spendable balance came from inside the Test Server, not App Store, Google Play, Fragment, @PremiumBot, or a refunded/disputed top-up awaiting clawback. A balance number alone is no longer enough.
• One invoice per run, enforced transactionally. The wrapper mints a run id; the server takes a Postgres advisory lock on it and consumes the run's single-invoice budget in the same transaction as the order insert. The budget is spent even if the provider rejects the call — a retry demands a fresh wrapper run and a fresh balance attestation, instead of silently risking another StoreKit sheet.

billing_telegram_test_server_runtime_required                → not in the restricted runtime: refuse
billing_telegram_test_server_amount_must_be_one              → any amount ≠ 1 XTR: refuse
billing_telegram_test_server_existing_balance_not_confirmed  → no attested balance: refuse
billing_telegram_test_server_non_store_balance_not_confirmed → no non-store provenance: refuse
billing_telegram_test_server_invoice_budget_consumed         → second invoice in one run: refuse

The design assumption is not “the operator learned their lesson”. It is that someone — a human at midnight, or an agent following a stale runbook — will hold the wrong belief again, and that for any invoice going through the guarded server path, the belief should cost at most 1 XTR whose non-store provenance was separately attested. The guard's honest limit: it governs invoice creation on our side. A manual top-up tapped directly in the client never touches our server, so no server code can cap it — which is exactly why “never top up during a test” stays a standing operating rule next to the code, not inside it.

The incident review paid twice, as they usually do. Reproducing the flow shook out two ordinary server bugs: lapsed subscriptions couldn't be cancelled or refunded, because the provider's explicit SUBSCRIPTION_NOT_ACTIVE rejection was treated as a hard failure; and the subscription period was hardcoded to 30 days, which made a repeatable renewal test impossible — it's now a test-only override accepting exactly the 60 s and 300 s periods Telegram's test environment supports, failing closed under production-strength config.

If you're building on Stars — or any wallet-funded IAP

1. Enumerate the ledgers before you test. Invoice, wallet, funding source — write them down and ask “which of these did my sandbox flag actually reach?” For Telegram Stars the honest answer is: the invoice, nothing else.
2. Treat every in-client purchase sheet as real money, whatever environment you believe you're in. A Face ID prompt or a price in your local currency mid-test means cancel first, investigate second.
3. Price test SKUs at the provider minimum. Our test invoice was 1,000 XTR because it mirrored the production price; nothing about the test needed it. 1 XTR proves the same loop.
4. Enter with a provenance-checked non-store test balance. Receive it inside the Test Server and verify its transaction history. If the balance is short or its source is unclear, stop — don't feed the wallet.
5. Encode the rules as fail-closed code, not runbook prose. Our prose said “this cannot charge real money” and was wrong. The guard doesn't assert — it refuses.
6. If real money does move: freeze, correlate, tell the truth. Preserve the purchased goods untouched, build the minute-exact two-ledger timeline immediately, put everything in one support case, and never dress an accident up as fraud.

None of this is Telegram-specific. Any system where a sandboxed charge draws from a balance that production money can fill — game currencies, platform credits, token wallets — has the same seam in the same place.

Recap

1. What happened: a Telegram Stars payment test on the Bot API's /test environment triggered S$1,077 of real Apple charges. No external system malfunctioned — the sandbox isolates the invoice, and the Stars that paid it were bought with real money.
2. Root cause: our own runbook asserted “cannot charge real money” and nobody verified it. What made the sentence false: test mode is a per-ledger property, and the flow crossed three ledgers with the flag on one. “Point local at Apple's sandbox” was never on the table: that switch lives in Telegram's build, not in any merchant's hands.
3. The money came back: goods frozen for reversal, a minute-exact timeline across both ledgers, one truthful support case — refunded in full the same day.
4. The fix is structural: a fail-closed server guard — restricted runtime, 1 XTR only, separate balance and non-store-provenance attestations, one invoice per run under an advisory lock. Worst-case repeat through the guarded invoice path ≈ one provenance-checked test Star; the client-side top-up sheet stays covered by the standing “never top up” rule.
5. The takeaway: your sandbox covers exactly the ledger it belongs to. Count the ledgers — then check that every “this is free” sentence has been verified at a real payment sheet.