The webhook is the only writer

We found the bug in a CSV. A finance audit of gateway collections turned up payments that had been booked twice — same student, same fee, two Payment Entries. Nobody had written a “book it twice” feature. We had written two “book it once” features, and they raced.

The anatomy of the race is worth spelling out, because every payment integration has it. When a gateway payment succeeds, you hear about it twice: once in the browser, when the gateway redirects the payer back to your success URL, and once on the server, when the gateway’s webhook lands. Both callbacks carry enough information to create the Payment Entry. So, naturally, both did — each one guarded by an “is this already paid?” check that passed for both, because both checked before either wrote.

Demote one writer

The fix was not a smarter lock. It was a design rule: the webhook is the only writer. The server-to-server callback — signed, verifiable, retried by the gateway until acknowledged — creates the Payment Entry and posts to the ledger. The browser redirect was demoted to display-only: it may show a success screen, poll for status, thank the payer. It writes nothing.

Once a single code path is authoritative, the remaining hard problems get much smaller, because they all live in one place:

• Idempotency.Gateways redeliver webhooks — that is a feature, not a bug. The handler treats the gateway’s transaction ID as the idempotency key and checks it inside the same transaction that writes, not before it.
• Verification. One place to verify HMAC signatures — with a constant-time compare, because a timing oracle on a payment endpoint is a bad week waiting to happen.
• Amounts.The webhook’s settled amount is authoritative, not what the UI thought the payer owed. If a concession or late-fee waiver was applied mid-session, the handler reconciles the difference before posting; if the document was already paid, it emits a reconciliation-required audit log with the exact delta instead of guessing.
• Evidence. The raw webhook is logged and committed immediately, before any downstream processing — so if the handler throws, you still hold the gateway’s side of the story. Credentials and sensitive headers are redacted first.

And when the webhook never comes?

A single writer needs a safety net, because sometimes the gateway genuinely fails to deliver. Ours is a scheduled reconciliation sweep: query the gateway’s transaction API for recent payments, compare against what the ledger knows, and process anything missed through the same handler — same idempotency key, same code path. The sweep and the webhook can race each other safely, precisely because the already-paid check lives inside the write.

The audit CSV that started all this became our regression test. The double-bookings stopped — not because we got more careful, but because the design stopped requiring care. That is the standard worth aiming for in money code: correctness that survives an inattentive Tuesday.