Webhooks fire twice. Your invoices shouldn't.
Notes on payment idempotency and invoice integrity from building a multi-tenant commerce platform.
On a multi-tenant commerce and CRM platform with 257 REST endpoints, the bugs that worry me most aren't crashes. They're the quiet ones: a customer charged twice, or an invoice that disagrees with the payment it belongs to.
Nearly all of them come from one fact: every network operation can happen more than once.
Where duplicates come from
- A client retries because the response timed out, even though the request succeeded.
- A payment provider redelivers a webhook because your endpoint answered slowly.
- A queue worker crashes after doing the work but before acknowledging it.
None of these is an edge case. At scale, all of them happen every day.
The rules I build by
1. Every mutating request carries an idempotency key. The server stores the key with the result. A repeat of the same key returns the stored result instead of doing the work again.
2. Webhooks are deduplicated by the provider's event ID. The event ID gets a unique constraint in the database. A second delivery hits the constraint and becomes a no-op.
3. Uniqueness is enforced by the database, not the application. An if (!exists) insert check in code is a race condition. A unique index isn't.
4. Invoice state moves forward only. Transitions are explicit (draft → issued → paid) and guarded, so a late or replayed event can never move an invoice backwards.
The mindset shift
Stop asking "what if this runs twice?" and assume it will. Once retries are designed in, they stop being incidents and become ordinary traffic.