Idempotency and retry in a payment core: building operations that cannot be duplicated and cannot be forgotten
1. Introduction: why “repeat the request” is not the same as “repeat the operation”
Anyone who has worked with payments knows this scenario. The user pays, the browser spins, the server stays silent longer than usual. The user clicks the button again — and a minute later finds two charges instead of one. Or none: the payment “went through”, but the status never came back, and what happened to it is unknown to both the user and the system.
Outwardly this looks like a failure. In fact, it is normal operation of a distributed system: the network does not guarantee message delivery — packets are lost, connections break, timeouts happen. A request can reach the server and be processed, but the response can be lost on the way. The client cannot tell “it did not arrive” from “processed, but the response was lost” — and retries. The server receives a repeated request and does not know whether to process it again. This is the classic two-generals problem: over an unreliable channel, the parties cannot reliably agree on whether an operation was performed. The only practical way to reach agreement is idempotent processing of repeats — which is what this article is about.
The difference between “repeat the request” and “repeat the operation” is the difference between a double charge and a correct repeated notification. A payment core must be designed so that a repeated request does not create a new operation but returns the result of the existing one.
But duplication is only half the problem. The other half is operations that get lost. A gateway turned out to be unavailable, retries produced nothing, the payment stayed unprocessed — and was forgotten. Or it was processed with a big delay, when the fine’s discount period (50% off in the first 20 days) had already expired. For the company this is a direct loss: a fine that could have been paid at a discount goes out at full price. That is why the payment core contract is strict: every payment must be processed exactly once — not 0 and not 2 times.
The cost of an error here is higher than in most other systems, and it has three layers.
Financial. A double charge, or paying 100% of a fine instead of 50% — because the service guarantees payment — is money that has to be refunded: disputes, chargebacks, lost interest. Every duplication costs real money, and the higher the volume, the more it costs.
Reputational. A user who was charged twice does not dig into architectural causes: they simply remember the service as “unreliable”. In B2B payments that means lost contracts; in B2C, churned users.
Regulatory. Payment systems are required to ensure the correctness of operations: errors in processing payments mean complaints, audits, fines. The regulator does not care whether you had a “network problem”.
On the payment core I developed for РОСШТРАФЫ, processing ran at more than 20,000 successful transactions a day at 99.99% uptime. At that scale, a percentage of errors stops being an abstraction: every percent of erroneous operations is hundreds of incidents a day. So the task was not “fix a bug” but design operations so that network failures and retries are safe by construction.
This article is about the pair of mechanisms — idempotency and retry — and what they delivered on a real core: a 25% drop in erroneous and fraudulent operations and roughly a 10% increase in successful payment conversion.
2. The approach: a payment core as a state machine
Any payment operation is not a “request” that either executed or not. It is a path with state: the operation is created, passes through stages, changes statuses, and at any moment can be interrupted — by the network, a timeout, a service crash, a redelivered request.
If the idea “operation = request” leads to chaos on any failure, the idea “operation = state machine” turns failures into managed transitions.
The contract is simple and strict: an operation must be processed exactly once (exactly-once). Zero times is a lost payment and a missed discount period; twice is a double charge, refunds, and disputes. A state machine with statuses solves both: state prevents losing an operation, idempotency prevents processing it twice.
An operation as a set of states and transitions
A payment has predictable states: created, processing, completed, rejected, in reconciliation, needs manual review. Each transition between states is a separate action, and it has one key requirement: the transition must be idempotent. Re-applying the same transition must not change the result.
This means: if an operation has already moved to “completed”, a repeated attempt to move it there must not create a second charge. It must detect that the state has already been reached and return it — as a fact, not as a new action.
Three levels where idempotency lives
Idempotency is not a property of one method but of the whole chain. It must be ensured on three levels, and if even one is missed — duplicates leak through the others.
API level. A repeated request with the same operation identifier returns the same result as the first one. The client can safely retry without thinking about the consequences.
Handler level. Before every action, the payment logic checks: hasn’t this already been done? Not “create a new charge”, but “find the operation by this key and return its state”.
Storage level. Unique constraints on operation keys are the last line of defense that stops the database from creating a duplicate even if the application errs. Storage does not forgive, and that is good.
One business meaning — one identifier
The key principle the whole design rests on: one business meaning of an operation = one identifier (idempotency key) that survives all retries. The client generates the key once for the whole operation — not per request, but for its entire lifecycle. Retries send the same key. The server finds the existing operation by the key and returns its result.
This works like the real world. A bank transfer is not a “message” that can be sent twice: it is an order with a number, and the bank sees by it: the operation is already done, here is its status. Likewise, re-registering a hotel guest with the same passport does not create a second room — the system finds the existing record and returns it.
A payment core where “operation = state machine with idempotent transitions and a single key” can survive any repeated attempts — from the network, clients, gateways. Next, the two mechanisms separately.
3. Mechanism 1: idempotency
Idempotency is a property of an operation: no matter how many times you run it, the result is the same as after the first run. In payments this property is not a luxury but a necessity, because redelivery of requests is not an accident — it is the norm. Queues deliver messages at-least-once, gateways retry, users double-click buttons.
On the РОСШТРАФЫ core, idempotency mechanisms were introduced together with the ГИС ГМП and acquiring integrations: every external call, every processing operation — with a key that lets us understand: “this already happened”.
How to check that an operation has already been processed
Before performing an action, the handler looks up the operation by key:
- key found, operation in a terminal status — return its result as the answer to the current request. Do not process again, do not create a duplicate;
- key found, operation in an intermediate status — return “operation in progress”; the client can keep waiting and retrying;
- key not found — create a new operation and perform the action.
Simple logic, but it is exactly what turns a “repeated request” from a source of double charges into a safe repeated notification.
Designing the response
An important nuance: a repeated request must return the original result, not a “new” one. That means the same status, the same identifier, the same details. If the first request failed, the repeated one must return the same error with the same identifier — not a “success” — otherwise the client cannot tell a real rejection from a network failure.
The “elevator button” analogy
Idempotency is an elevator button. No matter how many times you press it, the elevator comes once. The button remembers that the call has already been accepted, and repeated presses do not create new calls. That is exactly how a payment service should behave on repeated requests: acknowledge that the operation has already been accepted and return its result.
4. Mechanism 2: retry and queues
Idempotency answers “what happens if the request repeats?”. Retry answers “should we repeat at all — and how?”. On their own these mechanisms do not work: retry without idempotency is a factory of double charges; idempotency without retry is lost operations. The client got a timeout and left, the operation hung in limbo. They work only as a pair.
A payment is two operations, not one
The user-facing part of a payment is usually two operations, not one. First, the system blocks funds on the card — authorization (hold): the amount is frozen but not yet charged. Then the payment goes to processing in ГИС ГМП, and only after the external system confirms does the authorization become a capture — the money is actually taken from the card.
For retry this is fundamental: between the block and the capture there is a window in which a repeated attempt is doubly dangerous. A repeated block without releasing the previous one means funds frozen twice. A repeated capture without a status check means a double charge. So each of the two operations gets its own idempotency key and its own retry rules: the block is idempotent on its own, and the capture happens exactly once and only after the external system confirmed processing.
Retry on the synchronous leg
The synchronous leg is the client-facing part of the payment: initialization, blocking funds, lightweight checks. Here calls are retried by rules, not “as it happens”:
- a bounded number of attempts — not an infinite loop, but a fixed limit after which the operation goes to the dead-letter queue for manual review;
- exponential backoff — the interval between attempts grows, so we do not hammer an external system at the moment it is struggling;
- timeouts per attempt — each attempt is bounded in time, otherwise retry itself becomes a source of hangs;
- jitter — so simultaneous client retries do not land at the same point.
The synchronous leg has another important property: security checks happen here. The operation has not charged money yet — funds are only blocked, and the capture happens after the external system confirms. So at this stage the payment can be safely cancelled without financial loss: if the transaction looks suspicious or the user’s behavior is anomalous — the block is released, no money was taken, the user lost nothing. The further an operation moves down the pipeline, the more expensive a cancellation is: after capture it is a refund with disputes and chargebacks. Hence, the rule: security checks go at the entrance, where a rejection costs nothing.
The rule that turns retry from a danger into a mechanism: retry only idempotent operations. A repeated call is exactly as safe as safely repeatable its action is. So first — keys and checks, then — retries.
The async leg: ГИС ГМП
The second part of the loop — working with the ГИС ГМП gateways — is async by construction. Gateways cannot handle a synchronous stream of requests, so fine-payment and money-transfer requests go only to a queue and are processed by workers: request accepted → queued → executed → status recorded → user notified.
Here the full set of mechanics is at work: load balancing across gateways, shifting load from one gateway to another on degradation, retries by rules, per-operation execution control, and a status model for fine postings — the operation always knows which stage it is at. Stuck and problematic operations do not wait to be stumbled upon: the payment inspector periodically runs through all operations in a period, finds the problematic ones, and hands the list over for manual review. Posting reports give visibility over the whole flow.
A failure in this loop must not turn into financial loss. If a notification is lost — the user did not learn about the status in time, but the money is unharmed, and the queue will redeliver the message. The reconciliations run on a schedule also survive failures: a missed cycle is caught up by the next one.
Asynchrony takes everything that can wait off the critical path: postings, notifications, reconciliations. What stays synchronous is what determines the result on the user’s side: initialization and blocking funds.
When retry does not save you: balancing across gateways
A single acquirer is a single point of failure. If the gateway is down, no retries will help: the operation will repeat uselessly, accumulate failures, and delay the payment — exactly the “payment did not go through in time” scenario. That is why the core worked with several providers in the “Strategy” pattern — support for 5+ payment providers with rotation: if one gateway degrades or returns unstable responses, traffic moves to another. This raised payment acceptance reliability to 99.8% and removed the dependence on a single vendor. A prioritized provider list cut commission costs: traffic goes to the live provider with the lowest commission. The net effect: higher payment acceptance conversion.
Balancing across gateways is retry at the architecture level: we repeat not the request to an unavailable gateway but the routing to an available one. Third-party latencies are also taken into account: the decision where to send a payment is made not by the “favorite” provider but by the current state of each — timeouts, error rate, response speed.
Recurring payments as separate operations
Recurring payments are regular charges the system runs on a schedule without user involvement. Typical examples: service subscriptions, auto-payments — utilities, fines, loans, installments — and regular contributions. The user consents to charges once, and then the payment core processes the payment for each period on its own: day, month, quarter.
The main value of recurring payments is conversion. The user enters payment details and confirms them (3DS) once, and subsequent payments go through “automatically”: no re-entering details, the system charges on its own. Every period in which the user would have to pull out the card and go through confirmation again is a chance to lose the payment. Recurring removes that chance: the first payment pays for itself across the whole series that follows.
Hence, an important routing rule: if a recurring payment for a user already exists through a specific gateway — subsequent charges should go through that same gateway for as long as possible. There is already a confirmed token and a working pairing; switching to another gateway breaks the recurring link, and the user will most likely have to enter details and confirm again. Switching between gateways is for new payments and emergencies, not for recurring ones.
For the core, recurring remains a series of ordinary idempotent operations: each period’s payment is an independent operation with its own idempotency key — a redelivery will not create a second payment for the period, and a missed period will not be lost — scheduled retry will carry it through. A separate layer is subscription management: pause, cancel, change of payment method, card-expiry notifications. These events must also be idempotent: a repeated notification or a repeated cancellation request must not change the subscription state twice.
The “courier” analogy
Retry is a courier who, on a failed delivery, tries again on a schedule: waited, came back, tried once more, with a growing interval. But he never brings two identical parcels — because the parcel is idempotent: it has one number, and by the number you can see whether it has been delivered. Retry is responsible for “try again”, idempotency for “do not do it twice”.
5. Observability and control
The mechanisms exist — but how do you know they work instead of masking a problem? Retry without observation is a black box: operations repeat, nobody knows why, and incidents surface in user complaints.
Monitoring and alerting on key stages
The payment cycle is monitored at key stages — initialization, blocking funds, processing, capture confirmation, status check. Counters are collected at each stage and live in Grafana: how many operations started, processed, rejected, stuck, required retries. In SRE terms these are SLIs — service level indicators — and each has a target level (SLO): the share of successful operations, gateway response time, the share of operations that reach the end without manual intervention.
But counters show only quantity — the second dimension is time: how long each step-to-step transition takes. Initialization → fund blocking in the gateway → processing in ГИС ГМП → capture confirmation: each transition has an average time, and it lives next to the counters. Time metrics are the first to notice degradation: if the average fund-blocking time in a payment gateway shoots up, it is not statistical noise but a signal of gateway trouble and a reason to switch traffic to another.
But a metric without an alert is an archive. Alerts are set on atypical values: a sharp rise in retries, a drop in the share of successful operations, a spike of “needs review” statuses. The goal is to catch a problem at its birth, before it becomes an incident: when the deviation is still visible in a counter rather than in user complaints.
Retry metrics deserve special attention. A rise in retries is a signal, not noise: it means an external system is degrading, or timeouts are misconfigured, or something in our loop has started to slow down. If retries grew while processed operations fell — retry is not working; it is masking the problem.
Status distribution as diagnostics
The fastest way to gauge the core’s health is to look at the distribution of operations across statuses. The normal picture: the vast majority successfully processed, a small share rejected by business rules, and a small tail of “needs review”. If the tail grows — that is a reason for an investigation, not for hoping it “resolves itself”.
Reliability is controlled failure handling
99.99% uptime on a core with 20,000+ transactions a day is not “nothing ever fails here”. It is “failures happen and are handled so that users do not notice them”. The network will lose packets, external systems will delay responses, services will restart. The question is not how to avoid it but what the system does when it happens: idempotency prevents duplication, retry prevents loss, observability prevents blindness.
6. Pitfalls and traps
The mechanisms are simple, but in practice the details decide everything. Here are the traps people step on most often.
Fake idempotency
A key built from data that changes between attempts is not idempotency. If the key includes a recomputed amount or a moving timestamp, a repeated request will create a new operation. The key must be built from stable business-meaning data, not from technical request attributes. And it must be generated by the client — once per operation, not per attempt.
Retry without idempotency — duplication; idempotency without retry — losses
Two sides of the same coin. Retry without keys — we get duplicates. No retry at all — operations are lost: the user got a timeout, left, and the payment never went through. The mechanisms must be introduced as a pair, and in this order: first idempotency, then retry on top of it.
Reconciliations as a safety net
No matter how well you design, some discrepancies will be caught by neither retries nor idempotency: an external system processed an operation, and we did not learn about it. The safety net is periodic reconciliations with external systems: we compare our records with theirs and find discrepancies the mechanisms missed. Reconciliation is not a replacement for idempotency but a second line of defense that runs on a schedule.
On the РОСШТРАФЫ core, analytics and reconciliations rely on ClickHouse: daily financial reports that used to take hours now build in 3–5 minutes. Fast analytics is not just convenience: the faster a reconciliation, the earlier a discrepancy is found and the cheaper it is to fix.
Fraud patterns disguised as technical failures
Some “strange repeats” are not the network but deliberate actions: fraud patterns. Repeated attempts can be an attempt to run an operation again or probe the system’s boundaries. It is important to distinguish technical retries from anomalous sequences — and here fraud monitoring decides: engineers working with the security team. I do not reveal the internal fraud-detection rules — but the architectural takeaway is simple: retry logic must be visible and measurable, otherwise anomalies drown in the noise.
For this, statistical data about anomalous user behavior during payment is collected: patterns that do not look like technical retries — anomalous operation frequency, uncharacteristic sequences, suspicious samples. The statistics are collected on the fly and allow decisions in the moment rather than after an incident review. Architecturally this means one thing: data for analysis must be collected at every payment stage, not only when suspicion has already arisen.
What not to do
Do not store sensitive data in logs and do not log full payment data (PAN, CVV) — these are direct PCI DSS requirements. The payment loop is an area where “just in case” logging turns into data compromise. Log identifiers and statuses, not contents.
7. Result metrics
−25% erroneous and fraudulent operations. The ГИС ГМП and acquiring integrations with idempotency and retry mechanisms cut the level of erroneous and fraudulent operations by 25% — per the security team. This is the effect of the “mechanisms → fewer errors” chain: duplicates stopped being created, lost operations stopped being lost, anomalies became visible.
+~10% conversion of successful payments and −15% average processing time. Optimizing the key processes — payment initialization and processing — raised successful payment conversion by about 10% and cut average transaction processing time by 15%.
20,000+ successful transactions a day at 99.99% uptime. The scale on which all of this operates.
How to read the chain. Fewer erroneous operations — fewer abandoned attempts and repeated clicks — a higher share of payments completed to the end. I do not claim a direct causal link between each mechanism and each number: the numbers have several sources, and part of the effect comes from accompanying process optimization. But the direction is unambiguous: core reliability directly affects business metrics, not just technical ones. One number = one fact from the resume, without inventing intermediate dependencies.
And about how hypotheses were tested: constant A/B testing through feature flags. A change — payment step order, timeouts, failure behavior, gateway routing — was rolled out behind a flag and enabled on real clients: the sample was defined on live traffic, control and experimental groups ran in parallel, and only a data-confirmed hypothesis went to production. Synthetic tests did not work here: effects were measured in fractions of a percent of conversion, and only statistics on real payments could see them, not a lab run. This is discipline, not a process: conversion grew not “from a good idea” but from decisions validated on live samples.
8. Takeaways and checklist
Idempotency and retry are not “nice to have” and not “someday later”. They are the baseline reliability of a payment core: without them, any network, any timeout, any repeated click is a potential financial incident.
The rollout order is not accidental: first idempotent operations, then retry on top of them, then observability, and finally reconciliations. Each step builds on the previous one: you can only retry what is safe to repeat; you can only observe what is measurable; you can only reconcile what is recorded.
Checklist for reviewing a payment loop
- Does every operation have a single idempotency key, generated once and surviving all retries?
- Does a repeated request with the same key return the original result — the same status, the same identifier?
- Are unique constraints on keys in place at the storage level — as the last line of defense?
- Is retry bounded in attempts, with exponential backoff, timeouts, and jitter?
- Are only idempotent operations retried — can a retry create a duplicate?
- Are retry metrics and status distribution visible — is a retry rise impossible to miss?
- Do reconciliations with external systems run on a schedule — as a second line of defense?
- Do sensitive data never reach the logs?
If the answer to any item is “no” — it is a candidate for the nearest sprint. Not because “it should look nice”, but because the cost of an error in payments is money.