Hybrid sync/async: how to generate documents under load without hitting timeouts
1. Introduction: the problem of heavy operations in a synchronous world
Anyone who has worked with B2B platforms knows this scenario. The user clicks “Generate report”, and the browser hangs for tens of seconds. The request went to the server, the server collects data, renders a PDF or Excel — and all this time the user stares at a spinner, unable to tell whether the system is working or not. After a while they click the button again. Then again. And get three identical reports, two of which they did not want, and one they never got at all, because everything crashed.
This is not a bug. This is design: a heavy operation runs synchronously inside an HTTP request.
On the high-load B2B logistics platform I designed and developed for Gazpromneft Supply, document generation was one of the critical operations: the full document-flow cycle — from building large reports to signing through integrations with third-party systems. Here the cost of a timeout is higher than in an ordinary web application: a document is not interface decoration but a client’s business process. A missed generation deadline means a missed signing deadline, which means a delayed deal and losses for both sides.
Why operations become heavy
Synchronous generation stops coping for three reasons, and all of them act at once.
First — data volume per operation grows. A report is not a “SELECT from one table”. It involves queries across dozens of entities, aggregations, formatting, rendering. The more data the system accumulates, the longer each generation takes. An operation that took a second yesterday takes ten today — simply because there is more data.
Second — the number of users grows. More clients — more simultaneous generations. Each request holds a web-server process for the whole processing time. Ten heavy generations at once — and the lightweight requests that should answer in milliseconds queue up behind them.
Third — integrations with external systems. The full document-flow cycle does not end on our side: a document must be sent, signed, and confirmed. External services are slow, and their APIs are underdocumented. When a user’s HTTP request hangs waiting for another system’s answer, we hand over execution-time control to a service we do not control.
What happens when it breaks
The consequences of the synchronous approach are visible on five levels.
UX. The user sees a “frozen” screen, cannot tell whether work is in progress, and repeats the action. Repeated clicks spawn repeated generations and duplicate documents — and people have to sort them out.
Business. For a B2B client, a document is part of their own process. Every delay is their missed deadlines, their contract penalties, their lost deals. Trust in the platform falls faster than functionality grows.
Infrastructure. Processes occupied with heavy generations do not serve lightweight requests. One long request eats the resource a hundred fast ones need. The system degrades as a whole, not just on the “heavy” endpoint.
Cascading failures. Overload breeds growing wait queues on the server, client timeouts, automatic retries — and retries add even more load. The system enters a tailspin from which it only exits after manual intervention.
The team. Incidents instead of features, firefighting instead of planned work. The team puts out fires — and the capacity that should have gone into product development burns up.
Why “treating the symptoms” does not work
When the system starts choking, the first impulse is to raise timeouts. That is not treatment but postponement: the user waits longer, connections hang, load accumulates. The second impulse is to add servers. But the synchronous path scales linearly and expensively: every request holds a resource for the whole processing time, so we pay for peak load, not average. The peak lasts an hour — yet the servers are paid for around the clock.
The third “simple” solution — move everything to the background — does not work either: the user needs not a “task status” but a result — a finished document. A client’s business process does not end with sending to a queue. If we simply remove the operation from the request without giving the user a way to learn when it completed — we replaced a timeout with confusion.
So the task is formulated as: fewer timeouts + guaranteed execution + visible status. The solution we arrived at is a hybrid synchronous-asynchronous task-processing architecture. More on that next.
2. The approach: hybrid synchronous-asynchronous processing
The hybrid principle is simple: the fast part runs synchronously, the heavy, long, and unreliable part runs asynchronously through a queue. The user waits exactly as long as it takes for the system to say “accepted, the task is in the queue, here is its status”. Everything else — generation, integrations, retries — happens in the background.
The main question the architect decides is where the boundary lies. Which operations does the user wait for synchronously, and which can go to a queue with a status? You can err in both directions: send everything to the queue — and get delays and complexity everywhere; keep everything synchronous — and get the original pain.
Criteria for “to the queue”
In practice, one of these signs is enough to make an operation a candidate for asynchrony:
- execution time exceeds the user’s waiting threshold (if the user is willing to watch a spinner for less time than the operation takes — it is a queue candidate);
- the operation depends on external systems with an unknown SLA;
- the operation’s cost grows with data volume;
- the operation repeats for every order or document — its total cost multiplies by business volume;
- the user does not need the result immediately — a readiness notification is enough.
What always stays synchronous
Asynchrony is not about “moving everything to the background”. What stays synchronous: request acceptance, validation, saving data, reading task status, any lightweight operation within hundreds of milliseconds. The rule is simple: the UI waits only for “decision accepted”, not for “work done”. The user must get an instant response that their request is accepted and being processed — and see the execution status in real time.
Why RabbitMQ
For the async part we chose a RabbitMQ task queue. The reasons are engineering, not fashionable: delivery guarantees, built-in retry mechanics, no task loss on restarts. The queue gives us a buffer between load peaks: a burst of requests does not land on generation synchronously but accumulates and is processed by workers as capacity allows.
“Fast today ≠ fast tomorrow”
The boundary deserves a separate discussion because it is not static. An operation that takes 100 milliseconds today can become a bottleneck tomorrow: data grows, users multiply, new requirements appear, seasonality arrives. If we miss the moment the boundary shifted, the consequences are unpleasant: the operation becomes unavailable, cannot be processed, and synchronous paths are harder to scale — they hit resource limits faster than async ones.
How to know in advance what will become a problem? Four tools:
- Trend monitoring. Watch p50/p95/p99 latency per operation, data volume, growth rate. The trend matters more than the absolute value: if p95 grows quarter over quarter, the boundary is shifting.
- Capacity planning. Extrapolate growth, run stress and load tests to find the limit before we hit it in production.
- SLA and business requirements. Record what is critical and what response time is acceptable; revisit when product requirements change.
- Architectural indicators. Operations growing with data volume (linear in data), operations with external dependencies, operations with locks and resource contention — candidates for boundary revision.
At the same time, avoid the opposite extreme. There are operations that will not become a problem for years: stable in volume, fast, no external dependencies. They can stay synchronous. Optimizing in advance what does not hurt is premature optimization: the solution becomes more complex than the problem. But premature optimization has a flip side — mandatory measurability. We do not build async infrastructure “just in case”, but we are obliged to have metrics that say “it’s time” in time. Metrics are insurance against blindness, not a pretext for construction.
The economics of the decision
Any architectural choice is an investment, and it must be counted on a multi-year horizon: adoption (development, infrastructure, training), support (incidents, monitoring, on-call), maintenance (updates, tech debt, enhancements). The approach pays off when the cost of the solution is less than the cost of the problem: real timeouts, lost business, SLA breaches. Count in person-days and money, not in “architectural beauty”. In our case the math was simple: a RabbitMQ queue cost a multiple less than the downtime of document generation during peak periods.
3. The mechanism: a task queue with state
Now, how it works inside — at a level reusable in any project, regardless of the specific broker.
The task lifecycle
A task follows the path: acceptance → queuing → worker processing → user notification of readiness. The user initiates generation, the system validates the request, saves the task, publishes it to the queue — and answers the user: “accepted, status is such-and-such”. Then a worker picks up the task, performs the generation, and changes the status. The user sees the result through the status interface — without repeated clicks and without guessing.
A task is not only a queue message
The key architectural decision: a task is an object with a status in the database, not just a message in a queue. A task has a table with state: new → processing → done / failed / retry / cancelled. The queue is transport and buffer; the database is state, history, and control.
Why two stores? A queue without a table cannot answer “what is happening with all tasks right now?” — it can only deliver messages. A table without a queue gives no delivery guarantee or buffering. Together they cover each other: the queue delivers, the database remembers. If a worker dies — the task stays in the table with the processing status, and the control mechanism returns it to work. This is the “guaranteed document generation even under partial system failures”: a task survives the failure of an individual component and gets completed.
Statuses and transitions: who moves them and when
The status model solves three classic problems of distributed processing: races, reprocessing, and “stuck” tasks.
- new → processing: a worker atomically “claims” the task (an atomic UPDATE with a status condition). Two workers cannot claim one task — this rules out race conditions at the database level.
- processing → done / failed: the worker itself, by the processing outcome. Failed with retry potential goes to retry.
- processing → new: if a worker died during processing, after a timeout (TTL) and heartbeat control the task returns to the queue. There are no “stuck” tasks — there are tasks whose processing time expired.
- new → retry: after a failure with exponential backoff; after the attempt limit — to the dead-letter queue for manual review, not into an infinite loop.
Idempotency is a mandatory property of handlers: reprocessing a task must not create duplicate documents. Queues deliver at-least-once, so redelivery is the norm, not an accident. One defense: unique keys and a result check before creating something new.
Producer and consumer
On the producer side, two things matter. First — publishing with confirmation (publish confirm), so a message is not lost if the broker crashes. Second — atomicity of task write and publication: you cannot write a task to the database but not publish it to the queue (or vice versa). For this we use the transactional outbox pattern: the event is written to an outbox table in the same transaction as the business data, and a separate relayer publishes it to the queue — only after a successful commit, not during the write. This eliminates the “database and queue diverged” desync — the classic dual-write problem.
On the consumer side: acknowledgment (ack) is sent only after processing fully completes; broken messages do not return to the loop but go to the dead-letter queue; a worker does not grab more messages than it can process (prefetch).
Timeouts and control
Every task has a deadline — an SLA on processing. Control rests on three mechanisms: a watchdog for stuck tasks (processing longer than normal → re-claim), worker heartbeats (is the handler alive), and queue metrics (depth, message age, error count). Without this, “async” turns into “lost”: if a queue silently accumulates, the user learns about the problem last.
These mechanisms — control and monitoring of document generation — became the platform’s reliability guarantee for critical business processes: a document is either ready, or it has a clear status and a date when it will be ready. Together with horizontal worker scaling, this kept the platform stable under a 100% peak load increase without growing critical-operation response time.
4. Behavior under load: what grows the load and what to do about it
The hybrid scheme solves the timeout problem but opens the next question: what happens when load grows? The answer must be known in advance, because a queue is a buffer, and buffers fill up.
What grows the load
Load growth must be considered in two planes, and they must not be confused.
First — load within one message. One task gets heavier: the report has more rows, the generation logic grew more complex, an external service answers slower, the database holds locks longer. With a constant number of tasks, each one takes longer to process — so the system’s overall throughput falls.
Second — the number of messages grows. More users, more operations per user, seasonal peaks. A separate scenario is a retry avalanche: if an external service starts failing, retries add messages to the queue faster than workers can process them. A snowball that is easy to underestimate.
How to estimate capacity
A useful rough calculation: the required number of workers is approximately the product of the message arrival rate and the average processing time, divided by the desired utilization. Queue depth grows when the arrival rate exceeds the consumption rate. These two numbers — rate and time — must be known for every critical operation, otherwise capacity is estimated “by eye”, and “by eye” does not work in highload.
How to measure
The main queue KPIs are depth and message age (lag): how many tasks are waiting and how old the oldest one is. Plus per-worker metrics: how many tasks processed per hour, how many errors, what the average processing time is. Plus broker and worker resources. Only together do these numbers give the picture “the queue is accumulating because consumption lags” — rather than just “something is wrong”.
Resilience mechanisms
From system-design practice, proven mechanisms work here:
- horizontal worker scaling: load growth should mean more handlers, not more timeouts;
- backpressure with prefetch tuning and batching;
- partitioning for parallel processing;
- the queue itself as a buffer between load peaks;
- priorities — important work is processed first;
- per-worker concurrency limits and processing timeouts;
- isolation: different task types in different queues, so a heavy type does not starve a light one;
- caching heavy computations;
- worker autoscaling by queue depth.
And it is important to remember the limits of these mechanisms:
- horizontal worker scaling runs into the database and external systems — you cannot scale them as easily;
- backpressure runs into the user — tasks start waiting;
- priorities — into starvation of low-priority tasks;
- cache — into invalidation.
Knowing the limits is as valuable as knowing the mechanisms.
How to test before production
- capacity is verified, not declared;
- load testing at two to three times the expected peak;
- a worker or broker failure test — what happens when a component dies;
- an “external service is down” simulation — the queue must accumulate, not lose.
That is exactly how the declared property of the architecture is verified: stability under peak load growth without increasing critical-operation response time.
On the platform, these mechanisms — horizontal worker scaling within a High Availability architecture — produced a measurable result: stable operation under a 100% peak load increase without growing critical-operation response time.
5. Integrations and external systems: the queue as a data bus
The full document-flow cycle does not end on our side: a document must be sent, signed, and confirmed by third-party systems. Integrations are a separate layer of design, and here asynchrony shows its best side.
Interaction patterns
Services communicate in different ways, and choosing a pattern is choosing by the nature of the operation:
- synchronous request-response (REST, gRPC) — when the result is needed immediately;
- fire-and-forget through a queue — “accepted, we’ll do it”;
- request/reply through a queue with a correlation ID — an async request that still needs an answer;
- pub/sub — “an event happened, whoever wants reacts”;
- saga — a chain of steps with compensations across services;
- webhooks and polling — for external systems that have no queues of their own.
The queue as a data bus
The queue turns point-to-point integrations into a data bus: producers publish, consumers receive, with no direct dependency between them. This gives loose coupling, peak buffering, retries, and independent service lifecycles — each can update and scale on its own schedule. Asynchrony also protects against slow and underdocumented external APIs: an external service does not hold up a user’s request; a task quietly waits for its attempt in the queue.
Choosing a broker: RabbitMQ or Kafka
When the question “why not Kafka?” comes up, here is my answer. They are different tools for different jobs. RabbitMQ — smart routing (exchanges with different exchange types), complex interaction patterns, delivery guarantees; it is a task-oriented broker, natural for processing tasks and RPC-like exchanges. Kafka — an event log: high throughput, partitioning, the ability to replay history from the beginning, event retention, and a whole ecosystem around it. For event streams and analytics — Kafka; for tasks with guarantees and routing — RabbitMQ. In practice, systems with both classes of tasks often use both: RabbitMQ for tasks, Kafka for events. In our architecture, tasks live in RabbitMQ — matching the class of problems being solved.
Contracts and versioning
Integrations are contracts, and contracts are versioned. A message schema (JSON Schema or Avro), backward compatibility — we add fields but never remove them; breaking changes only through a new contract version and consumer migration. Without this, “we extended a field” turns into “we broke a consumer” at the worst possible moment.
Handling external system unavailability
External systems fail — that is the norm, not an emergency. So calls get timeouts, failed services get a circuit breaker (do not hammer a downed service with endless requests), pools get isolation (bulkhead). Retries with exponential backoff — only for idempotent calls, otherwise a retry becomes a double operation. And separately — channel security: encryption between services and the broker, broker authorization, no secrets in messages, sanitization of data coming from outside.
Channel liveness monitoring
A data bus requires its own monitoring. Health checks for channels and the broker, queue metrics (depth, age, rate), alerts on delay and accumulation. And most importantly — response procedures: when an alert fires, what do we do. A standby broker and failover, load redistribution between workers, a circuit breaker on consumers. A communication channel is an engineering object with an SLA, not “something between services”.
6. Pitfalls and traps: what we hit so you do not have to
An async architecture pays for its advantages with a set of traps. I collected the most painful ones — those that surface not in the first month but when the system is already “working”.
Handler idempotency. Reprocessing a task must not create duplicate documents. The broker delivers at-least-once: a handler crashed after execution but before acknowledgment — the task will come again. This is not a bug; it is a property. One defense — idempotency and statuses in the database.
Timeouts inside a worker ≠ timeouts in the UI. By moving an operation to the background, we move the problem rather than solve it. If the worker also fails, you need retries and a dead-letter queue — otherwise “async” turns into “lost with a delay”.
Poison messages. One broken message loops a worker: crash → retry → crash. The solution — an attempt counter and a dead-letter queue. It is important to distinguish a temporary error (we retry) from a broken one (to DLQ).
Retry storms. Retries without exponential backoff and jitter turn a partial failure of an external service into an avalanche of requests. That is a “polite” way to kill a service that is still alive.
Thundering herd. After a crash or at startup, N workers all grab tasks at once — a hit on the database and external systems. Treated with concurrency limits and prefetch tuning.
Claim without heartbeat. A worker claimed a task and died — the task hangs in processing forever. TTL return and heartbeat are mandatory: a task must return to the queue if the handler shows no signs of life.
Non-atomic write and publish. Wrote to the database but did not publish — or published but did not write. The classic database-queue desync, solved by the outbox pattern: the event is written in the same transaction as the business data.
Order and parallelism. Parallel workers break processing order; dependent documents (one generated from another) start conflicting. The solution — an ordering key or per-task-key concurrency limits.
Side effects on retry. A retry must not repeat user notifications, writes to external systems, or repeated integrations. An idempotency key for every side effect.
DLQ without monitoring. A dead-letter queue silently accumulates — a quiet loss of tasks. An alert on DLQ growth is mandatory; DLQ review is a regular procedure, not “someday”.
“Async” as an excuse. Moving everything to a queue without designing guarantees means moving the problem and adding delay. Statuses, metrics, SLA on processing — not an option but the condition of honest asynchrony.
Message bloat. Putting large documents and binary data into messages degrades the queue. The solution — Claim Check: a reference in the queue, data in storage.
Implicit deadlines. A task has an SLA, but the worker processes “stale” tasks — the user already cancelled or repeated the action. A freshness check before execution is mandatory.
Unbounded queue growth. On consumer failure, a queue grows forever — disk and memory run out. Message TTL, depth limits, overflow policies.
Duplicates at the entrance. The user clicked “generate” twice — two tasks. Deduplicate at the entrance by an idempotency key from the request context.
7. Result metrics: how to know it all works
An async system requires async control: while a task is in the queue, “everything works” is not a fact but a hypothesis. Metrics turn the hypothesis into a fact.
Direct queue metrics
Queue depth, message age (lag), processing rate, processing time, error count, dead-letter queue size. This is the basic pulse: if depth grows at a stable consumption rate — consumption is lagging.
Indirect problem signals
The most valuable are indirect parameters that signal a problem before it becomes an incident. p95/p99 processing time growing. Retry count growing. Tasks in processing longer than normal — stuck. External service degradation by latency and error rate. Queue depth growing at a stable consumption rate. And — an often underestimated signal — growth in repeated user actions: people do not click twice without a reason, and the reason is usually on our side.
Time-based execution control
Besides real-time dashboards, periodic checks are needed: daily and weekly exports of tasks for the period, analysis of unfinished, old, and failed ones. This is a “reconciliation with reality”: what should have been done versus what was done. And an SLA on processing — a task has an execution-time norm, and violating it is an alert, not statistics.
Control mechanisms (what works most often)
Dashboards with threshold alerts (Prometheus/Grafana and similar). Worker heartbeats and “stuck” detection. Retries with backoff, DLQ with dedicated monitoring. A task-status journal — an audit trail that answers “what happened to this document” in a minute, not a day. And regular queue reports — daily and weekly. These control and monitoring mechanisms for document generation are exactly what made the process predictable: a document is either ready, or it has a status and a deadline.
8. Takeaways and selection criteria
The hybrid synchronous-asynchronous architecture is not a compromise between “badly synchronous” and “complexly asynchronous” but a separate scheme with clear guarantees. The selection rule is simple: synchronous — what the user waits for and what must be fast; asynchronous — everything heavy, long, and dependent on external systems. The UI waits for “decision accepted”, not “work done”.
The economics of the decision
Any choice is an investment, and it must be counted on a multi-year horizon: adoption, support, maintenance. The approach pays off when the cost of the solution is less than the cost of the problem. Do not build async infrastructure for a pain you have not measured: first a baseline, then a solution.
Comparing approaches: what to choose
| Criterion | Synchronous | Async (queue) | Hybrid |
|---|---|---|---|
| Adoption | Low: minimal new code | High: broker, statuses, DLQ | Medium: two paradigms |
| Support | Low: few moving parts | High: monitoring, on-call | Medium-high |
| Maintenance | Simple | Expensive: broker, tech debt | Medium: boundary discipline |
| Risks | Timeouts under growth | Task loss, duplicates | Blurred boundary |
| When to choose | ≤100–200 ms, not data-bound | Heavy, long, externally dependent | The user waits for part of the result |
Adoption checklist
Map operations by time → decide the sync/async boundary → a queue with retry and monitoring → idempotent workers → queue metrics. Five steps, each verified separately. If after adoption a metric has not moved — you built the wrong thing.
On the platform this path produced a measurable result: fewer timeouts in the web interface, guaranteed background execution and document generation even under partial system failures — at a 100% peak load increase without growing critical-operation response time. This is not “beautiful architecture”; these are business metrics you can show in numbers.