Background jobs that survive restarts: queues, retries, and idempotency for business web applications

Every business web application eventually has work that does not belong inside a web request: sending an order confirmation, generating a monthly report, syncing an invoice, or calling a partner webhook. The work itself can take seconds, fail halfway, or depend on a service that is briefly down. Run inline, a slow email provider becomes a slow checkout, and a failed webhook becomes a lost update. Background jobs break that coupling, but a queue on its own does not make work reliable. Reliability comes from three decisions made together: what goes on the queue, how failures are retried, and how handlers behave when they run twice.

Why background work fails quietly

Inline work fails loudly: the user sees an error page. Background work fails quietly, which is worse. A job that crashes after sending half its emails, a worker that restarts mid-report, or a webhook that times out after the remote side already processed it — none of these produce a visible error, and all of them produce the worst kind of bug: work half done, duplicated, or silently missing. This is the same class of problem as observability for asynchronous workflows: once work leaves the request, you lose its natural feedback loop and must build a replacement — job status, attempt counts, and alerts on stalled jobs.

The first design rule follows from that: a background job must be a durable record, not a function call with a delay. If the worker process dies, the job must still exist somewhere it can be picked up again. In-memory timers and fire-and-forget promises do not survive a deploy or a restart; a persistent queue — a database table, Redis, or a dedicated broker — is what turns "we tried" into "we will finish, or we will tell you why we did not."

A queue is a contract, not a waiting room

A useful job record carries four things: the payload, an identifier unique to that unit of work, a scheduled time, and a status. The payload should be the minimum needed to re-derive the work — usually an entity ID — so a job that runs an hour late still acts on current data. Without the unique identifier, a retry is indistinguishable from a brand-new request.

Keep job handlers small and single-purpose. A handler that sends an email, updates three tables, and calls two external APIs is three jobs pretending to be one. Small handlers also make connection pooling and resource limits easier to reason about: each handler holds its resources briefly instead of one long handler hogging a connection while it waits on a slow third party. Small handlers also keep the user-facing request inside its performance budget, because the request only enqueues and returns.

Retries: backoff, limits, and a dead letter

Most background failures are transient: a network blip, a brief provider outage, a database under momentary load. Retrying immediately and forever is wrong for both. Immediate retries hammer a service that is already struggling, and unlimited retries turn a permanent failure — a deleted account, a malformed payload — into an infinite loop that consumes workers.

The standard shape is exponential backoff with jitter and a hard attempt limit: wait seconds, then minutes, with random jitter so workers do not retry in lockstep and create a thundering herd. After the limit, the job goes to a dead-letter queue — a visible pile of jobs needing human decision, not a silent drop. A dead-letter queue is also an operational input to a deployment pipeline that proves what it ships — a spike in dead letters after a release is often the first signal that the release changed behaviour.

One distinction matters more than the backoff curve: retry only what can succeed on retry. A timeout is retryable; a validation error is not. Classifying errors as transient or permanent keeps retries from becoming a denial-of-service attack on your own dependencies.

Idempotency: design handlers that can run twice

Retries only work if running a handler twice is safe. In a distributed system, "exactly once" delivery is not something a queue can promise; at-least-once is the honest guarantee, so duplicates will happen. A worker can crash after charging a card but before marking the job complete, and the retry will run the charge again. Idempotency is the answer: make the effect of a job depend on a key, not on how many times it runs.

The practical pattern is simple. Every job carries an idempotency key — usually the job's unique ID or a business key like an order number plus an action name. Before doing the side effect, the handler claims the key in the same transaction that records the work, so claiming and doing are atomic. If the key is already claimed, it skips the side effect and reports success. The same idea protects recovery scenarios: after a restore, replayed jobs must not re-apply effects that already happened.

-- Claim the key and record the effect in one transaction.
-- A second run with the same key finds the claim and skips the work.
BEGIN;
INSERT INTO processed_jobs (idempotency_key, job_type, result)
VALUES ('order-4821:send-invoice', 'send-invoice', 'sent')
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING idempotency_key;
-- If no row is returned, the work was already done: exit successfully.
COMMIT;

Where the external system supports it, pass the key through: payment providers and many APIs accept an idempotency key natively, moving deduplication to the side that performs the effect. Where it does not, keep the claim on your side. Either way, never let "did we already do this?" be answered by memory or by hoping the first attempt failed cleanly.

The go-live checklist

Before a job system carries real business work, check it against this list:

• Every job is a durable record surviving worker restarts and deploys.
• Payloads reference entities by ID, not snapshots that go stale.
• Retries use exponential backoff with jitter, an attempt limit, and a dead-letter queue.
• Errors are classified as transient or permanent before any retry decision.
• Every handler with a side effect is idempotent, keyed on a business-stable identifier.
• Job progress, attempt counts, and dead-letter depth are visible on a dashboard, with alerts.
• Workers shut down gracefully: they finish the current job or return it to the queue.
• Long-running jobs have timeouts; timed-out jobs are retried or dead-lettered, never left running.

What to build first

If your application has no job system yet, start small: a durable queue table, one worker, backoff with a dead-letter table, and idempotency keys on every handler that touches money, email, or an external system. That moves email, reports, and integrations out of the request path without introducing silent failures. Priorities, scheduled jobs, and per-tenant rate limits can wait until the boring parts are proven.

Where background work lives is also an architecture decision: choose boundaries you can operate, and a queue is often the cheapest boundary, letting two parts of the system evolve and fail independently. If your team is weighing that change, our cloud and DevOps work covers queue and worker infrastructure, and custom software development covers the application side. When you are ready to move slow, flaky work out of your request path, talk to us.