Why did the night report not leave?

Most often because a script is launched at a fixed time by the system and the server was restarting, or the script failed without anyone being alerted. The construction that settles this: a scheduled task in a queue, a start-up catch-up that reruns missed times, and an alert if the report has not left by the planned time plus a margin.

Why did the SMS go out twice?

Because two servers or two clicks launched the same send, or a retry resent messages already sent. The construction: a lock with lease in the database so a task is taken only once, and an idempotency key per message so a message already sent never leaves again, even after a retry.

What is a work queue?

A task table in the database, with a state, a priority, a due time, a payload, a key and a lease, and workers that take the next task, lock it, run it and record the outcome. Everything that must not block the screen goes through it: sends, exports, recomputes, reports, reminders, imports.

How do you retry without making the provider outage worse?

With a wait that doubles between attempts, drawn at random within a window so a thousand tasks do not restart at the same second, and capped. That is the recommendation published by the Amazon Web Services architecture team. After the last attempt, the task goes to the failure queue with an alert.

What happens to a task that fails for good?

It goes to the failure queue, with its cause, its payload and its attempt count, and an alert goes to the manager. A human reads that queue every morning, retries or fixes. It is never deleted silently.

How many workers are needed?

For most SMEs, two are enough: one for heavy tasks, one at a time, and one for short tasks in parallel; you add more when the queue grows, and it is a number you measure rather than guess. Where those workers run depends on the hosting chosen.

How long does it take to set this up?

Six weeks in the typical calendar: one for the task inventory, one for the queue and workers, one for retries and the failure queue, one for scheduling, one for migrating existing scripts, one for measurement. The duration depends on the number of existing scripts and sending providers.

“The night report never left, and nobody knew”: what gets built for the background tasks of an application in Algeria in 2027, work queue, lock, retries, idempotency, scheduling, failure queue

Short answer. Everything that must not block the screen, sends, exports, recomputes, reports, reminders, becomes a task in a work queue stored in the database, with a priority, a due time, a payload and an idempotency key. A worker takes the task by locking it with a lease of a few minutes, renews it while it progresses, and the lease falls if it dies; two workers never take the same one. When the task fails, it retries after a wait that doubles, drawn at random within a window so a thousand tasks do not restart at the same second, and capped; the key means the part already done is not redone. After the last attempt, the task goes to a failure queue with an alert, and a human reads it every morning. Fixed-time tasks have a catch-up at start-up for missed times, a rate for sends, and a log the manager reads at 07:00. Six weeks, the first without code.

This guide is for the manager whose reminders go out twice or not at all, the founder settled in France waiting for a night report that no longer arrives, and the person who has seen an accounting export crash three times in a row. It does not repeat which channel carries which message, described in the notifications guide, nor how a field application queues its writes without network, described in the offline guide; it describes the machine that, server side, runs everything that has no screen, and it extends our software engineering page.

Background tasks that hold, in five layers

A script launched by the system at midnight is one layer out of five. The four others are what makes it succeed, or lets you know when it fails.

Figure 1 · The accented layer is what distinguishes a work queue from a script launched at midnight: failure there is a planned state, with a wait, a key and an exit. Without it, an SMS provider outage at 06:30 loses the day’s reminders, or sends them twice.

Requests. Everything that must not block the screen: sending 1,200 reminders, exporting 50,000 rows, recomputing stock, the Monday report, payment reminders; the screen answers in a second by creating a task with a priority and a due time, and the task does the work behind. Work queue. A task table in the database, with state, priority, due time, payload, idempotency key, attempt count and lease; the queue is backed up with the database, following the backups guide, and visible in the application. Execution. A worker locks the task with a lease of a few minutes, runs it with a maximum duration, renews the lease while it progresses, and releases it if it dies; two workers never take the same task, because the database holds the lock.

Retries and failures. This is the accented layer: a failure triggers a retry after a wait that doubles, drawn at random, capped; the idempotency key means the part already done is not redone; after the last attempt, the task goes to the failure queue and someone is alerted. Scheduling and measurement. Fixed-time tasks have a start-up catch-up; sends have a smoothed rate; a log says what ran, a board shows late, failed and waiting tasks, and an alert goes out if the queue grows, in the spirit of the monitoring guide.

What a task must be able to do

Six properties of a well-built task, and the third is the one that avoids the worst.

Figure 2 · The third point is accented because it avoids the worst: two workers taking the same task and sending twice. The first point is what makes all the others easy: a task of a hundred sends retries in ten seconds, a task of twelve hundred retries in two minutes and redoes what was done.

Be small: 1,200 sends are twelve tasks of a hundred; an export is batches; a short task retries fast and redoes little. Carry its key: the same task taken twice does the work once, and a message already sent does not leave again, following the mechanics described in the offline guide. Lock itself: a lease renewed, released if the worker dies; this is what avoids two reminders per customer. Fail cleanly: the cause is written, and the wait before the next attempt doubles, with jitter and a cap.

Respect a rate: one message per second to the messaging provider, nine batches for an export, never a burst, because the provider cuts off the account that sends 1,200 messages in ten seconds; channel details are in the WhatsApp agent guide. Leave a trace: start, end, outcome, attempts, readable by the manager, not only by the developer.

One task, from the queue to success or to the failure queue

The figure follows a task “send 1,200 WhatsApp reminders” while the messaging provider is unavailable.

Figure 3 · A task “send 1,200 reminders” fails three times because the provider is unavailable, then succeeds at the fourth attempt; waits double and are drawn at random within a window, so that a thousand failed tasks do not all restart at the same second, and are capped. The 380 messages sent at the first attempt do not leave again, thanks to the idempotency key. After the last attempt, the task would go to the failure queue with an alert. Durations are a constructed example.

The task is taken and locked; the first attempt fails because the provider answers unavailable, but 380 messages had already left. The second attempt waits thirty seconds, plus or minus ten drawn at random; the third two minutes, plus or minus thirty seconds; the fourth eight minutes, plus or minus two, and succeeds: 1,200 sent, zero duplicates, because the idempotency key, reminder plus appointment plus date, skips the 380 already sent. Had a fifth attempt been needed, the wait would have been capped at thirty minutes, and beyond that the task would have gone to the failure queue with an alert to the manager.

Jitter in the wait is not a detail: when a provider goes down, a thousand tasks fail at the same second, and without jitter they all retry at the same second, which knocks it down again. The Amazon Web Services architecture team published a recommendation on this point: capped exponential backoff, with jitter, which spreads retries and clearly reduces the number of calls. The failure queue is read every morning by a human, who retries or fixes; it is never emptied silently. Durations in the figure are a constructed example.

«A failing task is neither lost nor replayed forever: it retries at intervals that double, drawn at random, capped, and ends in a queue a human reads.»

Six real situations, and what the application does

The six situations found in almost every application we take over, and the construction that settles them.

Figure 4 · The first two rows are settled by the catch-up and the lock; the next two by batches and retries; the last two by the rate and the maximum duration. None is settled by a developer getting up at night.

The first two rows are settled by the start-up catch-up and by the lock with lease; the second is the costliest, because two reminders per customer are seen by 1,200 customers. The next two are settled by batches and retries: a background export in batches stores a file and sends a link, and a provider outage becomes a ten-minute delay instead of a loss. The last two are settled by the smoothed rate and the maximum duration with expired lease. None is settled by a developer getting up at night to relaunch a script.

A server's night, on two workers

The figure shows what runs between 22:00 and 07:00 on an SME's server, on two workers, and three things that would have gone wrong without the queue.

Figure 5 · A server’s night, on two workers: the backup holds the “database” lock and the cleanup waits; the accounting export leaves in batches; the Monday report is missed by a restart at 02:04 and caught up at 02:15; the 1,200 appointment reminders leave at 06:30 in batches of one hundred, one message per second; the report leaves at 07:00 with what ran. Times are a constructed example.

Worker A does the heavy tasks one at a time: the database backup at 22:00, which holds the “database” lock; the stock recompute right after, following the stock guide; the accounting export in batches at midnight, to the format the accountant expects following the integration guide; the Monday report preparation at 02:00, missed because the server restarted at 02:04 and caught up at 02:15; the 1,200 same-day appointment reminders at 06:30, in batches of a hundred, one message per second, following the booking guide; the report sent to the manager at 07:00. Worker B does the short tasks in parallel: the temporary-file cleanup, which waits for the backup to finish because the lock is held; payment reminders at 23:30; next week's slot recompute at 05:30.

Three things would have gone wrong without the queue: the 02:00 report would not have existed and nobody would have known; the cleanup would have run during the backup; the 1,200 reminders would have left in ten seconds and the account would have been blocked. How many workers, on which machine, and where, is a decision described in the hosting guide; times in the figure are a constructed example.

«A scheduled task has a time, but also a catch-up if the server was off, a lock so it does not run twice, a rate so it does not get cut off, and a log that says at 07:00 what ran.»

Seen elsewhere: AWS's recommendation on retries, and what carries over

Three facts carry over as they are. Exponential backoff with jitter: the Amazon Web Services architecture team showed, by simulation, that doubling the wait between attempts reduces contention little as long as clients retry at the same time, and that the remedy is to add jitter, which in their hundred-client simulation cut the number of calls by more than half; what carries over is a wait schedule with jitter and a cap, written in the first week. At-least-once delivery: a queue that guarantees a task will run thereby guarantees it may run twice, after a cut at the wrong moment; what carries over is that every task carries a key and its work is idempotent, like the payment webhooks described in the online payment guide. The failure queue, or dead letter, present in every messaging system; what carries over is that a human reads it, not only that it is stored.

What those facts do not decide for you: how many attempts, which cap, which rate to your provider, who is alerted and when the report leaves; the project's first week answers that, and the manager signs.

The rules of background tasks that hold

Six rules, written in the first week. Nothing long runs in the screen: everything becomes a task. Every task carries a key and its work is idempotent. A task is locked by a lease, renewed or released. Failure has a schedule: wait that doubles, jitter, cap, then failure queue and alert. Sends have a rate written per provider. What ran is readable by the manager at 07:00, with what did not run.

And one rule of method: existing night scripts are not rewritten at once; they become tasks one by one, starting with the one whose failure costs the most, and every switch comes with its log, following the delivery without breakage guide.

Six companies, and the task that was missing

Six situations composed from projects we meet; the names are generic and the figures are examples.

A medical practice in Algiers, appointment reminders. Reminders went out twice some mornings, because two servers launched the same script. The construction: tasks of a hundred sends, lock with lease, key per reminder. Duplicates stopped on the first day.
A wholesaler in Sétif, accounting export. The month-end export crashed in the browser past ten thousand rows. The construction: background task in batches, file stored, link sent to the accountant, resume at the batch; the format follows the integration guide.
A founder in Lyon, online shop in Algiers. The night report stopped arriving after a server update, and they found out three weeks later. The construction: start-up catch-up, alert if the report has not left by 07:15; remote management follows the shop from France guide.
A delivery platform in Oran, notifications. The SMS provider cut the account after a burst of two thousand messages. The construction: smoothed rate per provider, batches, retries with jitter when it is unavailable.
A distributor in Constantine, stock recompute. A stuck recompute held the queue all night and nothing else ran. The construction: maximum duration, expired lease that releases and flags the task, queue that moves.
An SME in Annaba, supplier imports. Price imports ran in the screen and crashed on a cut. The construction: import as a batched task with resume and key, following the Excel import guide.

The 2027 calendar

Six weeks for an application that already has night scripts and sends; the first week passes without code.

Figure 6 · Six weeks, and the first without code: the inventory of everything that runs without a screen is what nobody ever wrote, and it says how many tasks, which rates and who is alerted. The third week is accented because retries and the failure queue are what make a provider outage a delay and not a loss.

Week 1, task inventory: everything that runs without a screen, sends, exports, recomputes, reports, reminders, with priority, due time, rate, maximum duration and who is alerted; signed by the manager. Week 2, queue and workers: task table in the database, workers, lock with lease, maximum duration, task board in the application. Week 3, retries and failures: wait schedule with jitter and cap, idempotency key, failure queue, alert, manual retry.

Week 4, scheduling: fixed-time tasks, start-up catch-up, locks between tasks, smoothed rate for sends. Week 5, script migration: existing night scripts become tasks, one by one, with their log; sends go through the queue. Week 6, measurement: morning log, late and failed tasks, queue size, first failure cause of the week, on the dashboard following the real-time dashboard guide. Every week ends with a demo.

What makes background tasks fail

Five causes, seen in applications we take over. The midnight script without a trace: it fails silently for three weeks. The send triggered by a click: the employee clicks twice, or two employees click. The export in the browser: the connection drops, all is lost. The immediate, infinite retry: the fallen provider is hammered until it blocks the account. The queue without a human: the failure queue grows and nobody reads it.

And one cause of method: treating every task as a unique case, with its script and its retry logic. A work queue is a machine built once that every task uses.

Frequently asked questions

The questions managers and founders ask when they describe sends that go out twice and reports that do not go out.

Why did the night report not leave?: Most often because a script is launched at a fixed time by the system and the server was restarting, or the script failed without anyone being alerted. The construction that settles this: a scheduled task in a queue, a start-up catch-up that reruns missed times, and an alert if the report has not left by the planned time plus a margin.
Why did the SMS go out twice?: Because two servers or two clicks launched the same send, or a retry resent messages already sent. The construction: a lock with lease in the database so a task is taken only once, and an idempotency key per message so a message already sent never leaves again, even after a retry.
What is a work queue?: A task table in the database, with a state, a priority, a due time, a payload, a key and a lease, and workers that take the next task, lock it, run it and record the outcome. Everything that must not block the screen goes through it: sends, exports, recomputes, reports, reminders, imports.
How do you retry without making the provider outage worse?: With a wait that doubles between attempts, drawn at random within a window so a thousand tasks do not restart at the same second, and capped. That is the recommendation published by the Amazon Web Services architecture team. After the last attempt, the task goes to the failure queue with an alert.
What happens to a task that fails for good?: It goes to the failure queue, with its cause, its payload and its attempt count, and an alert goes to the manager. A human reads that queue every morning, retries or fixes. It is never deleted silently.
How many workers are needed?: For most SMEs, two are enough: one for heavy tasks, one at a time, and one for short tasks in parallel; you add more when the queue grows, and it is a number you measure rather than guess. Where those workers run depends on the hosting chosen.
How long does it take to set this up?: Six weeks in the typical calendar: one for the task inventory, one for the queue and workers, one for retries and the failure queue, one for scheduling, one for migrating existing scripts, one for measurement. The duration depends on the number of existing scripts and sending providers.