We have a backup every night, is that not enough?

It depends on two figures only you can write: how many minutes of data you accept to lose, and how many hours of downtime. A nightly copy loses up to a day and comes back in a few hours, if it is elsewhere than on the server and someone has already restored it. To hold fifteen minutes you need continuous archiving of the database logs; to hold two hours you need a drill that has already done it.

What is the restore drill for if backups are automatic?

To prove that they restore, and in how long. Most recovery failures we see come from unreadable, incomplete or encrypted copies or lost passwords, and no automation detects that: only a full restore, on an empty server, by someone who did not write the procedure, shows it. The timed report is also what lets the manager read a verdict without understanding the technology.

How do you protect against ransomware that also encrypts the backups?

With copies it cannot reach: on a second site, at another provider, versioned, and locked for a written duration during which nobody, not even the administrator account, can erase or overwrite them. The compromised server is then recreated clean from the code repository, and the database and files come back from the locked copies.

What happens if the supplier vanishes with the access?

If the code repository, hosting accounts, domain and copies are in your company’s name, secrets are in a manager and two people at your end know, the disappearance is an inconvenience: another supplier takes over from the repository and the copies. If everything was in the supplier’s name, it is another story, and the guide on the blocked application applies.

How much data do you really lose?

With continuous archiving of the database logs, a few minutes at most: you come back to the point just before the failure. With a nightly copy, up to a day. For files, kept versions allow coming back to yesterday, or to the previous hour if the store allows it. The two figures are written per system, because a till and a catalogue do not have the same tolerance.

How long to put all this in place?

Eight weeks: one for the inventory and the two figures, two for the database and its first restore to the minute, one for files, one for the recreatable server, secrets and accounts, one for the first full drill, two to install the monthly drill and the alerts. On an existing application, the first three weeks are often enough to go from “we have backups” to “we have restored”.

If the server dies tomorrow, what is left of your application in Algeria in 2027? Backups, tested restore, maximum loss, recovery time and immutable copy

Short answer. You first write two figures per system: how many minutes of data you accept to lose, and how long you accept to be down. For a management application we write fifteen minutes and two hours by default. Then you build what holds them: code and configuration in a repository off the server, to recreate a new server in minutes; the database with a regular copy and continuous archiving of its logs to another site, to come back to any minute; files in a versioned store on a second site, with a period during which nothing can be erased, even by an administrator; secrets in a manager, accounts and domain in the company’s name, two people who know; and a full restore drill every month, on an empty server at another provider, timed, recorded, read by the manager. A backup never restored is not a backup.

This guide is for the manager who has already lost a day of sales, the founder settled in France who will only learn by phone that the server is dead, and the IT manager who must answer “what if…”. It does not repeat where to place servers, described in the hosting guide, nor how to take over an application whose supplier vanished, described in the dedicated guide; it describes what is built so that the failure is an ordinary Tuesday morning, and extends our page on cloud and infrastructure.

What the documentation says

The database can come back to the minute, but not with just any backup. PostgreSQL’s documentation describes continuous archiving of transaction logs combined with a base backup: “this technique supports point-in-time recovery: it is possible to restore the database to its state at any time since your base backup was taken”. It also warns that logical dumps, the ones made every night with an export, “do not produce file-system-level backups and cannot be used as part of a continuous-archiving solution”, that you need “a continuous sequence of archived WAL files” going back at least to the start of the backup, and that you are “advised to monitor the archiving process”. What this imposes: the nightly export keeps its place, but what holds a fifteen-minute objective is continuous archiving, to another site, watched by an alert when it stops.

Files can become impossible to erase, even by you. The object-storage documentation of a large provider describes a “write-once-read-many” lock: in compliance mode, “a protected object version can’t be overwritten or deleted by any user, including the root user in your AWS account”, for a retention period that cannot be shortened; and it names as a use “a configurable recovery window against ransomware or accidental deletion”. What this imposes: file and database copies are versioned and locked for a written duration, thirty days for instance, and an attacker who has taken the server cannot erase them; nor can a hurried administrator.

The rule security agencies repeat fits in three figures: three copies, on two media, one elsewhere; we add a fourth word, immutable, and a fifth, restored. What we did not find: a public statistic of data losses suffered by companies in Algeria, or of average recovery time; what we see on the applications we take over is a nightly backup on the same server in most cases, and a restore never attempted in almost all, and we say so as an observation.

The day the server dies, on the clock

Recovery is not understood on a principle but on a morning, minute by minute.

Figure 1 · One hour twenty-one of downtime and one minute of data lost, on the clock: the database came back to the 9:13 point because its logs were archived continuously, the files from a copy on a second site, and the server itself from the code repository. Nothing was improvised, because the previous month’s drill had played every step. Example built from real cases.

9:14, the production server’s disk dies. 9:16, the alert reaches two phones, not one. 9:20, the decision is taken and the recovery document opened: it says who does what, in what order, and where the copies are. 9:25, a new server exists at the backup provider, created from the code repository that describes the server; nobody installs anything by hand. 9:40, the database is restored to the 9:13 point: the last base copy plus the logs archived continuously up to the minute before the failure. One minute of data lost, for a written objective of fifteen.

10:05, files, photos, documents, exports, are back from the versioned copy on the second site; it is the longest step, which is why it is measured every month. 10:20, the domain name points to the new server. 10:30, twelve business checks pass: an order, an invoice, a payment, an export, a login. 10:35, customers are working again. One hour twenty-one of downtime for a two-hour objective. Nothing was improvised: every step had been played during the previous month’s drill, and the founder, in Lyon, followed the morning on the same document as the team. The example is built from real cases; the two objectives are the ones we write by default.

«A backup never restored is not a backup; the drill makes it real, and the report makes it verifiable.»

What it takes to survive your server, in five layers

Behind that morning, five layers, and only one that proves the other four exist.

Figure 2 · The accented layer is where the gap between “we have backups” and “we can go back to 9:13” is widest: a logical copy every night loses up to a day, continuous log archiving loses a few minutes. But without the fifth layer, none of the first four is proven.

Code and configuration: the application’s code, the server description, settings and deployment scripts live in a repository off the server; a new server is recreated from that repository in minutes, and nothing is “installed by hand” once, as the delivery guide describes. The database: a regular base copy and continuous archiving of logs to another site, to come back to any minute; a replica if activity requires never stopping at all. Files: photos, documents and exports in a versioned object store on a second site, with a period during which nothing can be erased.

Secrets and access: passwords, keys and certificates in a secrets manager, never on the server or in the code; hosting accounts and the domain in the company’s name, with two people who know, so that neither the dead server nor the vanished supplier takes anything away; it is the discipline described in the rights guide. Drill and supervision: a full restore every month on an empty server, timed and recorded, and alerts if a backup is missing or archiving stops. It is the only layer that proves the other four exist: a backup never restored is a hypothesis.

Six ways to back up, and what you really lose

“We have backups” covers six very different realities, and the server dying at three in the afternoon tells them apart.

Figure 3 · The third row is the only one that holds a fifteen-minute maximum-loss objective; the fourth is fine as long as the provider and the account exist, which is exactly what no longer exists in two of the guide’s six failures. The last row is the most common of all.

The nightly logical copy on the same server dies with the disk: you lose everything, and come back from an old copy found by chance, or never. The nightly logical copy sent elsewhere loses the day, sales since two in the morning, and comes back in a few hours if someone knows how to restore and the copy is readable; it is the most widespread of the honest methods. The base copy plus continuously archived logs loses a few minutes at most and comes back in under an hour if the drill was done; it is the only row that holds a fifteen-minute objective.

The provider snapshot alone loses nothing as long as the provider exists and the account is open, which is exactly what no longer exists in two of the guide’s six failures. Files copied without versions lose every file deleted or encrypted before the next copy, and yesterday’s copy may itself have been encrypted. All of that without a drill: the loss is unknown, the delay is unknown, and you find out on the day of the failure; it is the most common row of all, and the reason for this guide.

The monthly restore drill, timed and recorded

The drill is the piece that turns backups into recovery, and it reads in eight lines.

Figure 4 · The drill is played by the person who did not write the procedure, on an empty server at another provider, on the first Tuesday of the month. Every step is timed, the total is compared with the written objective, and every deviation is opened: here file restore doubled, and the reason will be known before the next drill. The report is read by the manager, who needs to understand none of the eight lines to read the verdict.

On the first Tuesday of the month, on an empty server at another provider, the person who did not write the procedure plays it. Server created from the repository, four minutes. Secrets retrieved by the second person, three: no password was on the dead server. Database restored to the chosen point, eighteen: base copy plus logs up to 14:02 the day before, the point is arbitrary and changes every month. Files restored from the second site, twenty-two: twice as long as last month, and the deviation is opened. Application started, three. Twelve business checks, nine. Domain name switched in test then reverted, two. Report written and signed, five.

Sixty-six minutes for an objective of one hundred and twenty: met. But the report also says what doubled, and why it will be known before the next drill; that is how you learn that files have grown, that the second site changed bandwidth, or that a manual step crept into the procedure. The manager reads the verdict without understanding a single one of the eight lines, and that is intended; the founder in France receives it by e-mail the same day. It is the same principle as remote acceptance: you do not believe what you have not seen done.

Six failures, and what makes recovery possible

Six failures we have seen or taken over, and for each what made, or would have made, recovery possible.

Figure 5 · The six failures do not share a cause but share the answer: copies elsewhere, versioned, immutable, a return point to the minute, and a drill that has already played the scenario. Two of the six are not technical at all, the supplier and the account; they are the ones that cost most.

The disk dies: server recreated from the repository, database restored to the failure point, files from the second site; it is the morning of the first figure. The bad query: a script wipes three thousand customers at 15:12, nobody sees it before 17:00; the database is restored to the 15:11 point on a server alongside, customers are reinjected, and the audit log says exactly what; it is the case where continuous archiving is worth most, because the nightly copy would already have lost the day. Ransomware: the server and the backups attached to it are encrypted the same night; the second-site copies, versioned and locked for thirty days, cannot be, and the server is recreated clean from the repository.

The vanished supplier: they had the access, the passwords and the repository; if the repository, accounts and domain are in the company’s name, secrets are in a manager and two people know, the recovery is an inconvenience, not a disaster; otherwise the dedicated guide begins. The closed hosting account: card refused at renewal, account suspended, snapshots inaccessible; copies live at a second provider, and recovery is played there, exactly as in the drill. The deleted file: an employee deletes the shared contracts folder and synchronisation propagates the deletion; versions are kept, and the folder comes back to yesterday’s date in minutes. Two of the six failures are not technical at all, and they are the ones that cost most.

«Two of the six failures are not technical at all, the supplier and the account; they are the ones that cost most.»

Seen elsewhere: the reference documentation, and what transposes

Two sources say the essentials. PostgreSQL’s documentation says continuous log archiving allows coming back to any point since the base copy, that a logical export is not enough, and that archiving is to be monitored; what transposes is the fifteen-minute objective and the alert when archiving stops. The object-lock documentation says that in compliance mode nobody, not even the root account, can erase a version during its retention, and names ransomware as a use case; what transposes is the immutable copy for thirty days, at a second provider.

What these sources do not say: how many minutes your company accepts to lose, and how many hours it accepts to wait; those two figures are yours, they are decided in week one, and they decide everything else.

The rules of an application that survives its server

Six rules, written in week one. Two figures per system, maximum loss and recovery time, signed by the manager. Three copies, two media, one elsewhere, and the copy elsewhere is versioned and immutable for a written duration. The database archives continuously to another site, and an alert rings when archiving stops. The server is recreated from the repository; nothing is installed by hand. Secrets, accounts and domain are off the server, in the company’s name, known to two people. One full drill every month, timed, recorded, played by someone else.

A seventh rule concerns the quote: “daily backups” is not a requirement, it is a sentence; the requirement reads “maximum loss and recovery time written per system; base copy and continuous log archiving to a second site with alert; files in a versioned store on a second site with thirty-day lock; server recreatable from the repository; secrets in a manager, accounts and domain in the client’s name, two holders; recovery document; monthly full restore drill on an empty server at another provider, timed, report delivered”, and it is checked as the specification guide recommends. A quote that writes “automatic backup included” with no loss or delay figure has written the sixth row of the table.

Six companies, and the figure they wrote

Six companies we often equip, and for each the two figures written and what holds them.

The shop run from France, described in the dedicated guide. Fifteen minutes and two hours; the founder receives the drill report every month and knows, from Lyon, that the Algiers till will restart.
The in-store point of sale, described in the dedicated guide. It keeps taking payments without a server thanks to the offline mode of the synchronisation guide; server recovery therefore has four hours, but zero minutes of lost sales.
The property developer, described in the dedicated guide. Buyers’ contracts and receipts are in the versioned store locked for a year; no signed file can disappear.
The dental practice, described in the dedicated guide. Five minutes and one hour for patient records; the drill is played on a Saturday, when the practice is closed.
The SaaS platform, described in the dedicated guide. One objective per client written in the contract, and the drill restores one client at a time to prove isolation.
The company migrating, described in the data migration guide. The first restore drill is played before the switch-over, because switch-over week is the worst time to discover you cannot restore.

The 2027 calendar

The calendar is that of an application made able to survive its server in eight weeks, the first without a byte copied.

Figure 6 · Eight weeks, and the first passes without a byte copied: you first write what you accept to lose and how long you accept to wait, per system, because those two figures decide everything else. The monthly drill starts in week six and never stops.

Week 0 to 1 takes the inventory and writes the two figures: what exists and where, maximum loss and recovery time per system, signed by the manager. Weeks 1 to 3 handle the database: base copy and continuous log archiving to another site, and a first restore to the minute verified. Week 3 to 4 handles files: versioned object store on a second site, immutability period, restore test of one folder. Week 4 to 5 makes the server recreatable from the repository, puts secrets in a manager, and moves accounts and domain into the company’s name, with two holders.

Week 5 to 6 plays the first full drill on an empty server at another provider, timed, and corrects the procedure. Weeks 6 to 8 install the monthly drill, on the first Tuesday, the alerts if a backup is missing, and the report read by the manager. The cost of these copies and of the drill enters the yearly budget described in the guide on keeping an app alive; and when an AI model runs on your servers, the private deployment guide adds its own copies to the same drill.

What makes a recovery fail

The recoveries we have seen fail almost always did for one of these reasons.

The backup on the same server. It dies with the disk, and you find out that morning.
Never restored. The copy exists, it is unreadable, incomplete or encrypted, and nobody knew.
Passwords on the dead server. The database is restored, and nobody can connect to it.
A single holder. They are on leave, or they are the vanished supplier.
Erasable copies. The ransomware encrypted them along with the server.
The hand-installed server. Three days to rebuild it, and nobody remembers everything.
No written figure. Everyone had a different objective in mind, and the recovery took as long as it took.

Frequently asked questions

The questions we receive from managers, founders and IT managers.

We have a backup every night, is that not enough?: It depends on two figures only you can write: how many minutes of data you accept to lose, and how many hours of downtime. A nightly copy loses up to a day and comes back in a few hours, if it is elsewhere than on the server and someone has already restored it. To hold fifteen minutes you need continuous archiving of the database logs; to hold two hours you need a drill that has already done it.
What is the restore drill for if backups are automatic?: To prove that they restore, and in how long. Most recovery failures we see come from unreadable, incomplete or encrypted copies or lost passwords, and no automation detects that: only a full restore, on an empty server, by someone who did not write the procedure, shows it. The timed report is also what lets the manager read a verdict without understanding the technology.
How do you protect against ransomware that also encrypts the backups?: With copies it cannot reach: on a second site, at another provider, versioned, and locked for a written duration during which nobody, not even the administrator account, can erase or overwrite them. The compromised server is then recreated clean from the code repository, and the database and files come back from the locked copies.
What happens if the supplier vanishes with the access?: If the code repository, hosting accounts, domain and copies are in your company’s name, secrets are in a manager and two people at your end know, the disappearance is an inconvenience: another supplier takes over from the repository and the copies. If everything was in the supplier’s name, it is another story, and the guide on the blocked application applies.
How much data do you really lose?: With continuous archiving of the database logs, a few minutes at most: you come back to the point just before the failure. With a nightly copy, up to a day. For files, kept versions allow coming back to yesterday, or to the previous hour if the store allows it. The two figures are written per system, because a till and a catalogue do not have the same tolerance.
How long to put all this in place?: Eight weeks: one for the inventory and the two figures, two for the database and its first restore to the minute, one for files, one for the recreatable server, secrets and accounts, one for the first full drill, two to install the monthly drill and the alerts. On an existing application, the first three weeks are often enough to go from “we have backups” to “we have restored”.