Short answer. An application is monitored on four signals, errors, latency, traffic, saturation, to which we always add two rows: external probes that do what a customer does, every minute, from Algiers and from Paris, and a business signal, paid orders per hour, which the manager understands without explanation. Each signal has a written threshold before launch and a destination: the on-call phone for what demands immediate action, the team channel for what can wait until morning, the report for the rest. Alerts fire on the symptom, what customers see, never on the cause; the cause is read in structured logs and grouped errors. Every alert that wakes someone has a runbook, written calmly. An availability objective is written in the contract, 99.5 % per month by default, and the Monday report says whether it was met, what fell, for how long, and which threshold changes. Six weeks, one of them spent inducing failures to check that everything rings.
This guide is for the manager who learns about outages from customers, the founder settled in France who will know nothing of the Algerian night unless warned, and the person who must compare two maintenance contracts. It does not repeat how you come back after the failure, described in the backups guide, nor what an application costs every year, described in the guide on its life; it describes what is built so that you know, and extends our page on cloud and infrastructure.
What reliability teams say
Monitoring has a vocabulary, and it comes from the teams that run the largest services. Google’s reliability engineering reference book names “the four golden signals of monitoring: latency, traffic, errors, and saturation”, and asks the question every monitoring system must be able to answer: “what’s broken, and why?”, the what being the symptom and the why the cause. What this imposes: you alert on the symptom, payments failing, and keep causes, a full disk, a gateway down, for the logs and the runbook; an alert on every possible cause wakes someone ten times for nothing.
The same book says what an alert that wakes someone must be: “every time the pager goes off, I should be able to react with a sense of urgency”, “every page should be actionable”, and “every page response should require intelligence”, that is, not be a mechanical reaction a script could perform. It also warns against averages: “it’s tempting to design a system based upon the mean of some quantity”, when it is the tails, the 99th percentile, that say a service is saturating. What this imposes: latency thresholds are written at the 99th percentile, not the mean; and every alert that rang without demanding an action is removed at the next report, because a phone that rings for nothing ends up on silent.
What we did not find: a public measure of the availability of business applications in Algeria, or of the mean time between an outage and someone noticing it. What we see on the applications we take over is an outage learnt from a customer in most cases, and monitoring reduced to pinging the server in almost all; we say so as an observation, and it is the reason for the two rows we add to the four signals.
What is measured, and where it rings
Monitoring fits in a six-row table, and that table is written in week one.
Errors: the share of failed requests, per screen and per function; above 2 % over five minutes for a critical function, the phone; above 0.5 %, the channel. Latency: at the 99th percentile, per route; above 3,000 ms, the channel; above ten seconds, the phone, because at that point customers give up. Traffic: requests and orders per hour compared with the same hour last week; a drop by half for fifteen minutes in business hours wakes someone, because it is often the only sign of an upstream outage, at the operator or the SMS provider. Saturation: disk, memory, connections, queue; 80 % the channel, 90 % the phone.
External probes: every minute, from Algiers and from Paris, a full request that opens, logs in and reads an order; two consecutive failures from both cities wake someone, from one only it is the local network and goes to the channel. Business: zero paid orders during a business hour wakes someone; it is the alert the manager understands without explanation, and the one that would have rung at 2:16 in the example that follows. The thresholds are the ones we write by default; they are discussed in week one, and reread every month.
«Alert on the symptom, what customers see; read the cause in the logs. An alert on every possible cause wakes someone ten times for nothing.»
One night incident, seen by monitoring
Monitoring is not understood on a table but on a night, minute by minute, on two curves.
2:14, the payment gateway stops answering: payment requests wait, then fail; the error rate goes from 0.3 % to 12 %, 99th-percentile latency from 420 ms to eight seconds. 2:16, the 2 % errors-over-five-minutes threshold is crossed and the on-call phone rings; the alert says what, since when, and which runbook to open. 2:19, acknowledged; the “payment errors” runbook says where to look: grouped logs show timeouts towards the provider, not an error in our code. 2:27, the workaround planned in the runbook is applied: cash on delivery is offered instead, through a flag, without deployment, and errors drop, because customers ordering at that hour no longer see the gateway.
2:41, the gateway returns; errors fall back under 1 %. 2:45, the flag is removed, and the incident report is opened for Monday’s blameless review: twenty-seven minutes during which customers saw another payment method, not an error. Without monitoring, the same night would have been discovered at 9 by a WhatsApp message, and the gateway would have come back on its own without anyone knowing what had happened, or for how many customers. The example is constructed; the 2 % and 3,000 ms thresholds are the ones we write by default for payment, per the guide on its integration.
Monitoring an application, in five layers
Behind those two curves, five layers, and the one that separates monitoring from decoration.
External probes: a full request every minute from two cities, doing what a customer does; it is the only layer that sees what customers see, network included, and it is independent of the server it watches. The application: structured logs with an id per request, errors grouped by cause with their frequency, traces of a request’s path; this is where you read the why when the probes say the what. Infrastructure: CPU, memory, disk, connections, queue, and expiring certificates and domain; Saturday-night outages almost all come from here.
Alerts and on-call: thresholds written per signal, a destination per severity, a named on-call person with a second as backup, and a runbook per alert that wakes someone; every alert that wakes someone must demand an action, otherwise it joins the report. The report: on Monday, the month’s availability against the objective, incidents and their durations, open actions, alert noise; what the manager reads in two minutes, and what is reread every month to lower or raise thresholds; it is built on the same principles as the manager’s dashboard. Charts nobody looks at wake nobody: an alert is a sentence, a threshold, a destination and a runbook.
A month of availability, against the objective
The objective is written in the contract, and the month reads in thirty cells.
The objective: 99.5 % availability per month, measured by the external probes, that is three hours thirty-six of tolerated downtime; that figure is the month’s error budget, and it gets spent. Day 8, twenty-seven minutes: the gateway night, worked around by a flag. Day 19, one hour twenty-five: the database disk filled up, the saturation alert rang at 90 %, when it was already almost full, and the database stopped before on-call had finished freeing space. Budget consumed: one hour fifty-two out of three hours thirty-six; objective met, at 99.74 %.
The report’s action: the disk saturation threshold goes from 90 to 80 %, so that the alert arrives while there is still time to act; it is exactly the kind of line that comes out of a report and does not exist without it. An error budget has another virtue: when it is almost spent mid-month, you stop shipping new features and repair, and that is a decision the manager can take by reading three lines. The values are a constructed example; the 99.5 % objective is the one we write by default for a management application, and it is discussed according to what the hosting guide decided.
When the phone rings, in six steps
On-call is not a person who knows everything; it is a procedure written at two in the afternoon to be played at three in the morning.
The phone rings, on a written symptom and threshold; the alert says what, since when, and which runbook to open. Acknowledged in five minutes, otherwise the second person is called; the time is noted, because reaction time is part of the report. The runbook: one page per alert that can wake someone, where to look, what to check, which workarounds exist; nobody thinks well at three in the morning, and the runbook was written calmly. The workaround, not the fix: flag, failover, restart, re-queue; you stop the symptom first, understand afterwards.
Say what is happening: a banner in the app and a message to the manager, two lines, through the channels described in the notifications guide. The fix and the review: the fix ships in daytime, by the normal path of delivery, and the Monday review is blameless, with at least one action; the report says what rang, how fast you reacted, and what changes. When the failure touched the data, the backups guide takes over at step four.
«Monitoring never seen ringing is a hypothesis, like a backup never restored.»
Seen elsewhere: reliability teams, and what transposes
One source says the essentials. Google’s reliability engineering book names the four signals, separates symptom from cause, requires every alert that wakes someone to be urgent and actionable, and warns against averages in favour of percentiles; what transposes is the six-row table, thresholds at the 99th percentile, and the rule that an alert that rang for nothing is removed. What does not transpose as is, is scale: a management application does not need a night team, it needs one person, a second as backup, and written documents.
What this source does not say: which functions are critical at your end, and what availability percentage you accept to pay for; the project’s first week answers that, and the manager signs.
The rules of monitoring that warns before customers
Six rules, written in week one. Six signals, the golden four plus external probes and business. Thresholds written before launch, at the 99th percentile for latency. Alert on the symptom, and read the cause in the logs. Every alert that wakes someone demands an action and has a runbook; otherwise it goes to the report. An availability objective in the contract, and an error budget spent knowingly. The Monday report, read by the manager, and thresholds reread every month.
A seventh rule concerns the quote: “24/7 monitoring” is not a requirement, it is a sentence; the requirement reads “external probes from two cities every minute; structured logs with request id, grouped errors, traces; thresholds written per signal with phone, channel or report destination; named on-call person and backup, acknowledgement in five minutes; runbook per alert that wakes someone; monthly availability objective and error budget; induced failure before launch; weekly report with availability, incidents, durations, actions, noise”, and it is checked as the specification guide recommends. A quote that writes “monitoring included” without saying who is woken, by what, and how fast, has written the server ping.
Six applications, and the alert that saved them
Six applications we often monitor, and for each the alert that rang before the customers.
- The shop run from France, described in the dedicated guide. Zero sales taken during a business hour on a Tuesday: the Algiers till had lost the network; the founder knew in Lyon before the manager in Algiers.
- Ticketing, described in the dedicated guide. Traffic on the evening tickets go on sale triples; the queue saturation alert rang at 80 %, and the queue was widened before tickets sold twice.
- The delivery platform, described in the dedicated guide. The SMS provider stops answering; the drop in SMS sent woke someone, and the second provider took over by failover.
- The payment app, described in the dedicated guide. The night of figure 2: twenty-seven minutes, one flag, no customer lost.
- The SaaS platform, described in the dedicated guide. One objective per client in the contract; the Monday report breaks down per client, and a single slow client is seen before they call.
- The cold chain, described in the dedicated guide. The temperature probes are themselves monitored: a probe silent for fifteen minutes wakes someone, because silence is not good news.
The 2027 calendar
The calendar is that of an application put under monitoring in six weeks, one of them to induce failures.
Week 0 to 1 writes what is monitored and the objective: critical functions named, the availability objective, thresholds per signal, who is on call and who is backup. Week 1 to 2 installs the external probes from Algiers and Paris and the health check of every service. Week 2 to 3 structures the logs with an id per request, groups errors, adds traces, and writes the retention. Week 3 to 4 writes the alerts with their destinations and a runbook per alert that wakes someone.
Week 4 to 5 induces failures: cut the database, fill the disk, stop the payment gateway, expire a certificate in test; every alert must ring, every runbook must serve, and what did not ring is fixed. Week 5 to 6 installs the Monday report and the monthly threshold review. Monitoring never seen ringing is a hypothesis, like a backup never restored; and when the app is slow rather than broken, the speed guide takes over, with the same percentiles.
What makes monitoring fail
The monitoring we take over almost always failed for one of these reasons.
- The server ping as the only probe. The server answers, the app refuses every payment.
- Charts without alerts. Everything was visible on a screen nobody watched at 2 am.
- Alerts on causes. Fifty possible alerts, ten rings a night, and the phone on silent after a week.
- The average instead of the percentile. The average was fine; one customer in twenty waited ten seconds.
- No runbook. The alert rang, and the person spent an hour looking for what to check.
- A single on-call person. They were asleep, or on leave, or had left the company.
- Never induced. The first real outage was also the first test of the alerts.
Frequently asked questions
The questions we receive from managers, founders and IT managers.
- Why do we learn about outages from customers when the server is monitored?
- Because what is monitored is often the server, not the application: it answers the ping, and yet payments fail or login no longer works. You need probes that do what a customer does, from outside, every minute, and a business signal, paid orders per hour, that rings when it drops to zero in business hours.
- Who is woken, and by what?
- A named on-call person, with a second as backup if the acknowledgement does not arrive in five minutes. They are woken by the symptoms that demand immediate action: errors of a critical function above 2 % over five minutes, external probes failing from two cities, zero paid orders during a business hour, saturation at 90 %. Everything else goes to the team channel or the report.
- What is a runbook?
- One page per alert that can wake someone, written calmly: where to look, what to check, which workarounds exist, who to notify. Nobody thinks well at three in the morning; the runbook makes the reaction independent of the on-call person’s memory. It is reread after every incident and corrected.
- What does 99.5 % availability mean?
- Three hours thirty-six of tolerated downtime per month, measured by the external probes. It is an error budget: every incident spends it, and the Monday report says what is left. When it is almost exhausted, you stop new features and repair. The objective is written in the contract and discussed according to what the app does: a till and a catalogue do not have the same tolerance.
- How do you keep the phone from ringing for nothing?
- By alerting on symptoms rather than causes, writing thresholds at the 99th percentile rather than the average, and removing at the next report every alert that rang without demanding an action. An alert that wakes someone must be urgent and actionable; otherwise it joins the channel or the report. Alert noise is a line of the Monday report.
- How long to put an application under monitoring?
- Six weeks: one to write what is monitored and the objective, one for probes and health, one for logs and traces, one for alerts and runbooks, one to induce failures and check everything rings, one for the Monday report. On an existing application, the first two weeks are often enough to stop learning about outages from customers.