Monitoring is not alerting

You find out the site is down because a customer texts you about it. Meanwhile there's a monitoring dashboard, and it has graphs on it, and as far as anyone knows the graphs were fine.

They probably were. Dashboards are honest — they just don't wake anybody up. Somewhere between "we have monitoring" and "we found out before the customer did" there's a second system that most small setups never build.

Two different jobs

Monitoring is collecting the state of your systems and putting it somewhere a person can look: CPU, disk, response times, queue depth. It answers questions you already thought to ask.

Alerting is a machine deciding that something is wrong and interrupting a human about it. It answers the question you weren't awake to ask.

A dashboard is a tool for investigating. It is not a safety net, because its entire operation depends on a person choosing to look at it — and the moment you most need someone looking is 2am on a holiday weekend. If the only thing standing between a failure and your customers is human attention, you have monitoring and no alerting.

An alert that always fires is not an alert

The first thing that kills an alerting setup is volume. Disk hits 81 percent and emails you every night. A backup job warns about the same skipped file for six months. Within a few weeks everyone has a mail filter, and the one alert that mattered lands in the same folder as the noise.

The rule I hold to: every alert that reaches a phone has to name something a human would act on immediately. If nobody would get up for it, it's a report — put it in a weekly summary or on the dashboard where it belongs. And an alert with no named owner is just a dashboard with a louder delivery mechanism. In a good week, the number of alerts that reach a person should be zero.

A check that can't detect its own death

The second thing that kills it is quieter. Most small setups work by having an agent on each machine push metrics somewhere. If that agent stops — the process dies, the disk fills, the host is powered off — the graph goes flat.

A flat line looks like calm. Nothing is over a threshold, because nothing is being reported at all. Systems fail silently and dashboards render silence as good news.

So absence of data has to be its own alarm. Whatever you use should raise an alert when it hasn't heard from a host or a job in a set window — the pattern is usually called a dead man's switch, and every serious monitoring tool supports some version of it. Same for scheduled work: a nightly backup that doesn't check in by morning should page you exactly as loudly as one that fails.

The part in-house setups almost always skip

Here's the turn, and it's the one I'd fix first if I could only fix one thing. An agent running on your server cannot see most of the ways your service breaks for customers.

The agent reports that the web server process is up, memory is fine, disk is fine — all true — while the outside world gets nothing. Because the failure was in front of the box: a TLS certificate that expired, a domain registration that lapsed, a nameserver that stopped answering, a firewall rule someone tightened, a CDN pointed at the wrong origin, a service listening on the loopback address instead of the public one. The agent lives on the far side of every one of those. It is reporting on a machine, not on a service.

The only check that proves your business is reachable is one that starts on a network you don't own and does what a customer does, in order: resolve the domain, complete the TLS handshake, request the real page, and read what comes back. All four steps, from outside.

And read what comes back, not just the status code. A parked domain returns 200. A default web-server welcome page returns 200. A maintenance page returns 200. Assert on a string that only your actual page contains — a product name, a phone number, a form label — and the check starts failing for the reasons you care about.

What to do

1. Write down the failures that would cost you money in the first hour. Site down, email not delivering, phones not ringing, payments not clearing. That short list is your alert list; everything else is a graph.
2. For each one, build an outside-in check from a network you don't control. Resolve, handshake, fetch, verify content.
3. Add expiry alarms for the two things that fail on a calendar rather than at random: TLS certificates and domain registrations. Warn weeks out, not days.
4. Add a heartbeat so that silence — from a host, an agent, or a backup job — is treated as failure.
5. Route the real alerts to something that actually interrupts a person, and name who that person is.
6. Once a month, delete every alert that fired without anyone doing anything. That's the maintenance that keeps the rest trustworthy.

Uptime checks, alert routing, and someone on the other end of them is most of what NetCare is.