Skip to content

Uptime monitoring without false alarms: why locations matter

Most false uptime alarms come from the path between one check location and your server, not from the server. Check from at least three independent locations, alert only when the majority sees the failure, and check for a keyword on the page rather than only the status code. That removes most noise without hiding real outages.

Most false uptime alarms have nothing to do with your server. They come from the network path between one check location and your site: a few lost packets, a slow DNS resolver, a routing change at a transit provider. The remedy is to check from at least three independent locations and alert only when the majority sees the failure, and to look for a keyword on the page instead of trusting the status code alone. This article explains why that works and how to verify a suspected outage yourself.

Why does a single check location produce false alarms?

An HTTP check involves more than your web server. The checker resolves the name through DNS, opens a TCP connection across several networks, negotiates TLS, sends the request and waits for the answer within a timeout. Any link in that chain can fail briefly without a single visitor noticing:

  • The checker's DNS resolver times out once.
  • A router between two providers drops packets for thirty seconds.
  • The checker's own data centre has a network problem.
  • A firewall or rate limit on your side blocks the checker's address but nobody else's.

From one location, all of these look exactly like "site down". If the check runs every minute, even a path that works 99.9 % of the time produces more than one failed check per day. Sent as alerts, that is enough to teach a team that the monitor cries wolf.

The opposite error exists too. A check running in the same data centre as the server, or on the server itself, keeps reporting "up" while the site is unreachable from the internet.

How does a majority decision help?

With three locations, each round produces three results. The rule is simple: the check counts as failed only when at least two of three locations see a failure in the same round.

Location ALocation BLocation CResultTypical cause
okokokupnormal operation
failokokupnetwork problem at or near location A
failfailokdownsite or a regional network down
failfailfaildownsite down

A real outage is seen by all locations at once, so it is not delayed. A problem local to one checker is outvoted. Three is the practical minimum: with two locations, "one says up, one says down" cannot be decided.

Two refinements make the rule robust:

  • Independence. Three checkers in the same data centre share the same uplink and fail together. Useful locations differ in network and, ideally, in country.
  • Confirmation over time. Requiring the failure in two consecutive rounds filters out a restart or deployment that takes a few seconds. This is called debouncing; it is one of the techniques described in alert fatigue, quiet hours and escalation.

The single location that disagrees is still useful information. If one location fails repeatedly while the others are fine, look at geo-blocking, a firewall rule or a DNS problem that affects only some networks.

Why is the status code alone not enough?

A status of 200 means the web server answered. It does not mean the page works. Typical cases where the status is fine and the site is not:

  • The application shows a friendly error page but sends 200.
  • A cache or CDN delivers an empty or outdated page.
  • The web server answers with its default site or another customer's site after a configuration change.
  • The domain has expired and now shows a registrar's parking page.
  • A consent, login or maintenance page replaces the content.

A keyword check closes that gap. It downloads the page and looks for a text that is only present when the page really works: a product name, a menu entry, the copyright line in the footer. Choose text that comes from the application and, if possible, from the database, not from a static header.

The reverse form is just as useful: fail when a certain text appears, such as "Fatal error" or "Service Unavailable".

How do I verify an outage by hand?

When an alert arrives, three commands answer most questions. First, the status code, the total time and the address that actually answered:

curl -sS -o /dev/null -w "%{http_code} %{time_total}s %{remote_ip}\n" https://example.com
200 0.128693s 104.20.23.154

If the request fails, curl names the stage: could not resolve host (DNS), connection refused or timed out (network, firewall, web server stopped), or a certificate problem (TLS). For the last one, see monitoring TLS certificate expiry.

Second, the keyword. grep -c prints the number of matching lines; 0 means the text is missing:

curl -fsS --max-time 10 https://example.com | grep -c "Example Domain"
1

-f makes curl fail on HTTP errors from 400 upwards instead of printing the error page, and --max-time keeps the command from hanging.

Third, where the time is spent. If the site is slow rather than down, the breakdown shows whether DNS, the connection, TLS or the application is responsible:

curl -sS -o /dev/null -w "dns %{time_namelookup}s  connect %{time_connect}s  tls %{time_appconnect}s  first byte %{time_starttransfer}s\n" https://example.com
dns 0.003642s  connect 0.022001s  tls 0.045717s  first byte 0.065489s

The values are cumulative. A large jump between tls and first byte points to the application or database; a large dns value points to the name servers.

Useful variations: curl -4 and curl -6 test IPv4 and IPv6 separately, because a forgotten AAAA record breaks the site only for part of the visitors. dig +short example.com @1.1.1.1 asks a public resolver directly. And run the test from a second network, such as a phone hotspot, before you conclude anything from a single vantage point.

Which settings matter?

  • Interval. One minute is a sensible default for production sites. Longer intervals delay detection and make the availability figure coarse.
  • Timeout. Around ten seconds for a full page request is common. Much shorter, and slow moments count as outages; much longer, and a hanging site is detected late.
  • Redirects. Decide whether the check follows redirects. A check on http:// that follows a redirect to https:// tests two things; a check that does not follow it tests only the redirect.
  • What to check. The home page is often cached. A page that needs the application and the database, such as a search result or a login form, says more.
  • Not only HTTP. Mail servers, databases and VPN gateways are checked through TCP, ping or DNS checks. Jobs that run on a schedule cannot be checked from the outside at all; they need to report in themselves, as described in cron job monitoring with heartbeats.

Common mistakes

  • Checking only from the own network or the same data centre. That measures the internal path, not what visitors experience.
  • Alerting on the first failed request. One failure from one location is a measurement, not an incident.
  • Blocking the checkers. Rate limits, bot protection and geo-blocking rules often catch monitoring requests. Allow the checker addresses or user agent explicitly.
  • Using a keyword that is always present. A word in a static header or in the error page itself proves nothing.
  • Checking a static file. robots.txt stays reachable long after the application has stopped.
  • Ignoring response time. A page that takes fifteen seconds is down for practical purposes, even with status 200.

Checklist

  1. Check every important site from at least three independent locations.
  2. Alert only when the majority of locations sees the failure.
  3. Require confirmation by a second measurement before notifying anyone.
  4. Add a keyword that only appears when application and database work.
  5. Set a timeout of about ten seconds and an interval of one minute.
  6. Make sure firewalls and bot protection let the checkers through.
  7. Keep the curl commands above at hand to verify an alert in a minute.

How Lynx Heartbeat helps

Lynx Heartbeat checks websites from four locations (Nuremberg, Helsinki, Ashburn in the USA and Singapore) and raises an alert only when they agree: every location that is currently reporting has to see the error. Besides plain HTTPS checks there are keyword, certificate expiry, domain expiry, ping, TCP and DNS checks. Failures are debounced before a notification goes out, and the recovery notice states how long the incident lasted. The check types are listed on the features page; plans and included monitors are on the pricing page.

Frequently asked questions

Why does my uptime monitor report outages that nobody else noticed?

A single check location reports every problem on its own network path as an outage of your site: packet loss, a slow DNS resolver, a routing change at one provider. With several locations and a majority rule, such local problems no longer trigger an alert.

How many check locations do I need?

Three is the practical minimum. With two locations a disagreement cannot be decided; with three, two out of three form a majority. The locations should not all share the same network or data centre.

What is a keyword check?

A keyword check fetches the page and looks for a text that only appears when the page works, such as a product name or the footer text. It catches pages that answer with status 200 but show an error, an empty page or the wrong site.

How often should a website be checked?

Once a minute is a good default for sites that matter. Combined with a rule that a failure must be confirmed by a second measurement, a real outage is reported after about two minutes.

Does a multi-location check slow down the detection of real outages?

Hardly. A real outage is seen by all locations in the same round, so the majority is reached immediately. Only failures seen by a single location are held back.

Websites

Related articles

All articles