Online shop down at night: notice it before customers do
In short
A shop can answer 200 on the home page while the cart, the checkout or the payment step is broken. Check the pages that earn money for a text that must appear, and for one that must not, measure from more than one location so that a network hiccup wakes nobody, and watch certificate and domain expiry. Decide in advance who is woken, after how long, and what that person does first.
To notice a shop outage at night before customers do, you need more than a check that the home page answers. Check the pages that make money, such as cart and checkout, for a text that must be there; measure from more than one location; watch certificate and domain expiry; and agree in advance who gets woken and what they do first. This article covers each part with commands you can try from any Linux machine.
Why is "the home page returns 200" not enough?
A status code says that the web server answered, not that customers can buy. Common failures that keep the home page green:
- The front page comes from a cache. A reverse proxy or a full-page cache serves it while PHP or the database behind it is down.
- Cart and checkout depend on more. Sessions, the database, stock queries and the payment provider are only needed there, so these pages break first.
- Maintenance mode answers 200. Many shop systems show their maintenance page with a normal status code. After a failed update at 02:00 the shop stays closed until someone looks.
- Error pages answer 200. An application that catches its own exceptions often renders "Something went wrong" as a regular page.
So check at least one page that needs the full application, and check its content, not just its status.
How do I check a shop page by hand?
Start with status and response time. -f makes curl fail on 4xx and 5xx, -sS hides the progress bar but shows errors:
curl -fsS -o /dev/null -w '%{http_code} %{time_total}\n' https://shop.example.com/checkout200 0.412A broken page looks like this:
curl: (22) The requested URL returned error: 502
502 30.004Then look for a text that must be on the page. Choose something that only appears when the application really works, for example the label of the order button or the heading of the cart, not the shop name in the footer:
curl -fsS --max-time 10 https://shop.example.com/cart \
| grep -q "Proceed to checkout" && echo "ok" || echo "MISSING"And the other way round, a text that must not be there:
curl -fsS --max-time 10 https://shop.example.com/ \
| grep -qi "maintenance" && echo "MAINTENANCE PAGE" || echo "ok"Some details matter for both:
- Set a timeout. Without
--max-time, curl waits a long time for a server that accepts the connection but never answers. A shop that needs 25 seconds for the checkout is down for the customer. - Do not test with a login. A check should use the pages an anonymous visitor sees. Cart pages usually work without an account.
- Do not place orders. A check that submits forms every minute creates fake orders, sends mails and may trigger the payment provider. Check that the checkout page loads; test a full order by hand or in a staging system.
Which expiry dates take a shop offline?
Two kinds of outage need no failure at all, just a date.
The TLS certificate: when it expires, browsers show a full-page warning and almost nobody buys. Read the date with openssl:
echo | openssl s_client -connect shop.example.com:443 -servername shop.example.com 2>/dev/null \
| openssl x509 -noout -enddatenotAfter=Mar 14 08:12:03 2027 GMTopenssl x509 -checkend 1209600 returns an error code if the certificate expires within 14 days, which is useful in a script. Automatic renewal fails quietly more often than one would like: a changed DNS record, a blocked port 80 or a renewal hook that no longer runs.
The domain: if it is not renewed, the name stops resolving and the shop disappears for everyone at once. Check the expiry date at the registrar and make sure the payment details stored there are current.
Why check from more than one location?
A check from one place cannot tell "the shop is down" from "the path between this checker and the shop is broken". At night, when routes are changed and maintenance happens, such false alarms are common, and after two or three of them people start to ignore the phone.
The rule that works: the shop counts as down only when several independent locations see the failure at the same time, and only after it failed more than once in a row. A single slow answer is noise; a confirmed failure from every location is an incident.
What breaks under a load peak?
Night is not always quiet. Newsletters go out early, campaigns start at midnight, bots crawl the catalogue. Two bottlenecks show up first in a shop:
- PHP-FPM runs out of workers. Requests queue, the checkout becomes slow, then the web server answers 502 or 504. How to find and size the limit is explained in PHP-FPM: server reached pm.max_children.
- The database runs out of connections. Slow queries hold connections until new ones are refused.
In both cases the home page from the cache may still be fast. This is exactly why the checkout check with a timeout matters.
Who gets woken, and when?
An alert at night needs an owner. Write down:
- Who is on call this week. One person, not a group chat where everybody waits for somebody else.
- What counts as urgent. Shop unreachable or checkout broken: yes. Certificate expires in 12 days: no, that is a message for the morning.
- Escalation. If the person on call does not acknowledge within 15 minutes, the next person is notified, then a third.
- Quiet hours. Hold back minor warnings at night, but let a confirmed outage through.
How to keep alerts rare enough that people still react is described in Alert fatigue: debounce, quiet hours and escalation.
A short runbook for the night
Keep it to one page and store it where the person on call can open it from a phone.
- Confirm the outage from a second network, for example the mobile phone, with the curl commands above.
- Log in to the server and check the basics:
uptime,df -h,free -m. - Check the services:
systemctl status nginx php8.3-fpm mariadb(orapache2,mysql, with your PHP version). - Read the latest errors:
journalctl -u php8.3-fpm --since "30 min ago"and the error log of the web server. - Fix the obvious cause: free disk space, restart a hung PHP-FPM pool with
systemctl restart php8.3-fpm, end a runaway query. Note what you see before restarting, because a restart erases the evidence. - If the fix takes longer, post a short notice on the status page.
- Write three lines for the morning: what happened, what you did, what is still open.
Avoid larger changes at night: no updates, no configuration experiments, no restore of the database unless it is clearly needed.
Common mistakes
- Only the home page is checked. The cache keeps it green while checkout is down.
- The keyword is too generic. The shop name also appears on the error page.
- No timeout. A hanging page counts as "still loading" instead of down.
- One location. Its network problems become your alarms.
- Everybody is notified. Then nobody feels responsible.
- Certificate renewal is trusted blindly. It works until the day it does not.
Checklist
- Choose one page that needs the full application, usually cart or checkout.
- Pick a text that must appear and one that must not.
- Check with a timeout of about 10 seconds.
- Monitor certificate and domain expiry with warnings weeks ahead.
- Check from several locations and confirm before alerting.
- Name the person on call and set an escalation.
- Write and store the runbook.
How Lynx Heartbeat helps
Lynx Heartbeat checks your pages over HTTPS from four locations in Nuremberg, Helsinki, Ashburn (USA) and Singapore, with a keyword that must be on the page or a text that must not be there. A shop counts as down only when every location that is reporting sees the failure and the check failed twice in a row. Certificates are flagged 14 and 3 days before they expire, domains 30 and 7 days ahead where the registry publishes the date. Quiet hours can let serious incidents through, and if nobody acknowledges, the next step is notified after 15 and 30 minutes. A monitor loads pages but does not submit forms or place test orders; that part stays a manual test. Details are on the page Website monitoring.
Frequently asked questions
Why is checking the home page of a shop not enough?
The home page is often served from a cache and stays up while the application behind it fails. Cart, checkout and payment need the database, sessions and external services, so they break first. A maintenance page or an error page that a shop system renders itself can also come with status 200.
What is a keyword check?
A check that loads a page and looks for a specific text in it, for example the label of the button that completes an order. If the text is missing, the check fails even if the status code is 200. The opposite form looks for a text that must not appear, such as a maintenance notice or a database error.
Why should uptime checks run from several locations?
A single location sees its own network problems as an outage of your shop. If the shop only counts as down when several independent locations see the failure, a short routing problem at one of them does not wake anyone at three in the morning.
Who should be woken when the shop goes down at night?
One person on call, not the whole team. If nobody acknowledges the incident within a fixed time, the next person is notified. Quiet hours hold back minor warnings at night but should let a confirmed outage of the shop through.