Skip to content

Monitoring client servers as an agency or hosting provider

Monitoring servers for many clients works when each client has a separate area with its own servers, alert channels and status page, people get only the role they need, and agents are rolled out in small waves with one-time tokens. Add consistent names, alert routing and maintenance windows per client, a weekly report, and a defined offboarding that stops monitoring and deletes the client's data.

Monitoring servers for many clients stays manageable when four things are fixed from the start: every client has their own area with their own servers, alert channels and status page; every person gets only the role they need; agents are rolled out in small waves with one-time tokens; and there is a written process for when a client leaves. Naming conventions, alert routing per client, maintenance windows and a weekly report then fit into that frame. This article goes through each part and ends with a checklist.

Why one area per client?

The temptation is to put all servers into one account and sort them with tags. That works until the first mistake: an alert for client A goes to client B's channel, a client login shows someone else's server, a status page lists a monitor that belongs to another contract.

A separate area per client prevents this structurally:

  • Servers and monitors belong to exactly one client.
  • Alert channels are defined per client, so a rule cannot accidentally notify the wrong people.
  • Status pages draw only from the client's own monitors.
  • Logins for the client can see only their own area.

Your own team needs the opposite view: one place that shows across all clients which one needs attention right now. Look for a setup where the separation runs one way (you see all clients, a client sees only themselves) and is enforced by the system, not by a filter in the interface that someone can forget to set.

Which roles do people need?

Least privilege applies to monitoring like to any other admin tool. Monitoring holds host names, versions and the weak points of every client, which makes it an attractive target. A simple split:

RoleTypical personMay
Ownerhead of operationsmanage users, install agents, settings
Technicianon-call engineeracknowledge incidents, change alert rules
Readerproject manager, supportsee everything, change nothing
Client (end customer)contact person at the clientsee their own servers and incidents, read-only

Three rules keep this clean: personal logins only (no shared "ops" account), two-factor authentication for everyone with write access, and a review of all logins every quarter. When someone leaves your company, their access ends on their last day, not when somebody notices.

How do I roll out an agent to many servers?

Installing by hand works for five servers. For fifty, you want a list, a pre-check and waves. Whether you use a configuration management tool or a plain SSH loop, the principles are the same.

First, check that you can reach every server non-interactively and know what is running on it:

while read -r host; do
  printf '%-20s ' "$host"
  ssh -n -o BatchMode=yes -o ConnectTimeout=5 "$host" \
    '. /etc/os-release && echo "$PRETTY_NAME $(uname -m)"' || echo "UNREACHABLE"
done < wave-1.txt
web-01               Ubuntu 24.04.1 LTS x86_64
web-02               Debian GNU/Linux 12 (bookworm) x86_64
db-01                UNREACHABLE

-n keeps ssh from consuming the rest of the list on standard input, and BatchMode=yes makes it fail instead of asking for a password. Fix every UNREACHABLE before you go on.

Then install, with these precautions:

  • One token per server, valid once and for a short time. If a token leaks, it is useless after it has been used or expired.
  • Never put the token on the command line. Command lines appear in ps, in shell history and in logs. Pass it over standard input into a file with mode 0600 that the installer reads and deletes.
  • Waves. One or two servers first, then ten, then the rest. Check after each wave that the new servers report data.
  • Stop at the first unexplained failure. A rollout that continues after errors produces fifty half-installed agents.
  • Keep a rollback. Know the one command that removes the agent again, and test it on the first server.
# token for this one server, kept out of the command line
ssh -o BatchMode=yes web-01 'umask 077; cat > /root/agent.token' < token-web-01.txt

The installer then reads /root/agent.token; how exactly depends on your agent.

Warning: A loop over SSH with root access runs the same command on every server in the list. Test it against one server, read the list twice, and never run a loop that changes things without the pre-check above.

How should servers be named and grouped?

Names should tell the person on call at 3 a.m. which client, which role and which server it is, without opening a spreadsheet. A pattern such as <client>-<role>-<number> does that: shop-a-web-01, shop-a-db-01, clinic-b-mail-01. Keep the client part identical in server names, monitor names and channel names.

Within a client, groups by role (web, database, mail) or by environment (production, staging) help with rules: staging can have quieter alerting than production. Avoid groups by person ("Tom's servers"); people change, servers stay.

How do alerts reach the right people?

Every client needs an answer to three questions, written down: who is notified, through which channel, and what happens if nobody reacts.

  • Channels per client. Your on-call channel always; the client's own contact only if they asked for it and know what to do with an alert.
  • Escalation. If nobody acknowledges an incident, notify a second person after a fixed time, then the team. See alert fatigue, quiet hours and escalation for timing and pitfalls.
  • Service levels. A client with a support contract around the clock gets different rules from one with office hours only. Put the difference into the rules, not into people's heads.

Maintenance windows, status pages and reports

Maintenance windows per client, per server or per monitor keep planned work from paging anyone. Measurement should continue during the window, so you can see afterwards whether the server came back properly.

A status page per client answers "is it down or is it me?" without a phone call. It should show the client's services and their state, announced maintenance and a way to subscribe, and nothing about the internal setup.

A weekly report shows the client what you did for them: availability, incidents and their duration, and what expires soon, such as certificates and domains. It also helps you notice a client whose servers produce far more incidents than others.

What happens when a client leaves?

Offboarding is the part most often forgotten. A client who left two years ago and still has an agent running, alert channels pointing to former employees and a status page online is a data protection problem and a source of confusing alerts. Write the steps down once:

  1. Agree on the end date and on whether data has to be kept or handed over.
  2. Remove the agent from every server of the client, or have the new provider do it.
  3. Stop all external checks for the client's websites.
  4. Revoke the client's logins and remove their alert channels.
  5. Hand over the status page domain or switch the page off.
  6. Delete the stored data once any agreed retention period has passed.
  7. Record what was deleted and when.

Common mistakes

  • All clients in one area, separated only by tags.
  • Shared logins for the whole team.
  • Install tokens in shell history or a ticket.
  • Rolling out to all servers at once.
  • Sending every alert to the client as well as to the team.
  • No offboarding, so former clients stay monitored for years.

Checklist

  1. One separate area per client, with its own channels and status page.
  2. Personal logins with the smallest role that works, second factor for write access.
  3. Pre-check SSH access and operating systems before any rollout.
  4. One-time tokens per server, passed over standard input.
  5. Rollout in waves, stop at the first unexplained failure.
  6. A naming pattern with the client in every name.
  7. Written alert routing and escalation per client.
  8. Maintenance windows, status page and weekly report per client.
  9. A written offboarding process, including deletion of data.

How Lynx Heartbeat helps

In Lynx Heartbeat each client is a sub-tenant: a closed area with its own servers, monitors, alert channels and status page. Your agency sees every sub-tenant, a sub-tenant never sees the agency or another client, and this is enforced in the database. Clients can get their own read-only login for their servers and incidents, or a status page on their own domain with announced maintenance and an email subscription. The sub-tenants belong to the Agency plan and are set up for you when you request access. When a client leaves, one command on each server removes the agent again; the rest of the offboarding stays your process. More on the agencies page.

Frequently asked questions

How should an agency separate clients in its monitoring?

Give each client a separate area with its own servers, monitors, alert channels and status page. Your team sees all areas; a client sees only their own. Ideally the separation is enforced by the system itself, not only by naming or filters.

How do I install a monitoring agent on many servers safely?

Check SSH access and the operating system first, then install in small waves, starting with one or two servers. Use a separate one-time token per server, pass it over standard input instead of the command line, and stop the rollout at the first unexplained failure.

Should clients get access to the monitoring?

Only if they need it, and then read-only and limited to their own servers. Most clients are better served by a status page and a regular report. Security findings and internal details usually stay with the agency.

What do I do when a client leaves?

Remove the agent from their servers, stop external checks, revoke the client's logins and alert channels, hand over or switch off the status page, and delete the stored data once any agreed retention period has passed. Write down what was deleted and when.

Operations

Related articles

All articles