AI root cause analysis for server outages: what works
In short
A language model is useful for root cause analysis when you give it a timeline, the metrics around the incident, recent changes and short log excerpts, and ask for ranked hypotheses with a way to verify each one. Redact secrets and customer data first, never let the model run commands itself, and check every claim against the evidence line it is based on.
A language model can shorten root cause analysis after a server outage, but only as well as the evidence you give it. Hand it a timeline, the metrics around the incident, recent changes and short log excerpts, and ask for ranked hypotheses, each with a way to verify it and the risk of the next step. Redact secrets and customer data first, keep a person in charge of every command, and check each claim against the line of evidence it rests on. Used this way, it saves you the first twenty minutes of searching. Used as an oracle, it sends you down the wrong path with confidence.
What does a language model actually need?
A model knows a lot about Linux and nothing about your server. Everything it says about your incident comes from what you paste. Four kinds of evidence do most of the work:
- A timeline. When did the first symptom appear, when was the alert, what did people do and when. Times with time zone, ideally from the same clock.
- Metrics around the incident. CPU, load, memory, swap, disk usage and I/O for an hour or two before the start and a few minutes after. Trends matter more than single values: "memory available fell from 3 GB to 80 MB within 40 minutes" is more useful than "memory is low".
- Recent changes. Deployments, package updates, configuration changes, new cron jobs, traffic campaigns. Many outages follow a change, and the model cannot guess that one happened.
- Short log excerpts. The lines around the start, from the affected service and the kernel. Twenty relevant lines are better than two thousand irrelevant ones.
What it does not need: your whole log directory, a full process dump or the configuration of unrelated services. More text does not mean a better answer; it buries the signal and costs more.
What must the model never see?
Before anything leaves the server, go through it once with one question: would I paste this into a public ticket? If not, redact it.
- Secrets. Passwords, API keys, tokens, private keys, session cookies. They appear in more places than expected: in environment files, in connection strings such as
mysql://app:secret@db-01/shop, in command lines shown byps, in debug logs. - Environment variables. Never paste the output of
env,printenvor a.envfile. If a variable matters, name it and describe its value ("DB_HOSTpoints to the replica"). - Customer and personal data. Email addresses, names, order numbers, IP addresses of visitors in access logs. Replace them with placeholders such as
customer-1or203.0.113.10. - Internal names, if your policy requires it. Host names and domains can be replaced consistently (
web-01,example.com) without losing meaning.
A quick filter before copying helps, but it is no guarantee:
grep -inE 'pass(word)?|secret|token|api[_-]?key|authorization' excerpt.txtIf this prints anything, look at each line before you send the file.
How do I gather evidence on Linux quickly?
These commands are read-only and work on current Debian and Ubuntu. Run them as root or with sudo, and save the output to a file so you can redact it before sharing.
The service log for the window of the incident:
journalctl -u mysql --since "2026-11-17 02:00" --until "2026-11-17 02:45" --no-pagerKernel messages, which show the OOM killer, disk errors and hung tasks:
dmesg -T | tail -n 30
journalctl -k --since "2026-11-17 02:00" --no-pager[Tue Nov 17 02:31:07 2026] Out of memory: Killed process 1432 (mysqld) total-vm:5918032kB, anon-rss:3412880kB, file-rss:0kB, shmem-rss:0kB, UID:110 pgtables:7348kB oom_score_adj:0A search for the OOM killer in the classic log file. On Debian 12 and later there is often no /var/log/syslog, because rsyslog is not installed by default; use journalctl -k there. The RHEL family writes to /var/log/messages.
grep -i "out of memory" /var/log/syslogThe current state of disks and memory:
df -h
free -m total used free shared buff/cache available
Mem: 7940 7702 95 12 142 81
Swap: 2047 2047 0 0And whether the server rebooted or was shut down:
last -x | head
journalctl --list-boots | tail -n 3last -x lists reboots and shutdowns from wtmp; journalctl --list-boots shows each boot with its first and last journal entry, which is useful if the server died without a clean shutdown. For a full disk, start with finding what is using the space.
What should the prompt look like?
A loose question ("why did my server crash?") gets a loose answer. A fixed structure gets a usable one:
Symptom: web-01 (Ubuntu 24.04, 8 GB RAM) stopped answering HTTP at 02:31 CET.
Restarted by hand at 02:44, fine since.
Timeline:
02:00 nightly import job starts (runs daily, unchanged for months)
02:20 memory available falls below 500 MB
02:31 HTTP check fails, alert sent
02:44 server restarted
Recent changes: yesterday 16:00 deployment, MySQL buffer pool raised from 2 GB to 4 GB.
Evidence:
[kernel lines, service log excerpt, free -m output, redacted]
Task:
1. List the three most likely causes, ranked.
2. For each, quote the evidence line it is based on.
3. For each, give one read-only command that confirms or rules it out.
4. For any fix you suggest, state the risk and whether it needs a restart.
Say clearly if the evidence is not enough.The last line matters. A model asked for an answer will produce one; asking it to say when the data is insufficient gives it a way out other than guessing.
Why must a person run every command?
During an incident the server is already in a fragile state. A model that suggests systemctl restart mysql does not know that the database is replaying its log and a restart now costs another twenty minutes, or that the "large old files" it wants removed are binary logs a replica still needs.
Keep the roles clear: the model proposes, a person decides and runs. That means no agent with shell access, no automation that pipes the answer into a terminal, and no copying a command you do not understand. Read-only commands first, changes after the cause is confirmed, and anything destructive only with a backup and a second pair of eyes.
How do I catch made-up answers?
Language models produce fluent text whether or not the facts support it. Typical failures in root cause analysis:
- Invented evidence. "The log shows repeated connection timeouts" when it shows nothing of the kind.
- Wrong option names. A configuration directive that does not exist in your version.
- The plausible standard story. "Probably a memory leak" fits many incidents and explains none.
The countermeasure is the one already in the prompt: every claim needs a quoted line or metric from your data. Then check the top hypothesis yourself with a read-only command. If the quote is not in what you pasted, discard the claim. For error patterns in logs, a rising error rate is often a better guide than single lines.
What about costs and data protection?
If you use a model through an API, use an account and key that belong to you or your company, so you know who processes the data and who pays. Check the provider's terms on data retention and whether inputs are used for training. In the EU, sending server data that may contain personal data to a provider usually needs a data processing agreement; this is general information, not legal advice.
Costs depend on the amount of text. Condensed evidence (summaries, minimum and maximum, the relevant lines) is cheaper and gives better answers than raw dumps.
When should I not use it?
- While the outage is still running and the fix is obvious. Restore service first, analyse afterwards.
- For security incidents. A compromised server needs forensics and a clean copy of the evidence, not text pasted into another service.
- When you cannot redact the data in the time available.
- When nobody can verify the answer. If the person on call cannot judge the suggested commands, escalate to someone who can.
Common mistakes
- Pasting whole log files instead of the window around the start.
- Leaving out the recent change because it "cannot be related".
- Pasting
envoutput or connection strings with passwords. - Running a suggested command without reading it.
- Accepting the first hypothesis without a confirming check.
- Treating the analysis as finished: write down the confirmed cause and the fix.
Checklist
- Write the timeline with times and time zone.
- Collect service log, kernel log,
df -h,free -mand reboots for the window. - Add metrics from before the start and the list of recent changes.
- Redact secrets, environment values and customer data.
- Ask for ranked hypotheses, each with an evidence line, a check and a risk.
- Verify the top hypothesis with a read-only command.
- Decide and run changes yourself.
- Record cause and fix with the incident.
How Lynx Heartbeat helps
Lynx Heartbeat has an AI incident analysis on the incident page that does the gathering for you: it sends Claude what Lynx measured around the incident, including condensed metrics from two hours before the start, the checks that failed, the names of the largest processes and earlier incidents on the server. Claude returns the likely cause with its certainty, the evidence from the data, steps with command and risk, and what to check if it is not that. It runs nothing; the answer is text, and you decide what to execute. The analysis uses your own Anthropic API key, is off until the account owner switches it on, and never sends file contents, command lines, user names or credentials, which also means it does not read your log files. Details are on the AI incident analysis page.
Frequently asked questions
Can a language model find the root cause of a server outage?
It can suggest likely causes quickly if you give it good evidence: a timeline, the metrics around the incident, recent changes and short log excerpts. It cannot see anything you did not give it, so its answer is a ranked list of hypotheses to verify, not a diagnosis.
What should I never paste into an AI assistant during an incident?
Passwords, API keys, tokens, private keys, environment files, connection strings with credentials, and personal or customer data such as email addresses or order details. Redact them or replace them with placeholders before you send anything.
Should an AI assistant be allowed to run commands on my server?
Not during an incident. Let the model suggest commands with their risk, then decide and run them yourself. A wrong command on a server that is already in trouble can turn a short outage into data loss.
How do I know whether the AI's answer is made up?
Ask it to quote the exact evidence line or metric behind every claim. A claim without a source in the data you provided is a guess. Then verify the top hypothesis with a read-only command before you change anything.