On this page
An incident runbook should answer the questions that are hardest to answer under pressure: what failed, which services depend on it, where a known-good backup lives, who can authorize recovery and how to prove that customers are served again. A long document without tested commands is not enough. This guide shows a practical structure for a Linux web application and explains how to exercise it safely. Replace every illustrative path, service name and owner with your actual inventory.
Prerequisites and recovery objective
List the application owner, on-call contact, hosting account, DNS authority, database, queues, external providers and backup location. Define recovery time and recovery point objectives with the business owner; do not invent a guarantee based on an untested backup schedule. Distinguish a single-service restart from a host rebuild and from a regional outage. Each scenario needs a different decision threshold and communication path.
Keep privileged credentials in an approved secrets manager, never in the runbook. The document may name the credential location and access procedure without copying the secret. Store an offline or separately accessible copy of the runbook so a failed application does not hide its own recovery instructions.
Start with a read-only incident snapshot
Capture time, symptoms, scope and current health before changing anything. Read-only Linux commands can show whether the outage is a single unit, database dependency, disk exhaustion or memory pressure. Limit log collection to what is necessary and protect customer data in any shared incident channel.
date -u
uptime
df -h
free -h
systemctl --failed --no-pager
journalctl -b -p err --no-pager -n 80
These commands are examples, not a complete diagnosis. A healthy homepage may conceal a failed queue worker, broken login or payment callback. Record an application-specific health baseline covering public pages, authentication, database, background jobs and external dependencies. Note which checks are safe to run during an incident and which might create orders or customer resources.
Map dependencies and choose the recovery path
Document the actual service graph: reverse proxy, application process, database, cache, queue, scheduler, object storage and provider APIs. Identify which nodes share a host and which are external. If the database is unavailable, restarting the proxy is unlikely to help. If the disk is full, investigate the consumer before deleting logs or backups. If a deployment introduced the failure, a code rollback may be safer than a system reboot, but database migrations can make code-only rollback unsafe.
For each scenario, write a decision table with a trigger, an owner, an action and a validation. Example: “If the website returns 502 and the application socket is absent, inspect the app service log and recent deployment; restart only the failed website service after checking dependencies.” Keep the actual restart command in the internal version of the runbook once the unit name is verified. Do not publish a generic command that could restart an unrelated service.
Restore from a tested backup
Describe how to locate the last known-good database backup, its timestamp, encryption key reference and checksum. Restore first into an isolated environment. Verify schema, record counts and application behavior before considering production replacement. For customer data, assess the gap between backup time and incident time; a restore can discard legitimate orders or payments. Protect financial and external provisioning events by reconciling authoritative provider state before replaying jobs.
sha256sum /path/to/verified-backup.dump
This is an integrity check example, not a restore instruction. A matching checksum establishes file integrity against a trusted recorded value; it does not prove the backup is complete or restorable. Include the exact database-specific restore procedure only after it has been tested for your schema and permissions. Document the expected duration and the rollback if the restore fails.
Validate customer journeys, not only processes
After a fix, run smoke tests that reflect actual business use: homepage, login, account page, a read-only order view, a support-ticket view and queue processing of a safe idempotent job. Check both the direct origin and the public hostname if a proxy or CDN is involved. Verify that no unexpected 403, 429 or 500 responses appeared in logs. If payments or provisioning were interrupted, reconcile events before retrying; never blindly replay non-idempotent work.
Record the recovery timestamp, affected customers, data-loss assessment, residual risk and follow-up tasks. Communicate the verified state, not an optimistic “all clear.” Schedule a post-incident review that improves alerts, dependency ordering, backups or deploy checks. A runbook becomes useful when drills expose its missing assumptions.
Exercise and maintain the runbook
Run a tabletop exercise with operations, support and the business owner. Then perform isolated restore drills and safe failure simulations without touching production customer resources. Assign a reviewer to refresh commands after operating-system, hosting or provider changes. Track last drill date and any unresolved issue. Archive obsolete procedures so they do not compete with the current recovery path.
FAQ: Should every failure trigger a full restore? No; restore only when evidence shows that narrower recovery cannot safely repair the state. FAQ: Can a successful backup job replace a drill? No; the restore path, access and data consistency must be tested. Expected result: an authorized on-call engineer can diagnose the failure, select a safe recovery path and demonstrate restored customer workflows without improvising dangerous commands.
Related reading: Linux backup planning and restore tests and Linux health monitoring. East4Serv server management can help review recovery procedures and operational ownership.
Primary references: https://csrc.nist.gov/glossary/term/disaster_recovery_plan and https://nvlpubs.nist.gov/nistpubs/legacy/sp/nistspecialpublication800-34r1.pdf.