EAST4SERV / SEARCH

ما الذي تبحث عنه؟

InsightsOperations
Operations

Control Linux Journal Growth Without Losing Incident Evidence

Control Linux journal growth without losing incident evidence

MEASURE LOGS to SET BUDGET to VERIFY HISTORY workflow diagram
Original East4Serv technical workflow for Control Linux Journal Growth Without Losing Incident Evidence
Share this article
Control Linux journal growth without losing incident evidence

A full root filesystem can interrupt databases, package updates and even logging itself. On a systemd host, the journal may be one contributor, but deleting journal files blindly can destroy the evidence needed to explain an outage. This guide shows how to measure journal use, distinguish persistent from volatile storage, set a deliberate retention budget and verify the result. Examples apply to modern systemd-based Linux hosts; check the local systemd version and logging policy first.

Prerequisites and retention policy

You need administrative access and agreement on how long incident records must remain available. Some organizations ship logs to a protected central collector; others rely on local journals. A size cap cannot promise a fixed number of days because log volume changes with traffic and incidents. Decide what matters more: a minimum historical window, a maximum disk budget, or both. Record any compliance or incident-hold requirement before reducing retention.

Start with read-only measurements and inspect the filesystem that actually contains journal data:

df -h / /var/log
journalctl --disk-usage
du -sh /var/log/journal /run/log/journal 2>/dev/null
systemd-analyze cat-config systemd/journald.conf

The final command is available on many modern systemd releases; if it is unavailable, inspect /etc/systemd/journald.conf and its drop-ins directly. journalctl --disk-usage includes active and archived journal files. A file manager may show a different value if it measures another filesystem or namespace. Do not infer that the journal is the cause of a full disk until you compare other consumers.

Understand where records are stored

Storage=volatile uses runtime storage under /run, which generally does not survive reboot. Persistent storage uses /var/log/journal. Storage=auto depends on whether persistent storage is available according to the local configuration. Early boot records may initially be held in volatile storage. Read the actual settings instead of assuming every distribution has the same default. A container may forward logs to a runtime collector rather than retain a host journal in the container.

journalctl -b -p warning --no-pager
journalctl --list-boots
journalctl -u nginx --since '1 day ago' --no-pager

The unit name nginx is only an example. If --list-boots shows only the current boot, verify storage and rotation before promising post-reboot history. Logs can contain IP addresses, paths and application errors; restrict access and avoid pasting raw journals into public issues.

Set a measured disk budget

Systemd offers SystemMaxUse= and SystemKeepFree= for persistent journals, with RuntimeMaxUse= and RuntimeKeepFree= for volatile storage. The service applies the tighter applicable limit. Use a drop-in under /etc/systemd/journald.conf.d/ only after checking whether your release supports it and whether another managed configuration already controls the values. A proposed setting is shown below for review, not a universal recommended size:

# /etc/systemd/journald.conf.d/60-local-retention.conf
[Journal]
SystemMaxUse=1G
SystemKeepFree=2G

Size the numbers for the real filesystem and log rate. A small VM and a large logging host need different budgets. Back up the old configuration, review the merged settings, then apply the vendor-documented reload or restart procedure in a planned window. Do not restart logging during a live incident merely to obtain a prettier disk graph. A cap can discard the oldest archived records as new records arrive, so confirm the retention you actually obtain over time.

Reclaim space only after preserving evidence

journalctl --vacuum-size and --vacuum-time remove archived journal files. They do not erase records from the active file, so the reported total may remain above the requested size. A rotation step may make the current file archived, but that action should be reviewed against the incident-evidence requirement. Prefer exporting necessary evidence to a secure location before vacuuming. Avoid manually removing *.journal files from a running system.

journalctl --disk-usage
journalctl --verify

These checks are read-only. If the filesystem remains full after a reviewed retention adjustment, inspect application logs, caches, crash dumps, container layers and orphaned files. A log flood often reflects a real recurring application error; size limits prevent disk exhaustion but do not fix the source.

Troubleshooting and validation

If the configured limit seems ignored, check whether the journal is persistent or volatile, whether a drop-in overrides your setting, and whether active files account for the discrepancy. Examine permissions and mount state. If journal messages suddenly disappear, check Storage=, log rate limits, service-level output settings and whether a collector is receiving the records. Validate that at least one recent and one older expected event can still be retrieved after a change.

FAQ: Is journalctl --vacuum-size=1G a permanent cap? No; it is an immediate cleanup of archived files, while journald configuration governs ongoing storage. FAQ: Can I delete journal data before a backup? Only after confirming the records are no longer required for incident response or retention. Expected result: disk use remains within a documented budget while enough history remains to diagnose service failures.

Related reading: Diagnose a systemd service after reboot and Linux disk-space diagnosis. East4Serv server management can help review a logging and alerting policy.

Primary references: https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html and https://www.freedesktop.org/software/systemd/man/latest/journalctl.html.

journaldLog retentionLinux operations