Small Linux habits that can save hours when something goes wrong

Administering Linux isn’t just about memorizing commands. Many of the differences between resolving an issue in ten minutes or spending an hour testing things often come down to small habits: checking before touching, verifying what has changed, saving evidence before restarting, and confirming the outcome of each action. These simple practices are useful whether managing a home server or overseeing production systems.

The keys to good Linux habits in 20 seconds

• Before restarting a service, it’s wise to review its status and recent logs.
• journalctl, ss, findmnt, or vmstat help to understand what’s actually happening.
• Destructive commands should be tested with options like --dry-run first.
• In automation, errors should be checked, and reliance on the terminal environment should be minimized.
• Documenting incidents prevents investigating the same problem twice.

Applying these practices doesn’t require installing an observability platform; many of the necessary tools are included in any modern distribution.

The core idea is to change a very human tendency: stop testing solutions before understanding the problem.

Before fixing Linux, ask it what’s happening

When a website stops responding, a database appears locked, or a service crashes, rebooting often seems like the quickest fix.

For example:

systemctl restart nginx

Sometimes it works.

The issue is that this also changes system state and may cause useful information to disappear—details that help identify the root cause.

Before rebooting, it’s much more informative to run:

systemctl status nginx

Then, review recent messages:

journalctl -u nginx --since "-15 min"

And check if any process is actively listening on the expected ports:

ss -lntp

A quick look at storage and memory completes an initial diagnosis:

df -h

free -h

Within less than a minute, you can gather a substantial amount of information.

This approach introduces a simple rule: observe first, then intervene.

Asking what changed often saves a lot of time

A server that worked yesterday and doesn’t today rarely changed by magic.

Likely causes include an update, deployment, certificate change, configuration modification, or firewall rule.

So, start with:

git status

git diff

If configuration is under version control, review recent commits:

git log --since="2 days ago" --oneline

On Debian or Ubuntu servers, also check APT logs and dpkg; RPM-based distributions have similar histories.

In professional environments, deployment histories via CI/CD, Ansible, or other automation platforms can be even more revealing.

Asking “what has changed?” is often a better strategy than immediately searching for a complex cause.

journalctl is much more useful when filtered

A common mistake is running:

journalctl -xe

and encountering hundreds or thousands of lines.

The issue might be present but hidden.

It’s more effective to narrow the search.

Only logs for nginx, for example:

journalctl -u nginx

The last twenty minutes:

journalctl -u nginx --since "-20 min"

Current boot errors:

journalctl -p err -b

Kernel messages:

journalctl -k -b

And there’s a particularly useful option right after a server has rebooted:

journalctl -b -1

This allows examining the previous boot, helping to understand what was occurring before the system restarted.

Checking failed services takes seconds

Another small command that can reveal hidden issues is:

systemctl --failed

It shows systemd units that have failed.

This might include auxiliary services, mounts, startup tasks, or components that haven’t yet indicated failure to users.

Investigate before resetting their status with:

systemctl reset-failed

Again, don’t erase a clue before understanding it.

Network, disks, and performance: check what’s really happening

Application configuration indicates how it should behave.

Linux can show how it is actually functioning.

This difference is crucial.

ss confirms which ports are listening

If an app supposed to run on port 443, verify with:

ss -lntp

This displays TCP sockets in listening state and, with sufficient permissions, their related processes.

For UDP:

ss -lunp

And for active TCP connections:

ss -tnp

This helps identify various issues.

If nothing appears on :443, the service isn’t listening.

If it appears as:

127.0.0.1:443

but should be accessible from another machine, it might be bound only to localhost.

Before suspecting routers, external firewalls, or cloud providers, it’s wise to verify what’s happening directly on the server itself.

df doesn’t tell the whole storage story

Almost any sysadmin knows:

df -h

It’s excellent for checking space usage.

But modern systems can have partitions, LVM volumes, cloud disks, NFS, bind mounts, or container storage.

To inspect devices:

lsblk

To examine mounts:

findmnt

And to answer specific questions like “Where is /var/lib/docker really located?”:

findmnt --target /var/lib/docker

This combination helps distinguish between physical disk, partition, filesystem, and mount point, related but not identical concepts.

It also avoids dangerous errors when multiple disks share similar names.

A snapshot of CPU usage doesn’t always explain a problem

top shows what’s happening right now.

But a machine can have an issue that occurs for five seconds every minute.

Tools like:

vmstat 1

provide, each second, data on CPU, memory, processes, swap, and I/O activity.

If sysstat is installed, another helpful tool is:

iostat -xz 1

which offers detailed info on storage device activity.

The difference is straightforward:

top indicates what’s happening now.

A series of measurements can reveal the pattern causing the problem.

When the application doesn’t explain the failure, check the kernel

Some failures only report that something went wrong.

The kernel can provide the missing piece.

For example:

journalctl -k -b

may reveal issues with disks, memory, drivers, or network interfaces.

If a process disappears without explanation, it’s also worth checking for signs of memory pressure or the OOM killer.

A problem seemingly caused by an application might actually be due to the operating system.

The most important habits happen before pressing Enter

Linux allows doing a lot with a single line.

That’s an advantage but also a risk.

For example:

find /backup -type f -mtime +30 -delete

this deletes all matching files directly.

It’s better to test the selection without deleting anything first:

find /backup -type f -mtime +30 -print

Once confirmed, add -delete.

Similarly with rsync:

rsync -a --delete /source/ /destination/

It’s wise to run a dry run first:

rsync -a --delete --dry-run /source/ /destination/

--dry-run shows what would happen without making actual changes.

This small habit can prevent significant data loss.

Validate configuration before restarting

Suppose nginx has been modified.

Instead of editing and restarting directly, test first:

nginx -t

Then, if everything checks out, reload:

systemctl reload nginx

This prompts a better workflow sequence:

edit
validate
compare
reload
verify

rather than:

edit
restart
wait

Whenever a service supports reload, prefer it over a full restart when possible.

kill -9 should be the last resort

When a process seems stuck, it might be tempting to just do:

kill -9 PID

But SIGKILL terminates immediately, without cleanup.

Start with:

kill PID

which sends SIGTERM.

Before using either, confirm which process you’re dealing with:

ps -fp PID

Additional info can be retrieved from:

cat /proc/PID/status

Sometimes, kill -9 is genuinely necessary.

The habit is to avoid using the most aggressive intervention as the first choice.

Good automation involves more than just scripting

Many scripts work fine when run manually but fail in cron or systemd.

The typical cause is the environment.

An interactive terminal can have a PATH, environment variables, SSH keys, or a working directory that doesn’t exist when run automatically.

Instead of:

python backup.py

use:

/usr/bin/python3 /opt/scripts/backup.py

and explicitly define the environment needed by the script, avoiding reliance on personal aliases.

Check if a command succeeded

Bash uses exit codes.

After a command, check its exit status with:

echo $?

A 0 usually indicates success.

In scripts, it’s better to check the command directly:

if rsync -a /data/ /backup/data/; then
    echo "Backup completed"
else
    echo "Backup failed" &&2
    exit 1
fi

This integrates the result into the script’s logic.

A backup shouldn’t be considered successful just because the script finished.

set -e doesn’t automatically fix all errors

Many scripts include:

set -e

to stop execution upon failure.

It can be helpful but Bash has several exceptions.

A common configuration is:

set -Eeuo pipefail

pipefail is especially useful in pipelines.

For example:

grep "error" archivo-inexistente | sort

Without pipefail, errors in the first command might go unnoticed, and the pipeline could return the result of sort.

Still, no set of options replaces explicit error handling in important automation.

Better to automate checks than fixes

Not all automation is about corrections.

Sometimes, simple notifications are enough.

For example:

systemctl is-active --quiet nginx || echo "nginx is not active"

This can be extended to automatically verify:

• free disk space;
• recent backups;
• expiring certificates;
• essential mounts;
• failing services;
• listening ports;
• RAID status;
• time synchronization.

Automatically restarting a service whenever a problem appears can hide issues for months.

Detecting and logging the state helps investigate the root cause.

Save a snapshot of the system before making changes

A diagnostic script can quickly gather essential data:

date
hostname
uptime
free -h
df -h
lsblk
systemctl --failed
ss -lntp
journalctl -p err -b --no-pager

And save it with:

/opt/tools/capture-state.sh > "/var/tmp/state-$(date +%Y%m%d-%H%M%S).log" 2>&1

If a restart is needed later, you still have a record of the previous state.

This is extremely useful in complex incidents.

The habit that doesn’t require a terminal: document what you’ve learned

Every solved problem becomes a piece of reusable knowledge.

A simple record should include:

• what happened;
• when it started;
• what changed;
• which logs were reviewed;
• which commands helped;
• the root cause;
• how it was fixed;
• how to detect a similar issue earlier.

No need for a special platform.

A private Git repository with Markdown documents can over time become a mini knowledge base about real infrastructure problems.

An important advantage is that six months later, no one has to rediscover the same solution exactly.

A good administrator’s experience isn’t just in their memory; it’s also in the procedures they’ve documented.

Linux values method over speed

Two people may know the exact same commands.

One approach:

restart
test
change
test again

The other:

observe
save evidence
define boundaries
form hypotheses
test
make minimal changes
verify

The second doesn’t necessarily know more Linux, but works with a better method.

This way of managing systems remains useful even with Prometheus, Grafana, OpenTelemetry, Kubernetes, or sophisticated observability platforms behind the scenes.

When something fails, there’s always a point where you need to ask concrete questions: which process exists, what port it listens on, what changed, which filesystem is full, or what the kernel logs just before failure.

Commands help to find those answers.

Good habits determine whether you ask the right questions before modifying the system.

Frequently asked questions

What should be checked first when a Linux server fails?

Depending on the issue, reviewing systemctl status, journalctl, systemctl --failed, ss, df, free, and findmnt provides a quick initial snapshot of the system’s state.

Why shouldn’t a service be immediately restarted?

Because restarting might change system state and add new logs, making it harder to identify the original cause. It’s better to gather evidence first.

What is the purpose of --dry-run?

It simulates certain operations without applying changes, helping to see what would happen. Tools like rsync use it to show files that would be copied or deleted before executing.

Is set -e enough to make Bash scripts safe?

No. It can help stop scripts on certain errors, but Bash has exceptions. For critical automation, explicitly checking results and understanding options like pipefail is recommended.

Sources:

• Official systemd documentation: systemctl, journalctl, systemd-analyze.
• GNU Bash Reference Manual.
• Util-linux project: documentation for lsblk and findmnt.
• iproute2: documentation for ss.
• procps-ng and sysstat: diagnostic and performance tools.