Skip to content

Diagnosing High CPU and Memory Usage

"The server is slow" is not a diagnosis — it's a starting point. This article covers the standard toolchain for finding out what is actually consuming CPU, memory, or I/O, in order, rather than guessing.

How it works

Start with top (or htop), but read it correctly

$ top -b -n 1 | head -15
top - 14:22:01 up 12 days,  3:41,  2 users,  load average: 2.14, 1.98, 1.76
Tasks: 210 total,   2 running, 208 sleeping,   0 stopped,   0 zombie
%Cpu(s): 12.3 us,  4.1 sy,  0.0 ni, 81.9 id,  1.2 wa,  0.0 hi,  0.5 si,  0.0 st
MiB Mem :  16024.0 total,    528.0 free,   9216.0 used,   6280.0 buff/cache
MiB Swap:   8192.0 total,   3382.0 free,   4810.0 used.   6102.0 avail Mem

Load average is the number of processes runnable or waiting on uninterruptible I/O, averaged over 1/5/15 minutes — it is not a percentage. A load average of 2.14 on a 4-core machine means the system is roughly half as busy, on average, as it could be; the same 2.14 on a 1-core machine means it's over double what it can handle.

%Cpu(s) breakdown matters more than the single load number: us (user-space processes), sy (kernel/syscalls), wa (waiting on I/O — a high wa with low us/sy points at a storage bottleneck, not a CPU one, even though top is a "CPU tool"), si/hi (soft/hard interrupts).

available memory, not free, is the number that actually matters. Linux aggressively uses spare RAM for disk cache (buff/cache) because unused RAM is wasted RAM — that cache is reclaimed instantly if an application needs it. A system showing 528MB free but 6GB available is not low on memory; it's using RAM efficiently.

$ vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0 3485536 528292 1293700 5136604    1    2   175  1239    2    1  9  6 76  5  5
 0  0 3485536 525352 1293704 5136692    0    0     0  7672 5709 9402  3  3 89  6  0

(vmstat 1 5 prints every 1 second, 5 times.) The columns to watch:

  • r — processes actually running or runnable right now, competing for CPU. If this number is consistently higher than your core count, you have genuine CPU contention.
  • b — processes blocked, uninterruptibly waiting (almost always on disk I/O).
  • si/so — swap in/out. Any sustained non-zero so means the system is actively paging memory out to disk — a strong signal of real memory pressure, distinct from merely having "low free memory" (which, per above, is often fine).

Finding the specific offending process

ps aux --sort=-%cpu | head -10     # top CPU consumers
ps aux --sort=-%mem | head -10     # top memory consumers
pidstat 1 5                         # per-process CPU over time (needs sysstat package)

Narrowing down I/O wait specifically

iostat -x 1 5

The %util column per device is the key one — close to 100% means that device is the bottleneck, regardless of how "idle" the CPU looks in top.

When you need to go deeper: perf

sudo apt install linux-tools-common linux-tools-$(uname -r)   # Debian/Ubuntu
sudo perf top                                                   # live, symbol-level CPU profiling

perf top shows exactly which functions (kernel or userspace, if the binary has symbols) are consuming CPU cycles right now — the next level down from "which process" to "which function inside that process."

Ubuntu/Debian vs. RHEL-family differences

top, vmstat, ps, free are part of procps, identical everywhere. pidstat/iostat come from the sysstat package on both families, under the same names. perf's installation package name differs (linux-tools-<kernel-version> on Debian/Ubuntu vs. perf directly on RHEL-family), but usage is identical once installed.

Practical exercise

  1. Generate artificial CPU load safely with yes > /dev/null & (pipes an endless stream of y characters to nowhere, pinning one core) and observe it in top — find its PID.
  2. Confirm with ps aux --sort=-%cpu | head -5 that the same process appears at the top.
  3. Kill it (kill <PID>) and confirm load drops back down within a few seconds in vmstat 1 5.

How to verify you did it correctly

  • top's %Cpu(s) us figure should jump to roughly 100/<core count>% while the yes process runs (e.g., ~25% on a 4-core machine for one such process), and the process should be visible at the top of ps aux --sort=-%cpu.
  • After kill, vmstat's r column should return to its prior baseline within a couple of seconds.

Common mistakes

  • Treating free memory as the metric that matters. Chasing "why is free memory so low" on a Linux system that's actually healthy (high buff/cache, high available) is a very common wasted-effort pattern.
  • Diagnosing "high load" as "high CPU usage" without checking wa. A load average of 8 with 90% wa and 5% us is an I/O problem, not a CPU problem — adding CPU cores would not help.
  • Using a single top snapshot instead of watching it (or vmstat) over time. Transient spikes look identical to sustained problems in a single snapshot.

⚠ Dangerous commands

  • kill -9 (SIGKILL) on a process holding a database lock or mid-write can corrupt data — prefer a graceful kill (SIGTERM, the default) or the application's own shutdown mechanism first, and reserve -9 for processes genuinely unresponsive to SIGTERM.
  • Running stress-testing tools like stress or a CPU-pinning loop (as in the exercise above) directly on a production system, without first confirming you have a safe way to stop it, can cause a real outage — always have the kill command ready in another window before starting.

Review questions

  1. Why can a system show very little "free" memory and still be perfectly healthy?
  2. What does a high wa percentage in top actually indicate, and why wouldn't more CPU cores fix it?
  3. What's the difference between the r and b columns in vmstat output?
  4. Why is a single top snapshot insufficient to diagnose an intermittent performance problem?