Debugging Linux Performance with /proc, PSI, and Cgroups
When a Linux application becomes slow, the first instinct is often to inspect application code, check CPU usage, or restart the affected process.
But Linux performance problems are not always caused by application-level code.
A container may be experiencing CPU throttling even when the host has available CPU capacity. A process may have low CPU utilization but spend time waiting for I/O. A system may show apparently sufficient memory while applications are experiencing memory pressure.
The important part of performance debugging is understanding which resource is constrained, which process or container is affected, and whether the available measurements actually support the suspected cause.
Here is a practical workflow for investigating these problems.
1. Start by identifying the scope
Before collecting metrics, identify where the problem is occurring.
There are three useful levels to investigate:
- Host level: Overall CPU, memory, pressure, disk, and network activity.
- Container level: Resource limits, CPU throttling, and memory constraints.
- Process level: CPU consumption, memory usage, I/O activity, and scheduling behavior.
These levels are related, but they are not interchangeable.
For example, high memory usage across the host does not automatically mean that one particular process is responsible.
Likewise, low host CPU utilization does not prove that a container has enough CPU capacity. Its cgroup may impose a much lower limit.
Always establish the scope before drawing conclusions.
2. Understand CPU measurements over time
Linux exposes CPU statistics through /proc/stat.
These counters are cumulative, so one isolated reading is not enough to understand current CPU activity.
A useful approach is to collect measurements over a consistent time interval and compare the changes.
Look at:
- User CPU time
- System CPU time
- Idle CPU time
- I/O wait
- Steal time
- Context switches
For a process, CPU time can be inspected through /proc/PID/stat and summarized using tools such as ps.
For example:
ps -p 1234 -o pid,comm,%cpu,%mem,nlwp,state The important distinction is between a cumulative counter and a rate calculated over an observation window.
A process that accumulated significant CPU time over several hours may currently be idle.
3. Investigate memory pressure, not just memory usage
Looking at free memory alone can be misleading.
Linux uses available memory for caching, and the amount of memory currently in use does not independently establish that the system is under pressure.
Start with:
grep -E 'MemTotal|MemAvailable|SwapTotal|SwapFree' /proc/meminfo Then inspect memory-related counters:
grep -E 'pgfault|pgmajfault|oom_kill' /proc/vmstat For additional context, Linux Pressure Stall Information provides visibility into resource stalls.
Check:
cat /proc/pressure/memory PSI helps identify whether tasks are experiencing delays because memory resources are constrained.
The useful approach is to correlate memory availability, page faults, memory pressure, and OOM-related events instead of relying on one number.
4. Inspect cgroups when containers are involved
Container resource limits introduce another layer of investigation.
A host can have available memory and CPU capacity while a container is constrained by its own limits.
For cgroup v2, inspect the relevant workload's cgroup directory.
Common files include:
cat /sys/fs/cgroup/cpu.max cat /sys/fs/cgroup/cpu.stat cat /sys/fs/cgroup/memory.current cat /sys/fs/cgroup/memory.max cat /sys/fs/cgroup/memory.events The workload's actual cgroup directory may be nested, so do not assume the root directory contains the correct values. Its membership can be inspected through /proc/PID/cgroup.
Pay particular attention to:
- CPU quota and available capacity
nr_throttledthrottled_usec- Memory usage relative to the configured limit
- OOM and other memory events
Cgroup v1 uses different file layouts and counter names, so collectors need to account for the version in use.
5. Investigate the target process
Once the affected process is identified, inspect its individual resource usage.
For example:
PID=1234 grep -E '^(Name|State|VmRSS|VmSwap|voluntary_ctxt_switches|nonvoluntary_ctxt_switches):' "/proc/$PID/status" Process-level information can help answer questions such as:
- Is the process using more CPU than expected?
- Is resident memory increasing?
- Is swap usage changing?
- Are major page faults increasing?
- Are context switches increasing?
- Is the process generating substantial I/O?
Where permissions allow, /proc/PID/io and /proc/PID/fd can provide additional information about I/O activity and open file descriptors.
Again, compare values across time where appropriate. Cumulative context-switch and I/O counters are more useful as changes or rates than as isolated totals.
6. Check disk and network activity
Resource bottlenecks are not always CPU or memory problems.
For block-device statistics, Linux exposes:
cat /proc/diskstats For network connections, inspect socket states and queue information:
ss -tan Pay attention to:
- Disk latency and accumulated I/O activity
- Socket states such as
CLOSE_WAIT,TIME_WAIT, andSYN_RECV - Non-zero
Recv-QandSend-Qvalues - TCP retransmission trends
A non-zero queue or an individual retransmission counter is not enough to prove a bottleneck. Correlate the observations with application latency and changes over time.
7. Use active profiling only when necessary
Passive telemetry should generally be the first step.
When the evidence indicates that more detailed investigation is needed, tools such as strace and perf can provide additional information.
They are useful for investigating system calls, scheduling, and process-level activity.
However, active profiling can introduce overhead and may require additional permissions or capabilities.
Keep profiling targeted to the affected process, use bounded durations, and understand the impact before enabling it.
8. Separate measured facts from hypotheses
This is one of the most important parts of a useful diagnostic workflow.
Consider these two statements:
Measured observation:
The process reported 30% CPU utilization during the observation interval.
Diagnostic hypothesis:
The process may be experiencing scheduling contention.
The first statement is a measurement. The second requires supporting evidence.
A good diagnostic report should distinguish between:
- Observed measurements
- Relevant counter changes
- Possible causes
- Confidence in a finding
- Recommended next diagnostic actions
This prevents an initial suspicion from being reported as a confirmed root cause.
9. A small example from a controlled workload
During a local test, I ran a controlled Python process and collected a process-level snapshot.
The sanitized output looked like this:
{ "process": "python3", "threads": 1, "cpu_percent": 30.0, "memory_rss_mib": 9.65, "swap_mib": 0, "context_switches": { "voluntary": 71, "nonvoluntary": 165 } } These values demonstrate the structure of a process-level diagnostic snapshot. They are from one controlled run, not a benchmark or a performance guarantee.
The next step is not to conclude that the process has a problem. It is to compare the measurements with the expected workload and investigate any relevant changes or pressure signals.
10. A practical debugging checklist
When investigating a Linux performance issue, I would follow this order:
- Identify whether the issue is host-wide, container-specific, or process-specific.
- Collect system CPU and memory measurements over consistent intervals.
- Inspect CPU, memory, and I/O pressure.
- Check cgroup limits and throttling when containers are involved.
- Inspect target-process CPU, memory, context switches, and I/O.
- Correlate disk and network observations with application behavior.
- Use active profiling only when passive evidence indicates that it is necessary.
- Keep measured facts separate from diagnostic hypotheses.
The main lesson is simple: measure first, correlate second, and diagnose third.
Linux exposes a substantial amount of useful performance information through /proc, /sys, cgroups, and PSI. The challenge is collecting the correct signals and interpreting them in the right context.
A disciplined evidence-first workflow makes Linux performance investigations more reproducible and helps avoid unnecessary changes based on assumptions.



