Linux Memory Pressure Explained: kswapd, Direct Reclaim, Compaction, and the OOM Killer

Introduction

When operating Linux servers, you may encounter situations such as a sudden increase in swap usage, degraded performance even though some memory is still available, or a process being forcibly terminated by the OOM Killer.

When these problems occur, it is easy to focus only on the result—insufficient memory. In reality, however, the Linux kernel performs a series of actions in stages to resolve memory shortages.

For example, when free memory decreases, kswapd first starts reclaiming unnecessary pages in the background. If that does not secure enough free memory, Direct Reclaim is performed by the process requesting memory. If the situation deteriorates further and memory still cannot be allocated, the OOM Killer terminates a process as a last resort to prevent the entire system from becoming unresponsive.

Understanding these mechanisms helps you interpret logs and performance data correctly during memory pressure and makes root-cause analysis and troubleshooting more efficient.

This article explains, in chronological order, how Linux detects a memory shortage, how it reclaims memory, and what it does when those efforts are insufficient. We will systematically examine the roles of kswapd, Direct Reclaim, Memory Compaction, and the OOM Killer, as well as the relationships among them.

Overall Flow During a Memory Shortage

Linux does not immediately invoke the OOM Killer merely because free memory has become low.

It responds to memory pressure in stages: first reclaiming unnecessary memory, and then moving to the next measure only if enough free memory still cannot be secured.

The overall flow is as follows.

First, when free memory falls below a certain level, the Linux kernel determines that the system is approaching a memory shortage and wakes a kernel thread called kswapd. kswapd reclaims unnecessary pages in the background and secures free pages in preparation for new memory requests.

If kswapd can reclaim enough pages, applications can continue running without being aware that page reclaim is taking place.

If memory consumption increases rapidly or kswapd cannot reclaim pages quickly enough, Direct Reclaim is performed by the process that requested memory. Because that process must wait until reclamation finishes, Direct Reclaim can cause slower application responses and processing delays.

When a high-order allocation is required but fragmentation prevents the allocation of a contiguous free region, Memory Compaction is performed. It attempts to eliminate fragmentation by relocating pages in physical memory and creating a contiguous free area.

If the required memory still cannot be secured, Linux determines whether the OOM Killer may be invoked. For a normal user-process allocation, the OOM Killer can forcibly terminate a process and release its memory. In contexts where the OOM Killer cannot be invoked, such as interrupt handling and certain kernel operations, the allocation itself may fail.

In this way, Linux first tries to secure free pages through kswapd and Direct Reclaim. When high-order pages are required, it may also use Memory Compaction to address fragmentation. If allocation is still impossible, Linux either invokes the OOM Killer or fails the allocation, depending on the context, so that the system as a whole can continue operating.

When Does Linux Decide That Memory Is Low?

The phrase “out of memory” may suggest that physical memory has reached zero. Linux, however, does not wait until all free memory is exhausted before taking action.

The kernel is designed to maintain a predetermined number of free pages. When available memory falls below that threshold, it decides that the system is approaching a memory shortage and begins reclaiming pages.

The thresholds used for this decision are called watermarks.

What Is a Watermark?

A watermark is a guideline for the number of free pages that should be maintained in each memory zone.

Linux divides physical memory into multiple zones according to purpose. A typical x86_64 system, for example, has DMA, DMA32, and Normal zones.

Each memory zone has the following three watermarks.

The following diagram illustrates the watermarks.

During normal operation, the number of free pages is maintained around the High watermark.

As applications consume large amounts of memory, the number of free pages gradually falls. Once it drops below the Low watermark, Linux decides that free memory may soon become insufficient.

In other words, Linux begins page reclaim while some headroom still remains, rather than waiting until free memory reaches zero.

Conditions That Trigger kswapd and Direct Reclaim

When the free-page count in a memory zone falls below the Low watermark, Linux wakes kswapd. This background kernel thread is responsible for increasing the number of free pages.

It reclaims page cache and unused memory pages and, when necessary, moves anonymous pages to swap so that free pages are available for new memory requests.

Because this work runs in the background, applications can usually continue without being aware of page reclaim.

When the number of free pages recovers to around the High watermark, kswapd stops reclaiming pages and returns to a waiting state. By securing free memory in advance, kswapd plays an important role in preventing applications from running into memory shortages.

However, if memory usage rises rapidly or reclamation cannot keep pace, kswapd may fail to secure enough free pages. Direct Reclaim is then performed.

Differences Between kswapd and Direct Reclaim

Normally, kswapd secures free pages in the background, so Direct Reclaim does not occur frequently.

When memory usage increases abruptly or reclamation cannot keep up, however, Direct Reclaim may occur and directly affect application performance.

Frequent Direct Reclaim can therefore be regarded as a sign that Linux is struggling to cope with memory pressure.

What Is kswapd?

kswapd is a Linux kernel thread that runs in the background to reclaim memory.

When the number of free pages falls below the Low Watermark, kswapd is woken up and starts reclaiming pages in the background. By reclaiming pages, kswapd increases the amount of free memory.

If memory becomes even more constrained, Direct Reclaim may occur, where the process requesting memory performs page reclaim itself. With Direct Reclaim, memory allocation may have to wait for page reclaim to complete, which can increase application latency.

This section explains what types of pages kswapd reclaims and how it frees memory.

Pages Reclaimed by kswapd

kswapd frees memory by searching for reclaimable pages.

Two common types of pages reclaimed by kswapd are File-backed Pages and Anonymous Pages.

File-backed Pages are backed by files and include pages used for the Page Cache. Anonymous Pages are not backed by files and include memory used for process Heaps and Stacks.

File-backed Pages

File-backed Pages are memory pages that hold data originating from files.

The reclaim process differs depending on whether a File-backed Page is a Clean Page or a Dirty Page.

Clean Pages

A Clean Page contains the same data as the corresponding data on storage.

Because the same data already exists on storage, the page can be released directly. If the data is needed again after the page has been reclaimed, it can be read from the file again.

Dirty Pages

A Dirty Page contains data that has been modified in memory but has not yet been written back to storage.

Because the modified data must be preserved, a Dirty Page cannot simply be released. Once Writeback to storage completes and the page becomes clean, it can be reclaimed.

Anonymous Pages

Anonymous Pages are memory pages that are not backed by files. Examples include memory used for process Heaps and Stacks.

Unlike File-backed Pages, the contents of Anonymous Pages cannot be retrieved again from a file.

Therefore, if their contents need to be preserved while reclaiming memory, the pages can be written to Swap space when Swap is available. This operation is called Swap Out.

When a swapped-out page is needed again, it is read from Swap space back into memory.

Reclaiming File-backed Pages and Anonymous Pages

Linux does not always reclaim either File-backed Pages or Anonymous Pages first.

Instead, it adjusts the balance between reclaiming File-backed Pages and Anonymous Pages based on factors such as page type, page state, and usage. One setting that influences this balance is vm.swappiness.

Clean File-backed Pages can be released directly because the same data already exists on storage. Dirty File-backed Pages can be reclaimed after Writeback.

Anonymous Pages cannot be retrieved from files, so if their contents need to be preserved, they must be moved to Swap before the memory can be reclaimed.

The reclaim methods can be summarized as follows:

kswapd reclaims pages while considering their type and state to maintain sufficient free memory.

Checking Page Reclaim Activity by kswapd

You can check page reclaim activity performed by kswapd using /proc/vmstat.

Use the following command to check pgscan_kswapd and pgsteal_kswapd:

grep -E “pgscan_kswapd|pgsteal_kswapd” /proc/vmstat

The counters have the following meanings:

These values are cumulative counters since system startup. Therefore, to check current page reclaim activity, collect the values at regular intervals and compare the differences.

If pgscan_kswapd increases rapidly over a short period, kswapd is actively scanning pages. If pgsteal_kswapd also increases, pages are actually being reclaimed from among the scanned pages.

By monitoring these /proc/vmstat counters over time, you can determine when and how actively kswapd is reclaiming memory.

Summary of Memory Reclaim by kswapd

What Is Direct Reclaim?

As explained in the previous section, Linux starts reclaiming pages through kswapd when the number of free pages falls below the Low watermark.

However, if memory consumption rises rapidly or page reclaim cannot keep pace, kswapd alone may be unable to secure enough free pages. Direct Reclaim is performed in this situation.

Direct Reclaim attempts to secure the required free pages by reclaiming memory in the execution context of the process that requested the allocation.

Conditions That Trigger Direct Reclaim

Normally, kswapd performs page reclaim in the background, allowing applications to continue without being aware of it.

In the following situations, however, kswapd may not be able to reclaim pages quickly enough.

• A large amount of memory is allocated in a short period
• Overall system memory usage is high
• kswapd cannot secure enough free pages

If a process requests more memory under these conditions, Linux makes the requesting process perform page reclaim itself.

Characteristics of Direct Reclaim

The defining characteristic of Direct Reclaim is that the process requesting memory performs page reclaim itself.

Normally, kswapd performs reclamation in the background, so applications can continue processing without being aware of it.

With Direct Reclaim, by contrast, the requesting process cannot continue its original work until page reclaim has completed. Reclamation is therefore performed synchronously in the same execution context as the application.

Page reclaim can involve discarding page cache, swapping out anonymous pages, and writing back dirty pages. These operations may generate disk I/O, so if reclamation takes time, application response times can deteriorate significantly.

Direct Reclaim can consequently cause performance degradation that is difficult to identify from CPU and memory utilization alone.

For example, an application’s response time may suddenly deteriorate even though CPU and memory utilization show no major change. In such a case, Direct Reclaim may be delaying memory allocation.

When investigating incidents in production systems, it is important to check not only memory utilization but also whether Direct Reclaim is occurring frequently.

Checking Direct Reclaim Activity

On Linux, Direct Reclaim activity can be checked in /proc/vmstat.

[root@almalinux ~]# grep -E "pgscan_direct|pgsteal_direct" /proc/vmstat
pgsteal_direct 64
pgscan_direct 64
pgscan_direct_throttle 0
[root@almalinux ~]#

The fields have the following meanings.

These values are cumulative counters, so a single reading is not sufficient for assessment.

For example, collect them once per minute and compare the differences. A large increase over a short period may indicate frequent Direct Reclaim.

An increase in pgscan_direct does not necessarily indicate a problem. A brief increase can be normal. What matters is whether the value continues rising during periods of degraded response time.

Summary of Memory Reclaim by Direct Reclaim

What Is Memory Compaction?

The previous section explained how Direct Reclaim reclaims pages. Even if enough free memory has been recovered, however, the required allocation may still fail because of memory fragmentation.

Memory fragmentation is a state in which free pages exist but cannot be allocated as one contiguous region. In this situation, Linux performs Memory Compaction, relocating pages to create a contiguous area of free memory.

Why Compaction Is Necessary

Linux manages physical memory in page-sized units. Even when enough pages are free in total, a large allocation may be impossible if those pages are scattered. This condition is called memory fragmentation.

In the following diagram, four pages are free, but they are not contiguous and therefore cannot satisfy a large memory allocation.

Linux therefore uses compaction to move movable pages together and consolidate free pages in one location.

This creates a contiguous range of free pages and allows the system to satisfy larger memory requests.

Compaction plays an important role when contiguous physical memory is required, such as for huge pages and high-order allocations.

Relationship to the Buddy Allocator

Memory Compaction is needed because the Buddy Allocator allocates contiguous physical pages. It may allocate not only one page, but groups of two, four, eight, or more contiguous pages.

As fragmentation progresses, a large contiguous area may no longer be available even though the total number of free pages is sufficient. High-order allocations may then fail.

High-order pages are used for purposes such as the following.

• Allocating huge pages
• DMA buffers used by some device drivers
• Large network and storage buffers

To satisfy such requests, Linux performs Memory Compaction when necessary and attempts to create contiguous free pages.

If it still cannot obtain a contiguous region, the kernel log may contain a “page allocation failure” message.

In production systems, this error is often caused by fragmentation rather than a shortage in the total amount of memory.

The Buddy Allocator is explained in detail below.
Linux Kernel Memory Management Explained: Page Frames, Memory Zones, Buddy Allocator, and SLUB

What Is the OOM Killer?

The previous section explained how Memory Compaction resolves memory fragmentation and creates contiguous free pages.

Even after page reclaim and Memory Compaction, however, the required memory may still be unavailable. If this condition persists, new memory cannot be allocated and the entire system may become unresponsive. Linux therefore invokes the OOM Killer as a last resort.

OOM stands for “Out Of Memory.” The OOM Killer forcibly terminates a process to release memory and prevent the entire system from stopping.

Conditions That Trigger the OOM Killer

Linux does not invoke the OOM Killer immediately when free memory becomes scarce. It first performs page reclaim through kswapd.

If enough free pages still cannot be secured, the process requesting memory performs page reclaim through Direct Reclaim.

If fragmentation prevents a contiguous free region from being allocated, Memory Compaction consolidates the free pages.

If the allocation still cannot succeed and the allocation context permits it, Linux may invoke the OOM Killer.

The OOM Killer is thus the final measure used only when page reclaim and memory relocation cannot resolve the problem.

How the OOM Killer Works

When the OOM Killer runs, Linux selects a process to terminate. One metric used in this decision is oom_score.

oom_score indicates how likely each process is to be selected by the OOM Killer. In general, memory consumption is an important factor in the score, together with oom_score_adj and other conditions.

Administrators can adjust this priority by configuring oom_score_adj.

For example, assigning a low value to an important system process and a high value to a process whose termination would have little impact changes their relative likelihood of being selected.

The current values can be checked in the following files.

/proc/<PID>/oom_score
/proc/<PID>/oom_score_adj

The OOM Killer uses these values as part of the process-selection decision.

As an example, let us check the values for Tomcat (PID 2546596) and PostgreSQL (PID 357466) on a Linux server that I operate.

[root@almalinux ~]# cat /proc/2546596/oom_score
793
[root@almalinux ~]# cat /proc/357466/oom_score
0
[root@almalinux ~]# cat /proc/2546596/oom_score_adj
0
[root@almalinux ~]# cat /proc/357466/oom_score_adj
-1000
[root@almalinux ~]#

Reading the Logs

When the OOM Killer runs, it writes messages to the kernel log. During incident investigation, check the kernel log first.

The following commands can be used. Example output is shown below.

[root@almalinux ~]# dmesg | grep -i oom
Out of memory: Killed process 12345 (java) total-vm:16777216kB, anon-rss:8388608kB, file-rss:10240kB, shmem-rss:0kB

[root@almalinux ~]# journalctl -k
Aug 03 15:42:18 almalinux kernel: java invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
Aug 03 15:42:18 almalinux kernel: CPU: 3 PID: 12345 Comm: java Not tainted 5.14.0-570.el9.x86_64
Aug 03 15:42:18 almalinux kernel: Out of memory: Killed process 12345 (java) total-vm:16777216kB, anon-rss:8388608kB, file-rss:20480kB, shmem-rss:0kB
Aug 03 15:42:18 almalinux kernel: oom_reaper: reaped process 12345 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

The log records the time at which the OOM Killer ran, the name and PID of the terminated process, and its memory usage.

You can use this log to determine the following.

• Which process was terminated
• When the OOM Killer ran
• Whether the incident was caused by a memory shortage

In production systems, service impact has often already occurred by the time the OOM Killer runs. It is therefore important to examine not only the log itself but also memory usage, swap usage, and page-reclaim activity before and after the event.

Conclusion

This article explained how the Linux kernel secures free pages when memory becomes scarce.

Linux does not immediately invoke the OOM Killer merely because free memory is low. It monitors memory using the watermarks configured for each zone and starts background page reclaim through kswapd when free pages become insufficient.

If kswapd cannot keep pace and the required free pages are unavailable when an allocation is requested, Direct Reclaim runs in the requesting process’s execution context. Because this reclaim is synchronous, frequent Direct Reclaim can cause slower application responses and processing delays.

When a high-order allocation is required but fragmentation prevents a contiguous region from being obtained, Memory Compaction relocates pages and attempts to create one. If this also fails, a page allocation failure may occur.

If the required memory still cannot be allocated after page reclaim and related measures, Linux may invoke the OOM Killer as a last resort. It selects a process using oom_score, oom_score_adj, and other factors, forcibly terminates that process, and releases its memory.

When investigating an incident caused by memory pressure, checking only memory utilization and swap usage is not enough. Examining watermark status together with counters such as pgscan_kswapd, pgsteal_kswapd, pgscan_direct, and pgsteal_direct reveals how much page reclaim was occurring inside Linux.

The key to understanding Linux memory shortages is to look beyond the result—low free memory—and understand the sequence involving kswapd, Direct Reclaim, Memory Compaction, and the OOM Killer. With this knowledge, you can more accurately determine what was happening inside Linux when performance deteriorated or an OOM event occurred.

If you’d like to learn more about Linux memory management, I’ve covered the topic in detail in the following book. Please check it out if you’re interested.

Learn How Linux Works Through Visual Explanations

This book explains how Linux memory management works through clear, visual illustrations, making the concepts easy to understand even for beginners.

View the Book on Kindle