Linux User Mode vs. Kernel Mode: System Calls Explained

Introduction

On Linux, applications and the Linux kernel run in different CPU execution modes. Applications normally run in user mode, while the Linux kernel runs in kernel mode.

Applications use system calls when they need functions managed by the kernel, such as file I/O or network communication. When a system call is executed, the CPU switches from user mode to kernel mode, and the kernel performs the required operation.

This article explains the differences between user mode and kernel mode and how system calls work, using practical examples with read() and strace.

What Are CPU Modes?

A CPU has mechanisms that restrict the functions and privileges available to a program based on its execution context. These different levels of CPU execution privilege are called CPU modes.

On Linux, CPU modes can broadly be divided into two categories:

• User mode
• Kernel mode

Ordinary applications run in user mode, while the Linux kernel runs in kernel mode.

What Is User Mode?

User mode is the CPU mode used to run ordinary applications. For example, the regular program code of Apache, PostgreSQL, Java applications, and commands such as ls normally runs in user mode.

Operations that could affect the entire system are restricted in user mode. An application cannot freely execute privileged CPU instructions or directly access kernel memory. If applications could perform such operations without restriction, a bug in one application could affect the Linux kernel or other processes.

Ordinary applications therefore operate in the restricted user mode.

What Is Kernel Mode?

Kernel mode is the CPU mode in which the Linux kernel runs. It provides greater access to CPU functions than user mode.

The Linux kernel uses these privileges to perform system-wide operations such as:

• Managing processes and threads
• Managing memory
• Processing file systems
• Controlling devices
• Processing network traffic

For example, when an application wants to read data from a file, the user-mode application does not operate the storage device directly. It requests the operation from the Linux kernel, which performs the necessary work in kernel mode.

Why Are CPU Modes Separated?

The primary reason for separating CPU modes is to protect the system. Running applications in user mode restricts the operations they can perform. Even if an application contains a bug, it cannot freely overwrite kernel memory or directly control hardware.

Linux maintains system security through the following division of responsibilities:

User mode: Runs applications with restricted privileges
Kernel mode: Runs the Linux kernel with elevated privileges

Applications still need to use Linux kernel functions such as file access and network communication. System calls provide a safe way for user-mode applications to request these operations from the kernel.

Differences Between User Mode and Kernel Mode

The main difference between user mode and kernel mode is the privilege level of operations that can be executed on the CPU. Ordinary applications run in user mode, where operations that could affect the entire system are restricted. The Linux kernel runs in kernel mode and has the elevated privileges required to manage the CPU, memory, devices, and other system resources.

The main differences are summarized below.

Different Instructions Can Be Executed

In addition to instructions that any program can execute, CPUs provide privileged instructions that can be executed only at an elevated privilege level.

Applications running in user mode cannot freely execute these privileged instructions.

Kernel mode, on the other hand, allows the operating system to execute the privileged instructions needed to manage the system.

This restriction prevents ordinary applications from arbitrarily changing the behavior of the CPU or the system as a whole.

Different Memory Regions Can Be Accessed

In the virtual address space visible to a Linux process, user space and kernel space are distinct.

An application running in user mode can access the user-space memory assigned to it, but it cannot freely access kernel space.

The Linux kernel runs in kernel mode and can access the memory required for kernel operations.

This mechanism prevents application bugs from directly modifying critical kernel data.

Applications Cannot Directly Control Hardware

User-mode applications generally do not directly control hardware such as storage devices and network interface cards.

For example, when an application reads a file, the application itself does not directly control the storage device. Instead, it asks the Linux kernel to perform the read operation.

The Linux kernel performs the required operation through the file system, device drivers, and other components, then returns the result to the application.

What Is a System Call?

A system call is the mechanism that allows an application running in user mode to use functions provided by the Linux kernel.

As explained above, ordinary applications run in user mode and cannot freely operate kernel-managed functions such as file systems, networking, and process management.

Applications therefore use system calls to request operations from the Linux kernel.

For example, to read data from a file, an application does not directly operate the storage device. It requests a read from the Linux kernel through a system call.

Common System Calls

Linux provides many different system calls. Common examples include:

When an application executes the read system call, for example, the Linux kernel reads data from a file or another source and returns the result to the application. What appears to the application as a simple file read relies on Linux kernel functionality behind the scenes.

Library Functions vs. System Calls

When an application uses a system call, the program does not necessarily invoke the system call directly. It generally uses a library function provided by a library such as glibc, which invokes a system call internally when required.

For example, when a program opens a file, a system call such as openat may be used behind the scenes.

However, calling a library function does not always cause a system call. Some functions complete entirely within the library.

The important point is that a system call is the interface through which a user program uses Linux kernel functions.

What Happens When a System Call Is Executed?

When a system call is executed, the CPU switches from user mode to kernel mode, and the Linux kernel performs the operation requested by the application.

We will use an application reading data from a file with the read system call as an example.

How the read() System Call Works

On Linux, an application uses the read() system call to read data from a file.

From the application’s perspective, it simply calls read(). Internally, however, Linux performs several operations, including switching from user mode to kernel mode, checking the page cache, and issuing storage I/O when necessary.

The following sections describe the general flow of read() for a file accessed through ordinary buffered I/O.

1. Application

First, the application attempts to read data from a file. It normally opens the file in advance using a function such as open(), then passes the resulting file descriptor to read().

The main arguments to read() are the file descriptor, a user-space buffer in which to store the data, and the number of bytes to read.

2. Call read()

The application runs in user mode and calls read() to read the file. A user-mode application cannot directly control hardware such as a storage device.

It therefore asks the Linux kernel to read the file through the read() system call.

3. Switch to Kernel Mode

When the read() system call is executed, the CPU switches from user mode to kernel mode. This allows the Linux kernel to process the read() request.

Importantly, the application itself does not move into kernel space. While the same task continues to execute, the CPU changes its execution mode from user mode to kernel mode and runs kernel code.

4. Check the File Descriptor

The kernel uses the file descriptor passed to read() to locate information about the target file. A file descriptor is simply an integer, but internally the kernel uses it to reach data structures such as the struct file associated with that file.

Using this information, the kernel identifies the target file and current file position, then continues through the VFS and the relevant file-system implementation.
Note: struct file is an internal kernel data structure used by Linux to manage a file opened by a process.

5. Check the Page Cache

Ordinary buffered I/O uses the page cache to read file data. The page cache stores file data from storage in memory.

The kernel checks whether the file data requested by the application is present in the page cache.

The subsequent processing follows one of two paths depending on whether the required data is available in the page cache.

6. Cache Hit

If the required data is already present in the page cache, the result is a cache hit. The kernel does not need to issue a read I/O request to a disk or SSD.

It can use the page-cache data already in memory. Because memory access is generally faster than storage access, the page cache plays an important role in improving Linux file-I/O performance.

7. Cache Miss

If the required data is not present in the page cache, the result is a cache miss. The kernel issues a read I/O request to storage and retrieves the required data from the disk or SSD.

If the task must wait for storage I/O to complete, the task that called read() may enter a waiting state and release the CPU. Another runnable task can use the CPU during that time.

The file data read from storage is placed in the page cache. If the same data is needed again, it may then be read from the page cache.

8. Copy Data from the Page Cache to User Space

Once the required data is available in the page cache, the kernel copies it to the user-space buffer specified in read().

For a cache hit, the kernel copies the data from the existing page-cache entry.

For a cache miss, the kernel first reads the data from storage into the page cache and then copies it to user space.

Thus, for both cache hits and cache misses, data from the page cache is ultimately delivered to user space.

9. Return the read() Result

After copying the data to the user-space buffer, the kernel returns the result of read() to the application.

When data is read successfully, the return value of read() is the number of bytes actually read.

The return value also indicates other outcomes: zero means the end of the file was reached, while -1 indicates an error.

10. Return to User Mode

When the kernel finishes processing read(), execution returns from the system call. The CPU changes from kernel mode back to user mode, and control returns to the application.

11. The Application Continues Processing

After returning to user mode, the application continues with the operation following read(). At this point, the file data is stored in the application’s user-space buffer. The application can analyze it, display it, or use it for another operation.

Summary of the read() System Call Flow

Although read() appears simple from the application’s perspective, Linux internally switches from user mode to kernel mode, checks file information, checks the page cache, performs storage I/O if required, copies data to user space, and finally returns to user mode.

An especially important point is that storage I/O does not occur every time read() is called. If the required data is present in the page cache, Linux can use the data in memory without accessing storage.

Inspecting System Calls on Linux

The strace command lets you inspect the system calls executed by a program on Linux. Even familiar Linux commands invoke many Linux kernel functions through system calls internally.

Information Available from strace

As an example, use strace to examine the system calls executed by the ls command. The following output is an excerpt.

[root@almalinux strace]# ls
test1.txt  test2.txt
[root@almalinux strace]#
[root@almalinux ~]# strace ls
Excerpt from the command output
------------------------------------------------------------------------
1. execve("/usr/bin/ls", ["ls"], 0x7ffe02b7bcc0 /* 24 vars */) = 0
2. openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
   read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\227\2\0\0\0\0\0"..., 832) = 832
   mmap(NULL, 2129872, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fa90e600000
3. openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 5
4. getdents64(5, 0x5556ed323970 /* 4 entries */, 32768) = 112
5. write(1, "test1.txt  test2.txt\n", 21test1.txt  test2.txt
)  = 21
6. exit_group(0)                           = ?
   +++ exited with 0 +++

1. Start the ls Program

First, execve() executes /usr/bin/ls.

execve("/usr/bin/ls", ["ls"], ...) = 0

Execution of the ls process begins here.

2. Load the Required Shared Libraries

To run ls, shared libraries such as libc are opened with openat() and mapped into the process’s virtual address space with mmap().

For libc.so.6, for example, the sequence openat() → read() → mmap() is visible.

This section represents the preparation performed when the program starts.

3. Open the Current Directory

This is an important part of the primary operation performed by ls.

openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 5

Because . represents the current directory, ls first opens the target directory.

4. Retrieve the Directory Entries

Next, getdents64() is executed.

getdents64(5, 0x5556ed323970 /* 4 entries */, 32768) = 112

getdents64() is a system call that retrieves entries from a directory.

5. Write the Result to the Screen

The retrieved filenames are written to standard output with write().

write(1, "ddagent-install.log jmxterm-1.0"..., 44) = 44

Here, 1 is the file descriptor for standard output (stdout). Because standard output is connected to a terminal in this example, the result of ls appears on the screen.

6. Terminate the Process

Finally, exit_group() terminates ls.

Tracing Only Specific System Calls with strace

strace can also restrict its output to selected system calls. To inspect only the write system call, for example, run:

[root@almalinux strace]# strace -e trace=write ls
write(3, "", 0)                         = 0
write(4, "{\n    \"runtime_id\": \"dc8caa21-df"..., 114) = 114
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1144668, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
write(1, "test1.txt  test2.txt\n", 21test1.txt  test2.txt
)  = 21
+++ exited with 0 +++
[root@almalinux strace]#

Displaying every system call can produce a very large amount of output. If you already know which operation you want to investigate, filtering the target makes the results easier to inspect.

Tracing a Running Process

You can also attach strace to an existing process. If the PID is 357466, for example, use strace -p 357466.

[root@almalinux strace]# strace -p 357466
strace: Process 357466 attached
pselect6(9, [6 7 8], NULL, NULL, {tv_sec=56, tv_nsec=945681111}, NULL) = ? ERESTARTNOHAND (To be restarted if no handler)
--- SIGUSR1 {si_signo=SIGUSR1, si_code=SI_USER, si_pid=357472, si_uid=26} ---
newfstatat(AT_FDCWD, "logrotate", 0x7fffc9e87000, 0) = -1 ENOENT (No such file or directory)
getpid()                                = 357466
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7ff37540ac10) = 1790873
rt_sigreturn({mask=[]})                 = -1 EINTR (Interrupted system call)
rt_sigprocmask(SIG_SETMASK, ~[ILL TRAP ABRT BUS FPE SEGV CONT SYS RTMIN RT_1], NULL, 8) = 0
openat(AT_FDCWD, "postmaster.pid", O_RDWR) = 9
read(9, "357466\n/var/lib/pgsql/data\n17669"..., 8191) = 102
close(9)                                = 0
getpid()                                = 357466

This lets you inspect the system calls executed by the process in real time. However, running strace adds overhead and may affect performance, so use it carefully in production environments. The impact can be especially significant for processes that make system calls at a high frequency.

The Difference Between user and system CPU Usage

When you inspect CPU usage on Linux, you see fields such as user and system. These values indicate the mode in which the CPU was executing work.

Understanding CPU modes helps you determine not only that CPU usage is high, but also whether the application or the Linux kernel is consuming most of the CPU time.

What Is user CPU?

user CPU is the time the CPU spent executing in user mode. In other words, it primarily represents CPU time used by the application’s own processing.

user CPU usage increases when user-space work consumes significant CPU time, including:

• Calculations performed by a Java application
• Data compression or encryption
• Internal database calculations

In top, it is primarily shown as us.

What Is system CPU?

system CPU is the time the CPU spent executing in kernel mode. When an application makes a system call, the CPU switches to kernel mode and the Linux kernel performs the operation.

system CPU usage increases when the Linux kernel performs operations such as:

• File I/O
• Network processing
• Memory management

In top, it is primarily shown as sy.

Comparing user and system

The difference between user and system can be summarized as follows:

Even when total CPU usage is 80%, what is happening inside the CPU differs greatly between us = 70%, sy = 10% and us = 20%, sy = 60%.

In the first case, the application itself may be responsible for most of the CPU usage.

In the second case, kernel-side operations such as system calls and network processing may be consuming most of the CPU time.

Checking with top and vmstat

You can inspect usage by CPU mode with commands such as top and vmstat.

Example top Output

The following example shows the result of running top on AlmaLinux 9.

The output shows the following:

33.9 us: Approximately 33.9% of CPU time is being used for user-mode processing
65.8 sy: Approximately 65.8% of CPU time is being used for kernel-mode processing

Example vmstat Output

The following example shows the result of running vmstat on AlmaLinux 9.

us: CPU time used in user mode. Approximately 30% is used for user-mode processing.
sy: CPU time used in kernel mode. Approximately 70% is used for kernel-mode processing.

Understanding CPU modes and system calls allows you to interpret the CPU usage reported by top and vmstat not merely as numbers, but in terms of the work on which the CPU is spending its time.

Conclusion

This article explained CPU modes and system calls in Linux.

To operate the system safely, Linux broadly separates CPU execution into user mode and kernel mode. Ordinary applications run in user mode, while the Linux kernel performs system-wide operations in kernel mode.

Applications use system calls when they need Linux kernel functions such as file I/O or network communication. A system call switches the CPU to kernel mode. After the kernel completes the required operation, the CPU returns to user mode.

For a file read performed with read(), the kernel checks the file information and page cache and performs storage I/O if necessary. It then copies the data to user space and returns control to the application.

You can use strace to inspect the system calls an application actually executes. This is useful for understanding Linux internals and investigating application behavior.

Understanding CPU modes also makes it easier to interpret the us and sy values displayed by top and vmstat:

us: CPU time used in user mode
sy: CPU time used in kernel mode

These values reveal the type of processing on which the CPU is spending its time.

System calls and CPU modes are fundamental mechanisms for understanding how applications and the kernel work together inside Linux. This knowledge is useful not only for understanding Linux behavior but also for analyzing CPU load and performance.