Introduction
Linux servers process massive amounts of network traffic every day. Whether it is web requests, remote SSH logins, or database communication, virtually all network traffic is managed by the Linux kernel.
While many engineers are familiar with configuring IP addresses and routing, relatively few have the opportunity to learn how the Linux kernel actually processes network traffic internally.
In this article, we will explore how the Linux kernel manages network communication by explaining the roles of the TCP/IP stack, sockets, NIC drivers, and related components.
How Network Communication Works in Linux
The Linux kernel is responsible not only for CPU and memory management but also for managing network communication. For example, when a web browser accesses a web server, the application does not communicate directly with the Network Interface Card (NIC). Instead, it sends a networking request to the Linux kernel, which performs the actual network processing.
Linux separates applications running in user space from the kernel running in kernel space. Within the kernel resides the TCP/IP stack, which implements networking protocols such as TCP and IP to enable network communication.
The major layers of the Linux networking stack are:
- Socket Layer
- TCP/UDP Layer
- IP Layer
- Device Driver Layer
When an application sends data, the data passes through these layers before being transmitted onto the network. Receiving data follows the reverse path.
The overall communication flow is shown below. Although the driver layer differs in virtualized environments, the overall architecture remains the same.
The Linux kernel manages each of these layers and acts as the bridge between applications and network devices. Let’s examine the role of each layer.
Socket Layer
In Linux, the socket provides the interface between applications and the network. It serves as the programming interface through which applications perform network communication. Most networking applications rely on the socket API.
Some of the most commonly used system calls are:
| System Call | Primary Purpose | Server/Client | Description |
|---|---|---|---|
socket() | Create a socket | Both | Creates a socket used as the endpoint of communication. |
bind() | Assign an address | Mainly Server | Associates an IP address and port number with a socket. |
listen() | Wait for connections | Server | Places a socket into the listening state. |
accept() | Accept a connection | Server | Accepts an incoming client connection. |
connect() | Initiate a connection | Client | Sends a connection request to a server. |
For example, a web server enters the listening state using listen() and accepts client connections using accept().
The Linux kernel maintains communication state for each socket, including the port number, peer address, and send/receive buffers. For TCP sockets, it tracks connection states such as LISTEN and ESTABLISHED. For UDP sockets, it manages the information required for packet transmission.
You can view the current socket status using ss command:
The options are:
-t: TCP sockets-u: UDP sockets-l: Listening sockets only-n: Do not resolve host names-p: Display associated processes
Example output:
[root@ ~]# ss -tulnp
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 127.0.0.1:323 0.0.0.0:* users:(("chronyd",pid=610,fd=5))
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=958261,fd=3))
tcp LISTEN 0 511 *:443 *:* users:(("httpd",pid=3491698,fd=8),("httpd",pid=3491483,fd=8),("httpd",pid=3491480,fd=8),("httpd",pid=3491479,fd=8),("httpd",pid=845637,fd=8))
This command displays the sockets currently managed by the Linux kernel and shows which processes are listening on which ports.
TCP/IP Layer
TCP is a protocol designed to provide reliable communication. The Linux kernel implements features such as retransmission, flow control, congestion control, and in-order packet delivery.
A TCP connection progresses through several states. Common states include:
- LISTEN – The server is waiting for incoming connections.
- SYN_SENT – The client has sent a connection request.
- ESTABLISHED – The connection has been established and data transfer is possible.
- FIN_WAIT – The connection is being terminated.
- TIME_WAIT – The connection has been closed, and the socket waits for a period before being released.
The state transitions during TCP connection establishment and termination are illustrated below.
You can display TCP connection states using “ss -ant” command.
Options:
-t: TCP sockets-a: Display all sockets, including listening sockets-n: Do not resolve host names
IP, on the other hand, is responsible for delivering packets to their destination through routing and IP addressing.
Example:
[root@ ~]# ss -ant
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
CLOSE-WAIT 1 0 127.0.0.1:36816 127.0.0.1:8080
ESTAB 0 0 127.0.0.1:5432 127.0.0.1:60876NIC Drivers and Virtual NIC Drivers
Packets generated by the TCP/IP layer are not transmitted directly onto the network. Instead, they are passed to a physical NIC driver or a virtual NIC driver before reaching the network device.
A driver acts as the interface between the Linux kernel and the network hardware. Rather than interacting directly with hardware-specific implementations, the kernel communicates with devices through standardized driver interfaces.
Typical NIC drivers used on physical servers include:
- ixgbe – Intel 10 Gigabit Ethernet adapters
- i40e – Intel Ethernet 700 Series adapters
- bnxt_en – Broadcom NetXtreme-E adapters
- e1000e – Intel Gigabit Ethernet adapters
Most Linux distributions include these drivers by default, and the appropriate driver is selected according to the installed NIC hardware.
In virtualized environments, virtual NIC drivers are used instead:
- virtio_net – VirtIO virtual NIC driver used by KVM, Nutanix AHV, and OpenStack
- vmxnet3 – VMware’s paravirtualized NIC driver for VMware ESXi
- e1000 – Intel E1000 emulation driver used by VirtualBox and VMware
- hv_netvsc – Virtual NIC driver for Microsoft Hyper-V and Azure
You can identify the driver in use with the ethtool command:
Example:
[root@ ~]# ethtool -i enp0s3
driver: e1000
version: 5.14.0-570.12.1.el9_6.x86_64
firmware-version:
expansion-rom-version:
bus-info: 0000:00:03.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: noIn this VirtualBox environment, the guest operating system uses the e1000 virtual NIC driver.
The primary responsibilities of a NIC driver include:
- Managing transmit (TX) queues
- Managing receive (RX) queues
- Transferring data to and from the NIC
- Handling interrupts
- Configuring DMA
- Utilizing hardware offload features
Virtual Switches
In virtualized environments, a virtual switch sits behind the virtual NIC.
Like a physical Layer 2 switch, a virtual switch forwards Ethernet frames between virtual machines and external networks.
Common virtual switches include:
- Linux Bridge – The standard virtual switch included with the Linux kernel
- Open vSwitch (OVS) – A feature-rich virtual switch used in Nutanix AHV, OpenStack, and Kubernetes
- VMware Standard vSwitch – The default virtual switch within a VMware ESXi host
- VMware Distributed Switch (VDS) – A centralized virtual switch managed by VMware vCenter across multiple ESXi hosts
A virtual switch typically performs the following functions:
| Function | Description |
|---|---|
| MAC Address Learning | Learns which MAC address belongs to which port, reducing unnecessary broadcasts. |
| Frame Forwarding | Forwards Ethernet frames based on the learned MAC address table or floods unknown destinations. |
| VLAN Management | Logically separates networks using VLAN tags. Different VLANs cannot communicate directly. |
| Port Management | Manages virtual ports connected to VM virtual NICs, including VLAN and security settings. |
| QoS | Controls bandwidth and traffic priority among virtual machines. |
For example, when two virtual machines residing on the same host communicate with each other, their traffic may remain entirely within the virtual switch without ever reaching the physical NIC.
Therefore, when troubleshooting network problems in virtual environments, engineers should verify not only the physical network but also the status of the virtual switch.
Physical NICs and Virtual NICs
A Network Interface Card (NIC) connects a server to a network. On physical servers, NICs are installed in PCIe slots and appear in Linux as interfaces such as:
[root@localhost ~]# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:92:22:b8 brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 08:00:27:54:0c:a9 brd ff:ff:ff:ff:ff:ff
[root@localhost ~]#In virtualized environments, physical NICs are replaced by virtual NICs provided by the hypervisor. From the guest operating system’s perspective, they function just like physical network adapters.
For example:
- virtio_net is widely used with KVM.
- vmxnet3 is commonly used with VMware.
Modern NICs and NIC drivers perform much more than simple packet transmission. Many processing tasks are offloaded to hardware.
| Feature | Description | Benefit |
|---|---|---|
| Checksum Offload | NIC calculates TCP/UDP/IP checksums. | Reduces CPU utilization. |
| TSO (TCP Segmentation Offload) | NIC segments large TCP packets into MTU-sized frames. | Reduces CPU overhead. |
| LRO / RSC | NIC combines multiple received packets before delivering them to the OS. | Reduces receive processing overhead. |
| RSS (Receive Side Scaling) | Distributes incoming packets across multiple receive queues and CPUs. | Improves receive performance on multi-core systems. |
| DMA (Direct Memory Access) | NIC transfers data directly to memory without CPU intervention. | Reduces CPU utilization and increases throughput. |
These hardware acceleration features significantly reduce CPU overhead while improving networking performance.
When troubleshooting Linux networking issues, engineers should examine not only the TCP/IP stack but also the NIC drivers, virtual switches, and hardware NIC capabilities.
Summary
Linux network communication begins when an application issues networking requests through sockets. The Linux kernel processes these requests using the TCP/IP stack before transmitting packets through NIC drivers and physical NICs. In virtualized environments, virtual NICs and virtual switches become additional components in the communication path.
When investigating network issues, it is important to examine more than just IP addressing and routing. Socket states, TCP connections, NIC drivers, virtual switches, and NIC hardware offload features can all affect network behavior. Understanding how the Linux kernel manages network communication enables engineers to isolate problems more efficiently and identify root causes more quickly.
The topics covered in this article provide the foundation of Linux networking. Building on this knowledge by studying technologies such as NAPI, SoftIRQ, RSS, socket buffers, and TCP tuning will lead to a much deeper understanding of Linux network internals.
English 
Japanese
コメント