CPU Cache Explained: L1, L2, L3 and Linux Performance

Introduction

The CPU executes the instructions that make up applications and the operating system at extremely high speed. However, main memory, which stores the data needed for processing, cannot operate at the same speed as the CPU.

If the CPU accessed main memory every time it needed data, it would spend more time waiting for that data to arrive. Even a powerful CPU cannot deliver its full performance if it cannot retrieve the required data quickly.

CPU cache bridges the speed gap between the CPU and main memory. It temporarily holds data that the CPU is likely to use and supplies it faster than main memory can.

This article explains the role of CPU cache, the differences between L1, L2, and L3 caches, cache hits and misses, cache lines, and the relationship between caches in multicore CPUs. It also explains how to inspect CPU cache configuration and cache misses in Linux.

What Is CPU Cache?

CPU cache is a high-speed storage area logically situated between the CPU and main memory. It temporarily holds recently used data and data that the CPU is likely to use next.

Typical CPUs have several cache levels: L1 cache, L2 cache, and L3 cache. In simplified form, their relationship looks like this.

Caches closer to the CPU are faster but smaller. Conversely, lower levels have larger capacities but take longer to access.

The Role of CPU Cache

The primary role of CPU cache is to reduce accesses to main memory and shorten the time spent waiting for data. If the required data is in a cache, the CPU can retrieve it without accessing main memory.

For example, if a program repeatedly reads the same variable and that data remains cached, subsequent accesses may be faster. CPU cache is essential for bridging the speed gap between the CPU and main memory and unlocking the CPU’s performance.

Differences Between L1, L2, and L3 Caches

CPU cache consists of several levels with different speeds and capacities. The most common are L1 cache, L2 cache, and L3 cache. The letter L stands for Level.

L1 Cache

L1 cache is closest to the CPU core and is usually the fastest cache. However, it has a smaller capacity than the other levels.

Many CPUs split L1 cache into an L1 instruction cache, which holds instructions, and an L1 data cache, which holds data. This allows the CPU to fetch instructions and the data being processed efficiently.

L2 Cache

L2 cache is larger than L1 cache but slower to access. On many CPUs, each CPU core also has its own L2 cache.

If the CPU cannot find the required data in L1 cache, it checks L2 cache next. If the data is in L2, there is no need to access L3 cache or main memory.

L3 Cache

L3 cache is generally larger than L1 and L2 caches. It is slower to access than either of them, but it is still usually faster than main memory.

In multicore CPUs, L3 cache is often shared among several CPU cores. Data missing from a core’s L1 and L2 caches may therefore be available in the shared L3 cache.

L1, L2, and L3 Cache Comparison

Note: Cache levels, capacities, and sharing arrangements depend on the CPU architecture. Not all CPUs have the same configuration.

How Does the CPU Use Its Caches?

When executing instructions, the CPU needs data from memory as well as from registers. To read data at a particular address, it first looks in the caches closest to the CPU.

If the required data is in a higher-level cache, the CPU does not need to access lower-level caches or main memory. Performance therefore depends heavily on how much of the CPU’s data can be supplied from cache.

What Is a Cache Hit?

Finding the required data in a cache is called a cache hit.

For example, finding the required data in L1 cache is an L1 cache hit. Even if the data is not in L1, finding it in L2 still lets the CPU retrieve it faster than from main memory.

What Is a Cache Miss?

Failing to find the required data in a particular cache is called a cache miss.

Conceptually, an L1 cache miss leads to a lookup in L2 and then lower cache levels. If none of the caches contains the data, it is ultimately read from main memory. The actual lookup mechanism depends on the CPU architecture.

Data retrieved from main memory is usually also placed in cache. The next access to the same data may therefore be served from cache.

What Is a Cache Line?

CPU caches do not manage each byte independently. Data is transferred between main memory and cache in fixed-size blocks called cache lines.

A cache line size of 64 bytes is common in modern CPUs. However, the actual size depends on the CPU architecture, so it is important to check the information exposed by Linux.

Why Fetch Nearby Data Together?

Programs often access nearby data after accessing a memory address. Fetching the entire cache line, rather than only the required byte, increases the chance that the next access can be served from cache.

For example, with a 64-byte cache line, reading a single 4-byte integer loads the 64-byte block containing that integer into cache.

A subsequent access to neighboring data in the same cache line may then be handled without accessing main memory again.

Why CPU Cache Improves Performance

CPU caches work effectively because programs exhibit locality in their memory accesses. The two main forms are temporal locality and spatial locality.

Temporal Locality

Temporal locality means that recently used data or instructions are likely to be used again in the near future.

For example, a loop repeatedly uses its control variables and the same instructions. Keeping them in cache saves the CPU from repeatedly fetching them from main memory.

Spatial Locality

Spatial locality means that accessing a memory address makes accesses to nearby data likely in the near future.

Reading an array sequentially from its beginning is a typical example of spatial locality. Array elements are normally stored consecutively in memory, so a cache line containing one element may also contain several subsequent elements.

As a result, sequentially scanning an array generally uses cache more efficiently than randomly accessing widely separated addresses.

Array Access Order Affects Performance

Even when processing the same amount of data, performance can vary with the order in which memory is accessed.

Processing an array sequentially lets the program keep using data from the cache line it has already loaded. Random accesses within a large array may require a different cache line each time, increasing cache misses.

CPUs may also have hardware prefetchers that predict future memory accesses and load data in advance. Sequential, regular access patterns are easier to predict and can benefit more readily from this mechanism.

Actual performance also depends on data size, access patterns, CPU architecture, compiler optimizations, and other factors. Arrays are not automatically fast; what matters is whether the access pattern makes effective use of cache.

Multicore CPUs and CPU Cache

Modern servers and PCs commonly use multicore CPUs. In a multicore CPU, not every cache is necessarily shared by all cores.

In a typical configuration, each CPU core has its own L1 and L2 caches, while L3 cache is shared among several cores.

Private L1 and L2 caches give each core fast access to the data it needs. Shared L3 cache can also efficiently retain data used by multiple cores.

When Multiple CPU Cores Use the Same Data

When several CPU cores read and write the same memory data, copies of that data may exist in each core’s cache.

For example, if Core 0 changes some data but Core 1 continues using an old cached value, the cores would see inconsistent data.

The mechanism that prevents this problem and keeps data consistent across caches is called cache coherency. The CPU exchanges information between caches so that, when one core updates data, other cores do not simply continue using stale copies.

MESI is one of the well-known protocols used to implement cache coherency. For now, it is sufficient to understand this as a mechanism that prevents conflicting cache contents across cores.

How CPU Cache Affects Performance

CPU cache has a major impact on application performance. CPU utilization is not the only consideration; where the CPU obtains its required data also matters.

When Cache Hits Are Frequent

With frequent cache hits, the CPU obtains the required data from L1, L2, or L3 cache. Fewer main-memory accesses mean less time waiting for data and more efficient instruction execution.

When Cache Misses Are Frequent

More cache misses lead to more accesses to lower-level caches and main memory. If the data is missing even from the last-level cache, the CPU must wait for it to arrive from main memory.

A condition in which waiting for required data impedes internal CPU processing is generally called a CPU stall. CPU utilization alone cannot show whether instructions are executing efficiently. Cache misses and other factors may increase stalls and reduce the number of instructions executed per cycle.

Cache efficiency may decline with random accesses to large datasets, repeated processing of data that greatly exceeds cache capacity, or frequent updates to the same cache line by multiple cores.

When investigating application performance, it is therefore important to examine cache misses and memory access behavior alongside CPU utilization.

Checking CPU Cache in Linux

Linux exposes CPU cache configuration through lscpu and sysfs. Cache configurations differ between CPUs, so check your actual environment.

Checking with lscpu

lscpu displays information such as CPU architecture, CPU count, core count, NUMA topology, and cache configuration.

Its output includes CPU cache information such as the following.

Depending on the lscpu version and environment, displayed capacities may represent the sum of all caches at a given level rather than the size of one cache. Check instance counts as well.

Checking with sysfs

Detailed per-CPU cache information is available under /sys/devices/system/cpu/ in sysfs. To inspect CPU0’s caches, refer to the following directory. Subdirectories such as index0, index1, index2, and index3 each represent a particular cache.

Use the following command to view information about each cache.

grep . /sys/devices/system/cpu/cpu0/cache/indexX/{level,type,size,coherency_line_size,shared_cpu_list}

For example, a coherency_line_size value of 64 means that the cache line size is 64 bytes. If an L3 cache’s shared_cpu_list contains several CPU numbers, that cache is shared among those logical CPUs.

Investigating Performance with CPU Cache in Mind

Linux performance investigations often begin by checking CPU utilization with top or mpstat. However, CPU utilization alone does not reveal how efficiently instructions are executing inside the CPU.

For example, as an application’s working dataset grows and becomes harder to fit in CPU cache, cache misses may increase. More main-memory accesses caused by cache misses mean the CPU spends more time waiting for data.

In this situation, CPU utilization alone cannot reveal internal execution efficiency. Waiting for memory accesses after cache misses may increase CPU stalls* and reduce the amount of work completed per unit of time.

* A CPU stall is a condition in which instruction execution cannot make sufficient progress because of factors such as waiting for data or dependencies between instructions. Common causes include memory access latency, data dependencies, branch mispredictions, and contention for execution resources.

Examine CPU Utilization Together with Cache Misses

To investigate a performance problem, first use top or mpstat to check overall and per-CPU utilization. If rising CPU load coincides with declining application performance, you may also be able to use perf stat to inspect instruction counts, CPU cycles, and cache misses, depending on the environment.

perf stat -e cycles,instructions,cache-references,cache-misses <command>

Examining cycles and instructions together provides clues about how many instructions were executed relative to CPU cycles. The hardware events represented by cache-references and cache-misses depend on the CPU architecture, so detailed analysis should also consider CPU-specific PMU events.

Conclusion

CPU cache is an essential mechanism that compensates for the speed gap between the CPU and main memory and gives the CPU fast access to the data it needs.

L1, L2, and L3 caches differ in speed and capacity. The CPU uses these caches to reduce main-memory accesses. Understanding cache lines, temporal locality, and spatial locality also makes it easier to see how CPU cache improves performance.

When investigating performance problems, consider cache misses and CPU stalls as well as CPU utilization. In Linux, lscpu and sysfs expose cache configuration, and perf may be available to inspect cache misses and other events.