In a standard CPU architecture, memory access latency increases as data moves further from the execution cores. An L1 cache reference is extremely fast, taking approximately $$1.5$$ ns (roughly $$4$$ cycles). If data is not in the L1 cache, an L2 cache hit takes about $$5$$ ns ($$12$$ to $$17$$ cycles). Retrieving data from the L3 cache incurs higher latency, ranging from $$16$$ ns ($$42$$ cycles) for an unshared cache, to $$25$$–$$29$$ ns if shared or modified in another core, and up to $$40$$ ns ($$100$$ to $$300$$ cycles) if the cache is located on a remote CPU socket.

Claude

Because the bandwidth of main memory (typically $$20$$ to $$40$$ GB/s) is often an order of magnitude lower than a modern processor's data consumption rate—for example, a $$2$$ GHz quad-core CPU executing $$256$$-bit AVX operations across its $$4$$ cores can consume $$128$$ bytes ($$4 \times 32$$ bytes) per clock cycle, requiring a transfer rate of $$256 \times 10^9$$ bytes per second—CPUs utilize a local cache hierarchy to prevent execution starvation. The hierarchy begins with extremely fast but tiny L1 caches, which are typically split into separate data and instruction caches. If data is not found in L1, the search progresses downward to larger but slightly slower L2 caches (often exclusive per-core), and finally to substantial L3 caches that are shared across multiple cores. This tiered structure minimizes access latency by keeping frequently used data close to the execution units.

CPU Cache Hierarchy

Dive into Deep Learning

CPU registers are discrete, high-speed memory locations built directly into the processor core that operate at full clock speed without any delay penalty. While not strictly classified as part of the cache hierarchy, registers are utilized by compilers and programmers to temporarily stage instructions and operands for immediate execution by the CPU.

CPU Registers

To maximize the efficiency of a CPU's cache hierarchy, software and algorithms should be designed to optimize memory access patterns. Because caching algorithms actively attempt to read ahead, traversing memory in a forward, sequential direction is highly advantageous. Furthermore, keeping memory access localized ensures that data remains within the cache boundaries, preventing expensive cache misses that stall the processor.

Memory Access Pattern Optimization

False sharing is a severe performance degradation scenario in multi-core CPU caching. It occurs when a memory location cached on one processor core is subsequently requested by a thread running on a different processor core. To resolve this conflict, the first processor must halt its operations, write the cached information back to main memory, and allow the second processor to read it. This forced synchronization causes both processors to wait, potentially resulting in parallel code executing more slowly than an efficient single-processor implementation.

False Sharing in CPU Caches

CPU Cache Latencies

To achieve maximum computational efficiency, deep learning algorithms should be specifically matched to the constraints of the target hardware, taking into account factors like available memory bandwidth and overall memory footprint. By deliberately co-designing the algorithm to fit its parameters entirely within the processor's fast local caches, developers can often realize performance speedups of several orders of magnitude compared to unoptimized approaches.

Learn Before

Related