Accessing main memory (RAM) is significantly slower than hitting the CPU cache hierarchy. For example, a $$64$$ MB memory reference on a local CPU takes approximately $$46$$ ns, while a larger $$256$$ MB reference takes about $$75$$ ns. If the memory must be accessed across a multi-socket system (a remote CPU reference), the latency increases substantially due to the interconnect hop, taking roughly $$70$$ ns for a $$64$$ MB reference and $$120$$ ns for a $$256$$ MB reference.

Claude

Random Access Memory (RAM) serves as the primary memory of a computer, providing fast access to data required during active computation. For deep learning tasks, RAM is essential for temporarily storing and retrieving training data, intermediate weight vectors, and neural network activations before they are processed by the CPU or GPU.

Random Access Memory (RAM) in Deep Learning

Dive into Deep Learning

Modern CPU RAM is typically of the DDR4 variety, offering $$20$$ to $$25$$ GB/s bandwidth per module with a $$64$$-bit-wide bus. To achieve higher performance, memory modules are usually paired to utilize multiple channels. CPUs generally feature between $$2$$ and $$4$$ memory channels, resulting in a peak memory bandwidth ranging from $$40$$ GB/s to $$100$$ GB/s. Additionally, there are often two memory banks per channel; for instance, high-end processors like AMD’s Zen 3 Threadripper provide $$8$$ memory slots to accommodate these configurations.

CPU RAM Architecture and Bandwidth

Reading from RAM requires first sending an address to the memory module, a setup process that takes roughly $$100$$ ns. Once established, subsequent data transfers are exceedingly fast, taking only $$0.2$$ ns per $$64$$-bit record. When reading a long contiguous sequence of records—known as a burst read—the initial high cost is amortized. Conversely, isolated random reads require the $$100$$ ns setup every time, making the first read $$500$$ times more expensive than subsequent burst transfers. Consequently, while a system might perform up to $$10,000,000$$ random reads per second, computational efficiency dictates avoiding random memory access whenever possible in favor of burst reads and writes.

Learn Before

Related