University of Michigan - Ann Arbor

In an edge system where both direct CPU expert evaluation and direct memory access (DMA) transfers across PCIe draw from the same host DRAM subsystem, a saturated PCIe transfer leaves a constrained amount of memory bandwidth for CPU processing. If $B_H$ denotes the measured host-side expert-processing bandwidth and $B_P$ represents the pinned expert-transfer bandwidth across PCIe, the residual host bandwidth $B_R$ available for concurrent in-place CPU expert computation is given by:

$$B_R = \max(B_H - B_P, 0)$$

When $B_H > B_P$, the CPU can concurrently execute residual missed experts using this leftover host memory bandwidth without contending with or throttling the saturated PCIe bus.

Residual Host Bandwidth Formulation for CPU Expert Execution

Order the steps to determine and utilize residual host bandwidth for CPU expert execution under a saturated PCIe bus.

Calculate the residual host bandwidth (B_R) available for concurrent in-place CPU computation, and explain whether the CPU can execute missed experts concurrently without contending with or throttling the PCIe bus.

To minimize exposed decode latency when serving $m$ missing experts of size $S$ bytes each, the execution times of the concurrent PCIe cache-fill branch and CPU execution branch are balanced. The cache-fill duration is $T_{\text{fill}}(q) \approx \frac{q S}{B_P}$, and the CPU execution duration on residual bandwidth is $T_{\text{cpu}}(m - q) \approx \frac{(m - q) S}{B_H - B_P}$. Equating the execution times of the two concurrent branches yields:

$$\frac{q}{m - q} \approx \frac{B_P}{B_H - B_P}$$

Solving for the optimal cache-fill count $q^*$ produces the closed-form policy:

$$q^* \approx m \cdot \frac{B_P}{B_H}$$

In runtime implementation, $q^*$ is rounded to an integer with a minimum of at least one fill retained to ensure continuous cache warming, dynamically degenerating to pure on-demand GPU caching as $B_H \to B_P$.

Learn Before

Related