In multi-machine distributed training, utilizing a single central parameter server creates a significant bandwidth bottleneck. Because the network bandwidth per server is finite and comparatively low, all machines must communicate with this single central point to synchronize gradients and receive updated parameters. If there are $m$ worker machines, the time required to send all gradients to the central server scales linearly, resulting in an update time of $\mathcal{O}(m)$. This bottleneck severely limits the scalability of synchronous distributed optimization, as the central server cannot efficiently handle the data transfer demands of many simultaneous workers.