To overcome the bandwidth bottleneck of a single central parameter server in multi-machine training, the system can be scaled by distributing the parameter synchronization workload across multiple servers. By increasing the number of parameter servers to $n$, each server is responsible for storing and updating only a fraction of the parameters, specifically $\mathcal{O}(1/n)$. Consequently, the total time required for parameter updates and optimization across $m$ workers is reduced to $\mathcal{O}(m/n)$. In practice, to achieve constant scaling time regardless of the total number of workers, systems often use the exact same machines simultaneously as both workers and servers.