Cross-Layer Parameter Sharing in BERT

Cross-layer Multi-head Attention

A team of engineers is designing a deep neural network for a resource-constrained environment, such as a mobile device. To reduce the model's size, they implement a design where the same computational block, with its entire set of weights, is reused at every layer of the network. What is the most significant trade-off the engineers must consider with this approach?

Analyzing a Novel Transformer Architecture

Comparing Parameter Sharing Strategies