Cross-Layer Parameter Sharing in Transformers

A team of engineers is building a deep neural network to analyze very long text sequences. They discover that the model's size is exceeding their hardware's memory capacity. As a solution, they modify the architecture to make multiple layers use the exact same set of learnable parameters. What is the primary trade-off the engineers must consider with this parameter-sharing approach?

Optimizing a Transformer for a Low-Resource Environment

A key strategy for creating more efficient neural networks involves reusing parts of the model. Analyze the following concepts related to this strategy and match each term to its most accurate description.