Two-Level Tile-Based Approach in Tensor Parallelism

A machine learning engineer is training a model with an exceptionally large layer. The weight matrix for this single layer is so large that it cannot fit into the memory of one GPU, causing an 'out-of-memory' error during the matrix multiplication step. Which of the following strategies directly addresses this specific memory bottleneck by parallelizing the problematic matrix multiplication itself across multiple devices?

Solving a Memory Bottleneck with Parallelism

Analyzing Distributed Matrix Multiplication Strategies

Example of Tensor Parallelism in an FFN Sub-layer