When a suitable initial parameter set $$\theta_0$$ is not available, but parts of $$\theta_0$$ are accessible from related pre-trained models, these parts can be aggregated to form a suitable $$\theta_0$$. This aggregated parameter set can then be used directly or fine-tuned using the training dataset $$D_{train}$$.

Aggregating a Set of Parameters for Fine-Tuning

In few-shot learning, simply tuning initial parameters ($$\theta_0$$) using gradient descent on a small dataset can easily lead to overfitting. To mitigate this, regularization techniques are used during fine-tuning. These techniques include early-stopping, selectively updating parameters, updating related parts of $$\theta_0$$ together, and employing a model regression network.

Fine-Tuning Existing Parameters by Regularization

In few-shot learning, when the initial parameter set $$\theta_0$$ alone is not enough to encode the input of a new task, new parameters can be added to the model. The model then fine-tunes the existing $$\theta_0$$ while simultaneously learning the new parameters.

Fine-Tuning Existing Parameters with New Parameters

When using a training dataset ($$D_{train}$$) to refine initial parameters ($$\theta_0$$) in few-shot learning, several techniques can be used to improve the results. These strategies include fine-tuning existing parameters by regularization, aggregating a set of parameters, and fine-tuning existing parameters with new parameters.

University of Notre Dame

Google

In few-shot learning, the strategy of refining existing parameters uses initial parameters ($$\theta_0$$) from pre-trained models related to the current task as a starting point. It then uses the training dataset ($$D_{train}$$) to refine and find better parameters.

Modify Search Algorithm by Refining Existing Parameters

Wang, Y., Yao, Q., Kwok, J. T., & Ni, L. M. (2020). Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3), 1-34.

Learn Before

Related

Learn After