Given the few-shot dataset and model, just tuning $\theta_0$ using gradient descent can easily lead to overfitting. Thus some regularization techniques are used, including early-stopping, selectively updating, updating related parts of $\theta_0$ together, and model regression network.

Fine-Tuning Existing Parameter by Regularization

When we don’t have a suitable $\theta_0$ to start with but has some part of $\theta_0$ available, we may aggregate these part together to form a suitable $\theta_0$, then directly use it or fine-tune it by $D_{train}$

Aggregating a Set of Parameters

When $\theta_0$ alone is not enough to encode the input of the new few-shot learning task, we may add a new parameter to the current $\theta_0$, and fine-tune $\theta_0$ while learning new parameters.

Fine-Tuning Existing Parameter with New Parameters

When using $D_{train}$ to refine $\theta_0$, there’re several techniques can be used to improve results:

1)	Fine-Tuning Existing Parameter by Regularization

2)	Aggregating a Set of Parameters.

3)	Fine-Tuning Existing Parameter with New Parameters.


University of Notre Dame

This strategy uses some $\theta_0$ from some pre-trained models related to the current task as a starting point and uses $D_{train}$ to find better $\theta_0$.  

Modify Search Algorithm by Refining Existing Parameters

Wang, Y., Yao, Q., Kwok, J. T., & Ni, L. M. (2020). Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3), 1-34.

Learn Before

Related

Learn After