The SR(Softmax Response) layer's output can be too confident so that it is unreliable as a measure of confidence and the performance of this method will thus be even worse than that of random sampling.

Model Uncertainty in Deep Learning for Active Learning

AL often relies on a small amount of labeled sample data to learn and update the model, while DL is often very greedy for data. Data for every iteration may not be enough to train a good model.

Insufficient Data for Labeled Samples.

The processing pipelines of AL and DL are inconsistent. Most AL algorithms focus primarily on the training of classifiers and the various query strategies utilized are largely based on fixed feature representations.

Processing Pipeline Inconsistency. 

1. Model uncertainty in Deep Learning. 
2. Insufficient data for labeled samples.
3. Processing pipeline inconsistency.

University of Michigan - Ann Arbor

Active learning (AL) attempts to maximize a model’s performance gain while annotating the fewest samples possible.

Active Learning

A Survey of Deep Active Learning

AL approaches can be divided into membership query synthesis, stream-based selective sampling, and pool-based AL from application scenarios.

Active Learning Categories

1. Uncertainty-based Strategy.
2. Diversity-based Strategy.
3. Expected Model Change.

Query Strategy Categories of Active Learning

Challenges of Combining Deep Learning and Active Learning

We define $U^n = {X,Y}$ as an unlabeled dataset with n labels. 
$X$ is the sample space. 
$Y$ is the label space.
$P(x,y)$ is a potential distribution.
$L^m = {X,Y}$ is the current labeled training set with m samples.
Query strategy is $Q$ which maps $U^n$ to $L^m$.


Query Strategy Formalization of Deep Active Learning 

1. Batch Mode DeepAL.
2. Uncertainty-based and Hybrid Query Strategies.
3. Deep Bayesian Active Learning. (DBAL)
4. Density-based Methods.
5. Automated Design of DeepAL.

Learn Before

Related

Learn After