This approach combines Connectionist Temporal Classification (CTC) and Encoder-Decoder architectures. During training, a hyperparameter $$\lambda$$ (tuned on a development set) is used to weight the two log-likelihood objectives:

$$L = \lambda \log P_{encdec}(Y \mid X) + (1-\lambda) \log P_{ctc}(Y \mid X)$$

During inference, the two models are combined using the learned weights.

San Jose State University

Google

Connectionist Temporal Classification (CTC) is a combination of an algorithm and a loss function that transform a sequence of acoustic sounds into a sequence of letters, i.e., speech recognition. Unlike the encoder-decoder structure which transforms a longer input sequence to a shorter output sequence, CTC first transforms the input sequence to an equal length sequence, then transforms the alignment to a shorter output sequence by merging consecutive duplicates and removing blanks. The output vector of the first step is called alignment, and the transformation in second step is called collapsing function.

Connectionist Temporal Classification

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

The alignment is a sequence of vectors where each vector represents a letter. It is the middle output of the Connectionist Temporal Classification (CTC) algorithm, and it has the same length as the input of CTC algorithm. 

Alignment in Connectionist Temporal Classification

Combining CTC and Encoder-Decoder

 RNN-Transducer is used to remove the conditional dependency of the CTC .

 RNN-T for improving CTC

A Connectionist Temporal Classification (CTC) based automatic speech recognition system uses a negative log-likelihood loss with a special CTC loss function:

$$L_{CTC} = \sum_{(X,Y) \in D} -\log P_{CTC}(Y \mid X)$$

Where $$L_{CTC}$$ is the loss for the entire dataset $$D$$.

$$P_{CTC} = \sum_{A\in B^{-1}(Y)} \prod_{t=1}^T p(a_t \mid h_t)$$

Where $$P_{CTC}$$ is the probability of output $$Y$$ given input $$X$$, and it is the highest sum of probability over all its assignments.

Learn Before

Related