Combination of architecture of both CTC and Encoder-Decoder.

With a $\lambda$ tuned on dev set the two losses can be weighed during training

$L = \lambda \log P_{encdec} (Y \mid X) - (1-\lambda ) \log P_{ctc}(Y \mid X)$

With the learned weights two language models are combined for inferecing


San Jose State University

Connectionist Temporal Classification (CTC) is a combination of an algorithm and a loss function that transform a sequence of acoustic sounds into a sequence of letters, i.e., speech recognition. Unlike the encoder-decoder structure which transforms a longer input sequence to a shorter output sequence, CTC first transforms the input sequence to an equal length sequence, then transforms the alignment to a shorter output sequence by merging consecutive duplicates and removing blanks. The output vector of the first step is called alignment, and the transformation in second step is called collapsing function.

Connectionist Temporal Classification

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

The alignment is a sequence of vectors where each vector represents a letter. It is the middle output of the Connectionist Temporal Classification (CTC) algorithm, and it has the same length as the input of CTC algorithm. 

Alignment in Connectionist Temporal Classification

CTC based automatic speech recognition system uses negative log-likelihood loss with a special CTC loss function.

$L_{CTC} = \sum_{(X,Y) \in D} -log P_{CTC}(Y \mid X) $
$L_{CTC}$ - loss for the entire dataset D

$P_{CTC} = \sum_{A\in B^{-1}(Y)} \prod_{t=1}^T p(a_t \mid h_t)$

$P_{CTC}$ - Probability of output Y given input X, and it is the highest sum of probability over all its assignments

Connectionist Temporal Classification Training

Combining CTC and Encoder-Decoder

 RNN-Transducer is used to remove the conditional dependency of the CTC .

Learn Before

Related