The collapsing function in Connectionist Temporal Classification (CTC) transforms alignment to the final output sequence.  It has two steps:

1) Remove any consecutive letters. For example, “dinner” is turned into “diner”

2) Remove blanks. For example, if we use “_” to denote blanks, “din_ner” is turned into “dinner”.

As a result, the collapsing function is a mapping from many to one. For example, “dinn_ner” and “diin_nne_r” both map to “dinner”.

Collapsing Function

In Connectionist Temporal Classification (CTC), we assume that the output vectors in a single sequence are all independent. Mathematically, let the input sequence be $X=\{x_1, x_2,..,x_n\}$ where $x_i$ are vectors, and let the output alignment be $A=\{a_1, a_2, …, a_n\}$ where $a_i$ are vectors, then $P_{CTC}(A|X)=\prod_{i=1}^{n}p(a_i|X)$.

Strong Conditional Independence Assumption 

The alignment is a sequence of vectors where each vector represents a letter. It is the middle output of the Connectionist Temporal Classification (CTC) algorithm, and it has the same length as the input of CTC algorithm. 

University of Notre Dame

Connectionist Temporal Classification (CTC) is a combination of an algorithm and a loss function that transform a sequence of acoustic sounds into a sequence of letters, i.e., speech recognition. Unlike the encoder-decoder structure which transforms a longer input sequence to a shorter output sequence, CTC first transforms the input sequence to an equal length sequence, then transforms the alignment to a shorter output sequence by merging consecutive duplicates and removing blanks. The output vector of the first step is called alignment, and the transformation in second step is called collapsing function.

Connectionist Temporal Classification

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Alignment in Connectionist Temporal Classification

CTC based automatic speech recognition system uses negative log-likelihood loss with a special CTC loss function.

$L_{CTC} = \sum_{(X,Y) \in D} -log P_{CTC}(Y \mid X) $
$L_{CTC}$ - loss for the entire dataset D

$P_{CTC} = \sum_{A\in B^{-1}(Y)} \prod_{t=1}^T p(a_t \mid h_t)$

$P_{CTC}$ - Probability of output Y given input X, and it is the highest sum of probability over all its assignments

Connectionist Temporal Classification Training

Combination of architecture of both CTC and Encoder-Decoder.

With a $\lambda$ tuned on dev set the two losses can be weighed during training

$L = \lambda \log P_{encdec} (Y \mid X) - (1-\lambda ) \log P_{ctc}(Y \mid X)$

With the learned weights two language models are combined for inferecing


Combining CTC and Encoder-Decoder

 RNN-Transducer is used to remove the conditional dependency of the CTC .

Learn Before

Related

Learn After