The vocoder for Tacotron2 is an adaption of the WaveNet vocoder. The goal of the vocoding process here will be to invert a log mel spectrum representations back into a time-domain waveform representation.

The spectrogram prediction encoder-decoder and the WaveNet vocoder are trained separately, then the sequence of ground truth-aligned spectral features and gold audio output is then used to train the vocoder.


Vocoding

The exact same architecture we described for ASR—the encoder-decoder with attention – can be used for the first component of TTS. 

The Tacotron2 architecture is an encoder-decoder that maps from graphmes to spectrograms, followed by a vocoder that maps to wavefiles. 

University of Michigan - Ann Arbor

Text to speech, or speech synthesis, is the inverse problem of automatic speech recognition. It is a sequence-to-sequence learning task where the input is a sequence of text and the output is a generated audio file, meaning the resulting output sequence is significantly longer than the input sequence.

Text to Speech

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

The first component is an encoder-decoder model for spectrogram prediction: it maps from strings of letters to mel spectrographs: sequences of mel spectral values over time. The second component maps from mel spectrograms to waveforms.

Components of TTS

Numbers, monetary amounts, dates, and other concepts that are verbalized differently than they are spelled.

Non-standard Words

Often the verbalization of a non-standard word depends on its meaning. The meaning is called Semiotic Class.

Semiotic Class

Rule-based normalization is done in two stages: tokenization and verbalization. 

Rule-based Normalization

Encoder-decoder-based Models have been shown to work better than rules for such transduction tasks, but do require expert-labeled training sets in which non-standard words have been replaced with the appropriate verbalization; such training sets for some languages are available

Encoder-decoder-based Normalization

Spectrogram Prediction

Speech synthesis systems are evaluated by human listeners. We evaluate the quality of synthesized utterances by playing a sentence to listeners and ask them to give a mean opinion score (MOS), a rating of how good the synthesized utterances are, usually on a scale from 1–5. 

Paired t-tests to test for significant differences can be used to compare systems on the same sentence. AB tests can be used to compare exactly two systems.

Learn Before

Related

Learn After