Scoring interpolates the score assigned by the language model with the encoder-decoder score used to create the beam, with a weight $\lambda$ tuned on a holdout set. Since most models prefer shorter sentences, ASR systems generally add a length factor, such as  normalizing the probability by the number of characters in the hypothesis $|Y|_c$. The following image contains a typical scoring function.

Scoring beams of hypothesized sentences

Getting a final beam of hypothesized sentences is the simplest way to add a language model to the ASR architecture. This beam is sometimes referred to as an n-best list. A language model can then be used to rescore each hypothesis on the beam.

Columbia University

An encoder-decoder model is basically a conditional language model, so encoder-decoders implicitly must learn a language model for the output domain of letters from training data. However, the training data, which is generally speech paired with text transcriptions, may not include as much text as you need to train a good language model, since it’s easier to find large amounts of pure text training data than it is to find text paired with speech. Therefore, a model for ASR can usually be improved by incorporating a very large language model. 

Adding a language model

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Learn Before

Related

Learn After