Let $P_n$ be relative frequencies from $n>0$ observations on a finite alphabet, and $Q$ a normalized probability mass function on that alphabet. An elementary deduction from the discrete KL definition is that $D_{\mathrm{KL}}(P_n\|Q)$ is finite exactly when $Q(x)>0$ for every observed outcome. Terms with $P_n(x)=0$ contribute zero, even if $Q(x)=0$; unobserved outcomes need not have positive model mass. Under an i.i.d. model, a zero model probability at an observed outcome makes the sample likelihood zero and negative log-likelihood infinite—an elementary deduction from the likelihood product formula.

Sources:

• https://ocw.mit.edu/courses/6-441-information-theory-spring-2016/5d8f16adc3385c9ff2975b121bd620e4_MIT6_441S16_course_notes.pdf
• https://www.deeplearningbook.org/contents/ml.html