 CFG rules impose an independence assumption on probabilities that leads to poor modeling of structural dependencies across the parse tree. In a CFD the expansion of a non-terminal is independent of the context, which means that it is not dependent on other nearby non-terminals in the parse tree. Similarly, in a PCFG, the probability of a particular rule is independent of the rest of the tree. Therefore, in a PCFG, we compute the probability of a tree by multiplying the probabilities of each non-terminal expansion. However, this results in poor probability estimates because in English the choice of how a node expands can actually depend on the location in the parse tree of that node. For example, it turns out that NPs which are syntactic subjects are more likely to be pronouns, and NPs which are syntactic objects are more likely not to be. There is no way to represent this contextual difference in the probabilities of a PCFG. 


Poor independence assumptions in PCFGs


CFG rules don’t model syntactic facts about specific words, leading to problems with subcategorization ambiguities, preposition attachment, and coordinate structure ambiguities. Words do play a role in PCFGs since the parse probability includes the probability of a word given a part-of-speech. Depending on how these probabilities are set, a PCFG will always either prefer NP attachment or VP attachment. As it happens, NP attachment is slightly more common in English, so if we trained these rule probabilities on a corpus, we might always prefer NP attachment, causing us to misparse this sentence. Probabilistic context-free grammars are incapable of modeling important structural and lexical dependencies. 

Lack of lexical conditioning in PCFGs


We can split NP non-terminals into two versions: one for subjects, and one for objects. Having two nodes ($NP_{subject}$ and $NP_{object}$) would allow us to correctly model their different distributional properties since we could have different probabilities for the rules $NP_{subject} \to PRP$ and $NP_{object} \to PRP$. The intuition of splits can be implemented by doing parent annotation, in which we annotate each node with its parent in the parse tree. A PCFG can also be improved by splitting pre-terminal POS nodes. Adding specific tags for parts of speech can be useful in improving PCFG modeling. 


Improving PCFGs by splitting NP non-terminals


The main problems of PCFGs are probability estimators are as follows:
- Poor independence assumptions
- Lack of lexical conditioning


Columbia University

Probabilistic augmentation of context-free grammars in which each rule is associated with a probability. 

$A \rightarrow B [p]$

A PCFG is $consistent$ if for all sentences, the probabilities for that sentence sum up to 1.

Probabilistic Context-Free Grammar (PCFG)

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

To resolve disambiguation, the PCFG will calculate a probability of each parse tree (derivation) $T$ of a given sentence $S$. 

The probability of tree $T$ given $S$ is defined as:

$P(T,S) = \prod_{i=1}^{n} P(RHS_i | LHS_i)$

From there, the PCFG will choose the parse tree with the greatest probability as the correct parse of $S$.

PCFGs for Disambiguation

PCFGs are useful in language modeling as they can predict the probability of a sentence or part of a sentence

PCFGs for Language Modeling

The goal of PCFGs is to propose the most likely parse *T̂* of a given sentence *S*.

Learn Before

Related

Learn After