- End to end evaluation by adding the component to the actual application to understand if the the component can improve task performance. 
- Compare performance of 2 language models by running 2 speech recognition systems

University of Michigan - Ann Arbor

The most important models are:
- state machines
- rule systems
- logic
- probabilistic models
- vector-space models

These models are all drawn from the standard toolkits of computer science, mathematics, and linguistics.

Models used in NLP

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Each of the other models (state machines, formal rule systems, and logic) can be augmented with probabilities. For example, the state machine can be augmented with probabilities to become the weighted automaton, or Markov model. The key advantage of probabilistic models is their ability to solve the many kinds of ambiguity problems that can be recast as “given N choices for some ambiguous input, choose the most probable one”.

Probabilistic models

Extrinsic Model Evaluation in NLP

- Since running end to end evaluation for big nlp systems can be expensive, it is suggested to measure the model's quality independent of application. We defined part of the corpus as held out corpora (test test) for the purpose of validating the model's performance, and exclude the set during the training process. 

Intrinsic Model Evaluation in NLP

When there are many correlated features, logistic regression will assign
a more accurate probability than naive Bayes, because logistic regression will distribute the weights of correlated features, while naive Bayes will multiply both features and overestimating the evidence. In general, logistic regression works better on larger documents. On the other hand, naive Bayes works extremely well on very small datasets, and it can still make correct classification decision.

Logistic Regression v.s. Naive Bayes in NLP Classification Tasks

$Yˆ = argmax_{Y∈y} P(Y|X)$
However, the CRF does not compute a probability for each tag at each time step. Instead, at each time step the CRF computes log-linear functions over a set of relevant features, and these local features are aggregated and normalized to produce a global probability for the whole sequence.
In a CRF, the function F maps an entire input sequence X and an entire output sequence Y to a feature vector. Let’s assume we have K features, with a weight wk for each feature $F_{k}$:
$p(Y|X) = \frac{exp(\sum_{k=1}^K w_{k}F_{k}(X,Y))}{\sum_{Y'∈y} exp(\sum_{k=1}^Kw_{k}F_{k}(X,Y'))}$
We’ll call these K functions $F_{k}(X,Y)$ global features, since each one is a property of the entire input sequence X and output sequence Y. We compute them by decomposing into a sum of local features for each position i in Y:
$F_{k}(X,Y) =\sum_{i=1}^n f_{k}(y_{i−1}, y_{i},X,i) $
This constraint to only depend on the current and previous output tokens $y_{i}$ and $y_{i−1}$ are what characterizes a linear chain CRF.  A general CRF allows a feature to make use of any output token, and are thus necessary for tasks in which the decision depend on distant output tokens. General CRFs require more complex inference, and are less commonly used for language processing.

Learn Before

Related