Tokenization is a way of separating a piece of text into smaller units called tokens.

University of Michigan - Ann Arbor

Text corpora may have different features that are needed to process differently due to 1)language; 2)genre; 3)the demographic characteristics of the writer (or speaker); 4)time. Proper text normalization is required to make sure the models are fit to the text data. Text normalization make sure that the text is changed into a suitable and excellent form for further processing.

Text normalization

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

The process of text normalization includes:
- Tokenizing (segmenting) words 
- Normalizing word formats (case folding/lemmatization/stemming)
- Segmenting sentences

Process of text normalization

- A regular expression is "an algebraic notation for characterizing a set of strings". They are particularly useful for searching patterns in texts.
- Regular repressions helps to represent text patterns to be extracted from a document.

Regular Expression

Done with the use of large scale pre-trained language models that use Transformer-based architecture 

Text Representation

Tokenization

It's the task of determining that two words have the same root, despite their surface differences.

Lemmatization

An utterance is the spoken correlate of a sentence.

Utterance

The larger the corpora we look at, the more word types we find.
Assume $|V|$ is the number of types and $N$ is number of tokens. k and $\beta$ are positive numbers and $0< \beta <1$.  Then
$|V|= k N^{\beta}$

Learn Before

Related