Tokenization is the process of converting a sequence of text into smaller units, known as tokens. It is a foundational step in Natural Language Processing, and there are numerous different methods and strategies for how a text can be tokenized.

Tokenization

Punctuation, like periods, question marks, and exclamation points, are the most useful sentence-boundary markers. However, punctuation like periods are ambiguous among a sentence boundary marker, a number marker, and a marker of abbreviations. For this reason, sentence tokenization and word tokenization may be addressed jointly.

Sentence segmentation

Word normalization is the task of unifying words/tokens to a single standard format. The methods below are widely used depending on the tasks.

- Case folding
- Lemmatization
- Stemming

Word normalization

Unix commands such as tr, sort, and uniq can be used for simple normalization, tokenization, and frequency computation.

Unix Tools for Crude Tokenization and Normalization


The process of text normalization includes:
- Tokenizing (segmenting) words 
- Normalizing word formats (case folding/lemmatization/stemming)
- Segmenting sentences

University of Michigan - Ann Arbor

Text corpora may have different features that are needed to process differently due to 1)language; 2)genre; 3)the demographic characteristics of the writer (or speaker); 4)time. Proper text normalization is required to make sure the models are fit to the text data. Text normalization make sure that the text is changed into a suitable and excellent form for further processing.

Text normalization

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Process of text normalization

- A regular expression is "an algebraic notation for characterizing a set of strings". They are particularly useful for searching patterns in texts.
- Regular repressions helps to represent text patterns to be extracted from a document.

Regular Expression

Done with the use of large scale pre-trained language models that use Transformer-based architecture 

Text Representation

Tokenization is a way of separating a piece of text into smaller units called tokens.

It's the task of determining that two words have the same root, despite their surface differences.

Lemmatization

An utterance is the spoken correlate of a sentence.

Utterance

The larger the corpora we look at, the more word types we find.
Assume $|V|$ is the number of types and $N$ is number of tokens. k and $\beta$ are positive numbers and $0< \beta <1$.  Then
$|V|= k N^{\beta}$

Learn Before

Related

Learn After