Stemming a simpler but cruder alternative to the lemmatization, which mainly chops off word-final affixes. The Porter Stemmer is the most widely used stemming algorithms.

Stemming

Case folding is mapping everything to lower case. Whether applying it depends on the tasks. In information retrieval or speech recognition, it is very helpful. However, for sentiment analysis and other text classification tasks, information extraction, and machine translation, keeping cases can help differentiate words. For example, US the country and us the pronoun.

Case folding

Lemmatization is to determine that if two words have the same root, despite their surface differences. For example, $am$, $are$, and $is$ have the shared lemma $be$; the words $cat$ and $cats$ both have the lemma $cat$.

Lemmatization

Word normalization is the task of unifying words/tokens to a single standard format. The methods below are widely used depending on the tasks.

- Case folding
- Lemmatization
- Stemming

University of Michigan - Ann Arbor

The process of text normalization includes:
- Tokenizing (segmenting) words 
- Normalizing word formats (case folding/lemmatization/stemming)
- Segmenting sentences

Process of text normalization

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Tokenization is the process of converting a sequence of text into smaller units, known as tokens. It is a foundational step in Natural Language Processing, and there are numerous different methods and strategies for how a text can be tokenized.

Tokenization

Punctuation, like periods, question marks, and exclamation points, are the most useful sentence-boundary markers. However, punctuation like periods are ambiguous among a sentence boundary marker, a number marker, and a marker of abbreviations. For this reason, sentence tokenization and word tokenization may be addressed jointly.

Sentence segmentation

Word normalization

Unix commands such as tr, sort, and uniq can be used for simple normalization, tokenization, and frequency computation.

Learn Before

Related

Learn After