Stemming means chopping off words' affixes. The Porter algorithm is a simple and efficient way to do stemming, but it lacks high accuracy.

University of Colorado at Boulder

Text normalization means transferring words to their standard forms. Text normalization tasks include word segmentation and normalization, sentence segmentation, and stemming.

Text Normalization

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Word segmentation, or word tokenization, means segmenting running text into separative words. In practice, since word segmentation is usually the first step in language processing, it needs to be very fast.


Word normalization basically means changing words into standard formats, which are used to stand for other forms with the same meaning.


Word tokenization and normalization are generally done by regular expression or finite automata.

Word segmentation and normalization

Sentence segmentation means cutting a text into sentences, and it often uses punctuation as the cue. Given the fact that many words also contain punctuation, like Mr., sentence segmentation needs to decide whether a punctuation belongs to a word or functions in the sentence.

Learn Before

Related