Case folding is one kind of the word normalization.
Mapping everything to lower case is often very helpful for generalization in many tasks, such as information retrieval or speech recognition.
For sentiment analysis and other text classification tasks, information extraction, and machine translation, by contrast, case can be quite helpful and case folding is generally not done.

Case Folding

Word segmentation, or word tokenization, means segmenting running text into separative words. In practice, since word segmentation is usually the first step in language processing, it needs to be very fast.


Word normalization basically means changing words into standard formats, which are used to stand for other forms with the same meaning.


Word tokenization and normalization are generally done by regular expression or finite automata.

University of Colorado at Boulder

Text normalization means transferring words to their standard forms. Text normalization tasks include word segmentation and normalization, sentence segmentation, and stemming.

Text Normalization

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Word segmentation and normalization

Sentence segmentation means cutting a text into sentences, and it often uses punctuation as the cue. Given the fact that many words also contain punctuation, like Mr., sentence segmentation needs to decide whether a punctuation belongs to a word or functions in the sentence.

Sentence segmentation

Stemming means chopping off words' affixes. The Porter algorithm is a simple and efficient way to do stemming, but it lacks high accuracy.

Learn Before

Related

Learn After