Word segmentation, or word tokenization, means segmenting running text into separative words. In practice, since word segmentation is usually the first step in language processing, it needs to be very fast.


Word normalization basically means changing words into standard formats, which are used to stand for other forms with the same meaning.


Word tokenization and normalization are generally done by regular expression or finite automata.

Word segmentation and normalization

Sentence segmentation means cutting a text into sentences, and it often uses punctuation as the cue. Given the fact that many words also contain punctuation, like Mr., sentence segmentation needs to decide whether a punctuation belongs to a word or functions in the sentence.

Sentence segmentation

Stemming means chopping off words' affixes. The Porter algorithm is a simple and efficient way to do stemming, but it lacks high accuracy.

Stemming

Text normalization means transferring words to their standard forms. Text normalization tasks include word segmentation and normalization, sentence segmentation, and stemming.

University of Colorado at Boulder

Claude

Text corpora may have different features that are needed to process differently due to 1)language; 2)genre; 3)the demographic characteristics of the writer (or speaker); 4)time. Proper text normalization is required to make sure the models are fit to the text data. Text normalization make sure that the text is changed into a suitable and excellent form for further processing.

Text normalization

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Dive into Deep Learning

The process of text normalization includes:
- Tokenizing (segmenting) words 
- Normalizing word formats (case folding/lemmatization/stemming)
- Segmenting sentences

Process of text normalization

- A regular expression is "an algebraic notation for characterizing a set of strings". They are particularly useful for searching patterns in texts.
- Regular repressions helps to represent text patterns to be extracted from a document.

Regular Expression

Done with the use of large scale pre-trained language models that use Transformer-based architecture 

Text Representation

Tokenization is a way of separating a piece of text into smaller units called tokens.

Tokenization

It's the task of determining that two words have the same root, despite their surface differences.

Lemmatization

An utterance is the spoken correlate of a sentence.

Utterance

The larger the corpora we look at, the more word types we find.
Assume $|V|$ is the number of types and $N$ is number of tokens. k and $\beta$ are positive numbers and $0< \beta <1$.  Then
$|V|= k N^{\beta}$

Herden's Law / Heaps’ Law

After loading raw text into memory, it is commonly preprocessed to standardize the format and reduce vocabulary complexity before feeding it into sequence models. A straightforward preprocessing strategy uses a regular expression to replace every non-alphabetical character (including punctuation, digits, and whitespace) with a single space, and then converts all remaining characters to lowercase. For instance, applying the pattern [^A-Za-z]+ followed by lowercasing transforms 'The Time Machine, by H. G. Wells [1898]' into 'the time machine by h g wells '. This produces a uniform stream of space-separated lowercase words, eliminating noise from punctuation and capitalization that is unnecessary for many language modeling tasks.

Learn Before

Related

Learn After