 $$ 
A \ command \ for \ tokenizing \ words, \ in \ increasing \ specificity: \ \newline
\newline
1) \ tr \ \text{--}sc \ ’A\text{--}Za\text{--}z’ \ ’\backslash n’ \ < \ sh.txt \newline
2) \ tr \ \text{--}sc \ ’A\text{--}Za\text{--}z’ \ ’\backslash n’ \ < \ sh.txt \ | \ sort \ | \ uniq \ -c \newline
3) \ tr \ \text{--}sc \ ’A\text{--}Za\text{--}z’ \ ’\backslash n’ \ < sh.txt \ | \ tr \ A\text{--}Z \ a\text{--}z \ | \ sort \ | \ uniq \ \text{--}c \newline
4) \ tr \ \text{--}sc \ ’A\text{--}Za\text{--}z’ \ ’\backslash n’ \ < sh.txt \ | \ tr \ A\text{--}Z a\text{--}z \ | \ sort \ | \ uniq \ \text{--}c \ | \ sort \ \text{--}n \ \text{--}r  \newline
\newline
Where \ in \ each \ additional \ step: \newline
1) \ Words \ are \ tokenized \ per\text{--}line \newline
2) \ Sorts \ words \ alphabetically, \ displays \ instance \ counts \newline
3) \ Collapses \ uppercase \ letters \ to \ lowercase \newline
4) \ Sorts \ by \ frequency
$$


Simple Unix Tokenization Commands

Unix commands such as tr, sort, and uniq can be used for simple normalization, tokenization, and frequency computation.

University of California, Santa Cruz

The process of text normalization includes:
- Tokenizing (segmenting) words 
- Normalizing word formats (case folding/lemmatization/stemming)
- Segmenting sentences

Process of text normalization

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Tokenization is the process of converting a sequence of text into smaller units, known as tokens. It is a foundational step in Natural Language Processing, and there are numerous different methods and strategies for how a text can be tokenized.

Tokenization

Punctuation, like periods, question marks, and exclamation points, are the most useful sentence-boundary markers. However, punctuation like periods are ambiguous among a sentence boundary marker, a number marker, and a marker of abbreviations. For this reason, sentence tokenization and word tokenization may be addressed jointly.

Sentence segmentation

Word normalization is the task of unifying words/tokens to a single standard format. The methods below are widely used depending on the tasks.

- Case folding
- Lemmatization
- Stemming

Learn Before

Related

Learn After