logo
How it worksCoursesResearch CommunitiesBenefitsAbout Us
Schedule Demo
Learn Before
Code

Simple Unix Tokenization Commands

A command for tokenizing words, in increasing specificity: 1) tr –sc ’A–Za–z’ ’\n’ < sh.txt2) tr –sc ’A–Za–z’ ’\n’ < sh.txt ∣ sort ∣ uniq −c3) tr –sc ’A–Za–z’ ’\n’ <sh.txt ∣ tr A–Z a–z ∣ sort ∣ uniq –c4) tr –sc ’A–Za–z’ ’\n’ <sh.txt ∣ tr A–Za–z ∣ sort ∣ uniq –c ∣ sort –n –rWhere in each additional step:1) Words are tokenized per–line2) Sorts words alphabetically, displays instance counts3) Collapses uppercase letters to lowercase4) Sorts by frequencyA \ command \ for \ tokenizing \ words, \ in \ increasing \ specificity: \ \newline \newline 1) \ tr \ \text{--}sc \ ’A\text{--}Za\text{--}z’ \ ’\backslash n’ \ < \ sh.txt \newline 2) \ tr \ \text{--}sc \ ’A\text{--}Za\text{--}z’ \ ’\backslash n’ \ < \ sh.txt \ | \ sort \ | \ uniq \ -c \newline 3) \ tr \ \text{--}sc \ ’A\text{--}Za\text{--}z’ \ ’\backslash n’ \ < sh.txt \ | \ tr \ A\text{--}Z \ a\text{--}z \ | \ sort \ | \ uniq \ \text{--}c \newline 4) \ tr \ \text{--}sc \ ’A\text{--}Za\text{--}z’ \ ’\backslash n’ \ < sh.txt \ | \ tr \ A\text{--}Z a\text{--}z \ | \ sort \ | \ uniq \ \text{--}c \ | \ sort \ \text{--}n \ \text{--}r \newline \newline Where \ in \ each \ additional \ step: \newline 1) \ Words \ are \ tokenized \ per\text{--}line \newline 2) \ Sorts \ words \ alphabetically, \ displays \ instance \ counts \newline 3) \ Collapses \ uppercase \ letters \ to \ lowercase \newline 4) \ Sorts \ by \ frequency

0

1

Updated 2021-09-19

Contributors are:

Len Morelos-Zaragoza
Len Morelos-Zaragoza
🏆 8

Who are from:

University of California, Santa Cruz
University of California, Santa Cruz
🏆 8

References

Tags

Data Science

Related