A regular expression cheat sheet

https://cheatography.com/davechild/cheat-sheets/regular-expressions/

Regular expression reference

The simplest kind of regular expression is a sequence of simple characters.
Due to the complexity and features of regular expressions, several signs are designed to better match characters while searching.

Basic Regular Expression Patterns

The order of regex operator precedence: Parenthesis > Counters > Sequences and anchors > Disjunction.

For example, the Kleene* operator applies by default only to a single character, not to a whole sequence. The whole pattern can be repeated zero or more times if we can add * after the search pattern in ().

Operator precedence hierarchy

Greedy matching: In the greedy mode (by default), regular expressions always match the largest string they can.
Non-greedy matching: The operator +? is a Kleene plus that matches as little text as possible.

Greedy/ non-greedy matching

False positive: strings that were incorrectly matched
False negative: strings that were incorrectly missed
Increasing precision and recall by fine-tuning the search patterns
(minimizing false positives or false negatives)


False positive and false negative errors for regex matching

A useful cheatsheet for regex in one picture

Regular Expression Cheatsheet

Use parentheses to store a pattern in memory.

Capture Group

Occasionally we might want to use parentheses for grouping, but don’t want to capture the resulting pattern in a register. In that case we use a non-capturing group, which is specified by putting the commands non-capturing group"?".

Non-capturing Group

There will be times when we need to predict the future: look ahead in the text to see if some pattern matches, but not advance the match cursor, so that we can then deal with the pattern if it occurs.


Lookhead Assertion

We say that the patterns are greedy in cases when regular expressions match the largest string they can.

Greedy Matching in Regular Expression

The patterns are non-greedy when regular expressions match as little text as possible. For example, the operator *? and +? both enable non-greedy matching.

Non-greedy Matching in Regular Expression

- A regular expression is "an algebraic notation for characterizing a set of strings". They are particularly useful for searching patterns in texts.
- Regular repressions helps to represent text patterns to be extracted from a document.

San Jose State University

University of Michigan - Ann Arbor

Text corpora may have different features that are needed to process differently due to 1)language; 2)genre; 3)the demographic characteristics of the writer (or speaker); 4)time. Proper text normalization is required to make sure the models are fit to the text data. Text normalization make sure that the text is changed into a suitable and excellent form for further processing.

Text normalization

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

The process of text normalization includes:
- Tokenizing (segmenting) words 
- Normalizing word formats (case folding/lemmatization/stemming)
- Segmenting sentences

Process of text normalization

Regular Expression

Done with the use of large scale pre-trained language models that use Transformer-based architecture 

Text Representation

Tokenization is a way of separating a piece of text into smaller units called tokens.

Tokenization

It's the task of determining that two words have the same root, despite their surface differences.

Lemmatization

An utterance is the spoken correlate of a sentence.

Utterance

The larger the corpora we look at, the more word types we find.
Assume $|V|$ is the number of types and $N$ is number of tokens. k and $\beta$ are positive numbers and $0< \beta <1$.  Then
$|V|= k N^{\beta}$

Herden's Law / Heaps’ Law

After loading raw text into memory, it is commonly preprocessed to standardize the format and reduce vocabulary complexity before feeding it into sequence models. A straightforward preprocessing strategy uses a regular expression to replace every non-alphabetical character (including punctuation, digits, and whitespace) with a single space, and then converts all remaining characters to lowercase. For instance, applying the pattern [^A-Za-z]+ followed by lowercasing transforms 'The Time Machine, by H. G. Wells [1898]' into 'the time machine by h g wells '. This produces a uniform stream of space-separated lowercase words, eliminating noise from punctuation and capitalization that is unnecessary for many language modeling tasks.

Simplified Raw Text Preprocessing

Text normalization means transferring words to their standard forms. Text normalization tasks include word segmentation and normalization, sentence segmentation, and stemming.

Learn Before

Related

Learn After