Vocabulary Size, Who the speaker is talking to, Channal and Noise, and Speaker-Class Characteristics

Dimensions of ASR Variation

LibriSpeech is a large open-source read-speech 16 kHz dataset with over 1000 hours of audio books from the LibriVox project.

LibriSpeech

The Switchboard corpus of prompted telephone conversations between strangers
was collected in the early 1990s.

The Switchboard corpus

The CALLHOME corpus was collected in the late 1990s and consists of 120 unscripted 30-minute telephone conversations between native speakers of English who were usually close friends or family.

The CALLHOME corpus

CORAAL is a collection of over 150 sociolinguistic interviews with African American speakers, with the goal of studying African American Language (AAL).

CORALL

The CHiME Challenge is a series of difficult shared tasks with corpora that deal with robustness in ASR.

CHiME

The HKUST Mandarin Telephone Speech corpus has 1206 ten-minute telephone conversations between speakers of Mandarin across China, including transcripts of the conversations, which are between either friends or strangers.

The HKUST Mandarin Telephone Speech corpus

The AISHELL-1 corpus contains 170 hours of Mandarin read speech of sentences taken from various domains, read by different speakers mainly from northern China.

The AISHELL-1 corpus

The first step in ASR is to transform the input waveform into a sequence of acoustic feature vectors, where each vector represents the information in a small time window of the signal.

Feature Vector

The conversion from the analog representations (first air pressure and then analog electric signals in a microphone) into a digital signal.

Analog-to-digital Conversion

The architecture for ASR is the encoder-decoder which is implemented with recurrent neural networks (LSTMs) or transformers. For this architecture, it is common to begin with log mel spectral features and map to letters. It’s also possible to sometimes map to induced morpheme-like chunks, such as word pieces or BPE (byte-pair encoding).

Automatic Speech Recognition Architecture

It's the standard evaluation metric for speech recognition systems. The word error rate is based on how much the word string returned by the recognizer (the hypothesized word string) differs from a reference transcription.

Word Error Rate

The MAPSSWE test is a parametric test that looks at the difference between the number of word errors the two systems produce, averaged across a number of segments.

The Matched-Pair Sentence Segment Word Error (MAPSSWE) test

In statistics, McNemar's test is a statistical test used on paired nominal data. It is applied to $2 \times 2$ contingency tables with a dichotomous trait, with matched pairs of subjects, to determine whether the row and column marginal frequencies are equal (that is, whether there is "marginal homogeneity").

McNemar's test

McNemar’s test is only applicable when the errors made by the system are independent, which is not true in continuous speech recognition, where errors made on a word are extremely dependent on errors made on neighboring words.

Disadvantage of McNemar's test

Intelligent assistants—such as Siri, Alexa, and Google Assistant—utilize artificial intelligence to accurately respond to spoken requests. A key driver behind their widespread adoption is advanced automatic speech recognition technology, which has improved sufficiently to achieve human parity in certain applications, enabling these systems to manage everything from simple smart home controls to complex conversational support.

Intelligent Assistants

Acoustic waves can be mathematically represented using sine and cosine functions:

$$y = A \sin(2\pi ft)$$

Two key characteristics of an acoustic wave are its amplitude ($$A$$) and frequency ($$f$$).

Acoustic Waves

Automatic speech recognition is a sequence-to-sequence learning task where the input sequence is an audio recording of a speaker and the output is a text transcript of the spoken words. A significant challenge in this domain is that there is no one-to-one correspondence between audio frames and text, as thousands of audio samples may correspond to a single word, making the input sequence much longer than the output sequence.

University of Michigan - Ann Arbor

Claude

Carleton College

A Frame-based Dialogue System defines a dialogue frame structure and the formatted involved important components.

Frame-based Dialogue System

There is no exact definition for Natural Language Processing (NLP), but we could catch a glimpse through its goal and academic classification.
- The goal of this field (NLP) is to get computers to perform useful tasks involving human language. (Speech and Language Processing, 2nd edition)
- NLP is an interdisciplinary field of linguistics, computer science, information engineering, electrical engineering, psychology, and artificial intelligence. (Speech and Language Processing, 2nd edition; Wikipedia)

Natural language processing gives a computer the ability to analyze and synthesize human languages. 

Natural language processing

Sequence-to-sequence learning is a specific subset of sequence learning problems where both the inputs and the outputs consist of variable-length sequences. In these tasks, the length of the input sequence does not necessarily match the length of the output sequence. Prominent applications include machine translation and speech-to-text transcription.

Sequence-to-Sequence Learning

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Dive into Deep Learning

It's with high human cost and hard to transfer to other domains.

Disadvantages of Frame-based Dialogue System

1. domain classification
2. intent determination
3. slot filling

Main Components Extracted by Frame-based Dialogue System

NLG Module of Frame-based System

Automatic Speech Recognition

- Information extraction
- Machine translation
- Discourse, dialogue and pragmatics
- Natural language generation
- Speech recognition
- Lexical semantics
- Phonology / morphology
- Language resources

Natural language processing in ACM Computing Classification

Book:
- Speech and Language Processing (3rd ed. draft) Written by Dan Jurafsky and James H. Martin
- Speech and Language Processing (2nd edition) Written by Dan Jurafsky and James H. Martin
Journal:
- A Survey on Dynamic Neural Networks for Natural Language Processing
- A Survey of Race, Racism, and Anti-Racism in NLP

NLP references

The most important models are:
- state machines
- rule systems
- logic
- probabilistic models
- vector-space models

These models are all drawn from the standard toolkits of computer science, mathematics, and linguistics.

Models used in NLP

Text corpora may have different features that are needed to process differently due to 1)language; 2)genre; 3)the demographic characteristics of the writer (or speaker); 4)time. Proper text normalization is required to make sure the models are fit to the text data. Text normalization make sure that the text is changed into a suitable and excellent form for further processing.

Text normalization

In corpus linguistics, part-of-speech tagging (POS tagging or PoS tagging or POST), also called grammatical tagging is the process of marking up a word in a text (corpus) as corresponding to a particular part of speech, based on both its definition and its context. A simplified form of this is commonly taught to school-age children, in the identification of words as nouns, verbs, adjectives, adverbs, etc. 

Part-of-speech Tagging

Sentiment analysis (also known as opinion mining or emotion AI) refers to the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study affective states and subjective information. 


Sentiment analysis is widely applied to voice of the customer materials such as reviews and survey responses, online and social media, and healthcare materials for applications that range from marketing to customer service to clinical medicine.

Sentiment Analysis

In machine learning and natural language processing, a topic model is a type of statistical model for discovering the abstract "topics" that occur in a collection of documents. 


Topic modeling is a frequently used text-mining tool for discovery of hidden semantic structures in a text body. Intuitively, given that a document is about a particular topic, one would expect particular words to appear in the document more or less frequently: "dog" and "bone" will appear more often in documents about dogs, "cat" and "meow" will appear in documents about cats, and "the" and "is" will appear approximately equally in both. A document typically concerns multiple topics in different proportions; thus, in a document that is 10% about cats and 90% about dogs, there would probably be about 9 times more dog words than cat words. The "topics" produced by topic modeling techniques are clusters of similar words. A topic model captures this intuition in a mathematical framework, which allows examining a set of documents and discovering, based on the statistics of the words in each, what the topics might be and what each document's balance of topics is.


Topic models are also referred to as probabilistic topic models, which refers to statistical algorithms for discovering the latent semantic structures of an extensive text body. In the age of information, the amount of the written material we encounter each day is simply beyond our processing capacity. Topic models can help to organize and offer insights for us to understand large collections of unstructured text bodies. Originally developed as a text-mining tool, topic models have been used to detect instructive structures in data such as genetic information, images, and networks. They also have applications in other fields such as bioinformatics and computer vision.

Topic Model

Parsing, syntax analysis, or syntactic analysis is the process of analyzing a string of symbols, either in natural language, computer languages or data structures, conforming to the rules of a formal grammar.



Within computational linguistics the term is used to refer to the formal analysis by a computer of a sentence or other string of words into its constituents, resulting in a parse tree showing their syntactic relation to each other, which may also contain semantic and other information (p-values). Some parsing algorithms may generate a parse forest or list of parse trees for a syntactically ambiguous input.



Simply speaking, parsing in NLP is the process of determining the syntactic structure of a text by analyzing its constituent words based on an underlying grammar (of the language).

Parsing

For natural language applications, we often want to use words as the fundamental unit of the output. However, because vocabularies can be very large (up to hundreds of thousands of words), this can be computationally expensive. Both the high memory cost and the high computational cost, which occur at training and testing time, make the standard way to approach this type of distribution have many difficulties.

High Dimensional Outputs

Natural Language Processing has been developing alongside machine learning techniques for at least since 1990. In 1986 Rumelhart et al. developed one of the first instances of symbolic encoding for a neural network like structure. In 1990 Deerwester et al. then adapted this idea of symbolic encoding into word encoding which could then in theory be used in an NLP neural network. 

From 1991 until 2001, however, encoded word processing was ignored in favor of attempts at creating a character recognizing network instead. Such a system would have to learn both grammatical structure and the spelling of every word it would use in that structure. As a result, in 2001 when Bengio et al. reintroduced word processing, it initially had a higher performance than most other character processing networks. 

Although word processing networks were initially more performant, over time both methods for creating NLP networks have improved, and continue to show signs that more improvement may be possible. As such, it is hard to determine which method will ultimately prove more effective than the other.

Historical Perspective: Natural Language Processing

Natural language documents are read and studied by computers so they can answer the questions posed.

Machine Reading and Comprehension

In NLP, we need to find out how similar the two strings are to each other. Minimum edit distance finds out the dissimilarity of two strings and counts the minimum number of modifications to transform a string to another. Levenshtein distance is the method that is used to measure the minimum edit distance.

Minimum Edit Distance

Pieces of text used to train and/or develop NLP models can vary in several ways that may need to be accounted for, such as: 
- The language the text is written in
- The type/genre of text
- The author/speaker of the text
- How old the text is

Variation Factors of Input Texts

A common NLP task that applies logistic regression- to determine if the period symbol "." is the end of a sentence or not. A feature  express a single property or a combination of properties, such as the capitalization of the current word or previous word.

Period Disambiguation

Features of the instances are designed by hand- by examining the training set with linguistic intuitions and literacy level on the domain. 

Features Design for NLP Classification Problems

Vectors semantics is the standard way to represent word meaning in NLP. The idea of vector semantics is to represent a word as a point in a multidimensional semantic space that is derived from the distributions of the term is sometimes more strictly applied only to dense vectors like word2vec, rather than sparse tf-idf or PPMI vectors. The word 'embedding' derives from its mathematical sense as a mapping form one space or structure to another, although the meaning has shifted.

Vector Semantics and Embeddings

Vector or distributional models of meaning are generally based on a co-occurrence matrix, a way of representing how often words co-occur. We'll look at two popular matrices: the term-document matrix and the term-term matrix.

Words and Vectors

While word classes do have semantic tendencies—adjectives, for example, often describe properties and nouns people— parts of speech are defined instead based on their grammatical relationship with neighboring words or the morphological properties about their affixes.

English Word Classes

Computational Desiderata for Representations:

Verifiability:

Verifiability is a system’s ability to compare the state of affairs described by a representation to the state of affairs in some world as modeled in a knowledge base. 

Unambiguous Representations:

The meaning representations itself cannot be ambiguous. The representation of an input’s meaning should be free from any ambiguity, so that the system can reason over a representation that means either one thing or the other in order to decide how to answer. Vagueness: in which a meaning representation leaves some parts of the meaning underspecified. 

Canonical Form:

The doctrine of canonical form says that distinct inputs that mean the same thing
should have the same meaning representation. Canonical form does complicate the task of semantic parsing. 


Logical Representations of Sentence Meaning

First-Order Logic (FOL) is a flexible, well-understood, and computationally tractable meaning representation language.


First-Order Logic

Information extraction is a fundamental task in NLP where many problems can be framed as extracting specific, structured information from unstructured text. This process involves identifying data points such as named entities, relationships, and events. The ultimate goal is to convert this raw textual data into a structured format suitable for analysis and utilization in various downstream applications.

Information Extraction

A sense (or word sense) is a discrete representation of one aspect of the meaning of a word.

Word Senses

Utilized as a way to express the role that arguments of a predicate take
in a event.

Semantic Roles: Labeling

Semantic roles focus about what a word actually means and is actually doing in the semantic context they do not care about the structure of the sentence.

Semantic Roles ( Thematic Roles )

Question Answering is a computer science discipline within the fields of information retrieval and natural language processing (NLP), which is concerned with building systems that automatically answer questions posed by humans in a natural language.

Question Answering

It is the process of obtaining information system resources that are relevant to an information need from a collection of those resources. 

Information Retrieval

Dialogue systems, or conversational agents, are programs that communicate with users, through text and/or speech, in natural language.

Dialogue Systems

Intricate components of natural human conversation that must be considered when creating a dialogue system

1) Turn Structure

2) Speech Acts

3) Grounding

4) Dialogue Structure

5) Initiative

6) Implicature

Properties of Human Conversation

Prompt tuning is a parameter-efficient adaptation method for Large Language Models. It involves incorporating a set of trainable vectors, or soft prompts, which are adjusted during training to steer the model for a specific task. A defining characteristic of this technique is that it only modifies the model's input embedding layer, leaving the rest of the pre-trained model's parameters frozen.

Prompt Tuning

1. Fully Supervised Learning(Non-Neural Network)
2. Fully Supervised Learning(Neural Network)
3. Pre-train, Fine-tune
4. Pre-train, Prompt, Predict

Types of NLP Model Paradigms

1. Standard Language Model(SLM)
2. Corrupted Text Reconstruction(CTR)
3. Full Text Reconstruction(FTR)

Types of Training Objectives of Pre-trained LM

1. Promptless Fine-tuning
2. Tuning-free Prompting
3. Fixed-LM Prompt Tuning
4. Fixed-Prompt LM Tuning
5. Prompt+LM Fine-tuning

Major Tuning Strategy Types

Articulatory Phonetics is an area of study that focuses on how throat, mouth and nose work together to modify the airflow from lung to produce phonetics,

Articulatory Phonetics

Phonetics is the field of study in which speech sounds from world languages are studied - how they are created in a vocal tract, how they materialize in an acoustic sense, and how they can be relayed or processed digitally. 

Phonetics

Rather than representing words as discrete variables, word embeddings map words into low-dimensional real-valued vectors. This continuous representation space makes it possible to compute the meanings of words and word $$n$$-grams. As a result of this distributed representation, language models are no longer burdened with the curse of dimensionality, allowing them to represent exponentially many $$n$$-grams via a compact and dense neural model.

Word embedding

https://arxiv-download.xixiaoyao.cn/pdf/2105.03075.pdf

A Survey of Data Augmentation Approaches for NLP

Data augmentation (DA) refers to strategies for increasing the diversity of training examples without explicitly collecting new data. Related methods are first explored in computer vision (CV), and DA’s adaptation for natural language processing (NLP) seems secondary and comparatively underexplored.

Data Augmentation in NLP

Spelling correction is often considered from two perspectives, non-word spelling correction and real word spelling correction.

Spelling correction and the noisy channel

Syntactic constituency is the idea that groups of words can act as single units, which are called constituents

Constituency

The categorization of texts into organized groups, which is usually implemented by supervised learning.
Applications: sentiment analysis, spam detection, news categorization, topic analysis, and more.

Text Classification

Information extraction aims to extract structural information from unstructured text, which is a key field within natural language processing. 

Information Extraction (IE)

https://dspace.mit.edu/bitstream/handle/1721.1/116314/10462_2017_9588_ReferencePDF.pdf?sequence=2&isAllowed=y

A Survey of Natural Language Based Financial Forecasting

Han, X., Gao, T., Lin, Y., Peng, H., Yang, Y., Xiao, C., … Zhou, J. (2020). More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extraction. doi:10.48550/ARXIV.2004.03186


More Data, More Relations, More Context and More Openness: A Review and Outlook for Relation Extraction

https://ieeexplore.ieee.org/abstract/document/9328413

A Survey of the State-of-the-Art Models in Neural Abstractive Text Summarization

From Standard Summarization to New Tasks and Beyond: Summarization with Manifold Information

Machine translation (MT) is the task of automatically translating from one language to another without human labor, which brings convenience and reduces labor cost in international exchange and cooperation.

Learn Before

Related

Learn After