The notation ${\mathbf{v}_{i-n_c+1}, ..., \mathbf{v}_i}$ represents a set of vectors that form a context window. In a sequence of vectors, this set includes the vector at the current position $i$ along with the preceding vectors up to a certain window size, denoted by $n_c$. The window starts at the index $i-n_c+1$ and ends at the index $i$. This notation is fundamental for models that consider local context to make predictions.

Context Window Vector Set Notation (${\mathbf{v}_{i-n_c+1}, ..., \mathbf{v}_i}$)

An algorithm processes a sequence of data (indexed from 0) by looking at a fixed-size window of items at a time. If the algorithm is currently focused on the item at index 12, and the window size is 7, what is the starting index of the window?

An algorithm processes a sequence of data by examining a fixed-size window of elements. If a particular window of data starts at index 25 and its final element is at index 31, what is the size of this window?

Determining Context Window Size

Using the formula for calculating the starting index of a context window, explain mathematically why the algorithm described in the case study is failing. What is the specific calculated starting index that causes the error?

Debugging a Sliding Window Algorithm

The formula $i - n_c + 1$ is used to calculate the starting index of a context window within a sequence. In this expression, $i$ represents the index of the current or final element of the window, and $n_c$ represents the size or length of the context window. This calculation is fundamental for algorithms that operate on fixed-size segments of sequential data, such as in sliding window attention mechanisms.

Google

Knowledge gained through sensation, experience, study, investigation or instruction, or the communication of attained knowledge.

Information

Reference of Foundations of Large Language Models Course

A theory is a broad explanatory framework that accounts for larger bodies of data and interprets phenomena. In psychological research, working with theories is a basic, essential ingredient, serving as the foundation from which specific hypotheses are derived and tested. Crucially, theories can be supported by evidence but can never be definitively proved, and disconfirming a single hypothesis does not necessarily mean that the broader theory has been disproved.

Theory

Concepts are mental categories used to group and organize various types of information, such as linguistic data, images, ideas, and memories. They function as cognitive structures or 'big ideas' that are formed by observing, categorizing, and combining specific details. This organizational system allows the mind to perceive relationships between different elements of experience, making information more organized and accessible. Concepts are influenced by semantic memory and are integral to every aspect of life, being particularly noticeable in educational settings where they are explicitly taught.

Concept

Information that is (verifiably) misleading, false, or incomplete. Misinformation can be created with or without the intent to misinform the public. 

Misinformation 

When an information seeker is presented with an abundance of possible sources which can be difficult to sift through. This can cause relevant information to a search query to be overlooked. 

Information Overload

Prototypes

- Reference Works
- Periodicals
- Websites
- OpenStax

General Knowledge References

Information References

The knowledge and skills needed for a specific activity or topic. There are multiple types of literacy including media, digital/technological, information, etc.

Literacy

There are three distinguished uses of the term:

1. Information-as-thing

2. Information-as-process

3. Information-as-knowledge

The Three Forms of Information

Information Disciplines

Information dissemination is to distribute or broadcast information. The distribution and the spreading of information of all kinds to the users.

Information Dissemination

In a straightforward implementation of a distributed summation, such as calculating $$\sum_{\mathbf{k}_{j'} \in \mathbf{K}^{[u]}} \exp(\beta_{i,j'})$$ across sequence segments, a two-step process is utilized. First, the individual summations are executed independently on their respective computing nodes (e.g., node $$u$$ computes the sum for its assigned keys). Subsequently, these partial results are gathered from the various nodes and aggregated together to produce the final, combined total.

Distributed Summation Implementation

The formula $\mathbf{k}'_i = \phi(\mathbf{k}_i)$ describes the transformation of a vector $\mathbf{k}_i$ into a new vector $\mathbf{k}'_i$ by applying a function $\phi$. In this equation, $\mathbf{k}_i$ represents the original vector, $\phi$ is the transformation function or map, and $\mathbf{k}'_i$ is the resulting transformed vector.

Vector Transformation Formula

Square brackets `[ ]` are a standard notational convention in mathematics and computer science used to enclose the elements of a matrix. A matrix is a rectangular array of numbers, symbols, or expressions organized into rows and columns. The elements are written inside the brackets to define the matrix.

Matrix Bracket Notation

Query, Key, and Value are fundamental components of attention mechanisms, which allow a model to focus on relevant parts of an input sequence.
- **Query**: Represents the current context or element for which attention is being calculated. In the provided diagram, these are the multiple blue squares.
- **Key**: A label or identifier for a piece of information stored in memory. It is compared with the query to determine relevance. In the diagram, this is the green square labeled 'key'.
- **Value**: The actual content or information associated with a key. When a key is deemed relevant by a query, its corresponding value is retrieved. In the diagram, this is the green square labeled 'value'.
The interaction involves comparing a query with a set of keys to compute weights, which are then applied to the corresponding values to produce an output.

Query, Key, and Value in Attention Mechanisms

The cumulative future reward, often called the return, represents the total reward an agent accumulates from a specific time step $t$ until the final time step $T$. It is a key metric in reinforcement learning for assessing the long-term value of actions or states. The return is calculated by summing the rewards from time step $t$ onwards, as shown by the formula: $\sum_{k=t}^{T} r_k$, where $r_k$ represents the reward received at time step $k$.

Cumulative Future Reward (Return)

In reinforcement learning, policy decisions operate under a causality constraint. This means that an action selected at a specific time step, $t$, can only impact rewards obtained at or after that time ($r_t, r_{t+1}, ...$). Rewards received prior to time $t$ are considered unchangeable or 'fixed' from the perspective of the action at $t$.

Causality in Reinforcement Learning

The 'less than' inequality is a fundamental mathematical relation that indicates one numerical value is smaller than another. It is denoted by the symbol '<'. For instance, the statement $a < b$ asserts that the value of 'a' is strictly less than the value of 'b'.

Less Than Inequality

In mathematical notation, a bar placed over a variable, such as $\bar{y}$, typically represents the average or mean value of that variable. This notation can be applied to scalars or vectors. For instance, $\bar{\mathbf{y}}$ denotes a mean vector, where the bold font indicates it is a vector. A subscript, as in $\bar{\mathbf{y}}_k$, is used to index a specific average value or vector within a set or sequence.

Average Value Notation ($\bar{y}$)

The notation $q(\hat{y}_{i+t})$ represents a function, denoted by $q$, applied to a predicted value. The argument of the function, $\hat{y}_{i+t}$, signifies the predicted value (indicated by the hat symbol over $y$) of a variable $y$ at a future time step, indexed by $i+t$.

Function of a Predicted Future Value Notation ($q(\hat{y}_{i+t})$)

A draft model is a smaller, computationally less expensive model used to generate candidate sequences or tokens. The probability distribution $Pr_q(\cdot)$ represents the likelihood of generating a specific output according to this draft model. This approach is often used in techniques like speculative decoding to accelerate inference in larger, more powerful models by having the large model only verify the draft model's predictions rather than generating tokens from scratch.

Draft Model Probability Distribution ($Pr_q(\cdot)$)

This mathematical expression defines $\mathbf{W}_h$ as a weight matrix. The notation $\in \mathbb{R}^{d \times d_h}$ indicates that this matrix is composed of real numbers and has dimensions of $d$ rows and $d_h$ columns. In the context of neural networks, such a matrix is typically used as a set of learnable parameters within a linear layer to transform input vectors.

Weight Matrix Definition ($\mathbf{W}_h \in \mathbb{R}^{d \times d_h}$)

The formula $i - n_c + 1$ is used to calculate the starting index of a sequence or window of a specific length, denoted by $n_c$, that ends at the current position $i$. For instance, this calculation determines the first element in the set of sequential vectors ${\mathbf{k}_{i-n_c+1}, \dots, \mathbf{k}_i}$.

Index Calculation for Sequence Start Position

The notation $\langle g_1 \rangle, \dots, \langle g_\kappa \rangle$ represents a sequence of $\kappa$ cyclic subgroups. Each term $\langle g_i \rangle$ denotes the cyclic subgroup generated by the element $g_i$. The ellipsis ($\dots$) indicates that the sequence continues from the first element to the $\kappa$-th element.

Sequence of Cyclic Subgroups Notation

The 'greater than' inequality, denoted by the symbol '>', is a mathematical expression indicating that one value is larger than another. For example, the expression $y_b > y_a$ states that the value of $y_b$ is greater than the value of $y_a$.

Greater Than Inequality

The notation $\hat{y}_{i+1}, \hat{y}_{i+2}, \hat{y}_{i+3}, \dots$ represents a sequence of predicted future values, often the output of a forecasting or sequence generation model. 

- The hat symbol ($\hat{\cdot}$) over the variable $y$ indicates that it is a predicted or estimated value, as opposed to an observed, ground-truth value.
- The subscript $i+t$ denotes the value at a future time step $t$ relative to the current step $i$. For example, $\hat{y}_{i+1}$ is the prediction for the next step in the sequence.

Sequence of Predicted Future Values Notation

The formula $Pr_{\hat{\theta}}(x_{i+1}|x_0, ..., x_i)$ represents the conditional probability of the next element in a sequence, $x_{i+1}$, given all the preceding elements, $x_0, ..., x_i$. The subscript $\hat{\theta}$ indicates that this probability is estimated by a model with parameters $\hat{\theta}$. This is a fundamental concept in autoregressive models, such as language models, which generate sequences one element at a time based on the history of previously generated elements.

Conditional Probability of the Next Element in a Sequence

The notation $\text{Softmax}_\mathbf{w}(\cdot)$ represents the softmax function parameterized by a set of weights, denoted by the subscript $\mathbf{w}$. This signifies that the function's output is dependent on these weights, which are typically learned during the training process of a machine learning model.

Weighted Softmax Function Notation

This formula represents a prediction function, denoted as $Predict$, which is parameterized by $\tilde{\omega}$. This function takes the output of a BERT model as its input. The BERT model itself is parameterized by $\tilde{\theta}$ and is applied to some input, indicated by the placeholder $(\cdot)$.

Parameterized Prediction Function Notation ($Predict_{\tilde{\omega}}(BERT_{\tilde{\theta}}(\cdot))$)

A large language model is trained by processing a massive collection of text from the internet. In this context, distinguish between the 'data' being used and the 'information' the model is expected to learn. Explain how one serves as the raw material for the other.

Data vs. Information in Model Training

The notation $k_{j'} \in \mathbf{K}^{[u]}$ is used to indicate that $k_{j'}$ represents a row vector within the matrix $\mathbf{K}^{[u]}$.

Row Vector Notation ($k_{j'} \in \mathbf{K}^{[u]}$)

A climate scientist reads ten peer-reviewed articles, synthesizes the data and arguments presented, and develops a new, deeper understanding of the acceleration of glacial melt. This new understanding within the scientist's mind best exemplifies which of the following?

Start Index Calculation for a Context Window

Vector prefix notation, commonly denoted as $\mathbf{X}_{\leq i}$, represents the sequence of all vectors from the beginning of a sequence up to and including the $i$-th position. For example, if $\mathbf{X}$ is a sequence of vectors, then $\mathbf{X}_{\leq i} = (\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_i)$. This notation is frequently used in autoregressive models to refer to the prefixes of key ($\mathbf{K}_{\leq i}$) and value ($\mathbf{V}_{\leq i}$) sequences that have been processed so far.

Vector Prefix Notation

The notation $\langle g_1 \rangle, \dots, \langle g_\kappa \rangle$ represents a finite sequence of $k$ elements. The elements are denoted by $g$ with subscripts from $1$ to $\kappa$. The angle brackets $\langle \cdot \rangle$ enclosing each element can have different meanings depending on the mathematical context, such as denoting an expected value, an inner product, or the subgroup generated by an element.

Sequence of Elements in Angle Brackets Notation

A user asks a large language model to explain a scientific concept. The model retrieves relevant data, synthesizes it, and generates a paragraph as a response. The user reads this paragraph and gains a new understanding. Which part of this scenario best exemplifies 'information-as-process'?

A policy, denoted by $\pi$, defines an agent's behavior by mapping states to actions. In stochastic policies, $\pi(a|s)$ represents the probability of taking action $a$ while in state $s$. Policies are central to reinforcement learning as they are the component that is optimized to maximize rewards.

Policy in Reinforcement Learning ($\pi$)

This notation, $p(\hat{y}_{i+t})$, represents the probability of a predicted future value. In this expression, $p$ denotes a probability distribution, $\hat{y}$ (y-hat) is the predicted value for a variable $y$, and the subscript $i+t$ specifies that the prediction is for a future time step $t$ relative to a starting point $i$. This notation is commonly used in sequence modeling and time-series forecasting to quantify the likelihood of a particular predicted outcome.

Probability of a Predicted Future Value Notation ($p(\hat{y}_{i+t})$)

In mathematical and statistical modeling, the notation $\hat{y}_{i+t}$ is used to represent the predicted or estimated value of a variable $y$ at a future point in time. The components are:
- $\hat{y}$: The 'hat' symbol over the variable $y$ indicates that it is an estimated or predicted value, not an observed one.
- $i$: Represents the current index or time step from which the prediction is made.
- $t$: Represents the number of steps or units of time into the future for which the prediction is made.
For example, $\hat{y}_{i+4}$ denotes the predicted value of $y$ four steps ahead of the current position $i$.

Predicted Future Value Notation ($\hat{y}_{i+t}$)

To simplify mathematical expressions, the complete two-stage process of a text classification model is often represented using the uncluttered shorthand $$ F_{\omega,\hat{\theta}}(\cdot) $$. This compact notation serves as a direct substitute for the expanded formula $$ \mathrm{Classify}_{\omega}(\mathrm{Encode}_{\hat{\theta}}(\cdot)) $$, efficiently representing a function defined by both the classifier's parameters, $$ \omega $$, and the encoder's parameters, $$ \hat{\theta} $$.

Uncluttered Notation for Encoder-Classifier Models

Data encompasses facts, figures, or information gathered for analysis, reference, or computation. It can be qualitative or quantitative and serves as the foundation for reasoning, discussion, or calculation across fields like scientific research, business, and technology. Data exists in diverse forms, including numbers, text, images, or sounds, and is processed to extract meaningful insights. In computing, data is the foundational element; all interactions revolve around reading, modifying, and creating it. Fundamentally, all computer data is stored as uniform sequences of bits, making it inherently similar regardless of its perceived form.

Learn Before

Related

Learn After