Positional Encoding, denoted as $PE(i)$, is a technique used to inject information about the relative or absolute position of tokens in a sequence. The encoding for a specific position $i$ is represented as a vector in a d-dimensional real number space. This is formally expressed as $PE(i) \in \mathbb{R}^d$, meaning each position is mapped to a unique high-dimensional vector.

Positional Encoding Vector

A data scientist is preparing a dataset to train a model that predicts customer churn. For each of the 50,000 customers in the dataset, they record three features: 'account age in months', 'average monthly spending', and 'number of support tickets filed'. If a single customer is denoted as a data point  $\mathbf{x}_i$, which of the following correctly describes its mathematical representation?

Based on the following scenario, break down the notation $\mathbf{x}_i \in \mathbb{R}^d$ and explain what $\mathbf{x}_i$, $d$, and $\mathbb{R}$ each represent in this specific context.

Analyzing a Text Representation

A language model is designed to understand text by converting each word into a numerical representation. For this particular model, every word is transformed into a list of 300 numbers, with each number capturing a different aspect of the word's meaning. Using the standard mathematical notation, describe the complete representation for a single, arbitrary word vector (denoted as $\mathbf{x}$) and briefly explain what each part of your notation signifies in this context.

Describing a Word Vector

To formally indicate that a vector $$\mathbf{x}$$ contains $$n$$ elements, it is written as $$\mathbf{x} \in \mathbb{R}^n$$. In this notation, $$n$$ defines the dimensionality of the vector, signifying that it belongs to the mathematical space of all $$n$$-dimensional vectors with real-valued components.

Google

Claude

Each of these terms are called *slack-variables*. 
For a given training row $$\bar{x}^{(i)}$$, if:

- If $$\xi_i = 0$$, then  $$\bar{x}^{(i)}$$ is correctly classified, and is outside of the margin.
- If $$\xi_i>0$$, then $$\bar{x}^{(i)}$$ violates the margin, and is considered a *support vector*.
- If $$\xi_i > 1$$, then $$\bar{x}^{(i)}$$ is incorrectly classified by the decision boundary. 

When solving this optimization problem with SGD, we notice that correctly classified points that lie outside of the margin don't effect the update to $\bar{\theta}$. This is why the marginal/misclassified points are known as *support-vectors*.

What is $$\xi_i$$?

A vector is an organism which transmits infection by conveying the pathogen from one host to another without causing disease itself, may be part of the infectious process.

Vector

Reference of Foundations of Large Language Models Course

Dive into Deep Learning

Dimensionality Notation for Real-Valued Vectors

Vector borne disease is a disease that is transmitted to humans, plants, and animals by any arthropod species. 

Vector Borne Diseases

Vector competence defines the ability of a particular arthropod to transmit a given virus. For example, the overall mosquito barriers and antiviral responses toward pathogens would contribute to the mosquito's vector competence.

Vector Competence

Vectorial capacity describes the basic reproductive rate of a vector borne pathogen by a particular vector species or population. The formula below describes the total number of future infectious bites from mosquitos biting an individual infectious host on a single day.

Vectorial Capacity

Climate is a important factor that determines the rate of transmission of a disease. Changes in climate can lead to:
- Geographic shifts of vector and reservoirs
- Changes in rates of development, survival, reproduction of vectors, reservoirs, and pathogens

Such changes can lead to an increase in transmission and expansion of diseases in to other regions.


Climate and Vector Borne Diseases

When vectors represent examples from real-world datasets, their scalar elements correspond to specific features or quantities with real-world significance. For instance, a vector representing a medical patient might have components corresponding to vital signs and cholesterol levels, while a vector for a loan applicant might encode income and employment history.

Real-World Significance of Vectors

The individual scalar values that comprise a vector are referred to as its elements. In mathematical and computational contexts, these are also interchangeably called entries or components.

Element of a Vector

To prevent ambiguity, deep learning terminology makes a strict distinction between 'order' and 'dimensionality'. The 'order' of a tensor refers to its number of axes, whereas 'dimensionality' exclusively refers to the number of components or elements along a specific axis (e.g., a vector's length).

Tensor Order vs. Dimensionality

In mathematics and deep learning, vectors are typically distinguished from scalars through specific typographical conventions. A standard approach is to denote a vector using a bold lowercase letter, such as $$\mathbf{x}$$, $$\mathbf{y}$$, or $$\mathbf{z}$$, which visually differentiates it from an unbolded scalar variable like $$x$$.

Typographical Conventions for Vectors

A vector, denoted by a bold lowercase letter such as $$\mathbf{x}$$, can be defined by explicitly listing its elements. This component-wise representation is an ordered sequence of scalars, commonly written as $$\mathbf{x} = (x_1, x_2, \dots, x_n)$$, where $$n$$ represents the vector's dimensionality.

Component-wise Notation for a Vector

By default, vectors are visualized as column vectors, where their elements are stacked vertically. In formal mathematical notation, a vector $$\mathbf{x}$$ with elements $$x_1, \ldots, x_n$$ is represented as a column matrix enclosed in brackets:
$$ \mathbf{x} = \begin{bmatrix} x_{1} \\ \vdots \\ x_{n} \end{bmatrix} $$
This is in contrast to row vectors, whose elements are stacked horizontally.

Learn Before

Related

Learn After