In machine learning, one of the optimization purpose is to reduce the error and get the best parameters for the machine learning model. By using a vectorized implementation in an optimization algorithm, we can make the process of computation much faster. The code also becomes simpler and easy to debug. A basic example of optimization algorithm is gradient descent.

Why is vectorization important in Machine Learning?

Given two vectors $$\mathbf{x}, \mathbf{y} \in \mathbb{R}^d$$, their dot product, commonly denoted as $$\mathbf{x}^	op \mathbf{y}$$ and also referred to as the inner product $$\langle \mathbf{x}, \mathbf{y} angle$$, is computed as the sum of the products of their elements at corresponding positions. This fundamental operation results in a scalar value and is mathematically defined by the formula: $$\mathbf{x}^	op \mathbf{y} = \sum_{i=1}^{d} x_i y_i$$.

Dot Product

The computational advantage of vectorization can be demonstrated by comparing the time taken to add two high-dimensional vectors. For instance, when adding two $$10,000$$-dimensional vectors of all ones, using an explicit element-by-element for-loop in Python is significantly slower than using the vectorized elementwise addition operator (e.g., $$+$$). The vectorized operation leverages optimized linear algebra libraries to perform the entire addition simultaneously, yielding an order-of-magnitude speedup and dramatically faster execution times compared to the loop-based method.

Example: Computational Speedup of Vectorized Addition

When training machine learning models, processing whole minibatches of examples simultaneously is highly desirable for computational efficiency. Achieving this concurrent processing efficiently requires vectorizing the calculations to leverage fast linear algebra libraries, rather than relying on sequentially executed and costly Python for-loops.

Vectorization for Minibatch Processing

Beyond computational speedups, vectorizing code yields significant software engineering advantages. By pushing complex mathematical operations to highly optimized linear algebra libraries, developers reduce the number of calculations they must write manually from scratch. Delegating these mathematics to libraries decreases the potential for implementation errors and increases the overall portability of the code.

Software Engineering Benefits of Vectorization

Vectorization is a computational technique used to replace explicit, element-by-element loops (such as Python for-loops) with optimized operations on entire data structures simultaneously. In machine learning, vectorization leverages highly optimized linear algebra libraries to perform mathematical operations—like calculating the elementwise sum of vectors or processing whole minibatches of examples—at once. This approach dramatically reduces execution time by yielding order-of-magnitude speedups, simplifies code, reduces the potential for bugs, and increases portability by offloading mathematics to underlying libraries.

University of Michigan - Ann Arbor

Claude

- Values 
- Operators & Operands
- Functions
- Statements
- Expressions
- Debugging
- Classes and Objects
- Data Structures

Python Programming Language Foundational Concepts 

In deep learning data representation, when data requires only a single axis, it is structured as a one-dimensional tensor. This specific form of a tensor is formally referred to as a vector.

Vector (1D Tensor)

You’ll use the linalg and sparse modules. Note that scipy.linalg contains and expands on numpy.linalg. 
```
from scipy import linalg, sparse
```

Linear Algebra

https://www.coursera.org/learn/neural-networks-deep-learning?specialization=deep-learning

Neural Networks and Deep Learning

Dive into Deep Learning

A value is any thing in the form of a number, string, character, etc. that can be manipulated. Values can also be referred to as objects. 

Values in Python

Operators are things that represent computational processes in Python. These include +, -, *, **, //, %, and /. 
- "+" = Addition
- "-" = Subtraction
- "*" = Multiplication

- "**" = Exponent
- "//" = Divide, but only include whole number in result
- "%" = Modulo, which only displays remainder of operands 

Operands are the values being acted on by the operator. 

Operators & Operands in Python

Function can be thought of as devices that take in material, perform a specified operation on the taken-in material, then output a resulting object. 

Inputs are referred to as arguments 
End result is referred to as output or return value. 

Functions in Python

A statement is any line of instruction that the Python interpreter can potentially execute. 

Types of statements include but are not limited to:
- Variable assignment statements 
- while statements
- for statements
- if statements
- import statements 

Statements in Python 

- Expressions are any combination of data types to be evaluated. Evaluating an expression results in values/objects. Expressions can also be evaluated into or set to values. 
- Asking Python to print an expression will result in the interpreter attempting to evaluate the expression and display the result

Expressions in Python 

- General Tips & Tricks
- Common Syntax Errors
- Error types
- Error Messages

Debugging in Python

A Class allows us to group together data and functions. It provides a template which can be instantiated to create objects. Classes and Objects help improve readability and allow code reuse. 

Classes are most often used in Object Oriented Programming, as they allow programmers to bundle sets of functions and data together. 

Python Classes and Objects

Data structures facilitate the organization of data in computer memory in an efficient manner which further support faster and more flexible data manipulations.


The python interpreter offers the following built-in data structures:


- List
- Set
- Tuple
- Dictionary


Data Structures in Python

https://www.geeksforgeeks.org/vectorization-in-python/

Vectorization in python

Vectorization

A norm is a mathematical function $$\| \cdot \|$$ that maps a vector to a scalar, informally measuring its overall size or magnitude. To be a valid norm, the function must satisfy three properties for any vectors $$\mathbf{x}$$ and $$\mathbf{y}$$ and scalar $$\alpha \in \mathbb{R}$$:

1. Scaled norm: $$\left\|\alpha \mathbf{x}ight\| = |\alpha| \left\|\mathbf{x}ight\|$$
2. Triangle inequality: $$\left\|\mathbf{x} + \mathbf{y}ight\| \leq \left\|\mathbf{x}ight\| + \left\|\mathbf{y}ight\|$$
3. Nonnegativity: $$\left\|\mathbf{x}ight\| > 0$$ for all $$\mathbf{x} 
eq 0$$.

Vector Norm

```
A = np.matrix(np.random.random((2,2)))

B = np.asmatrix(b)

C = np.mat(np.random.random((10,5)))

D = np.mat([[3,4], [5,6]])
```

Creating Matrix

•INVERSE
```
A.I #inverse

linalg.inv(A) #inverse
```
•TRANSPOSE and CONJUGATE
```
A.T #transpose matrix

A.H #conjugate
```
•TRACE
```
np.trace(A) #trace
```
•NORM
```
linalg.norm(A) #Frobenius norm

linalg.norm(A,1) #L1 Norm (max column sum)

linalg.norm(A,np.inf) #L inf Norm (max row sum)
```
•RANK
```
np.linalg.matrix_rank(C) #matrix rank
```
•DETERMINANT 
```
linalg.det(A) #determinant
```
•SOLVING LINEAR EQUATIONS
```
linalg.solve(A,b) #solver for dense matrices

E = np.mat(a).T #solver for dense matrices

linalg.lstsq(F,E) #least-squares solution to linear matrix equation
```

•GENERALIZED INVERSE
```
linalg.pinv(C) #Compute the pseudo-inverse of a matrix (least-squares solver)

linalg.pinv2(C) #Compute the pseudo-inverse of a matrix
(SVD)
```

Basic Matrix Routines

 F = np.eye(3, k=1) #Create a 2X2 identity matrix

G = np.mat(np.identity(2)) #Create a 2x2 identity matrix

C[[C > 0.5]] = 0

H = sparse.csr_matrix(C) #Compressed Sparse Row matrix

I = sparse.csc_matrix(D) #Compressed Sparse Column matrix

J = sparse.dok_matrix(A) #Dictionary Of Keys matrix

E.todense() #Sparse matrix to full matrix

sparse.isspmatrix_csc(A) #Identify sparse matrix

Creating Sparse Matrices

•INVERSE
```
sparse.linalg.inv(I) #inverse
```
•NORM
```
sparse.linalg.norm(I) #norm
```
•SOLVING LINEAR EQUATIONS
```
sparse.linalg.spsolve(H,I) #solver for sparse matrices
```

Sparse Matrix Routines

```
sparse.linalg.expm(I) #Sparse matrix exponential
```

Sparse Matrix Functions

•EIGENVALUES AND EIGENVECTORS
```
la, v = linalg.eig(A) #Solve ordinary or generalized eigenvalue problem for square matrix

l1, l2 = la #Unpack eigenvalues

v[:,0] #First eigenvector

v[:,1] #Second eigenvector

linalg.eigvals(A) #Unpack eigenvalues 
```
•SINGULAR VALUE DECOMPOSITION
```
U,s,Vh = linalg.svd(B) #Singular Value Decomposition (SVD)

M,N = B.shape

Sig = linalg.diagsvd(s,M,N) #Construct sigma matrix in SVD
```
•LU DECOMPOSITION
```
P,L,U = linalg.lu(C) #LU Decomposition
```

Decompositions

```
la, v = sparse.linalg.eigs(F,1) #Eigenvalues and eigenvectors

sparse.linalg.svds(H, 2) #SVD```

Sparse Matrix Decomposition

The numpy (np) and scipy.linalg modules provide a variety of functions for matrix operations:

- Addition & Subtraction: np.add(A, D), np.subtract(A, D)
- Division: np.divide(A, D)
- Multiplication: A @ D (matrix multiplication operator), np.multiply(D, A) (element-wise), np.dot(A, D) (dot product), np.vdot(A, D) (vector dot product), np.inner(A, D) (inner product), np.outer(A, D) (outer product), np.tensordot(A, D) (tensor dot product), np.kron(A, D) (Kronecker product)
- Exponential: linalg.expm(A) (matrix exponential), linalg.expm2(A) (Taylor series), linalg.expm3(D) (eigenvalue decomposition)
- Logarithmic: linalg.logm(A)
- Trigonometric: linalg.sinm(D), linalg.cosm(D), linalg.tanm(A)
- Hyperbolic: linalg.sinhm(D), linalg.coshm(D), linalg.tanhm(A)
- Sign: np.signm(A)
- Square Root: linalg.sqrtm(A)
- Arbitrary: linalg.funm(A, lambda x: x*x)

Learn Before

Related

Learn After