1Cademy - Broyden-Fletcher-Goldfarb-Shanno Algorithm Rational

How it works Courses Research Communities Benefits About Us

Learn Before

BFGS Optimization Algorithm

Concept

Broyden-Fletcher-Goldfarb-Shanno Algorithm Rational

Recall that Newton's update is given by $\theta^\ast=\theta_0-H^{-1}\nabla_{\theta}J(\theta_0)$ where H is the Hessian of J with respect to $\theta$ evaluated at $\theta_0$ .
The approach adopted by BFGS algorithm is to approximate the inverse with a matrix $M_t$ that is iteratively refined by low-rank updates to become a better approximation of $H^{-1}$ .
Once the inverse Hessian approximation $M_t$ is updated, the direction of descent $\rho_t$ is determined by $\rho_t=M_tg_t$ .
A line search is performed in this direction to determine the size of the step, $\epsilon^{\ast}$ , taken in this direction. The final update to the parameters is given by: $\theta_{t+1}=\theta_t + \epsilon^{\ast}\rho_t$ .

0

1

Updated 2021-06-24

Contributors are:

XZ

Who are from:

University of Michigan - Ann Arbor

University of Michigan - Ann Arbor

Tags

Data Science

Related