18 Aug 2019

Segue from Euclidean Gradient Descent to Natural Gradient Descent

A slight change in SGD formulation, in terms of maximization of local approximation, leads to an interesting general connection to NGD via mirror descent.

#math #ml #gradient #natural-gradient #deep-learning

08 Aug 2019

A Fascinating connection between Natural Gradients and the Exponential Family

The Exponential family provides an elegant and easy method to compute Natural Gradients and thus can be used for Variational Inference.

#math #ml #gradient #natural-gradient #deep-learning