The cost function for maximum likelihood is as follows:

J(θ) = −Ex,y∼ˆpdatalog pmodel(y | x). 

The specific parameters of the cost function may change depending on the model being used, specifically a form of logpmodel.

One advantage of using maximum likelihood for the cost function is that there's no burden of designing cost functions for each model because it's automatically determined.


South Dakota State University

There are a few different types of cost functions, but for the most part the same type of cost function is used for neural networks. We usually use the principle of maximum likelihood for most cases, such as when our parametric model uses a distribution p(y | x;θ), where the relationship between the input (training data) and output (model's predictions) of the model as the cost function. A sometimes simpler approach used, where we are predicting some statistic of y conditioned on x, instead of predicting a complete probability distribution over y. 

Deep Feedforward Network Cost Functions

Learning Conditional Distributions with Maximum Likelihood

When we want to learn only one conditional statistic of y given x, instead of learning a full probability distribution p(y | x;θ), we can use a cost functional instead of just a cost function. In this way, we can view learning as choosing a function instead of choosing a set of parameters and our cost functional can be designed to have its minimum occur at a desired specific function. 

Learn Before

Related