The universal approximation theorem means that regardless of what function we are trying to learn, we know that a large feedforward network will be able to represent this function. We are not guaranteed, however, that the training algorithm will be able to learn that function.

Even if the feedforward network is able to represent the function, learning can fail for two different reasons. First, the optimization algorithm used for training may not be able to find the value of the parameters that correspondsto the desired function. Second, the training algorithm might choose the wrong function as a result of overfitting. There is no universal procedure for examining a training set of specific examples and choosing a function that will generalize to points not in the training set.