Smoother version of the rectifier activation function. Altough it is differentiable everywhere the performance of rectifier is better compared to softplus and rectifier is preferred.

San Jose State University

There is a lot of research going on activation functions, new units/activation functions which vividly exhibit great performance  will get published in research papers. The common hidden units found in advanced deep learning models are. 

- Linear Hidden Units
- Softmax units
- Radial basis functions
- Softplus
- Hard tanh

Other hidden units

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Linear hidden units implies identity functions as activation units or absence of activation functions.

Linear Hidden Units

Softmax units are used in output as well as in hidden layers. These activation function represent probability distribution over a discrete variable with k possible outcomes.

Softmax Units

The function saturates for most values of x and is active when it reaches the template $W_{(;,i)}$
The function is very difficult to optimize

Radial Basis Functions

Softplus

Similar to tanh and rectifier but bounded.

Learn Before

Related