Linear hidden units implies identity functions as activation units or absence of activation functions.

Linear Hidden Units

Softmax units are used in output as well as in hidden layers. These activation function represent probability distribution over a discrete variable with k possible outcomes.

Softmax Units

The function saturates for most values of x and is active when it reaches the template $W_{(;,i)}$
The function is very difficult to optimize

Radial Basis Functions

Smoother version of the rectifier activation function. Altough it is differentiable everywhere the performance of rectifier is better compared to softplus and rectifier is preferred.

Softplus

Similar to tanh and rectifier but bounded.

Hard tanh

There is a lot of research going on activation functions, new units/activation functions which vividly exhibit great performance  will get published in research papers. The common hidden units found in advanced deep learning models are. 

- Linear Hidden Units
- Softmax units
- Radial basis functions
- Softplus
- Hard tanh

San Jose State University

In the chain structure example $f (x) = f ^{(3)}(f^{(2)}(f^{(1)}(x)))$, $f^{(1)}$ is called
the first layer of the network, f^{(2)} is called the second layer, and so on. The final layer of a feedforward network is called the output layer. 

During neural network training, we drive $f(x)$ to match $f^∗(x)$. The training data provides us with noisy, approximate examples of $f^∗(x)$ evaluated at different training points. Each example x is accompanied by a label $y ≈ f^*(x)$.

The training examples specify directly that the output layer must produce a value that is close to $y$ at each point $x$. However, the behavior of the other layers is not directly specified by the training data. The training data do not say what each individual layer should do. Instead, the learning algorithm must decide how to use these layers to best implement an approximation of $f^*$. Because the training data does not show the desired output for each of these layers, they are called hidden layers.

Layers of a Feed Forward Neural Network

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

In the chain structure of a feedforward network, Let's assume the input example is $x \in \mathbb{R}^d$, then the first layer is given by
$h^{(1)} = g^{(1)} (W^{(1)T}x + b^{(1)})$
the second layer is given by 
$h^{(2)} = g^{(2)} (W^{(2)T} h^{(1)}+ b^{(2)})$
and so on.
For the output layer,
$\hat{y} = g^{(n)}(W^{(n)T} h^{(n-1)}+ b^{(n)})$

where the matrix $W^{(k)}\in \mathbb{R}^{d\times h_k}$ is the weight parameter for layer $k$ and $g^{k}$ is the activation function for layer $k$. Whether transposing $W^{(k)}$ depends on the shape of the vector.

Connection between the Layers of Neural Network

We don't count the input layer and the number of layers is defined as the number of hidden layers plus one for the output layer.
For example, in the following MLP, there are two layers.

Learn Before

Related

Learn After