An approach for dealing with the pooling layer issue that works as follows: 

1. The stride of the max-pooling operation is constrained to be equal to the width of the pooling region. 
2. The maximum input within each pooling region is assumed to be the input in the upper-left corner. 
3. All nonmaximal inputs within each pooling region are assumed to be zero.

The inverse un-pooling operation allocates a tensor of zeros, then copies each value from spatial coordinate $i$ of the input to spatial coordinate $i\times k$ of the output, where $k$ is an integer that defines the size of the pooling region.

Un-pooling

Rich details must be added as the representation of the image to be generated moves through the network. Information is discarded primarily through the pooling layer, however, since the network needs to add information, we can't add the inverse of a pooling layer, so we instead increase the spatial size of the representation.

University of Michigan - Ann Arbor

A generator network using a convolutional structure is often useful for generating images. We utilize the "transpose" of the convolution operator, often yielding more realistic images using fewer parameters.

Convolutional Generative Networks

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Learn Before

Related

Learn After