The encoder part of our network is inspired by the VGG16 [1]} that consists of five convolutional blocks and three fully connected layers.
Since our network explicitly utilizes the extracted CNNs feature maps, we only consider convolutional
layers and omit the fully connected layers. In the standard VGG16 model, with an input image having a size of \(h \times w \times 3\) , the spatial dimensions of the features generated from the last convolution layer (conv5-3) is \(\frac{h}{32} \times \frac{w}{32}\)  which is relatively small for the saliency prediction task. For preserving more spatial information of the feature map, we modify the network
structure via removing the final pooling layer (pool5). This results in an output feature blob of spatial dimensions \(\frac{h}{16} \times \frac{w}{16}\)  after the last convolution layer.
