In ESPCN [1]}, Shi et al. proposed an efficient sub-pixel convolutional layer. Instead of increasing the resolution by directly increasing the number of LR feature maps, sub-pixel first increases the dimension of LR feature maps, i.e., the number of the LR feature maps, and then a periodic shuffling operator is used to rearrange these points in the expanded feature maps to obtain the HR output (Fig. REF ). In detail, the formulation of the sub-pixel convolutional layer can be defined as follow:
\(I_{SR}=f^L(I_x)=\mathcal {PS}(W_L*f^{L-1}(I_x)+b_L),\) 
