Scene understanding is an important yet challenging task in computer vision, which has significant contribution to visual simultaneous localization and mapping(vSLAM) system [1]}, robot navigation [2]}, augmented reality [3]} and other applications. Two fundamental goals of scene understanding are monocular depth estimation [4]}[5]}[6]}[7]} and semantic segmentation [8]}[9]}[10]}[11]}, which have been extensively researched by utilizing deep learning. Recently, some works [12]}[13]}[14]} noticed the interactions between these two tasks and utilized the common characteristics to improve each other, which achieved great performance. However, most of them used a deep structure as encoder such as ResNet-101 [13]}, ResNet-50 [16]}, SE-ResNet [17]}, introducing a large number of downsampling and stride operations, which had negative influence [18]} towards depth estimation and semantic segmentation where fine-grained information is crucial. Despite there are also some works adopting strategies like skip-connection [17]}, up-projection [14]} and multi-scale training loss [16]} to mitigate this problem, these schemes have great demands on computation and memory.
