Offset regression is a standard method for spatial localization in many vision tasks, including human pose estimation, object detection, and instance segmentation. However, 
if high localization accuracy is crucial for a task, convolutional neural networks will offset regression
usually struggle to deliver.   This can be attributed to the locality of the convolution operation, exacerbated by variance in scale, clutter, and viewpoint. An even more fundamental issue is the multi-modality of real-world images. As a consequence, they cannot be approximated adequately using a single mode model.   Instead, we propose to use mixture density networks (MDN) for offset regression, allowing the model to manage various modes efficiently and learning to predict full conditional density of the outputs given the input. On 2D human pose estimation in the wild, which requires accurate localisation of body keypoints, we show that this yields significant improvement in localization accuracy. In particular, our experiments reveal viewpoint variation as  the dominant  multi-modal factor. Further, by carefully initializing MDN parameters, we do not face any instabilities in training, which is known to be a big obstacle for widespread deployment of MDN. The method can be readily applied to any task with a spatial regression component. Our findings  highlight the multi-modal nature of real-world vision, and the significance of explicitly accounting for viewpoint variation, at least when spatial localization is concerned.
 Training deep neural networks is a non-trivial task in many ways. Properly initializing the weights, carefully tuning the learning rate, normalization of weights or targets, or using the right activation function can all be vital for getting a network to converge at all. From another perspective, it is crucial to carefully formulate the prediction task and loss on top of a rich representation to efficiently leverage all the features learned. For example, combining representations at various network depths has been shown to be important to deal with objects at different scales Newell et al. (2016) ; Lin et al. (2017) ; Liu et al. (2016) . For some issues, it is relatively straightforward to come up with a network architecture or loss formulation to address them -see e.g. techniques used for multi-scale training and inference. In other cases it is not easy to manually devise a solution. For example, offset regression is extensively used in human pose estimation and instance segmentation, but it lacks high spatial precision. Fundamental limitations imposed by the convolution operation and downsampling in networks, as well as various other factors contribute to this -think of scale variation, variation in appearance, clutter, occlusion, and viewpoint. When analyzing a standard convolutional neural network (CNN) with offset regression, it seems the network knows roughly where a spatial target is located and moves towards it, but cannot get precise enough. How can we make them more accurate? That's the question we address in this paper, in the context of human pose estimation. Mixture density models offer a versatile framework to tackle such challenging, multi-modal settings. They allow for the data to speak for itself, revealing the most important modes and disentangling them. To the best of our knowledge, mixture density models have not been successfully integrated in 2D human pose estimation to date. In fact, our work has only become possible thanks to recent work of Zhou et al. (2019a) proposing an offset based method to do dense human pose estimation, object detection, depth estimation, and orientation estimation in a single forward pass. Essentially, in a dense fashion they classify some central region of an instance to decide if it belongs to a particular category, and then from that central location regress offsets to spatial points of interest belonging to the instance. In human pose estimation this would be keypoints; in instance segmentation it could be extreme points; and in tracking moving objects in a video this could be used to localize an object in a future frame Zhou et al. (2019b) ; Neven et al. (2019) ; Novotny et al. (2018) ; Cui et al. (2019) . This eliminates the need for a two stage top-down model or for an ad hoc post processing step in bottom-up models. The former would make it very slow to integrate a density estimation method, while for the latter it is unclear how to do so -if possible at all. In particular, we propose to use mixture density networks (MDN) to help a network disentangle the underlying modes that, when taken together, force it to converge to an average regression of a target. We conduct experiments on the MS COCO human pose estimation task Lin et al. (2014) , because its metric is very sensitive to spatial localization: if the ground truth labels are displaced by just a few pixels, the scores already drop significantly, as shown in top three rows of Table 4 . This makes the dataset suitable for analyzing how well different models perform on high precision localization. Any application demanding high precision localization can benefit from our approach. For example, spotting extremely small broken elements on an electronic board or identifying surface defects on a steel sheet using computer vision are among such applications. In summary, our contributions are as follows: • We propose a new solution for offset regression problems in 2D using MDNs. To the best of our knowledge this is the first work to propose a full conditional density estimation model for 2D human pose estimation on a large unconstrained dataset. The method is general and we expect it to yield significant gains in any spatial dense prediction task. • We show that using MDN we can have a deeper understanding of what modes actually make a dataset challenging. Here we observe that viewpoint is the most challenging mode that forces a single mode model to settle down for a sub-optimal solution. We have shown mixture density models significantly improve spatial offset regression accuracy. Further, we have demonstrate that MDNs can be deployed on real world data for conditional density estimation without facing mode collapse. Analyzing the ground truth data and revealed modes, we have observe that in fact MDN picks up on a mode, that significantly contributes to achieving higher accuracy and it can not be incorporated in a single mode model. In the case of human pose estimation, it is surprising that viewpoint is the dominant factor, and not the pose variation. This stresses the fact that real world data is multi-modal, but not necessarily in the way we expect. Without a principled approach like MDNs, it is difficult to determine the most dominant factors in a data distribution. A stark difference between our work and others who have used mixture models is the training data. Most of the works reporting mode collapse rely on small and controlled datasets for training. But here we show that when there is a large and diverse dataset, just by careful initialization of parameters, MDNs can be trained without any major instability issues. We have made it clear that one can actually use a fully standalone multi-hypothesis model in a real-world scenario without the need to rely on an oracle or postponing model selection to a downstream task. We think there is potential to learn more finer modes from the dataset, maybe on the pose variance, but this needs further research. Specially, it will be very helpful if the role of training data diversity could be analysed theoretically. At the same time, the sparsity of revealed modes also reminds us of the sparsity of latent representations in generative models Xu et al. (2019) . We attribute this to the fact that deep models, even without advanced special prediction mechanism, are powerful enough to deliver fairly high quality results on the current datasets. Perhaps, a much needed future direction is applying density estimation models to fundamentally more challenging tasks like the very recent large vocabulary instance segmentation task Gupta et al. (2019) .
We use mixture density networks to do full conditional density estimation for spatial offset regression and apply it to the human pose estimation task.
{"src": "scitldr-AIC_train", "paper_id": "ByeYOerFvr"}