---
abstract: 'Recent work has shown that convolutional neural networks (CNNs) can be used to estimate optical flow with high quality and fast runtime. This makes them preferable for real-world applications. However, such networks require very large training datasets. Engineering the training data is difficult and/or laborious. This paper shows how to augment a network trained on an existing synthetic dataset with large amounts of additional unlabelled data. In particular, we introduce a selection mechanism to assemble from multiple estimates a joint optical flow field, which outperforms that of all input methods. The latter can be used as proxy-ground-truth to train a network on real-world data and to adapt it to specific domains of interest. Our experimental results show that the performance of networks improves considerably, both, in cross-domain and in domain-specific scenarios. As a consequence, we obtain state-of-the-art results on the KITTI benchmarks.'
author:
- 'Osama Makansi^\*^'
- 'Eddy Ilg^\*^'
- Thomas Brox
bibliography:
- 'egbib.bib'
title: |
    FusionNet and AugmentedFlowNet:\
    Selective Proxy Ground Truth\
    for Training on Unlabeled Images
---

18SubNumber[2834]{}

Introduction
============

Like all deep learning applications that follow the supervised learning paradigm, the success of learning optical flow estimation stands and falls with the availability and quality of training data. In case of optical flow, the creation of ground-truth annotation on real images is extremely tedious and virtually impossible on large datasets. For this reason, state-of-the-art networks for optical flow estimation, such as FlowNet 2.0 [@flownet2] and PWC-Net [@PWCnet] have been trained on synthetically rendered images. These networks tend to generalize comparatively well to real images – in contrast to semantic tasks, such as object detection or semantic segmentation. This is because correspondence estimation is different from recognition and does not depend so much on the content of the images. In fact, optical flow estimation is possible without any learning, thus not requiring any training data. There is a long history of unsupervised optical flow methods that implement the concept of correspondence. These classical methods perform equally well as a state-of-the-art optical flow network, yet with significantly higher runtimes.

The advantage of learning comes in when correspondences cannot be established easily and priors are needed to make decisions. Typical examples are areas in the image that have homogeneous color (aperture problem) or areas that are occluded in the other image. Works from the pre-learning era used handcrafted regularizers [@schunck; @meminperez] and corresponding optimization heuristics to hallucinate optical flow in these areas. Learning such priors is much more elegant and also more successful: networks tend to outperform these classical techniques especially in occluded areas. However, such learning of priors is no longer independent of the image content: while basic hallucination strategies for occluded regions can be estimated from synthetic data, the hallucinated content should ideally depend on the objects in the scene. Thus, there is a domain gap between synthesized training images and real images, just like in semantic tasks. Real images are required for training. Multiple strategies have been proposed to integrate real images into the training procedure. These span from using the same unsupervised training loss for the network as is used in variational methods [@ahmadi; @unflow], over multi-task learning with an auxiliary task that allows learning from unlabelled images [@SZB17], to training on pseudo-ground-truth obtained from running an (unsupervised) variational method [@guided_flow_17].

This paper comprises two contributions. First, we present an assessment network that learns to predict the error for each of a set of flow fields generated with various optical flow estimation techniques. Then, a fused optical flow field can be trivially obtained by selecting for each pixel the flow vector with the smallest predicted error. We show that this assessment network, which we call FusionNet combines the advantages of a potentially large set of techniques and avoids their limitations. As a consequence, FusionNet yields results that exceed the performance of all methods that produced its input. Independent on how the state of the art will improve in the future, FusionNet can always benefit from these improvements. However, this comes at the cost of very large runtimes, since a whole set of partially slow methods must be run on the test image for the assessment network to assemble the final flow field. This is a show stopper for most optical flow applications.

Thus, as a second contribution, we augment a FlowNet by training it on the flow obtained with the assessment network, which now serves as proxy ground-truth. This shifts the large runtimes to the training phase, while the final network is as fast as a regular FlowNet at test time. Training data can be generated on all sorts of unlabeled videos, which allows the augmented FlowNet to learn priors from real images. This yields the currently best accuracy-runtime trade-off and enables the specialization to target domains directly on real images without tedious modeling of synthetic scenes in such domains. We show-case this with state-of-the-art results on the KITTI benchmarks.

Related Work
============

**Traditional optical flow estimation.** Optical flow estimation goes back to the works of Lucas&Kanade [@lucaskanade] and Horn&Schunck [@schunck]. Both rely on a brightness constancy term combined with a local or global smoothness assumption. Especially the variational approach of Horn&Schunck was extended by many successive works [@meminperez; @Bro04a; @pocktvl1]. While variational methods are very precise in small displacement cases, they have deficits in case of large displacements. This was taken into account by Brox et al. [@ldof], who mixed the variational method with a simple nearest-neighbor matching of local descriptors. DeepMatching [@deepmatching] elaborated on the matching, and EpicFlow [@epicflow] improved the variational refinement. FlowFields [@flowfields] builds upon EpicFlow and elaborates on the matching using a random search strategy. The present state of the art is defined by DCFlow [@dcflow] and MRFlow [@Wulff:CVPR:2017]. The accuracy of these techniques is very high and on-par or even higher than with learning based techniques. However, the combinatorial search in state-of-the-art methods leads to quite large runtimes that do not allow for interactive frame rates.

**Optical flow with supervised learning.** End-to-end learning of optical flow was pioneered by the work of Dosovitskiy et al. [@flownet], which presented the two network architectures FlowNetS and FlowNetC. The former is purely convolutional, while the latter includes an explicit correlation. The networks were trained on a simplistic dataset made from Flickr and chair images to which affine transformations were applied (FlyingChairs). Mayer et al. [@dispnet] introduced a more sophisticated 3D dataset (FlyingThings3D). Ilg et al. [@flownet2] presented a stack of networks termed FlowNet 2.0 with high accuracy and fast runtime. Ranjan et al. [@spynet] presented a network architecture that contains a spatial pyramid and runs even faster than FlowNet 2.0, but at the cost of accuracy. Sun et al. [@PWCnet] extended this idea by introducing correlations at the different pyramid levels. Their network termed PWC-Net currently achieves state-of-the-art results. Other methods combine feature learning with traditional methods: FlowFieldsCNN [@FlowFieldsCNN] uses an improved hinge embedding loss to train a Siamese architecture for feature extraction, which is then used in combination with FlowFields. PatchBatch [@PatchBatch] shows that CNN features can even be improved to a level on which plain nearest-neighbor matching performs well. DeepDiscreteFlow [@DeepDiscreteFlow] combines a local network with a context network and discrete optimization. **Optical flow with unsupervised learning components.** Ahmadi et al. [@ahmadi] proposed an unsupervised learning approach by using the brightness constancy loss from variational approaches to train a CNN. In principle, their approach replaces the Gauss-Newton step in variational optimization with back-propagation on a network representation. While coming from a fully unsupervised approach, the resulting flow fields are inferior to those of unsupervised variational techniques. Meister et al. [@unflow] proposed an additional unsupervised loss based on forward-backward consistency to train the network termed UnFlow in a completely unsupervised manner.

Several other methods introduce unsupervised losses in addition to supervised training on synthetic data. Yu et al. [@backtobasics] and Ren et al. [@dstflow] use the loss from variational approaches to refine the decoder stages of a pre-trained FlowNet. Lai et al. [@NIPS2017_6639] use a GAN approach to distinguish the optical flow estimated by the generator network from ground-truth optical flow. Sedaghat et al. [@SZB17] proposed the self-supervised auxiliary task of next frame prediction as additional loss. Like the above-mentioned works, this allows them to improve FlowNet on real-world data. The guided optical flow proposed by Zhu et al. [@guided_flow_17] uses the flow computed by a traditional, unsupervised method as proxy ground-truth to train a network in the usual supervised manner. The final network is limited by the performance of the traditional method that provided the proxy ground-truth. In contrast, we anticipate this drawback by training on flow fields produced by multiple different methods and locally selecting the best. This way, the final network can yield better results than any single method that produced the training data.

**Optical flow fusion.** The principle to locally select the best flow vector from a set of flow fields has been implemented outside the scope of deep learning. Lempitsky et al. [@lempitskyCVPR08] proposed a combinatorial optimization approach to combine flow fields from multiple methods based on a smoothness loss. Their approach was also used in MDPFlow [@mdpflow], which locally combined multiple hypotheses from coarser pyramid levels and nearest neighbor matching. In contrast to these approaches, our selection among flow vectors is based on a deep network that learns to predict directly the optical flow error rather than just selecting based on smoothness priors.

FusionNet
=========

![ Overview of the FusionNet principle. Given the input images, the optical flow is estimated with various existing methods. Each method’s optical flow estimate is used to warp the second image. The two input images, the warped image, and the flow are fed into the proposed assessment network, which is trained on predicting the error of each flow field. Finally the flow fields are merged by locally choosing the flow vector with the minimum predicted error. \[fig:fusion\_net\] ](figures/FusionNet){width="75.00000%"}

We assume that various optical flow estimation methods have different strengths and weaknesses. This does not exclude that these methods may have also many difficulties in common. However, as long as there are differences, we want to exploit these differences to choose from the method that works best on a particular problem.

To this end, we propose an assessment network that predicts the errors of the optical flow estimated by a set of existing methods, as shown in Figure \[fig:fusion\_net\] and is trained on synthetic data with available ground-truth optical flow. On first glance, this training on synthetic images looks like we are back at square one. However, the task of assessment is different from the task of flow estimation itself. First, we benefit from the information contained in the various input flow fields. Second, the assessment task may generalize more easily to other domains than the task of optical flow estimation, since it must only find ways to predict errors rather than predicting the flow field itself. The assessment network uses a typical encoder-decoder architecture with skip-connections; the architecture details are as in FlowNetS [@flownet2]. It takes the two input images into account together with the flow estimate and the second image warped by that flow. This error map predicted by the assessment network is used to optimally combine the estimated flow fields. We refer to the complete setup, as shown in Figure \[fig:fusion\_net\], as FusionNet. We investigate two different loss functions: an L1 loss and a hinge loss.

L1 Loss
-------

For training the assessment network with an L1 loss, we let the network directly estimate the pixel-wise endpoint error. Let the estimated optical flow at a certain pixel be denoted $w=(u,v)$ with the x- and the y-components $u$ and $v$. The ground-truth endpoint error $e_\mathrm{gt}$ for a pixel location is: $$e_\mathrm{gt} = \sqrt{(u - u_{\mathrm{gt}})^2+(v - v_{\mathrm{gt}})^2}.$$ Let $e$ be the error predicted by the assessment network. To improve on the predicted error, we apply back-propagation with the L1 loss: $$\mathcal{L}_1(e) = |e_\mathrm{gt} - e|.$$ In principle, one could train a separate, specialized network for each input method to be assessed. However, since we want to improve on the generalization of the assessment network, we use the same network for assessing all input flows, and rather sample the mini-batches during training from the different methods. More training details are provided in Section \[training\_details\].

Hinge Loss
----------

Directly applying an L1 loss on the error makes the network estimate the error for each method. However, for the fusion we only need to know the input methods with the lowest error. That means, the L1 loss potentially solves a harder problem than necessary to reach the actual goal[^1]. A related problem to picking the input with the smallest error is the one of designing a distance metric to match patches. This metric only needs to reflect the ranking, e.g. “A is closer to B than A is to C” [@NIPS2003_2366; @weinberger2009distance]. Many feature learning algorithms use this as a triplet loss [@dcflow; @wang01; @tripletnet; @fastnet; @wohlhart01]. With the same motivation, we use the well-known multi-class hinge loss [@multihinge; @multisvm] $$\label{joint_ass_function}
            \mathcal{L}_{\rm Margin}(e_1,...,e_N) = \sum_{i\neq j} \max(0,m+e_j-e_i),$$

where $j$ is the index of the method with the lowest error according to the ground-truth, and $m$ is the minimum margin between the best estimate and the other estimates. If the predicted best error corresponds to the true index $j$ and all other errors are at least $m$ larger than $e_j$, this loss will be zero. Otherwise, each error that is above the allowed margin will contribute to the loss. Since the network is allowed to rescale the errors, we can set $m=1$ without loss of generality. Note that the errors predicted by the network with this loss do no longer correspond to the L1 error but may be rescaled. The rescaling factor may even be different for each pixel. Obviously, the hinge loss implies joint training of the assessment network while giving all $N$ methods as input.

![ Using our FusionNet to augment a FlowNet: FlowNet and FusionNet are trained on labeled data. Subsequently, FusionNet is used to augment FlowNet with large amounts of unlabeled data. \[fig:domain\_transfer\] ](figures/DataDomains){width="35.00000%"}

Augmented FlowNet
=================

Given the FusionNet from the last section, we can apply it to any unlabelled data to estimate high-quality optical flow. However, running FusionNet is very costly, since it requires running the various, partially very slow optical flow estimation methods. In order to have fast optical flow estimation at test time, we use the optical flow fields estimated with FusionNet as proxy ground-truth in order to finetune a FlowNet, for instance, to optimize it for a specific domain or to make it run better on general real-world videos. The principle is quite straightforward and illustrated in Figure \[fig:domain\_transfer\].

Experiments
===========

We evaluated the concept on the common optical flow benchmarks, where we can quantify the improvements by the fusion and by the augmentation of FlowNet directly. In addition, we demonstrate the effect of the augmentation in a motion segmentation context.

Training Details {#training_details}
----------------

For training the assessment network, we followed the same training schedule as proposed in Ilg et al. [@flownet2] for training FlowNet, i.e., we first train on FlyingChairs for 1.2m iterations and subsequently on FlyingThings3D for 500k iterations. The augmented FlowNet is initialized with a FlowNet trained on the same schedule. We also applied the same data augmentation mechanism, i.e., a set of spatial and color transformations. The networks were implemented using the Caffe framework. The code will be made publicly available upon publication.

Datasets
--------

We used the two publicly available synthetic datasets FlyingChairs[@flownet] and FlyingThings3D[@flownet2] to train the assessment network and the initial FlowNet before augmentation. These are the two datasets for which labeled training data is available. For the unsupervised fine-tuning, we use various unlabeled datasets that we grouped to two domains: animation movies and driving.

**Animation movies.** We collected several animation movies from the Blender project[@blender] and used them for unsupervised training. For such animation movies there is the potential option to derive ground-truth optical flow, as shown in Butler et al. [@Butler:ECCV:2012] and Mayer et al. [@MIFDB16], but we did not use this option here and rather used just the unlabeled videos for training. For the evaluation in this domain we used the official Sintel benchmark dataset [@Butler:ECCV:2012].

**Driving.** Driving scenes are a popular application domain for optical flow. Thus, we selected them to make a second evaluation domain. For unsupervised training, we took approximately 100k frames from the Frankfurt part of the publicly available Cityscapes dataset [@Cordts2016Cityscapes]. For the evaluation in this domain, we used the two publicly available KITTI2012 [@Geiger2012CVPR] and KITTI2015 [@Menze2015CVPR] benchmark datasets. **Motion Segmentation.** For indirect evaluation of the optical flow on a motion segmentation task, we used approximately 32k frames from the UdG-MS19 and UdG-MS20 datasets [@udg] for unsupervised training. We evaluated the motion segmentation on the FBMS benchmark dataset [@OB14b].

FusionNet
---------

We evaluated the FusionNet with the following optical flow estimation techniques as input: LDOF [@ldof], DeepFlow [@deepmatching], EpicFlow [@epicflow], FlowFields [@flowfields], and FlowNet2 [@flownet2]. There are some very recent methods with even better performance, such as DCFlow [@dcflow], PWC-Net [@PWCnet], and MR-Flow[@Wulff:CVPR:2017], but their code was not operational in time to include them for the experiments. A nice property of FusionNet is that new methods can be integrated trivially at any time to improve results further.

Table \[tab:fusiontable\] compares FusionNet to the state-of-the-art on the common benchmark datasets. FusionNet consistently outperforms each of the techniques that have been provided as input, which demonstrates that the assessment network is able to locally select the best optical flow vectors. As a consequence, this brings it close to the most recent state of the art and would most likely outperform it if these methods were also included for selection. Table \[tab:fusiontable\] also reports the results when selecting the flow vectors based on the ground-truth error. This oracle fusion is the lower bound that FusionNet can achieve with the respective optical flow fields given as input.

The predicted error when trained with an L1 loss is shown in Figure \[fig:qualitative\_single\_ass\]. One can observe that it matches the ground-truth error quite well. Thus, in hard cases, if one of the input methods is able to estimate the motion successfully, FusionNet is able to select the best estimate. The predicted error for the margin loss is not directly interpretable due to the local scaling. However, the resulting final optical flow field is as good as the one obtained with the L1 loss. Also quantitatively comparing the L1 loss against the hinge loss, there is no significant difference.

Augmented FlowNet
-----------------

While FusionNet yields excellent optical flow that combines the best from all available methods, it requires 84 seconds per frame. In contrast, FlowNet 2.0 runs at 8 frames per second [^2]. In this section we test how far we can transfer the good results from FusionNet to a FlowNet, thus inheriting also the runtimes of the latter.

Table \[tab:AugmentedFlowNetC\] first shows the influence of the choice of the proxy ground-truth when fine-tuning a basic FlowNetC. Augmenting the FlowNet with an optical flow field that is superior to the baseline improves results, whereas inferior flow fields can decrease the performance. When using just a single proxy method, there is the dilemma of which method to choose. Feeding a random mixture of samples from various methods during fine-tuning (Rand. Mix) does not yield the best of all involved methods, but approximately the average of those. In contrast, the use of FusionNet resolves the dilemma.

We also distinguish between augmentation for a specific domain and generic augmentation. In the first case, we augment the FlowNet only on data from the respective domain, i.e., animation movies in case of Sintel and driving videos in case of KITTI; in the second case, data from both domains is used for finetuning. Specialization to a certain domain is one of the big advantages of learning-based optical flow methods, and a particular advantage for those methods that do not require supervision in that domain, as in our case.

Table \[tab:AugmentedFlowNetC\] shows that domain-specific augmentation improves results considerably on KITTI, which is a very special scenario. The error is almost cut into half. However, also the generic augmentation is not much worse, as it also benefits from the training data from the special domain, even though it is now mixed with data from another domain. Obviously, the network can figure out automatically at test time from which domain the input is from, and applies the appropriate priors from that domain. Table \[tab:AugmentedFlowNetStacks\] extends the augmentation to a stacked FlowNet and compares it to UnFlow [@unflow]. UnFlow uses an unsupervised loss, thus it can be specialized conveniently to any domain. The table shows results for UnFlow trained on CityScapes or the unlabeled data from KITTI, which outperform the supervised baseline, which was trained on synthetic data outside this domain. For better comparison to our strategy, we also report results for a semi-supervised version of UnFlow, i.e., it is initialized with a FlowNet trained on synthetic data before the unsupervised training starts.

Results show that the domain adaptation with the augmented FlowNet is clearly superior to the one of UnFlow[^3]. As we already observed in Table \[tab:AugmentedFlowNetC\], there is no significant difference between domain-specific training and training on a joint set of domains. This is also true for the stacked network.

Table \[tab:Benchmark\_results\] compares the stacked augmented FlowNet to the state of the art. On the KITTI benchmarks, the augmented FlowNet sets the new state of the art after being finetuned also with the ground truth from the KITTI training set. But also the generic version, which has not been finetuned with ground truth data yields very good results. The direct comparison to FlowNet2 quantifies the improvement on stacked networks due to the augmentation.

Interestingly, the stacked augmented FlowNet often even outperforms the FusionNet proxy. This is due to the finetuning with ground truth in case of the domain-specific network. Sometimes, also the generic network is better than FusionNet, but not consistently.

Fig. \[fig:qualitative\_augmented\_FlowNet\_Sintel\] shows some qualitative examples of the augmentation by the proxy, yet with the smaller, non-stacked network.

Finally, we also evaluated the augmented FlowNet in an application scenario. We augmented the FlowNet on data from UdG-MS19 and UdG-MS20 [@udg] and fed its optical flow into the motion segmentation approach from Keuper et al. [@KB15b]. The motion segmentation performance was evaluated on the FBMS benchmark [@OB14b]. Table \[tab:augmentedFlowNet\_motion\_seg\] shows that the adaptation to real images clearly helps a small FlowNetC to improve motion segmentation results. For the larger, stacked network, we do not see a significant improvement due to the augmentation. We attribute this to some saturation effect: the optical flow of FlowNet2 is already very good, such that other parts of the motion segmentation pipeline dominate the final result.

Conclusion
==========

In this paper, we have presented two contributions: (1) We have presented a way to assemble a high-quality flow field from a set of input flow fields being computed in an unsupervised manner using existing optical flow estimation methods. This has been achieved by training an assessment network that learns to predict the errors of the input techniques. (2) We have shown that finetuning a FlowNet on such high-quality flow fields allows for unsupervised adaptation of the network to a specific domain. With this strategy, we obtained state-of-the-art results on the KITTI benchmarks. Moreover, we have shown that this strategy is more successful on domain adaptation than a fully unsupervised approach that does not make use of any synthetic data.

Acknowledgements {#acknowledgements .unnumbered}
================

We acknowledge funding by the German Research Foundation (grant ) and the EU Horizon 2020 project Trimbot2020.

[^1]: That said, the prediction of the error could be valuable on its own right for a series of other purposes not discussed in this paper, for instance, uncertainty estimation.

[^2]: FlowNet2 runtime is reported on an Nvidia GTX1080 GPU, while the classical methods run on the CPU.

[^3]: UnFlow does not require any supervision, which makes it biologically more plausible. From the engineering perspective, however, this is irrelevant.
