Deep Energy: Using Energy Functions for Unsupervised Training of DNNs

The success of deep learning has been due in no small part to the availability of large annotated datasets. Thus, a major bottleneck in the current learning pipeline is the human annotation of data, which can be quite time consuming. For a given problem setting, we aim to circumvent this issue via the use of an externally specified energy function appropriate for that setting; we call this the"Deep Energy"approach. We show how to train a network on an entirely unlabelled dataset using such an energy function, and apply this general technique to learn CNNs for two specific tasks: seeded segmentation and image matting. Once the network parameters have been learned, we obtain a high-quality solution in a fast feed-forward style, without the need to repeatedly optimize the energy function for each image.

the computer vision and image processing literature is replete with examples of such functions for solving e.g., depth from stereo [15,16], super-resolution [17], single-image dehazing [18] and optical flow estimation [19]. These energies typically incorporate terms which ensure fidelity to the observed data, as well as some sort of signal prior. With this in mind, the main thrust of this paper is as follows: given such an energy function, we show how to formulate learning on an entirely unlabelled dataset. Thus, one can save immensely on the cost of annotation, by dispensing with it entirely. Alternatively, with the proposed scheme, one could offer a possible fusion with supervised learning. The outcome of our method can be used as initial near-perfect automatic annotations, later to be manually adjusted in a much shorter time in order to create labeled DB for supervised learning. Also, if a small group of labeled data is given, this could be fused as well to boost the overall scheme.
In this context, a natural question might be: given the energy function, why bother learning at allwhy not simply optimize the energy at run-time? There are two main answers to this question. The first is speed: once the network has been learned, the solution can be obtained by an almost instant forward-pass, which is typically orders of magnitude faster than explicit optimization. The second answer relates to regularization, from two points of view. Just as in [20], the mere choice of a specific CNN architecture to represent the prediction acts as a successful regularizer. Further, having the network fit its solution to a variety of images during training acts as an additional regularization.
The main contributions of this paper are as follows. First, we propose the Deep Energy approach for learning on unlabelled datasets (Section 3.1). Second, we show how to apply this formalism to two example problems, seeded segmentation (Section 3.2) and image matting (Section 3.3). Third, we learn CNNs for each of these problems on unlabelled datasets, and show performance which often exceeds the analytical solution to the corresponding explicit energy minimization (Section 4).

Related Work
Training CNNs via Energy Minimization Our approach is inspired by recent work on image style transfer. Given "content" and "style" images, the objective of [21] is to create a new image of the same content, in the style of a famous painting. The key to this method is the use of an energy function, which includes two terms. The first, a data fidelity term that forces the new image to be close to the content image in terms of a perceptual loss. The second, a "prior" or "physical" term, which promotes closeness in terms of second-order statistics between the newly-created and style images. Instead of minimizing this energy function for each input image, the works reported in [22,23] propose training a fully convolutional network to do so. During the training, as opposed to being given pairs of regular and stylized images and training the network in a standard supervised fashion, the authors proceed in an unsupervised way, via direct minimization of the energy function previously described. By the end of the training process, the network has learned to transform any input image into a stylized artistic image, by a fast and simple forward-pass operation.
Another line of work which is somewhat related to our approach is joint training of neural networks and graphical models [24,25]. The work reported in [24] tackles the task of single image depth estimation, using a neural network that is trained to provide the unary and pairwise potentials for a continuous CRF (Conditional Random Field). As opposed to supervised training, which regresses a neural network over ground-truth depth maps, their training of the unary and pairwise sub-networks involves an explicit minimization of a log-likelihood expression, modeling the relations between neighboring pixels. At test time, the two sub-networks provide the potentials that are fed to a closed-form analytic solution of the objective function used during training. In [25], the authors propose a hybrid between CNNs and MRFs (Markov Random Fields) in order to solve the human pose estimation problem. Their solution consists of two networks, the first, a supervised feed-forward part-detector providing rough heat-map estimations, and the second, a high-level spatial-model to enforce global pose consistency, formulated via CRFs. While these methods concentrate on specific applications and energy functions, we provide a broad end-to-end formulation.
Seeded Segmentation Seeded segmentation has enjoyed a considerable amount of attention with the introduction of the popular graph-cuts [26,27] and random-walker [28] algorithms. With the emergence of deep learning and fully annotated datasets such as Pascal VOC 2012 [8] and Imagenet [6], the focus has shifted to fully automatic semantic segmentation [29]. A recent attempt to alleviate the burden of per-pixel image annotation has been proposed in [7]. They suggest weak supervision in the form of user-provided points for the task of semantic segmentation. Their approach however does not originate from an energy minimization perspective and the weak supervision is only applied during training. At test time, regular semantic segmentation is performed.
Image Matting There have been numerous proposals for tackling the image matting problem. Since this problem is ill-posed, an additional regularization in the form of user interaction is required. Usually the user must input an additional "trimap", a rough segmentation of the image containing foreground, background and unknown pixels. The unknown pixels lie in the border between the object and the background, and usually contain challenging features such as hair, fur, semi-transparent objects, etc. In order to successfully output an accurate image matte, some iterative, often non-linear, optimization is involved [30,31,32,33,34,35]. Several deep learning-based techniques for image matting [9,36,37] have been proposed over the past two years. The work in [36] uses a CNN to learn a trimap, to be later fed as input to an analytic solver. In [37], several analytic solvers are used to create output alpha mattes for supervised training of a CNN. As opposed to these works, we circumvent the use of an analytic solver and directly train the network via an appropriate objective function. In [9] a large dataset for supervised image matting is created, using a composition of interesting matting objects over natural scene backgrounds. Using this dataset, end-to-end image matting is performed. This method provides impressive results on simulated image-matting benchmarks, however it may suffer from domain-shift when applied to real-life images.

The General Approach
Suppose that given data x, our goal is to compute the corresponding label y = y(x); it would be instructive to think of the label y as another signal, rather than a discrete label as in classification. We assume the existence of an energy function E(x, y) for which This formulation is quite standard in signal processing, image processing, and computer vision; it is the essence of an inverse problem approach [38,39,40].
Our idea is to learn a network which will effectively solve problem (1). In particular, consider a network parameterized by parameters θ, such that the function y = y(x) is hypothesized to look like y = y p (x; θ), where p stands for "prediction". Thus, learning how to solve the inverse problem is equivalent to learning the parameters θ. Assuming that our unlabelled training data consists of N examples of the datum X , {x n } N n=1 , in order to learn θ we propose to solve: We essentially learn the network so that, averaged over all samples, the energy function is minimized (with respect to the network parameters). This is the deep energy approach: we learn a prediction function y p (x; θ) without seeing any labelled data, via an externally specified energy function.
The proposed scheme can serve any application that can be posed in terms of an informative energy function, for which there is a difficulty in collecting large-scale data for supervised learning. Examples for such tasks are single image dehazing [18], optical flow [41], single image depth estimation [42], image retargetting [43], Retinex [44] and more. We turn now to demonstrate two specific such applications of Deep Energy: seeded segmentation and image matting.

Application: Seeded Segmentation
Background The first application we consider is seeded segmentation, in which the input consists of both an image and user-provided seeds, and the output is a segmentation of the image. These seeds, in the form of points or scribbles, provide the algorithm with an additional cue in computing the segmentation. Seeded segmentation has enjoyed a certain amount of popularity within the medical imaging community [45,46], given the fact that it allows the user a degree of control over the final result. From our point of view, seeded segmentation is an interesting test-bed for our Deep Energy approach, given the fact that there are standard energy functions for performing this task.
In particular, we use the energy function of the well-known Random Walker algorithm of Grady et al. [28], which we now briefly detail. The image is represented as a weighted undirected graph G = (V, E, W ), where the vertices V are the pixels, and edges E are between each pixel and its 4-neighborhood. The weight of edge e ij is given by w ij = e −β(Ii−Ij ) 2 , where I i , I j are the grayscale values of the pixels i, j and β is a global scaling parameter.
The idea behind the random walker is the following: the probability of a random walk starting from a given pixel and reaching each one of the seeds/classes in the image, is also the final probability the pixel receives for that class. Formally, let y l ∈ R N be the probability of each pixel in the image to belong to the label of class l ∈ {1..L}, where N = |V | is the number of pixels in the image. We denote x l ∈ R N as the "seed image" for the l th class, whose elements are 0 or 1, indicating absence or presence of a seed respectively. Y, X ∈ R N ×L are their respective vectorized versions over L classes. Then, one can solve for the random walk probabilities by minimizing the following energy: where Q = diag( l x l ) ∈ R N ×N is a matrix whose diagonal elements indicate the presence of a seed (of any class) at a given pixel; and L is the Laplacian of the graph. The first term is a smoothness term, ensuring that the segmentation is spatially smooth, though discontinuities are penalized less where the underlying image is discontinuous. The second term encourages fidelity to the seeds.
Tensorization One can directly use the energy expression in Equation (3) as a loss function during training. We now show how to convert this function to a form which is friendlier for the tensorprocessing which is common in deep learning packages. We rewrite Equation (3) as using a common expansion of the Laplacian quadratic form. Now, concatenate C identical copies of the output Y ∈ R N ×L along the last dimension to createỸ ∈ R N ×L×C , where C is the number of neighbors (C = 4 in our case). Further, formỸ η ∈ R N ×L×C , as a concatenation of the C "neighbor images" of the output y l ; in the case of 4-neighborhoods, the neighbor images are simply the image y l shifted left, right, up and down. Finally, the weights can be represented as an N × C matrix; we then take L copies of this matrix, and put them in the 3-tensor W ∈ R N ×L×C . Then the energy function can be written as follows: where we have summed over the batch dimension, b = 1...B; and the powers are taken elementwise.

Application: Image Matting
Background A second energy function which will illustrate the Deep Energy approach comes from the problem of image matting. In this task, one extracts an object from its background by determining the opacity and color of each pixel in the foreground. The input is an RGB image I, which is a composition of the foreground image, F, and the background image, B. Each pixel i in the image can be described by the matting equation: where α i ∈ [0...1] is the opacity of each pixel, also called the alpha matte. Note that recovering the alpha matte from I is extremely ill-posed. One has to deduce seven quantities per each pixel (the R, G, and B values for F i and B i , and the value of α i ).
In [30] the authors propose a closed-form solution to the image matting problem, with the help of minimal user interaction, again in the form of seeds. They derive a cost function using local smoothness assumptions on the foreground and background images, and show that it is possible to entirely eliminate the dependence on these images, thus allowing a closed-form global solution of the alpha matte. In particular, one may minimize the following energy:  Figure 1: System architecture. At input, our fully-convolutional network receives concatenated pairs of images and seeds. Apart from the input and output layers, our network is a cascade of "dilated residual blocks" (denoted by conv_bx, where x is the maximum dilation factor), which gradually increases the receptive field. The network's prediction, along with the input image and seeds, are fed to the deep energy loss, which essentially approximates the task-specific objective function.
where Q is a diagonal matrix whose diagonal elements indicate the presence of a seed -either foreground or background -at a given pixel, and x ∈ R N is the seed image, whose elements are 0 or 1, indicating foreground or background respectively. Note that if a pixel i is not a seed, x i can take on any value, as it will be zeroed out by the multiplication by Q. Finally, the matrix L is a special Laplacian-like matrix, specific to the matting problem; as derived in [30], it is given by where p n is a patch around pixel n; µ n ∈ R 3×1 and Σ n ∈ R 3×3 are the mean and covariance of the patch; and Id 3 ∈ R 3×3 is the identity matrix.
Tensorization We again rewrite the energy function in Equation (7) in a tensor-friendly format. Rephrasing the first term in (7) in terms of weights, we have that where we sum over all overlapping patches around N pixels in the resulting alpha matte, as well as over all possible combinations of pixel pairs i, j in a given 3 × 3 patch, where the total number of combinations is (3 2 ) * (3 2 ) = 81. The weights, w n i,j , are derived from the Laplacian and are given as: We can then tensorize the matting term and add the tensorized data term to get the final loss function: where k ∈ [1...K = 81] indexes the pixel pairs i, j in a 3 × 3 patch and W ∈ R B×N ×81 is the matrix of weights.α I ,α J ∈ R B×N ×81 are repetitions of the alpha matte; the first represents the alpha mattes in index i → (1, .., 1, 2, ..., 2, ..., 9, ..., 9) ∈ R 81 and the second represents the alpha mattes in index j → (1, 2, ..., 9, 1, 2, ..., 9, ..., 1, 2, ...9) ∈ R 81 . The data term of the energy function is exactly the same as that in seeded segmentation in Equation (5), apart from the minor difference that there is no need for summation over the classes l since the resulting alpha matte is already 1-dimensional, and X F ∈ R B×N and X B ∈ R B×N are simply the foreground and background seed images.

Architecture
Our network structure, shown in Figure 1, is fully convolutional and based on the Context Aggregation Network (CAN) introduced in [47]. Aggregation is performed by dilated convolutions with gradually increasing rates, that increase the receptive field and spread out the sparse seed data. We incorporate Resnet-style [48] skip connections to allow for additional gradient flow through the network layers.
More specifically, the intermediate layers consist of a cascade of dilated residual blocks. Each such block contains 2 consecutive regular conv layers, then a single dilated-conv layer with a dilation factor of 2 d where d is the block number. We add skip connections between each block's input and output as shown in fig. 1. Similar to Resnet [48], we connect the input and output by a simple addition, keeping the width of the output intact. Apart from the final layer, all convolution layers have 3 × 3 filters with an output width of 32. All layers, apart from the output layer and each block's last dilated conv layer, are followed by batch-norm and ReLU nonlinearity. The dilated-conv layers in each block are followed by batch-norm. The final conv layer is a simple linear transformation performed with 1 × 1 convolution of width equal to the size of the desired output, L. In seeded segmentation, there is an additional softmax layer that converts the output scores into probabilities.

Results
We now present the results of our deep energy loss on the seeded segmentation and image matting tasks. The results showcase the ability of deep energy learning to mimic the behavior of random-walk segmentation [28] and closed-form image matting [30].
Dataset We use the 2,913 annotated training images in the Pascal VOC dataset [8] to create 80% training and 20% test images, for both tasks discussed. It is critical to note that although these images are fully annotated with pixelwise segmentation masks, we never make use of this information; rather, we simply use the masks to generate very simple seeds, which are inputs to both seeded segmentation and image matting as previously described. We now describe this process of seed generation.
We perform binary segmentation on each PASCAL VOC image, in which we isolate a single object from the rest of the image. If there exist several objects in the original image, we create several copies of the same image with different seeds in each distinct object. By looking at only one object at a time and eliminating images with very small objects, we create ∼ 5,000 train and ∼ 1,200 test images. For seeded segmentation, we perform data augmentation in the form of flips and rotations by 45 • and 135 • , increasing the number of training images to ∼ 20,000. To generate the seeds, we add a single line or circle marker for each object by first attempting to plant it in the center of mass, or if that fails, in a random location within the object. We resize the resulting images to either 64 × 64 or 128 × 128.
Note that the Pascal VOC is not intended for image matting. There exist smaller datasets [49] that are less suitable for deep learning applications, or carefully controlled simulated datasets [9] that may not fully capture the expected behavior in the wild. We chose instead to demonstrate our approach on natural images, which are easy to obtain; and to use the minimal seeds described above rather than the complex trimaps commonly used in image matting [49,9].

Implementation Details
We implement our technique in TensorFlow on a GTX Titan-X Nvidia GPU. We use the Adam [50] optimizer with an initial learning rate of 3 × 10 −4 with an exponential decay factor of 0.96 every 10 epochs. We use a batch size of 32 and train the network for 500 epochs over the training data. In seeded segmentation, we set β = 1000, and λ = 10. We train on 64 × 64 images and cascade 6 dilated residual blocks reaching a maximum dilation factor of 32. In image matting we set λ = 1, train on 128 × 128 images and again use a maximum dilation rate of 32. The training time for both seeded segmentation and image matting is ∼ 35 hours.

Qualitative Results
We show the results of seeded segmentation on 128 × 128 resized images from the test set in Figure 2. One can see that our network approximates the analytic solution well. There are even occasions where the network solution outperforms the analytic one, when compared with the ground truth; for example, see rows 1, 2 of Figure 2. In other cases (see row 6), the network does not seem to disperse the seeds properly, a possible result of cascading limited-sized convolutions.
Since Pascal VOC is not intended for image matting, we show results in Figure 3 on images from the commonly used alphamatting.com [49] dataset, resized to 512 × 512. As opposed to our training process where we are given seeds, this dataset includes heavier user assistance in the form of trimaps. We treat the trimap as a seed image of foreground and background, with the missing gray pixels to be completed by the algorithm. Despite the very different nature of the train and test sets, the network performs quite well, demonstrating a good generalization ability. In the bottom rows, one can see blurriness in the network alpha mattes, which is a result of using a rather long chain of  repeated convolutions. This can be remedied by an additional "refinement network" as suggested in [9], adapted to the deep energy loss. We leave this important direction for future work.
Proximity to Analytic Solution Since we train the network to minimize a given energy function, we would like to verify how close it comes to actually minimizing this function, by comparing the energy values obtained. In seeded segmentation, the analytic solution is the closed-form solution of the objective function in Equation (5). We plug in the analytic and network solutions to this equation, and get two sets of values, the analytic loss and the network loss. We repeat this process for image matting and compute the analytic solution of the energy function, given in Equation (7). For reference, we also add the average loss value for a solution provided via a freshly-initialized network with random weights. The results in Table 1 show that the network loss is close to the analytic one (compared to initial "random" loss value), averaged over both train and test sets. The small gap between the network and analytic losses might be reduced through selection of different architectures, better suited for these specific tasks; or through the use of less arbitrary, more human-like seeds.  Accuracy In seeded segmentation, since our final goal is providing a segmentation map, we compare the performance of the analytic and network solutions in terms of the mIOU (mean intersection over union) metric, computed against the ground-truth segmentations of the train and test sets. The actual segmentation is performed by an argmax operation over the class dimension (consisting of foreground-background) of the output probabilities. The results in Table 1 show that the network outperforms the analytic solution in terms of mIOU (the higher the better). This result shows that although the network's only job is to mimic the analytic solution by approximating the randomwalker energy function, it learns something about natural images which boosts the results. Note further the generalization ability of our approach: the segmentation performance of the test set as measured by mIOU is almost identical to that of the train set. This can be attributed to the fact that our training gains regularization through the use of the energy as a loss, rather than using a supervised formulation. In image matting, we evaluate the average matting quality on the training images from alphamatting.com versus the ground truth alpha mattes. To quantify the performance of both solutions, we use the MSE metric proposed by [49]. As one can see from the results in Table  1, the analytic and network solutions are further apart then in seeded segmentation. This can be attributed to the fact that the network has been trained on an entirely different dataset.
Speed We now compare the speed of the analytic vs. network solutions. The analytic solution is written in fully vectorized Numpy/Scipy code and the network is implemented in TensorFlow. Table  2 shows the average evaluation time of 32 images of varying sizes using the analytic and network solutions. One can see a clear benefit in terms of speed in favor of our approach. As image dimensions grow, this increases, up to a factor of 92× in seeded segmentation and 2,660× in image matting.

Conclusion
We have introduced a new method of training CNNs by direct minimization of energy functions, which are commonly used in image processing and computer vision. We have shown application of this training technique to seeded segmentation and image matting, and demonstrated results which are comparable to, and sometimes exceed, the analytic solution. Future research will focus on fusing the deep energy loss with a standard supervised loss, to learn on datasets with few labelled examples.