Continuous Normalizing Flows (CNFs) have emerged as promising deep generative models for a wide range of tasks thanks to their invertibility and exact likelihood estimation. However, conditioning CNFs on signals of interest for conditional image generation and downstream predictive tasks is inefficient due to the high-dimensional latent code generated by the model, which needs to be of the same size as the input data. In this paper, we propose InfoCNF, an efficient conditional CNF that partitions the latent space into a class-specific supervised code and an unsupervised code that shared among all classes for efficient use of labeled information. Since the partitioning strategy (slightly) increases the number of function evaluations (NFEs),  InfoCNF also employs gating networks to learn the error tolerances of its ordinary differential equation (ODE) solvers for better speed and performance. We show empirically that InfoCNF improves the test accuracy over the baseline  while yielding comparable likelihood scores and reducing the NFEs on CIFAR10. Furthermore, applying the same partitioning strategy in InfoCNF on time-series data helps improve extrapolation performance. Invertible models are attractive modelling choice in a range of downstream tasks that require accurate densities including anomaly detection (Bishop, 1994; Chandola et al., 2009 ) and model-based reinforcement learning (Polydoros & Nalpantidis, 2017) . These models enable exact latent-variable inference and likelihood estimation. A popular class of invertible models is the flow-based generative models (Dinh et al., 2017; Rezende & Mohamed, 2015; Kingma & Dhariwal, 2018; Grathwohl et al., 2018) that employ a change of variables to transform a simple distribution into more complicated ones while preserving the invertibility and exact likelihood estimation. However, computing the likelihood in flow-based models is expensive and usually requires restrictive constraints on the architecture in order to reduce the cost of computation. Recently, introduced a new type of invertible model, named the Continuous Normalizing Flow (CNF), which employs ordinary differential equations (ODEs) to transform between the latent variables and the data. The use of continuous-time transformations in CNF, instead of the discrete ones, together with efficient numerical methods such as the Hutchinson's trace estimator (Hutchinson, 1989) , helps reduce the cost of determinant computation from O(d 3 ) to O(d), where d is the latent dimension. This improvement opens up opportunities to scale up invertible models to complex tasks on larger datasets where invertibility and exact inference have advantages. Until recently, CNF has mostly been trained using unlabeled data. In order to take full advantage of the available labeled data, a conditioning method for CNF -which models the conditional likelihood, as well as the posterior, of the data and the labels -is needed. Existing approaches for conditioning flow-based models can be utilized, but we find that these methods often do not work well on CNF. This drawback is because popular conditioning methods for flow-based models, such as in (Kingma & Dhariwal, 2018) , make use of the latent code for conditioning and introduce independent parameters for different class categories. However, in CNF, for invertibility, the dimension of the latent code needs to be the same as the dimension of the input data and therefore is substantial, which results in many unnecessary parameters. These additional but redundant parameters increase the complexity of the model and hinder learning efficiency. Such overparametrization also has a negative impact on other flow-based generative models, as was pointed out by (Liu et al., 2019) , but is especially bad in the case of CNF. This is because the ODE solvers in CNF are sensitive to the complexity of the model, and the number of function evaluations that the ODE solvers request in a single forward pass (NFEs) increases significantly as the complexity of the model increases, thereby slowing down the training. This growing NFEs issue has been observed in unconditional CNF but to a much lesser extent (Grathwohl et al., 2018) . It poses a unique challenge to scale up CNF and its conditioned variants for real-world tasks and data. Our contributions in this paper are as follows: Contribution 1: We propose a simple and efficient conditioning approach for CNF, namely InfoCNF. Our method shares the high-level intuition with the InfoGAN (Chen et al., 2016) , thus the eponym. In InfoCNF, we partition the latent code into two separate parts: class-specific supervised code and unsupervised code which is shared between different classes (see Figure 1) . We use the supervised code to condition the model on the given supervised signal while the unsupervised code captures other latent variations in the data since it is trained using all label categories. The supervised code is also used for classification, thereby reducing the size of the classifier and facilitating the learning. Splitting the latent code into unsupervised and supervised parts allows the model to separate the learning of the task-relevant features and the learning of other features that help fit the data. We later show that the cross-entropy loss used to train InfoCNF corresponds to the mutual information between the generated image and codes in InfoGAN, which encourages the model to learn disentangled representations. We have developed an efficient framework, namely InfoCNF, for conditioning CNF via partitioning the latent code into the supervised and unsupervised part. We investigated the possibility of tuning the error tolerances of the ODE solvers to speed up and improve the performance of InfoCNF. We invented InfoCNF with gating networks that learns the error tolerances from the data. We empirically show the advantages of InfoCNF and InfoCNF with learned tolerances over the baseline CCNF. Finally, we study possibility of improving large-batch training of our models using large learning rates and learned error tolerances of the ODE solvers.
We propose the InfoCNF, an efficient conditional CNF that employs gating networks to learn the error tolerances of the ODE solvers  
{"src": "scitldr-AIC_train", "paper_id": "SJgvl6EFwH"}