Rectified linear units, or ReLUs, have become a preferred activation function for artificial neural networks. In this paper we consider the problem of learning a generative model in the presence of nonlinearity (modeled by the ReLU functions). Given a set of signal vectors $\mathbf{y}^i \in \mathbb{R}^d, i =1, 2, \dots , n$, we  aim to learn the network parameters, i.e., the $d\times k$ matrix $A$, under the model $\mathbf{y}^i = \mathrm{ReLU}(A\mathbf{c}^i +\mathbf{b})$, where $\mathbf{b}\in \mathbb{R}^d$ is a random bias vector, and {$\mathbf{c}^i \in \mathbb{R}^k$ are arbitrary unknown latent vectors}. We show that it is possible to recover the column space of $A$ within an error of $O(d)$ (in Frobenius norm) under certain conditions on the  distribution of $\mathbf{b}$. Rectified Linear Unit (ReLU) is a basic nonlinear function defined to be ReLU : R → R + ∪ {0} as ReLU(x) ≡ max(0, x). For any matrix X, ReLU(X) denotes the matrix obtained by applying the ReLU function on each of the coordinates of the matrix X. ReLUs are building blocks of many nonlinear data-fitting problems based on deep neural networks (see, e.g., [20] for a good exposition). In particular, [7] showed that supervised training of very deep neural networks is much faster if the hidden layers are composed of ReLUs. Let Y ⊂ R d be a collection of signal vectors that are of interest to us. Depending on the application at hand, the signal vectors, i.e., the constituents of Y, may range from images, speech signals, network access patterns to user-item rating vectors and so on. We assume that the signal vectors satisfy a generative model, where each signal vector can be approximated by a map g : R k → R d from the latent space to the ambient space, i.e., for each y ∈ Y, y ≈ g(c) for some c ∈ R k . In this paper we consider the following specific model (single layer ReLU-network), with the weight (generator) matrix A ∈ R d×k and bias b ∈ R d : The generative model in (2) raises multiple interesting questions that play fundamental role in understanding the underlying data and designing systems and algorithms for information processing. Here, we consider the following network parameter learning problem under the specific generative model of (2) . Learning the network parameters: Given the n observations {y i } i∈[n] ⊂ R d from the model (cf. (2)), recover the parameters of the model, i.e., A ∈ R d×k such that with latent vectors {c i } i∈[n] ⊂ R k . We assume that the bias vector b is a random vector comprising of i.i.d. coordinates with each coordinate distributed according to the probability density function 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. p(·). This question is closely related to the dictionary-learning problem [16] . We also note that this question is different from the usual task of training a model (such as, [11] ), in which case the set {c i } i∈[n] is also known (and possibly chosen accordingly) in addition to {y i } i∈[n] . Related works. There have been a recent surge of interest in learning ReLUs, and the above question is of basic interest even for a single-layer network (i.e., nonlinearity comprising of a single ReLU function). It is conceivable that understanding the behavior of a single-layer network would allow one to use some iterative peeling off technique to develop a theory for the generative models comprising of multiple layers. To the best of our knowledge, the network parameter learning problem, even for single-layer networks has not been studied as such, i.e., theoretical guarantees do not exist. Only in a very recent paper [22] the unsupervised problem was studied when the latent vectors {c i } i∈ [n] are random Gaussian. The principled approaches to solve this unsupervised problem in practice reduce this to the 'training' problem, such as the autoencoders [10] that learn features by extensive end-to-end training of encoderdecoder pairs; or use the recently popular generative adversarial networks (GAN) [9] that utilize a discriminator network to tune the generative network. The method that we are going to propose here can be seen as an alternative to using GANs for this purpose, and can be seen as an isolated 'decoder' learning of the autoencoder. Note that the problem bears some similarity with matrix completion problems, a fact we greatly exploit. In matrix completion, a matrix M is visible only partially, and the task is to recover the unknown entries by exploiting some prior knowledge about M . In the case of (3), we are more likely to observe the positive entries of the matrix M , which, unlike a majority of matrix completion literature, creates the dependence between M and the sampling procedure.
We show that it is possible to recover the parameters of a 1-layer ReLU generative model from looking at samples generated by it
{"src": "scitldr-AIC_train", "paper_id": "HkxW672q8B"}