Deep Neutral Networks(DNNs) require huge GPU memory when training on modern image/video databases. Unfortunately, the GPU memory as a hardware resource is always finite, which limits the image resolution, batch size, and learning rate that could be used for better DNN performance. In this paper, we propose a novel training approach, called Re-forwarding, that substantially reduces memory usage in training. Our approach automatically finds a subset of vertices in a DNN computation graph, and stores tensors only at these vertices during the first forward. During backward, extra local forwards (called the Re-forwarding process) are conducted to compute the missing tensors between the subset of vertices. The total memory cost becomes the sum of (1) the memory cost at the subset of vertices and (2) the maximum memory cost among local re-forwards. Re-forwarding trades training time overheads for memory and does not compromise any performance in testing. We propose theories and algorithms that achieve the optimal memory solutions for DNNs with either linear or arbitrary computation graphs. Experiments show that Re-forwarding cuts down up-to 80% of training memory on popular DNNs such as Alexnet, VGG, ResNet, Densenet and Inception net. The standard DNN training process consists of two alternated stages: forward and backward. FIG0 (a) illustrates an example of feed-forward neural networks. In the forward stage, the network takes an input tensor, [BatchSize × Channel × W idth × Height], and computes the tensors at each layer until producing the output. In the backward stage, difference between the output and ground truth is passed back along the network to compute the gradients at each layer. The regular training approach saves tensors at all layers during forward, because they are all needed to compute gradients during backward. The total memory cost is the sum of cost over all layers.In popular backbone DNNs for feature extraction of images, such as AlexNet BID13 ), VGG BID22 ) and ResNet BID10 ), the memory cost increases quadratically with the input image resolution and network depth. For example, given an median size input tensor of (32, 3, 224, 224) , ResNet101 requires around 5000 MB. In more challenging tasks, DNNs that detect small objects and large number of object categories require input image resolution of more than 600 × 600 BID18 ; BID23 ; BID17 ). The memory issue is worse for video-based DNNs, such as CDC BID21 ), C3D BID12 ) and 3D-ResNet BID9 ). To model complex activities in video, the input tensor may contain 64 frames. Moreover, DNN training takes much more memory than testing. In order to train DNNs with large databases and big learning rate, the batch size can be up to 64. In training DNN compositions, such as Generative adversarial networks (GANs), multiple generator and discriminator networks are simultaneously stored in GPU memory.Existing efforts to address memory issues presented three main approaches: (1) Better single GPUs. Recent GPUs provide larger memory at the expense of exponentially growing price and power consumption. For instance, from TitanXp, Quadro P6000 to Tesla V100, for 1-2.7 times increase in memory, the prices increase 2.8-8.5 times. (2) Parallelization among multiple GPUs BID8 ; BID20 ; ; BID15 BID16 ; BID27 ; BID2 ; BID1 ), which requires expensive The regular approach saves all tensors during forward, and uses these tensors to compute gradients during backward. (b) Reforwarding (our) saves a subset of tensors during the first forward, and conducts "Re-forward" to compute tensors for gradients during backward.clusters, introduces substantial I/O cost, and does not reduce the total memory cost. (3) Low-level heuristic techniques. Optimization of computation graphs BID3 ), which merges inplace operations into non-inplace operations to cut down memory. Liveness analysis BID3 ), which dynamically recycles garbage tensors in training epochs. These approaches are specific to certain DNN structures, data and tasks.To address above issues, we propose a fundamental approach that explores trade-off between memory and computation power of GPUs. Note that recent affordable GPUs, although limited in memory ( 12GB), provide exceptional improvement in GPU cores and FLOPS. Trading computational time for memory is a very attractive solution that make it possible to train very heavy DNNs with finite GPU memory. Our approach only saves tensors at a subset of layers during the first forward, and conduct only extra local forwards to compute the missing tensors needed during backward. We call the extra forward process as Re-forwarding. The total memory cost is the sum of (1) the cost at the subset of layers and (2) the maximum memory cost among local re-forwards. Training with Reforwarding, see FIG0 (b), leads to substantial memory reduction. We propose sophisticate theories and efficient algorithms that achieve the optimal memory solution of arbitrary computation graphs. Re-forwarding is a fundamental approach that explores trade-off between memory and computation power of GPUs. By saving tensors at a subset of layers during forward, and conducting extra local forwards for backward, Re-forwarding makes it possible to train very heavy DNNs with finite GPU memory. To our knowledge, our theoretical and algorithmic results are the first top-down work that achieve an optimal memory solution for arbitrary computation graphs in DNNs. Re-forwarding can be further embedded and optimized with any low-level techniques such as distributed computing, GPU/CPU swapping, computation graph optimization and liveness analysis. Same on v q , v q must be v j or v t . As s ⊂ [s ij ), ∀v 1 ∈ s, v 1 has no edge with v 2 ∈ [s ij ). As s kj is close, ∀v 1 ∈ s, v 1 has no edge with v 2 ∈ s kj . ∀v 1 ∈ s, v 1 can only have edge with v 2 ∈ [s]. Thus the independence of s is guaranteed. Therefore, s is closed set, v k is the splitting vertex of s ij . DISPLAYFORM0 Same on v j , v j is the splitting vertex of s kt Lemma 4. If s ij has n splitting vertices {v 1 , v 2 , ..., v n }, then s ij = s i1 ∪ s 12 ∪ ... ∪ s nj ∪ {v 1 , v 2 , ..., v n } Proof. If n = 2, the splitting vertices are DISPLAYFORM1 According to Lemma 3, v 1 is splitting vertex of s i2 and v 2 is splitting vertex of s 1j . Therefore, DISPLAYFORM2 For n > 2, the lemma can be proved by repetitively using the conclusion in n = 2. Lemma 6. Any member of a maximal split can not be the subset of another closed set s s ij .Proof . Suppose the source vertex of s is v 1 and target vertex is v 2 , a member s xy of the maximal split is inside s. Suppose a member s ab of the maximal split has its source vertex v a inside s and target vertex v b outside s. Then the boundary vertex (the vertex that has edges to the non-overlapping parts of both sets) must be v 2 , otherwise the independence of s will be violated. Notice that v 2 is inside s ab and the independence of s ab needs to be guaranteed, for ∀v p ∈ s, v p / ∈ s ∩ s ab , v q ∈ s ∩ s ab , v p has no edge with v q . Therefore, v a is a splitting vertex of s.Similarly, if s ba has its target vertex v a inside s and source vertex v b outside s, the boundary vertex must be v 1 and v a is a splitting vertex of s.For the closed set s, from the discussion above, we know that there are at most 2 members of the maximal split that can overlap with s. Other members must be either completely inside s or completely outside s. Let's discuss the number of members that overlaps with s.If there are 0 member that overlaps with s, s is the union of a subset of members of the maximal split, which violates the definition of maximal split.If there is 1 member that overlaps with s, suppose the corresponding splitting vertex is v b , and the boundary vertex is actually v 2 . Then s 1b is a closed set containing s xy and corresponds to the situation of 0 member overlapping. s 1b is the union of a subset of members of the maximal split, and violates the definition of maximal split.If there are 2 members that overlaps with s, suppose they generate two different splitting vertex v a and v b . Then s ab is a closed set containing s xy and corresponds to the situation of 0 member overlapping. s ab is the union of a subset of members of the maximal split, and violates the definition of maximal split.If they generate the same splitting vertex v b , from lemma 5, v b is also the endpoint vertex of at least 1 other member s ab which has to be inside s. Suppose the two overlapping members are s cb that contains v 1 , and s bd that contains v 2 . As the source vertex of s, v 1 has path to v b and v 1 has path to v a , which implies v b has path to v a . As the target vertex of s, v 2 has path from v b and v 2 has path from v a , which implies v b has path from v a . This conflicts with the fact that s is acyclic. Therefore, this case is not possible.Therefore, this lemma is proved.Lemma 7. If non-branched s ij has at least 1 vertex but has 0 splitting vertex, then its maximal split has length > 2 Proof. As s ij is not branched , the members of its maximal split cannot have the starting vertex as v i and the ending vertex as v j at the same time. If s ij has at least 1 vertex, and its maximal split has length 2, then its maximal split must be {[s ik ], [s kj ]}, and v k will be the splitting vertex of s ij , which violates that s ij has no splitting vertex.If s ij has at least 1 vertex without splitting vertex, it has at least 2 edges and cannot have a trivial length 1 maximal split. Therefore, its maximal split has length > 2
This paper proposes fundamental theory and optimal algorithms for DNN training, which reduce up to 80% of training memory for popular DNNs.
{"src": "scitldr-AIC_train", "paper_id": "BJMvBjC5YQ"}