Compressed forms of deep neural networks are essential in deploying large-scale
 computational models on resource-constrained devices. Contrary to analogous
 domains where large-scale systems are build as a hierarchical repetition of small-
 scale units, the current practice in Machine Learning largely relies on models with
 non-repetitive components. In the spirit of molecular composition with repeating
 atoms, we advance the state-of-the-art in model compression by proposing Atomic
 Compression Networks (ACNs), a novel architecture that is constructed by recursive
 repetition of a small set of neurons. In other words, the same neurons with the
 same weights are stochastically re-positioned in subsequent layers of the network.
 Empirical evidence suggests that ACNs achieve compression rates of up to three
 orders of magnitudes compared to fine-tuned fully-connected neural networks (88×
 to 1116× reduction) with only a fractional deterioration of classification accuracy
 (0.15% to 5.33%). Moreover our method can yield sub-linear model complexities
 and permits learning deep ACNs with less parameters than a logistic regression
 with no decline in classification accuracy. The universe is composed of matter, a physical substance formed by the structural constellation of a plethora of unitary elements denoted as atoms. The type of an atom eventually defines the respective chemical elements, while structural bonding between atoms yields molecules (the building blocks of matter and our universe). In Machine Learning a neuron is the infinitesimal nucleus of intelligence (i.e. {atom, matter} ↔ {neuron, AI}), whose structural arrangement in layers produces complex intelligence models. Surprisingly, in contrast to physical matter where molecules often reuse quasi-identical atoms (i.e. repeating carbon, hydrogen, etc.), neural networks do not share the same neurons across layers. Instead, the neurons are parameterized through weights which are optimized independently for every neuron in every layer. Inspired by nature, we propose a new paradigm for constructing deep neural networks as a recursive repetition of a fixed set of neurons. Staying faithful to the analogy we name such models as Atomic Compression Networks (ACNs). Extensive experimental results show that by repeating the same set of neurons, ACNs achieve unprecedented compression in terms of the total neural network parameters, with a minimal compromise on the prediction quality. Deep neural networks (DNN) achieve state-of-the-art prediction performances on several domains like computer vision Tan & Le, 2019) and natural language processing (Vaswani et al., 2017; Gehring et al., 2017) . Therefore, considerable research efforts are invested in adopting DNNs for mobile, embedded, or Internet of Things (IoT) devices (Kim et al., 2015) . Yet, multiple technical issues related to restricted resources, w.r.t. computation and memory, prevent their straightforward application in this particular domain Samie et al., 2016; Mehta et al., 2018) . Even though prior works investigate neural compression techniques like pruning or low-rank parameter factorization, they face fragility concerns regarding the tuning of hyperparameters and network architecture, besides struggling to balance the trade-off between compression and accuracy (Cheng et al., 2017) . • a novel compression paradigm for neural networks composed of repeating neurons as the atomic network components and further motivated by function composition; • compression rates of up to three orders of magnitudes compared to a cross-validated fullyconnected network on nine real-world vector datasets; • first work to achieve sub-linear model complexities measured in the number of trained parameters compared to connected architectures on several computer vision tasks. 2 RELATED WORK In this paper we presented Atomic Compression Networks (ACN), a new network architecture which recursively reuses neurons throughout the model. We evaluate our model on nine vector and three image datasets where we achieve promising results regarding the compression rate and the loss in model accuracy. In general ACNs achieve much tinier models with only a small to moderate decrease of accuracy compared to six other baselines. For future work we plan to include skip connections in the architecture and to extend the idea to CNNs and the sharing of kernel parameters as well as for the FC layers. Another interesting path of research is the combination of the ACN scheme with NAS methods to further optimize the efficiency and performance of the created architectures.
We advance the state-of-the-art in model compression by proposing Atomic Compression Networks (ACNs), a novel architecture that is constructed by recursive repetition of a small set of neurons.
{"src": "scitldr-AIC_train", "paper_id": "S1xO4xHFvB"}