Structure plays a key role in learning performance. In centralized computational systems, hyperparameter optimization and regularization techniques such as dropout are computational means to enhance learning performance by adjusting the deep hierarchical structure. However, in decentralized deep learning by the Internet of Things, the structure is an actual network of autonomous interconnected devices such as smart phones that interact via complex network protocols. Self-adaptation of the learning structure is a challenge. Uncertainties such as network latency, node and link failures or even bottlenecks by limited processing capacity and energy availability can significantly downgrade learning performance. Network self-organization and self-management is complex, while it requires additional computational and network resources that hinder the feasibility of decentralized deep learning. In contrast, this paper introduces a self-adaptive learning approach based on holarchic learning structures for exploring, mitigating and boosting learning performance in distributed environments with uncertainties. A large-scale performance analysis with 864,000 experiments fed with synthetic and real-world data from smart grid and smart city pilot projects confirm the cost-effectiveness of holarchic structures for decentralized deep learning.
General-purpose holonic designs that integrate the above principles are proposed via holonic multi-agent system platforms @cite_7 @cite_46 ; multi-level modeling and simulation approaches @cite_1 ; and hierarchical problem solving @cite_8 . Customised holonic designs are been applied to various domains, including hierarchical planning @cite_24 , traffic control @cite_28 , manufacturing @cite_11 @cite_43 and smart grids @cite_51 @cite_38 @cite_39 . The success of these applications motivate the adoption of similar holonic principles for designing deep learning systems for large-scale distributed environments.
@cite_38 @cite_7 @cite_8 @cite_28 @cite_1 @cite_39 @cite_24 @cite_43 @cite_46 @cite_51 @cite_11
19505421 2077930920 2548899748 2964088293
Properly architected holonic systems can enhance the ability of each set of players in the industrial automation and control market to deliver added value by encapsulating, reusing and deploying their specialized intellectual property at succeedingly higher levels of integration. Such an architecture expands on the HMS concept of cooperation domains to include both low-level control (LLC) and high-level control (HLC) domains. LLC refers to normal, non-holonic control and automation functions, while HLC refers to the integration of these functions into holons through the use of software agent technology. Function blocks, as defined in the International Electrotechnical Commission (IEC) 61499 series of standards, can be used for encapsulation, reuse, distribution and integration of both LLC and HLC functions, while HLC functionality can be standardized as defined by the Foundation for Intelligent Physical Agents (FIPA). Designing the control infrastructure of future “smart” power grids is a challenging task. Future grids will integrate a wide variety of heterogeneous producers and consumers that are unpredictable and operate at various scales. Information and Communication Technology (ICT) solutions will have to control these in order to attain global objectives at the macrolevel, while also considering private interests at the microlevel. This article proposes a generic holonic architecture to help the development of ICT control systems that meet these requirements. We show how this architecture can integrate heterogeneous control designs, including state-of-the-art smart grid solutions. To illustrate the applicability and utility of this generic architecture, we exemplify its use via a concrete proof-of-concept implementation for a holonic controller, which integrates two types of control solutions and manages a multiscale, multiobjective grid simulator in several scenarios. We believe that the proposed contribution is essential for helping to understand, to reason about, and to develop the “smart” side of future power grids. In this paper, we present HoloNet, a well-designed Convolutional Neural Network (CNN) architecture regarding our submissions to the video based sub-challenge of the Emotion Recognition in the Wild (EmotiW) 2016 challenge. In contrast to previous related methods that usually adopt relatively simple and shallow neural network architectures to address emotion recognition task, our HoloNet has three critical considerations in network design. (1) To reduce redundant filters and enhance the non-saturated non-linearity in the lower convolutional layers, we use a modified Concatenated Rectified Linear Unit (CReLU) instead of ReLU. (2) To enjoy the accuracy gain from considerably increased network depth and maintain efficiency, we combine residual structure and CReLU to construct the middle layers. (3) To broaden network width and introduce multi-scale feature extraction property, the topper layers are designed as a variant of inception-residual structure. The main benefit of grouping these modules into the HoloNet is that both negative and positive phase information implicitly contained in the input data can flow over it in multiple paths, thus deep multi-scale features explicitly capturing emotion variation can be well extracted from multi-path sibling layers, and then can be further concatenated for robust recognition. We obtain competitive results in this year’s video based emotion recognition sub-challenge using an ensemble of two HoloNet models trained with given data only. Specifically, we obtain a mean recognition rate of 57.84 , outperforming the baseline accuracy with an absolute margin of 17.37 , and yielding 4.04 absolute accuracy gain compared to the result of last year’s winner team. Meanwhile, our method runs with a speed of several thousands of frames per second on a GPU, thus it is well applicable to real-time scenarios. Recent works have shown that exploiting multi-scale representations deeply learned via convolutional neural networks (CNN) is of tremendous importance for accurate contour detection. This paper presents a novel approach for predicting contours which advances the state of the art in two fundamental aspects, i.e. multi-scale feature generation and fusion. Different from previous works directly considering multi-scale feature maps obtained from the inner layers of a primary CNN architecture, we introduce a hierarchical deep model which produces more rich and complementary representations. Furthermore, to refine and robustly fuse the representations learned at different scales, the novel Attention-Gated Conditional Random Fields (AG-CRFs) are proposed. The experiments ran on two publicly available datasets (BSDS500 and NYUDv2) demonstrate the effectiveness of the latent AG-CRF model and of the overall hierarchical framework.