This work proposes the first strategy to make distributed training of neural networks resilient to computing errors, a problem that has remained unsolved despite being first posed in 1956 by von Neumann. He also speculated that the efficiency and reliability of the human brain is obtained by allowing for low power but error-prone components with redundancy for error-resilience. It is surprising that this problem remains open, even as massive artificial neural networks are being trained on increasingly low-cost and unreliable processing units. Our coding-theory-inspired strategy, "CodeNet," solves this problem by addressing three challenges in the science of reliable computing: (i) Providing the first strategy for error-resilient neural network training by encoding each layer separately; (ii) Keeping the overheads of coding (encoding error-detection decoding) low by obviating the need to re-encode the updated parameter matrices after each iteration from scratch. (iii) Providing a completely decentralized implementation with no central node (which is a single point of failure), allowing all primary computational steps to be error-prone. We theoretically demonstrate that CodeNet has higher error tolerance than replication, which we leverage to speed up computation time. Simultaneously, CodeNet requires lower redundancy than replication, and equal computational and communication costs in scaling sense. We first demonstrate the benefits of CodeNet in reducing expected computation time over replication when accounting for checkpointing. Our experiments show that CodeNet achieves the best accuracy-runtime tradeoff compared to both replication and uncoded strategies. CodeNet is a significant step towards biologically plausible neural network training, that could hold the key to orders of magnitude efficiency improvements.
Because of their potential to go undetected, soft-errors are receiving increasing attention (sometimes even regarded as the scariest aspect of supercomputing’s monster in the closet'' @cite_33 ). Common causes for soft-errors include: (i) Exposure of chips to cosmic rays from outer space causing unpredictable bit-flips; (ii) Manufacturing and soldering defects; and (iii) Memory and storage faults etc. @cite_33 @cite_10 . Even for specialized nanoscale circuits, the for semiconductors predicts that as devices become smaller, thermal noise itself will cause systems to fail catastrophically during normal operation even without supply voltage scaling @cite_61 , thus increasing the need for fault-tolerant training.
@cite_61 @cite_10 @cite_33
2087042114 2033346530 2296204683
Abstract Noise abatement is the key problem of small-scaled circuit design. New computational paradigms are needed -- as these circuits shrink, they become very vulnerable to noise and soft errors. In this lecture, we present a probabilistic computation framework for improving the resiliency of logic gates and circuits under random conditions induced by voltage or current fluctuation. Among many probabilistic techniques for modeling such devices, only a few models satisfy the requirements of efficient hardware implementation -- specifically, Boltzman machines and Markov Random Field (MRF) models. These models have similar built-in noise-immunity characteristics based on feedback mechanisms. In probabilistic models, the values 0 and 1 of logic functions are replaced by degrees of beliefs that these values occur. An appropriate metric for degree of belief is probability. We discuss various approaches for noise-resilient logic gate design, and propose a novel design taxonomy based on implementation of the MR... This paper reviews the basic physics of those cosmic rays which can affect terrestrial electronics. Cosmic rays at sea level consist mostly of neutrons, protons, pions, muons, electrons, and photons. The particles which cause significant soft fails in electronics are those particles with the strong interaction: neutrons, protons, and pions. At sea level, about 95 of these particles are neutrons. The quantitative flux of neutrons can be estimated to within 3 times , and the relative variation in neutron flux with latitude, altitude, diurnal time, earth s sidereal position, and solar cycle is known with even higher accuracy. The possibility of two particles of a cascade interacting with a single circuit to cause two simultaneous errors is discussed. The terrestrial flux of nucleons can be attenuated by shielding, making a significant reduction in the electronic system soft-error rate. Estimates of such attenuation are made. As a child, were you ever afraid that a monster lurking in your bedroom would leap out of the dark and get you? My job at Oak Ridge National Laboratory is to worry about a similar monster, hiding in the steel cabinets of the supercomputers and threatening to crash the largest computing machines on the planet. The monster is something supercomputer specialists call resilience- or rather the lack of resilience. It has bitten several supercomputers in the past. A high-profile example affected what was the second fastest supercomputer in the world in 2002, a machine called ASCI Q at Los Alamos National Laboratory. When it was first installed at the New Mexico lab, this computer couldn’t run more than an hour or so without crashing.