The central idea of the adaptive neural network computers, called neurocomputers, is to simulate the information processing architecture of the brain by linking together a large number of highly interconnected processors and making them to work in parallel. It is also assumed that there exist certain types of the processor information exchange dynamics, which makes it possible for a neurocomputer to learn from experience. Although there are difficulties with programming of general purpose computers working on the neural network principles, the performance of the brain in solving the pattern recognition problem clearly indicates the validity of the neural networks in this particular area.
In recent years the theory of neural networks was greatly advanced, among others, by Grossberg [Human Neurobiology, vol. 5, p. 1, 1986 and references therein], Hopfield [Biological Cybernetics vol. 52, p. 141, 1985 and references therein], Rumelhart [ISC Report 8506, University of California, San Diego and references therein], and Cooper [DTIC technical report #AD-A151 776]. A separate approach to the neural network theory, which is based on dynamics of certain functions defined on cell complexes, was developed by the author of this invention [A. N. Jourjine, Ph. D. Thesis, MIT 1984, unpublished; Phys Rev D31(1985)1443; Phys Rev D34(1986)3058; Quantum Filed Theory of Random Codes, 1987].
Digital computer simulations of various neural network theories confirm the fact that the neurocomputers can learn from experience and are capable of pattern recognition. Error propagation techniques were successfully used by Sejnowski et al. in addressing the voice recognition problem [T. J. Sejnowski and C. R. Rosenberg, NETtalk: a parallel network which learns to read aloud. Johns Hopkins University, January, 1986]. In addition, neural networks are very useful in solving complex optimization problems and in building Content Addressable Memories as demonstrated by Hopfield et al. [2] and Psaltis et al. [Appl. Optics 24(1985)1469].
The progress in building of hardware implementations of the neural networks has been limited. Because of the high degree of parallelism involved, neural networks are very inefficient when simulated on a digital computer. In order to utilize the advantages of the neural networks, one has to have a device with internal information processing architecture reflecting the architecture of the neural network. If a hardware implementation of a neural network is to be useful in solving pattern recognition problems for realistic data inputs, such as voice or visual image data, it is necessary to build a device with the following properties:
(1) a large number of independent information processors are linked together in a single device;
(2) each processor is preprogrammed and is either an analog processor, or a digital processor with a small volume of internal memory, i.e. is, what we call, cellular automaton microprocessor.
(3) each microprocessor is directly connected to a large number of other microprocessors, i.e. the absolute degree of connectivity is high;
(4) the ratio of absolute degree of connectivity to the total number of processors is relatively low, i.e. most of the processors are hidden units, that is are not linked directly to input and output;
(5) the connectivity pattern among the microprocessors is determined dynamically;
(6) changing of the connectivity pattern and the magnitude of the information exchange is relatively simple and is done locally by adjusting inner states of microprocessors;
(7) relative compactness and low power consumption by the device.
To justify the demands it is useful to mention that the brain, an example of neural network, has about 10 billion analog processors, each of which is connected to about 10 thousands others. This assembly consumes few hundred watts of power and is confined in a relatively small volume. It has been experimentally shown that the strength of intersynaptic connections changes as learning occurs in the brain and that the effective connectivity pattern depends on the relative timing of the "on" and "off" states of the neurons.
Attempts have been made at building implementations of neural networks in the form of VLSI circuits. At AT&T a fixed network has been put on the chip for use as a prototype CAM [L. D. Jackel et. al, J. Vac. Sci. Technol., B4(1986)61]. The problem with the VLSI implementations is that the interconnection pattern among the processors is static and the electrical interconnections take much of the space on the chip. There are also attempts to base neural network on purely optical processes using holograms [D. Psaltis et al. Appl. Optics 24(1985)1469; Scientific American, March(1987)88]. The problem with hologram based neural networks is that there everything is connected to everything and it is hard to reserve the hidden units, which are essential for learning as shown by Rumelhart [3]. At the present there is no hardware implementation of the adaptive neural network satisfying the above mentioned properties (1)-(7). On the other hand the software simulations of the neural networks on digital computers show that neural network implementations have a great advantage over conventional digital computers in the areas of pattern recognition and other areas of Artificial Intelligence. Thus a strong need exists for hardware implementation of adaptive neural networks, which satisfy the above mentioned properties (1)-(7).