Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> Contents: Introduction. The Problem and the Line of Attack. Summation and Learning in Perception. Field Theory and Equipotentiality. The First Stage of Perception: Growth of the Assembly. Perception of a Complex: The Phase Sequence. Development of the Learning Capacity. Higher and Lower Processes Related to Learning. Problems of Motivation: Pain and Hunger. The Problem of Motivational Drift. Emotional Disturbances. The Growth and Decline of Intelligence. <s> BIB001 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> A previously proposed model for memory based on neurophysiolo gical considerations is reviewed. We assume that (a) nervous system activity is usefully represented as the set of simultaneous individual neuron activities in a group of neurons; (b) different memory traces make use of the same synapses; and (c) synapses associate two patterns of neural activity by incrementing synaptic connectivity proportionally to the product of pre- and postsynaptic activity, forming a matrix of synaptic connectivities. We extend this model by (a) introducing positive feedback of a set of neurons onto itself and (b) allowing the individual neurons to saturate. A hybrid model, partly analog and partly binary, arises. The system has certain characteristics reminiscent of analysis by distinctive features. Next, we apply the model to "categorical perception." Finally, we discuss probability learning. The model can predict overshooting, recency data, and probabilities occurring in systems with more than two events with reasonably good accuracy. <s> BIB002 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> Crick and Mitchison1 have presented a hypothesis for the functional role of dream sleep involving an ‘unlearning’ process. We have independently carried out mathematical and computer modelling of learning and ‘unlearning’ in a collective neural network of 30–1,000 neurones. The model network has a content-addressable memory or ‘associative memory’ which allows it to learn and store many memories. A particular memory can be evoked in its entirety when the network is stimulated by any adequate-sized subpart of the information of that memory2. But different memories of the same size are not equally easy to recall. Also, when memories are learned, spurious memories are also created and can also be evoked. Applying an ‘unlearning’ process, similar to the learning processes but with a reversed sign and starting from a noise input, enhances the performance of the network in accessing real memories and in minimizing spurious ones. Although our model was not motivated by higher nervous function, our system displays behaviours which are strikingly parallel to those needed for the hypothesized role of ‘unlearning’ in rapid eye movement (REM) sleep. <s> BIB003 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> Abstract : Many parallel information processing systems have ben proposed which are loosely based on architecture of the nervous system. Many of the testable predictions of these systems fall in the realm of cognitive science since they perform some 'computations' well and some poorly and have pronounced 'psychologies'. In such system, simple elements are connected to each other; the elements act like 'neurons' in that they sum excitation and inhibition from other elements. Information is represented as large state vectors of element activities, and it can be shown that simple 'synaptic' learing rules can serve to associate arbitrary state vectors using a matrix of connections. One version of this class of models has been shown,, in the past to form concepts and to perform classification computations. This paper discusses the general importance and structure of psychological concepts and describes three simulations of a simple neural model that attempts to reproduce a few of the important properties of human concept formation. Prototypes; Neural Models; Neural nets. <s> BIB004 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> The asymmetric modification of the Hopfield model of the associative memory is considered. It is shown that the asymmetry does not change the main properties of the model, but leads to the internal nonthermal noise. The modification of the Hopfield algorithm is proposed which can be used for storing the correlated patterns and its storage capacity is estimated. The hierarchical memory model is proposed and studied. <s> BIB005 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> The Hopfield model of neural networks is extended to allow for the storage and retrieval of hierarchically correlated patterns. The overlaps between these patterns form an ultrametric tree. Intermediate states, which serve as ancestors for the following levels, are generated at each level of the tree. The states belonging to each level are stored, by a modified learning rule, in a series of identical networks, one for each level. The retrieval of a particular pattern is preceded and assisted by the successive retrieval of its ancestors. The performance of this scheme is studied analytically and numerically. <s> BIB006 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> From the Publisher: ::: Substantial progress in understanding memory, the learning process, and self-organization by studying the properties of models of neural networks have resulted in discoveries of important parallels between the properties of statistical, nonlinear cooperative systems in physics and neural networks. <s> BIB007 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> A Hopfield-like neural network that can store hierarchically correlated patterns with low level of activity is studied. Three learning rules are proposed which enable to obtain nearly optimal storage capacity. These learning rules have different rate of biological relevancy and the restrictions they put upon the structure of hierarchical tree. By varying the value of the neural threshold, it is possible to climb up and down the hierarchical tree. <s> BIB008 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> The process of neural assembly formation in neural networks with a low level of activity is investigated. The experiments conducted show the possibility of forming neural assemblies with nucleus and fringe and of restoring both the nucleus and the nucleus with fringe. This proves that the neural network has additional functions: to form prototypes and to restore the assembly nucleus and assembly fringes from their parts. The tests performed have addressed the problem of decreasing neural network capacity. > <s> BIB009 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> In the present letter, the authors propose a model of hierarchical concept formation in auto-associative memory composed of neuro-window elements. When hierarchically correlated patterns are memorized, each center of the correlated memory patterns can be defined as a concept pattern which represents those similar patterns. It is shown that the concept patterns are formed under a Hebbian learning rule in the neural network where all the neuro-window elements are just mutually connected, and that the memory patterns and the concept patterns in each level of hierarchy can be retrieved selectively by adjusting the parameters of the neuro-window elements. Copyright 1996 Elsevier Science Ltd <s> BIB010 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> Abstract An architecture of a neural network with assembly organization is described. Such network architecture is applied to the problem of texture segmentation in natural scenes. The network is partitioned into several subnetworks. Each subnetwork is a column structure in which features are represented by means of “float” coding. Input data excite corresponding “floats” of neurons in the subnetworks. In the process of learning the weights of modifiable connections between excited neurons are changed so that Hebb's assemblies are formed in the column structures. All subnetworks are incorporated into a single network by a neural activity control system. Computer simulation of the proposed network has been performed. The results of computer simulations show the possibility of successful application of the assembly neural network to the problem of texture segmentation. <s> BIB011 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> We study unsupervised Hebbian learning in a recurrent network in which synapses have a finite number of stable states. Stimuli received by the network are drawn at random at each presentation from a set of classes. Each class is defined as a cluster in stimulus space, centred on the class prototype. The presentation protocol is chosen to mimic the protocols of visual memory experiments in which a set of stimuli is presented repeatedly in a random way. The statistics of the input stream may be stationary, or changing. Each stimulus induces, in a stochastic way, transitions between stable synaptic states. Learning dynamics is studied analytically in the slow learning limit, in which a given stimulus has to be presented many times before it is memorized, i.e. before synaptic modifications enable a pattern of activity correlated with the stimulus to become an attractor of the recurrent network. We show that in this limit the synaptic matrix becomes more correlated with the class prototypes than with any of the instances of the class. We also show that the number of classes that can be learned increases sharply when the coding level decreases, and determine the speeds of learning and forgetting of classes in the case of changes in the statistics of the input stream. <s> BIB012 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> The schemes for compositional distributed representations include those allowing on-the-fly construction of fixed dimensionality codevectors to encode structures of various complexity. Similarity of such codevectors takes into account both structural and semantic similarity of represented structures. We provide a comparative description of sparse binary distributed representation developed in the framework of the associative-projective neural network architecture and the more well known holographic reduced representations of T.A. Plate (1995) and binary spatter codes of P. Kanerva (1996). The key procedure in associative-projective neural networks is context-dependent thinning which binds codevectors and maintains their sparseness. The codevectors are stored in structured memory array which can be realized as distributed auto-associative memory. Examples of distributed representation of structured data are given. Fast estimation of the similarity of analogical episodes by the overlap of their codevectors is used in the modeling of analogical reasoning both for retrieval of analogs from memory and for analogical mapping. <s> BIB013 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> Hebbian heteroassociative learning is inherently asymmetric. Storing a forward association, from item A to item B, enables recall of B (given A), but does not permit recall of A (given B). Recurrent networks can solve this problem by associating A to B and B back to A. In these recurrent networks, the forward and backward associations can be differentially weighted to account for asymmetries in recall performance. In the special case of equal strength forward and backward weights, these recurrent networks can be modeled as a single autoassociative network where A and B are two parts of a single, stored pattern. We analyze a general, recurrent neural network model of associative memory and examine its ability to fit a rich set of experimental data on human associative learning. The model fits the data significantly better when the forward and backward storage strengths are highly correlated than when they are less correlated. This network-based analysis of associative learning supports the view that associations between symbolic elements are better conceptualized as a blending of two ideas into a single unit than as separately modifiable forward and backward associations linking representations in memory. <s> BIB014 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> The principles of recency and contiguity are two cornerstones of the theoretical and empirical analysis of human memory. Recency has been alternatively explained by mechanisms of decay, displacement, and retroactive interference. Another account of recency is based on the idea of variable context (Estes, 1955; Mensink & Raaijmakers, 1989). Such notions are typically cast in terms of a randomly fluctuating population of elements reflective of subtle changes in the environment or in the subjects' mental state. This random context view has recently been incorporated into distributed and neural network memory models (Murdock, 1997; Murdock, Smith, & Bai, 2001). Here we propose an alternative model. Rather than being driven by random fluctuations, this formulation, the temporal context model (TCM), uses retrieval of prior contextual states to drive contextual drift. In TCM, retrieved context is an inherently asymmetric retrieval cue. This allows the model to provide a principled explanation of the widespread advantage for forward recalls in free and serial recall. Modeling data from single-trial free recall, we demonstrate that TCM can simultaneously explain recency and contiguity effects across time scales. <s> BIB015 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> While neuroscientists garner success in identifying brain regions and in analyzing individual neurons, ground is still being broken at the intermediate scale of understanding how neurons combine to encode information. This book proposes a method of representing information in a computer that would be suited for modelling the brain's methods of processing information. Holographic reduced representations (HRRs) are introduced here to model how the brain distributes each piece of information among thousands of neurons. It has been previously thought that the grammatical structure of a language cannot be encoded practically in a distributed representation, but HRRs can overcome the problems of earlier proposals. Thus this work has implications for psychology, neuroscience, linguistics, computer science and engineering. <s> BIB016 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> This paper presents some techniques for analogical mapping using associative-projective neural networks (APNNs). Sparse binary distributed representations of constant high dimensionality are constructed on-the-fly for hierarchical structures of various complexity. Such representations encode both surface and structural similarity of analogical episodes. The introduced mapping approaches are illustrated using test analogies. <s> BIB017 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> We study an Attractor Neural Network that stores natural concepts, organized in semantic classes. The concepts are represented by distributed patterns over a space of attributes, and are related by both semantic and episodic associations. While semantic relations are expressed through an hierarchical coding over the attribute space, episodic links are realized via specific synaptic projections. Due to dynamic thresholds expressing neuronal fatigue, the network's behavior is characterized by convergence toward the concept patterns on a short time scale, and by transitions between the various patterns on a longer time scale. In its baseline, undamaged state, the network manifests semantic, episodic, and random transitions, and demonstrates the phenomen of priming. Modeling possible pathological changes, we have found that increasing the ‘noise’ level or the rate of neuronal fatigue decreases the frequency of semantic transitions. When neurons characterized by large synaptic connectivity are deleted, semantic transitions decay before the episodic ones, in accordance with the findings in patients with Alzheimer's disease. <s> BIB018 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> A neural network with assembly organization is described. The network is artificially partitioned into several sub-networks according to the number of classes that the network has to recognize. The features extracted from input data are encoded into activation of certain patterns of neurons in the sub-networks. During a process of primary learning, Hebb's neural assemblies are formed in the sub-networks by means of modification of connections' weights. A procedure of secondary learning, which is named as that of differentiation, is described. The procedure is intended to improve a recognition accuracy of the network by means of additional modification of connections' weights between the neurons of the same sub-networks. A computer simulation of the network is performed. The differentiation process is studied in a set of experiments on a character recognition task using two types of objects: Ukrainian letters and Arabic numerals of modified US National Institute of Standards and Technology (MNIST) database. <s> BIB019 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> The purpose of the paper is an experimental study of the formation of class descriptions, taking place during learning, in assembly neural networks. The assembly neural network is artificially partitioned into several sub-networks according to the number of classes that the network has to recognize. The features extracted from input data are represented in neural column structures of the sub-networks. Hebbian neural assemblies are formed in the column structure of the sub-networks by weight adaptation. A specific class description is formed in each sub-network of the assembly neural network due to intersections between the neural assemblies. The process of formation of class descriptions in the sub-networks is interpreted as feature generalization. A set of special experiments is performed to study this process, on a task of character recognition using the MNIST database. <s> BIB020 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> A common problem encountered in disciplines such as statistics, data analysis, signal processing, textual data representation, and neural network research, is finding a suitable representation of the data in the lower dimension space. One of the principles used for this reason is a factor analysis. In this paper, we show that Hebbian learning and a Hopfield-like neural network could be used for a natural procedure for Boolean factor analysis. To ensure efficient Boolean factor analysis, we propose our original modification not only of Hopfield network architecture but also its dynamics as well. In this paper, we describe neural network implementation of the Boolean factor analysis method. We show the advantages of our Hopfield-like network modification step by step on artificially generated data. At the end, we show the efficiency of the method on artificial data containing a known list of factors. Our approach has the advantage of being able to analyze very large data sets while preserving the nature of the data <s> BIB021 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> We provide an overview of Vector Symbolic Architectures (VSA), a class of structured associative memory models that offers a number of desirable features for artificial general intelligence. By directly encoding structure using familiar, computationally efficient algorithms, VSA bypasses many of the problems that have consumed unnecessary effort and attention in previous connectionist work. Example applications from opposite ends of the AI spectrum --visual map-seeking circuits and structured analogy processing --attest to the generality and power of the VSA approach in building new solutions for AI. <s> BIB022 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> The 1990s saw the emergence of cognitive models that depend on very high dimensionality and randomness. They include Holographic Reduced Representations, Spatter Code, Semantic Vectors, Latent Semantic Analysis, Context-Dependent Thinning, and Vector-Symbolic Architecture. They represent things in high-dimensional vectors that are manipulated by operations that produce new high-dimensional vectors in the style of traditional computing, in what is called here hyperdimensional computing on account of the very high dimensionality. The paper presents the main ideas behind these models, written as a tutorial essay in hopes of making the ideas accessible and even provocative. A sketch of how we have arrived at these models, with references and pointers to further reading, is given at the end. The thesis of the paper is that hyperdimensional representation has much to offer to students of cognitive science, theoretical neuroscience, computer science and engineering, and mathematics. <s> BIB023 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> The objective of this paper is to introduce a neural-network-based algorithm for word clustering as an extension of the neural-network-based Boolean factor analysis algorithm (Frolov , 2007). It is shown that this extended algorithm supports even the more complex model of signals that are supposed to be related to textual documents. It is hypothesized that every topic in textual data is characterized by a set of words which coherently appear in documents dedicated to a given topic. The appearance of each word in a document is coded by the activity of a particular neuron. In accordance with the Hebbian learning rule implemented in the network, sets of coherently appearing words (treated as factors) create tightly connected groups of neurons, hence, revealing them as attractors of the network dynamics. The found factors are eliminated from the network memory by the Hebbian unlearning rule facilitating the search of other factors. Topics related to the found sets of words can be identified based on the words' semantics. To make the method complete, a special technique based on a Bayesian procedure has been developed for the following purposes: first, to provide a complete description of factors in terms of component probability, and second, to enhance the accuracy of classification of signals to determine whether it contains the factor. Since it is assumed that every word may possibly contribute to several topics, the proposed method might be related to the method of fuzzy clustering. In this paper, we show that the results of Boolean factor analysis and fuzzy clustering are not contradictory, but complementary. To demonstrate the capabilities of this attempt, the method is applied to two types of textual data on neural networks in two different languages. The obtained topics and corresponding words are at a good level of agreement despite the fact that identical topics in Russian and English conferences contain different sets of keywords. <s> BIB024 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> The paper consists of two parts, each of them describing a learning neural network with the same modular architecture and with a similar set of functioning algorithms. Both networks are artificially partitioned into several equal modules according to the number of classes that the network has to recognize. Hebbian learning rule is used for network training. In the first network, learning process is concentrated inside the modules so that a system of intersecting neural assemblies is formed in each module. Unlike that, in the second network, learning connections link only neurons of different modules. Computer simulation of the networks is performed. Testing of the networks is executed on the MNIST database. Both networks directly use brightness values of image pixels as features. The second network has a better performance than the first one and demonstrates the recognition rate of 98.15%. <s> BIB025 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> We present an approach to similarity-based retrieval from knowledge bases that takes into account both the structure and semantics of knowledge base fragments. Those fragments, or analogues, are represented as sparse binary vectors that allow a computationally efficient estimation of structural and semantic similarity by the vector dot product. We present the representation scheme and experimental results for the knowledge base that was previously used for testing of leading analogical retrieval models MAC/FAC and ARCS. The experiments show that the proposed single-stage approach provides results compatible with or better than the results of two-stage models MAC/FAC and ARCS in terms of recall and precision. We argue that the proposed representation scheme is useful for large-scale knowledge bases and free-structured database applications. <s> BIB026 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> In this article we propose the usage of binary spatter codes and distributed data representation for communicating loss and delay sensitive data in event-driven sensor and actuator networks. Using the proposed data representation technique along with the medium access control protocol the mission critical control information can be transmitted with assured constant delay in deployments exposing below 0 dB signal-to-noise ratio figures. <s> BIB027 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> We present a new cognitive architecture named Associative-Projective Neural Networks (APNNs). APNNs have a multi-module, multi-level, and multi-modal design that works with an original scheme of sparse binary distributed representations to construct world models of varied complexity required for both task-specific and more general cognitive modeling. APNNs provide scalability and flexibility due to a number of design features. Internal representations of APNNs are sparse binary vectors of fixed dimensionality for items of various complexity and generality. Representations of input scalars, vectors, or compositional relational structures are constructed on-the-fly, so that similar items produce representations similar in terms of vector dot-products. Thus, for example, similarity of relational structures (taking into account similarity of their components, their grouping and order) can be estimated by dot-products of their representations, without the need to follow edges or to match vertices of underlying graphs. Decoding distributed representations through the input representations is also possible. Storage, retrieval, and decoding of distributed representations are implemented by efficient auto-associative memories; using distributed memories based on the idea of Hebb’s cell assemblies additionally provides a natural tool for emergence of generalization hierarchies. In addition, we consider how APNNs account for representation grounding, deal with recent challenges for distributed representations, and present some open problems. <s> BIB028 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> Most people have great difficulty in recalling unrelated items. For example, in free recall experiments, lists of more than a few randomly selected words cannot be accurately repeated. Here we introduce a phenomenological model of memory retrieval inspired by theories of neuronal population coding of information. The model predicts nontrivial scaling behaviors for the mean and standard deviation of the number of recalled words for lists of increasing length. Our results suggest that associative information retrieval is a dominating factor that limits the number of recalled items. <s> BIB029 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> The contribution of this article is twofold. First, it presents an encoding approach for seamless bidirectional transitions between localist and distributed representation domains. Second, the appr ... <s> BIB030 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> In this article we present a pattern classification system which uses Vector Symbolic Architecture (VSA) for representation, learning and subsequent classification of patterns, as a showcase we have used classification of vibration sensors measurements to vehicles types. On the quantitative side the proposed classifier requires only 1 kB of memory to classify an incoming signal against of several hundred of training samples. The classification operation into N types requires only 2∗N+1 arithmetic operations this makes the proposed classifier feasible for implementation on a low-end sensor nodes. The main contribution of this article is the proposed methodology for representing temporal patterns with distributed representation and VSA-based classifier. <s> BIB031 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> In this paper, we propose a new approach to implementing hierarchical graph neuron (HGN), an architecture for memorizing patterns of generic sensor stimuli, through the use of vector symbolic architectures. The adoption of a vector symbolic representation ensures a single-layer design while retaining the existing performance characteristics of HGN. This approach significantly improves the noise resistance of the HGN architecture, and enables a linear (with respect to the number of stored entries) time search for an arbitrary subpattern. <s> BIB032 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> This article presents a methodology for intelligent, biologically inspired fault detection system for generic complex systems of systems. The proposed methodology utilizes the concepts of associative memory and vector symbolic architectures, commonly used for modeling cognitive abilities of human brain. Compared to classical methods of artificial intelligence used in the context of fault detection the proposed methodology shows an unprecedented performance, while featuring zero configuration and simple operations. <s> BIB033 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> Abstract This paper proposes a simple encoding scheme for words using principles of Vector Symbolic Architectures. The proposed encoding allows finding a valid word in the dictionary for a given permuted word (represented using the proposed approach) using only a single operation– calculation of Hamming distance to the distributed representations of valid words in the dictionary. The proposed encoding scheme can be used as an additional processing mechanism for models of word embedding, which also form vectors to represent the meanings of words, in order to match the distorted words in the text to the valid words in the dictionary. <s> BIB034 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> This review considers methods and algorithms for fast estimation of distance/similarity measures between initial data from vector representations with binary or integer-valued components obtained from initial data that are mainly high-dimensional vectors with different distance measures (angular, Euclidean, and others) and similarity measures (cosine, inner product, and others). Methods without learning that mainly use random projections with the subsequent quantization and also sampling methods are discussed. The obtained vectors can be applied in similarity search, machine learning, and other algorithms. <s> BIB035 </s> Neural Distributed Autoassociative Memories: A Survey <s> GENERALIZATION IN NAMs <s> Modality corresponding to medical images is a vital filter in medical image retrieval systems. This article presents the classification of modalities of medical images based on the usage of principles of hyper-dimensional computing and reservoir computing. It is demonstrated that the highest classification accuracy of the proposed method is on a par with the best classical method for the given dataset (83% vs. 84%). The major positive property of the proposed method is that it does not require any optimization routine during the training phase and naturally allows for incremental learning upon the availability of new training data. <s> BIB036
The Hebbian learning in matrix-type distributed memories naturally builds a kind of correlation matrix where the frequencies of joint occurrence of active neurons are accumulated in the updated connection weights. The neural assemblies thus formed in the network may have a complex internal structure reflecting the similarity structure of stored data. This structure can be revealed as stable states of the network -in the general case, different from the stored data vectors. That is, it is possible for vectors retrieved from a network to not be identical to any of the vectors stored in the network (which is generally undesirable). Similarity preserving binary vectors. Similarity of patterns of active neurons (represented as binary vectors) are assumed to reflect the similarity of items (of various complexity and generality) they encode. The similarity value is measured in terms of the number or fraction of common active neurons (or overlap, i.e. normalized dot product of the representing binary vectors). Moreover, the similarity "content" is available as the identities of common active neurons (the IDs of the common 1-components of the representing binary vectors). Note that such data representation schemes by similarity preserving binary vectors have been developed for objects represented by various data types, mainly for (feature) vectors (see survey in BIB035 ), but also for structured data types such as sequences  BIB015 BIB030 BIB034 and graphs BIB013 BIB017  BIB026  BIB028 . A significant part of this research is developed in the framework of distributed representations  BIB023 BIB022 BIB016 BIB032 BIB036 BIB031  BIB033 BIB027 for examples of their applications). Complex internal structure of cell assemblies for graded connections. When binary vectors reflecting similarity of real objects are stored in a NAM by variants of the Hebb (or Hopfield) rule, the weights of connections between the neurons frequently activated together will be greater than the mean value of all the weights. On the other hand, rare combinations of active neurons will have smaller weights. Thus, neuron assemblies (cell assemblies in terms of Hebb BIB001 ) formed in the network may have a very complicated structure. Hebb and Milner introduced the notions of "cores" and "fringes" ( BIB001 pp. 126-134; for more recent research see  ) to characterize qualitatively such complex internal structure of assemblies. The notions of core (kernel, nucleus) and fringe (halo) of assemblies have attracted attention to the function of assemblies distinct from the function of associative memory. This different function is not just memorization of individual activity patterns (vectors), but emergence of some generalized internal representations, that were not explicitly presented to the network as vectors for memorization. Some assembly cores may correspond to "prototypes" representing subsets of attributes (encoded by the active neurons represented by the 1-components of the corresponding vectors) often present together in some input vectors. Note that some of these subsets of components/attributes may never be present in any single vector. Stronger cores, corresponding to stable combinations of a small number of typical attributes present in many vectors employed for learning, may correspond to some more abstract or general object category (class). Cores formed by more attributes may represent more specific categories, or object prototypes. However, object tokens (category instances) may also have strong cores if they were often presented to the network for learning. Note that some mechanisms may exist to prevent repeated learning of vectors that are already "familiar" to the network. Also, the rate of weight modification may vary based on the "importance" of the input vector. The representations of real objects have different degrees of similarity with each other. Similarities in various combinations of features often form different hierarchies of similarities that reflect hierarchies of categories of different degrees of generalization (object -class of objects -a more general class, etc.). So, assemblies formed in the network (cores and fringes of different "strength") may have a complex and rich hierarchical structure, with multiple overlapping hierarchies reflecting the structure of different contents and values of similarity implicitly present in the base of vectors used for the unsupervised network learning by the employed variant of the Hebb rule. Thus, many types of the category-based hierarchies (also known as generalization or classification or type-token hierarchies) may naturally emerge in the internal structure of assemblies formed in a single assembly neural network (NAM). Complex internal structure of a neural assembly allows a virtually continuous variety of hierarchical transitions. To reveal various types of categories and prototypes and instances formed in the network, the corresponding assemblies should be activated. To activate only stronger cores, higher values of threshold should be used. Lower threshold values may additionally activate fringes. Research of generalization function in NAMs. Additional stable states that emerge in NAM after memorizing random base vectors and do not coincide with the base vectors are known as false or spurious states or memories, e.g., BIB003  . In  they regarded the emergence of spurious attractors in the Hopfield networks as a side effect of the main function of distributed NAMs, consisting not in memorizing individual patterns, but in formation of prototypes. Such an interpretation is close to the earlier work BIB002  BIB004 that considered formation of concepts, prototypes, and taxonomic hierarchies as a natural generalization of correlated patterns memorized in a distributed memory. Research of hierarchically correlated patterns and states in Hopfield networks has been initiated by physicists who studied the "ultrametric" organization of ground states in spin-glasses (e.g.,  BIB005 BIB006 ; see also BIB007 and its references). While these earlier works required explicit representation of patterns at various hierarchical levels to be used in the learning rule, more neurobiologically plausible and practical Hebb and Hopfield rules applied to (hierarchically) correlated sparse binary patterns themselves (obtained with some simple correlation model) were considered, e.g., in BIB008 BIB018 . BIB008 theoretically showed "natural" formation of stable cores and fringes as well as traveling through different levels of hierarchies by uniform changing of the threshold. More complex probabilistic neuron dynamics and threshold control expressing neuronal fatigue was modeled in BIB018 . Dynamics of transitions between stable memory states that models human free recall data and can also be used with hierarchically organized data was considered in BIB029 . The "neurowindow" approach of BIB010 may be considered as using multiple thresholds to activate cores or fringes. Revealing the stable states corresponding to emergent assemblies is used for data mining (binary factor analysis) in BIB021 BIB024 . Generalization in NAMS with binary connections. The Willshaw learning rule does not form assemblies with the complex internal structure needed for generalization functions, such as emergence of generalization (type-token) hierarchies. The Willshaw learning rule causes the connectivity of an assembly (corresponding to a vector) to become full after a single learning act (vector storage) and not change thereafter. To preserve the capability of forming assemblies with a non-uniform connectivity in NAMs with binary connections, a stochastic analogue of the Hebbian learning rule for binary connections was proposed in   : where ij x is a binary random variable equal to 1 with the probability that determines the learning rate. The connectivity value for some set of neurons is determined here by the number of their 1-weight connections. Neurons that have more than some fraction of 1-weight connections with the other neurons of the same assembly may be attributed to the core part of the assembly. In BIB009 they experimentally studied formation of assemblies with cores and fringes using the above mentioned "stochastic Willshaw" rule (D = 4096, pD = 120 -200, about 60 neurons in the core and 60-140 neurons in the fringe). Tests have been performed on retrieving a core by its part; a core and its full fringe by the core and a part of the fringe (the most difficult test); and, a core by a part of its fringe. As expected, experiments with correlated base vectors have shown a substantial decrease of storage capacity compared to random independent vectors. A special learning rule was proposed to increase the stability of fringes. Formation of prototypes with the stochastic Willshaw rule was also investigated in  BIB012 ; a model of paired-associate learning in humans is considered in BIB014 . Generalization in modular NAMs. A modular structure of neural networks where the Hebb assemblies are formed inside the modules was proposed and developed in BIB011 BIB019  BIB020  BIB025  . The modular assembly neural network is intended for recognition of a limited number of classes. The network is artificially partitioned into several modules (sub-networks) according to the number of classes that the network is required to recognize. Each module network is full-connected, connections are graded. The features extracted from all objects of a certain class are encoded into activation of the patterns of neurons within the corresponding sub-network. After learning, the Hebb assemblies are formed in each module network. In this modular structure, the network acquires the capability to generalize the description of each class within the corresponding module (subnetwork), i.e. separately and completely independently from all other classes. In  it was shown that the number of connections in each module can be reduced without loss of the recognition capability.