Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> A generalisation of the Hopfield model which includes interactions between p()2) Ising spins is considered. The exact storage capacity behaves as Np-1/2(p-1)! ln N when the number of nodes, N, is large. In the limit p to infinity , the thermodynamics of the model can be solved exactly without using the replica method; at zero temperature, a solution which is completely correlated with the input pattern exists for alpha < alpha c where alpha c to infinity as p to infinity and this solution has lower energy than the spin-glass solution if alpha < alpha 1=1/4 ln 2 where the number of patterns n=(2 alpha /p!)Np-1. For finite values of p, the correlation with the input pattern is not complete; for p=3 and 4, approximate values of alpha c and alpha 1 are obtained and for p to infinity the replica symmetric approximation gives alpha c approximately p/4 ln p. <s> BIB001 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> We discuss the thermodynamic phases and storage capacity of models which are extensions of the Hopfield-Little model of associative memory, generalized to allow for arbitrary polynomial Hamiltonians. The storage capacity increases as an exponential function of the highest power appearing in the polynomial Hamiltonian. We include the effects of replica symmetry breaking in our computations. <s> BIB002 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> Recent results on the memory storage capacity of higher order neural networks indicate a significant improvement compared to the limited capacity of the Hopfield model. However, such results have so far been obtained under the restriction that only a single iteration is allowed to converge. This paper presents a indirect convergence (long-term attraction) analysis of higher order neural networks. Our main result is that for any /spl kappa//sub d/<d!2/sup d-1//(2d)!, and 0/spl les//spl rho/<1/2, a Hebbian higher order neural network of order d with n neurons can store a random set of /spl kappa//sub d/n/sup d//log n fundamental memories such that almost all memories have an attraction radius of size /spl rho/n. If /spl kappa//sub d/<d!2/sup d-1//((2d)!(d+1)), then all memories possess this property simultaneously. It indicates that the lower bounds on the long-term attraction capacities are larger than the corresponding direct convergence capacities by a factor of 1/(1-2/spl rho/)/sup 2d/. In addition we upper bound the convergence rate (number of iterations required to converge). This bound is asymptotically independent of n. Similar results are obtained for zero diagonal higher order neural networks. <s> BIB003 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> A Bloom filter is a simple space-efficient randomized data structure for representing a set in order to support membership queries. Bloom filters allow false positives but the space savings often outweigh this drawback when the probability of an error is controlled. Bloom filters have been used in database applications since the 1970s, but only in recent years have they become popular in the networking literature. The aim of this paper is to survey the ways in which Bloom filters have been used and modified in a variety of network problems, with the aim of providing a unified mathematical and practical framework for understanding them and stimulating their use in future applications. <s> BIB004 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> While neuroscientists garner success in identifying brain regions and in analyzing individual neurons, ground is still being broken at the intermediate scale of understanding how neurons combine to encode information. This book proposes a method of representing information in a computer that would be suited for modelling the brain's methods of processing information. Holographic reduced representations (HRRs) are introduced here to model how the brain distributes each piece of information among thousands of neurons. It has been previously thought that the grammatical structure of a language cannot be encoded practically in a distributed representation, but HRRs can overcome the problems of earlier proposals. Thus this work has implications for psychology, neuroscience, linguistics, computer science and engineering. <s> BIB005 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> Quantitative expressions of long-term memory storage capacities of complex neural network are derived. The networks are made of neurons connected by synapses of any order, of the axono-axonal type considered by Kandel et al. for example. The effect of link deletion possibly related to aging, is also considered. The central result of this study is that, within the framework of Hebb's laws, the number of stored bits is proportional to the number of synapses. The proportionality factor however, decreases when the order of involved synaptic contact increases. This tends to favor neural architectures with low-order synaptic connectivities. It is finally shown that the memory storage capacities can be optimized by a partition of the network into neuron clusters with size comparable with that observed for cortical microcolumns. <s> BIB006 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> Many network solutions and overlay networks utilize probabilistic techniques to reduce information processing and networking costs. This survey article presents a number of frequently used and useful probabilistic techniques. Bloom filters and their variants are of prime importance, and they are heavily used in various distributed systems. This has been reflected in recent research and many new algorithms have been proposed for distributed systems that are either directly or indirectly based on Bloom filters. In this survey, we give an overview of the basic and advanced techniques, reviewing over 20 variants and discussing their application in distributed systems, in particular for caching, peer-to-peer systems, routing and forwarding, and measurement data summarization. <s> BIB007 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> We present a new cognitive architecture named Associative-Projective Neural Networks (APNNs). APNNs have a multi-module, multi-level, and multi-modal design that works with an original scheme of sparse binary distributed representations to construct world models of varied complexity required for both task-specific and more general cognitive modeling. APNNs provide scalability and flexibility due to a number of design features. Internal representations of APNNs are sparse binary vectors of fixed dimensionality for items of various complexity and generality. Representations of input scalars, vectors, or compositional relational structures are constructed on-the-fly, so that similar items produce representations similar in terms of vector dot-products. Thus, for example, similarity of relational structures (taking into account similarity of their components, their grouping and order) can be estimated by dot-products of their representations, without the need to follow edges or to match vertices of underlying graphs. Decoding distributed representations through the input representations is also possible. Storage, retrieval, and decoding of distributed representations are implemented by efficient auto-associative memories; using distributed memories based on the idea of Hebb’s cell assemblies additionally provides a natural tool for emergence of generalization hierarchies. In addition, we consider how APNNs account for representation grounding, deal with recent challenges for distributed representations, and present some open problems. <s> BIB008 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> In this paper, we propose a new approach to implementing hierarchical graph neuron (HGN), an architecture for memorizing patterns of generic sensor stimuli, through the use of vector symbolic architectures. The adoption of a vector symbolic representation ensures a single-layer design while retaining the existing performance characteristics of HGN. This approach significantly improves the noise resistance of the HGN architecture, and enables a linear (with respect to the number of stored entries) time search for an arbitrary subpattern. <s> BIB009 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> A model of associative memory is studied, which stores and reliably retrieves many more patterns than the number of neurons in the network. We propose a simple duality between this dense associative memory and neural networks commonly used in deep learning. On the associative memory side of this duality, a family of models that smoothly interpolates between two limiting cases can be constructed. One limit is referred to as the feature-matching mode of pattern recognition, and the other one as the prototype regime. On the deep learning side of the duality, this family corresponds to feedforward neural networks with one hidden layer and various activation functions, which transmit the activities of the visible neurons to the hidden layer. This family of activation functions includes logistics, rectified linear units, and rectified polynomials of higher degrees. The proposed duality makes it possible to apply energy-based intuition from associative memory to analyze computational properties of neural networks with unusual activation functions - the higher rectified polynomials which until now have not been used in deep learning. The utility of the dense memories is illustrated for two test cases: the logical gate XOR and the recognition of handwritten digits from the MNIST data set. <s> BIB010 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> Bloom filters are a data structure for storing data in a compressed form. They offer excellent space and time efficiency at the cost of some loss of accuracy (so-called lossy compression). This work presents a yes–no Bloom filter, which as a data structure consisting of two parts: the yes-filter which is a standard Bloom filter and the no-filter which is another Bloom filter whose purpose is to represent those objects that were recognized incorrectly by the yes-filter (that is, to recognize the false positives of the yes-filter). By querying the no-filter after an object has been recognized by the yes-filter, we get a chance of rejecting it, which improves the accuracy of data recognition in comparison with the standard Bloom filter of the same total length. A further increase in accuracy is possible if one chooses objects to include in the no-filter so that the no-filter recognizes as many as possible false positives but no true positives, thus producing the most accurate yes–no Bloom filter among all yes–no Bloom filters. This paper studies how optimization techniques can be used to maximize the number of false positives recognized by the no-filter, with the constraint being that it should recognize no true positives. To achieve this aim, an Integer Linear Program (ILP) is proposed for the optimal selection of false positives. In practice the problem size is normally large leading to intractable optimal solution. Considering the similarity of the ILP with the Multidimensional Knapsack Problem, an Approximate Dynamic Programming (ADP) model is developed making use of a reduced ILP for the value function approximation. Numerical results show the ADP model works best comparing with a number of heuristics as well as the CPLEX built-in solver (B&B), and this is what can be recommended for use in yes–no Bloom filters. In a wider context of the study of lossy compression algorithms, our research is an example showing how the arsenal of optimization methods can be applied to improving the accuracy of compressed data. <s> BIB011 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> A Bloom filter is a widely used data-structure for representing a set $S$ and answering queries of the form "Is $x$ in $S$?". By allowing some false positive answers (saying "yes" when the answer is in fact `no') Bloom filters use space significantly below what is required for storing $S$. In the distance sensitive setting we work with a set $S$ of (Hamming) vectors and seek a data structure that offers a similar trade-off, but answers queries of the form "Is $x$ close to an element of $S$?" (in Hamming distance). Previous work on distance sensitive Bloom filters have accepted false positive and false negative answers. Absence of false negatives is of critical importance in many applications of Bloom filters, so it is natural to ask if this can be also achieved in the distance sensitive setting. Our main contributions are upper and lower bounds (that are tight in several cases) for space usage in the distance sensitive setting where false negatives are not allowed. <s> BIB012 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> This article presents a modification of the recently proposed Holographic Graph Neuron approach for memorizing patterns of generic sensor stimuli. The original approach represents patterns as dense ... <s> BIB013 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> Deep neural networks (DNNs) trained in a supervised way suffer from two known problems. First, the minima of the objective function used in learning correspond to data points (also known as rubbish... <s> BIB014 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> In Krotov et al. (in: Lee (eds) Advances in Neural Information Processing Systems, Curran Associates, Inc., Red Hook, 2016) Krotov and Hopfield suggest a generalized version of the well-known Hopfield model of associative memory. In their version they consider a polynomial interaction function and claim that this increases the storage capacity of the model. We prove this claim and take the ”limit” as the degree of the polynomial becomes infinite, i.e. an exponential interaction function. With this interaction we prove that model has an exponential storage capacity in the number of neurons, yet the basins of attraction are almost as large as in the standard Hopfield model. <s> BIB015 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> Hyperdimensional (HD) computing is a promising paradigm for future intelligent electronic appliances operating at low power. This paper discusses tradeoffs of selecting parameters of binary HD representations when applied to pattern recognition tasks. Particular design choices include density of representations and strategies for mapping data from the original representation. It is demonstrated that for the considered pattern recognition tasks (using synthetic and real-world data) both sparse and dense representations behave nearly identically. This paper also discusses implementation peculiarities which may favor one type of representations over the other. Finally, the capacity of representations of various densities is discussed. <s> BIB016 </s> Neural Distributed Autoassociative Memories: A Survey <s> NAMs WITH HIGHER-ORDER CONNECTIONS AND WITHOUT CONNECTIONS <s> We study an indexing architecture to store and search in a database of high-dimensional vectors from the perspective of statistical signal processing and decision theory. This architecture is composed of several memory units, each of which summarizes a fraction of the database by a single representative vector. The potential similarity of the query to one of the vectors stored in the memory unit is gauged by a simple correlation with the memory unit's representative vector. This representative optimizes the test of the following hypothesis: the query is independent from any vector in the memory unit versus the query is a simple perturbation of one of the stored vectors. Compared to exhaustive search, our approach finds the most similar database vectors significantly faster without a noticeable reduction in search quality. Interestingly, the reduction of complexity is provably better in high-dimensional spaces. We empirically demonstrate its practical interest in a large-scale image search scenario with off-the-shelf state-of-the-art descriptors. <s> BIB017
Neural networks in the previous sections have connections of order 2 n = (а connection is between two neurons). In this section we consider values of n other than 2, for the NAMs with the structure of the Hopfield network (unlike Section "Hopfield NAMs" where we only considered the case 2 n = ). Neural networks with higher-order connections. In the higher-order (order 2 n > ) generalization of the Hopfield network, n neurons are connected by single connection instead of just two (for example, BIB006  BIB001 BIB002   BIB003 BIB010 BIB014 BIB015 ). For the neuron with states from {-1, + 1} ( p = 1/2), the network dynamics can be defined as . As in NAMs with connections between pairs of neurons, crit a depends on the specific type of learning rule and network dynamics. crit a does not exceed 2 and decreases with increasing n  . For the absence of errors (with a probability approaching 1), the number of stored vectors (for example, BIB001 BIB003 BIB010 BIB015 ). In BIB010 Fu and make a comparable contribution to the energy. For n ®¥, the main contribution to the energy is given by the memory vector y with the largest , áñ zy . For intermediate n , a large contribution is made by several nearest memory vectors. In BIB015 it is proved that for ()exp() Fuu = this memory allows one to retrieve exp() ND a = randomly distorted vectors (within Ham dist/2 D < from the stored vectors) by a single step of the sequential dynamics, for some 0ln2/2 a << , depending on the distortion, with probability converging to 1 for D ®¥. In BIB010 they consider the operation of such a network in the classification mode, where each stored base vector corresponds to one of the categories to be recognized. In particular, to classify the handwritten digits of the MNIST base into 10 classes, in addition to the "visible" neurons to which 28x28 images (with pixel values in [-1, + 1]) are input, there are 10 "classification" neurons. The value of the output is obtained by a non-linearity () gs applied to the input sum s , for example, tanh() s (instead of the sign() s function used in the memory mode). The outputs of visible neurons are fixed, and the outputs of classification neurons are determined by a single step of the dynamics. Vectors of N memory states are formed by learning on the training set. The N = 2000 memory vectors minimizing the classification error for the 60,000 images of the MNIST training set were obtained with the stochastic gradient descent algorithm. For a single step of the dynamics this structure is equivalent to a perceptron with one layer of N hidden neurons BIB010 . The nonlinearity at the output of the visible neurons is ()() fuFu = , and that at the output of hidden neurons is () gu . The learned memory vectors (with components normalized to [-1, + 1]) are encoded in the weight vectors of the connections between the visible neurons and the hidden neurons. It is shown that when n changes the visualized memory vectors change. For small n , the memorized vectors correspond to the features of the digit images, and for large n they become prototypes of individual digits BIB010 BIB014 . Neural networks with first-order connections. In a neural network with connection order 1 n = , each neuron is connected only with itself. They can be considered as networks without connections, where learning changes the state of the neurons themselves (with "neuron plasticity"  ). Thus, memory is a single vector of the dimension of the vectors of the base. For binary connections, we get the Bloom filter (see the reviews BIB004 BIB007 ), which exactly recognizes the absence of an undistorted query vector in the stored database by absence of at least one of its 1-components in the memory vector. If the 1-components of the query vector are a subset of 1-components of the memory vector, the vector is recognized as the base vector, but it is necessary to check this, since there is a false positive probability due to "ghosts" (vectors not from the base, the 1-components of which belong to the memory vector). Ghosts can be considered as analogous to spurious memories (Subsection "Research of generalization function in NAMs"). An analysis of the probability of their appearance under certain restrictions on stored random vectors is given in  . In BIB011 , they reduce the probability of false positives. In BIB012 , a Bloom filter version is analyzed which recognizes the absence of distorted query vectors. The autoscaling Bloom filter approach proposed in  suggests a generalization of the counting Bloom filter approach based on the mathematics of sparse hyperdimensional computing and allows elastic adjustment of its capacity with probabilistic bounds on false positives and true positives. In BIB013 , the formation of sparse memory vectors (with an additional operation of context-dependent thinning BIB008 ) is considered, and in BIB016 the probability of correct recognition is estimated. The use of graded connections (the formation of the memory vector is done by addition), including subsequent binarization, and the classification problem for vectors not from the base, are considered in BIB009 BIB016 . For real-valued vectors and connections, the recognition of random undistorted vectors is analyzed in  BIB005 . In BIB017 they allow distortion of vectors. In BIB005 BIB017 , the analysis of non-random base vectors is given.