The Gripon-Berrou neural network (GBNN) is a recently invented recurrent neural network embracing a LDPC-like sparse encoding setup which makes it extremely resilient to noise and errors. A natural use of GBNN is as an associative memory. There are two activation rules for the neuron dynamics, namely sum-of-sum and sum-of-max. The latter outperforms the former in terms of retrieval rate by a huge margin. In prior discussions and experiments, it is believed that although sum-of-sum may lead the network to oscillate, sum-of-max always converges to an ensemble of neuron cliques corresponding to previously stored patterns. However, this is not entirely correct. In fact, sum-of-max often converges to bogus fixed points where the ensemble only comprises a small subset of the converged state. By taking advantage of this overlooked fact, we can greatly improve the retrieval rate. We discuss this particular issue and propose a number of heuristics to push sum-of-max beyond these bogus fixed points. To tackle the problem directly and completely, a novel post-processing algorithm is also developed and customized to the structure of GBNN. Experimental results show that the new algorithm achieves a huge performance boost in terms of both retrieval rate and run-time, compared to the standard sum-of-max and all the other heuristics.
Another line of research focuses on efficient implementations of GBNNs. Jarollahi al demonstrate a proof-of-concept implementation of using (FPGA) in @cite_14 , though the network size is constrained to 400 neurons due to hardware limitations. The same authors implement in @cite_4 and runs 1.9 @math faster than @cite_14 , since bitwise operations are used in place of a resource-demanding module required by . @cite_0 , the same group of authors also develop a content addressable memory using GBNNs which saves 90 al @cite_10 implement an analog version of the network which consumes @math less energy but is @math more efficient both in the surface of the circuit and speed, compared with an equivalent digital circuit. However, the network size is further constrained to @math neurons in total. After analyzing the convergence and computation properties of both and , Yao al @cite_29 propose a hybrid scheme and successfully implement GBNNs on a GPU. An acceleration of 900 @math is witnessed without any loss of accuracy.
@cite_14 @cite_4 @cite_29 @cite_0 @cite_10
1977602574 1553143087 2055611966 2079849934
Associative memories are alternatives to indexed memories that when implemented in hardware can benefit many applications such as data mining. The classical neural network based methodology is impractical to implement since in order to increase the size of the memory, the number of information bits stored per memory bit (efficiency) approaches zero. In addition, the length of a message to be stored and retrieved needs to be the same size as the number of nodes in the network causing the total number of messages the network is capable of storing (diversity) to be limited. Recently, a novel algorithm based on sparse clustered neural networks has been proposed that achieves nearly optimal efficiency and large diversity. In this paper, a proof-of-concept hardware implementation of these networks is presented. The limitations and possible future research areas are discussed. Associative memories store content in such a way that the content can be later retrieved by presenting the memory with a small portion of the content, rather than presenting the memory with an address as in more traditional memories. Associative memories are used as building blocks for algorithms within database engines, anomaly detection systems, compression algorithms, and face recognition systems. A classical example of an associative memory is the Hopfield neural network. Recently, Gripon and Berrou have introduced an alternative construction which builds on ideas from the theory of error correcting codes and which greatly outperforms the Hopfield network in capacity, diversity, and efficiency. In this paper we implement a variation of the Gripon-Berrou associative memory on a general purpose graphical processing unit (GPU). The work of Gripon and Berrou proposes two retrieval rules, sum-of-sum and sum-of-max. The sum-of-sum rule uses only matrix-vector multiplication and is easily implemented on the GPU. The sum-of-max rule is much less straightforward to implement because it involves non-linear operations. However, the sum-of-max rule gives significantly better retrieval error rates. We propose a hybrid rule tailored for implementation on a GPU which achieves a 880-fold speedup without sacrificing any accuracy. A low-power Content-Addressable Memory (CAM) is introduced employing a new mechanism for associativity between the input tags and the corresponding address of the output data. The proposed architecture is based on a recently developed clustered-sparse network using binary-weighted connections that on-average will eliminate most of the parallel comparisons performed during a search. Therefore, the dynamic energy consumption of the proposed design is significantly lower compared to that of a conventional low-power CAM design. Given an input tag, the proposed architecture computes a few possibilities for the location of the matched tag and performs the comparisons on them to locate a single valid match. A 0.13μm CMOS technology was used for simulation purposes. The energy consumption and the search delay of the proposed design are 9.5 , and 30.4 of that of the conventional NAND architecture respectively with a 3.4 higher number of transistors. Encoded neural networks mix the principles of associative memories and error-correcting decoders. Their storage capacity has been shown to be much larger than Hopfield Neural Networks'. This paper introduces an analog implementation of this new type of network. The proposed circuit has been designed for the 1V supply ST CMOS 65nm process. It consumes 1165 times less energy than a digital equivalent circuit while being 2.7 times more efficient in terms of combined speed and surface.