Transient Dynamics of Sparsely Connected Hopfield Neural Networks with
  Arbitrary Degree Distributions
Transient Dynamics of Sparsely Connected
Hopfield Neural Networks with Arbitrary
Degree Distributions
Pan Zhang and Yong Chen ∗
Institute of Theoretical Physics, Lanzhou University, Lanzhou 730000, China
Abstract
Using probabilistic approach, the transient dynamics of sparsely connected Hopfield
neural networks is studied for arbitrary degree distributions. A recursive scheme is
developed to determine the time evolution of overlap parameters. As illustrative
examples, the explicit calculations of dynamics for networks with binomial, power-
law, and uniform degree distribution are performed. The results are good agreement
with the extensive numerical simulations. It indicates that with the same average
degree, there is a gradual improvement of network performance with increasing
sharpness of its degree distribution, and the most efficient degree distribution for
global storage of patterns is the delta function.
Key words: neural networks, complex networks, degree distribution, probability
theory
PACS: 87.10.+e, 89.75.Fb, 87.18.Sn, 02.50.-r
As a tractable toy model of associative memories and can also be viewed as
an extension of the Ising model, Hopfield neural networks [1] received lots
of attention in recent two decades. Equilibrium properties of fully-connected
Hopfield neural network have been well studied using spin-glass theory, es-
pecially the replica method [2,3]. Dynamics is also studied using generating
functional method [4] and signal-to-noise analysis [5,6,7] .
Given the huge number of neurons, there is only small number of intercon-
nections in human brain cortex (∼ 1011 neurons and ∼ 1014 synapses). In
order to simulate a biological genuine model rather than the fully-connected
networks, various random diluted models were studied, including extremely
diluted model [8,9], finite diluted model [10,11], and finite connection model
⋆ Physica A 387, 1009(2008)
∗ Corresponding author. Email address: ychen@lzu.edu.cn
Preprint submitted to Elsevier 19 November 2018
http://arxiv.org/abs/0704.1007v2
[12,13]. But neural connectivity is suggested to be far more complex than fully
random graph, e.g. the networks of c.elegans and cat’s cortical neural were re-
ported to be small-world and scale-free, respectively [14,15]. To go one step
closer to more biological realistic model, many numerical studies are carried
out, focusing on how the topology, the degree distribution, and clustering co-
efficient of a network topology affect the computational performance of the
Hopfield model [16,17,18,19]. With the same average connection, random net-
work was reported to be more efficient for storage and retrieval of patterns
than either small-world network or regular network [17]. Torres et al. reported
that the capacity of storage is higher for neural network with scale-free topol-
ogy than for highly random diluted Hopfield networks [18]. However, to our
best knowledge, there are no any theoretical results of either dynamics or
statics yet.
The goal of this paper is to analytically study the dynamics of Hopfield model
for a sparsely connected topology whose degree distribution is not restricted
to a specific distribution (e.g. Poisson) but can take arbitrary forms. Another
question investigated in this paper is how the degree distribution of connection
topology influences the network performance, especially whether there exists
an optimal degree distribution given a fixed number of nodes and connections.
Let us consider a system of N spins or neurons, the state of the spins takes
si (t) = ±1 and updates synchronously with the following probability,
Prob[si (t + 1) |hi (t)] =
eβsi(t+1)hi(t)
2 cosh (βhi (t))
, (1)
where β is the inverse temperature and the local field of neuron i is defined
hi (t) =
Jijsj (t) . (2)
We store q = αN random patterns ξµ = (ξ
1 , . . . , ξ
N) in networks, where α is
called the loading ratio. The couplings are given by the Hebb rule,
Jij =
j , (3)
where Cij is the adjacency matrix (Cij = 1 if j is connected to i, Cij = 0
otherwise). In contrast to spin glasses or many other physical systems, the
interactions between biological neurons are not symmetric: neuron i may in-
fluence neuron j even if neuron j has no influence on neuron i. So in our model,
Cij and Cji are chosen independently. Degree of spin i, ki =
j=1Cij, denotes
the number of spins that are connected to i. We consider the case that neurons
are sparsely connected, it means that N → ∞, ki → ∞ but ki/N → 0. For
example, we can take ki = O(lnN). And in this paper, the degrees of neurons
are set as an arbitrary distribution p (ki = k).
We use g (·) to express the transfer function,
si (t+ 1) = g (hi(t)) . (4)
Without loss of generality, let us consider the case to retrieve ξ1. We define
m (t) as the overlap parameter between network state s (t) and the first pattern
ξ1 as
m (t) =
ξ1i si (t) . (5)
Then the local field at time t can be represented by
hi (t) =
j 6=i
j sj (t) +
j 6=i
j sj (t) , (6)
where the first term is the signal from ξ1 and the second one is crosstalk noise
from other patterns. Our aim is to determine the form of the local field in
the thermodynamic limit N → ∞. We apply the law of large numbers to
the signal term and find that it converges to ξ1i
m (t) in the thermodynamic
limit. To show this point intuitively, we can simply replace the signal term by
its average,
j 6=i
j sj (t) =ξ
j sj (t)
. (7)
This formula is exact in the thermodynamic limit because the whole system is
assumed to be self-averaging. Since Cij and ξ
1 are independent of each other,
we can write the average of product as the product of average,
j sj (t)
= ξ1i 〈Cij〉
ξ1j sj (t)
. (8)
Using definition of ki together with Eq. (5), we have 〈Cij〉 = ki/N and m (t) =
ξ1j sj (t)
. So we have following formula,
j 6=i
j sj (t) = ξ
m (t) . (9)
Taking a closer look at the second term in Eq. (6), if all the terms in the
sum (with regard to µ) are independent, we are able to apply the central
limit theorem to it. As pointed out in Ref. [8], two conditions are essential for
the independence of terms in the sum: first is that almost all feedback loops
are eliminated, and the second is that with probability 1, any two neurons
have different clusters of ancestors, i.e. they will remain independent because
they receive inputs from two trees which have no neurons in common. In our
model, because of the sparsely connected architecture together with the high
asymmetry of synaptic connections, two conditions are both satisfied. Thus the
second term in Eq. (6) converges to a zero-mean Gaussian form N
(q−1)ki
where
(q−1)ki
is the variance of Gaussian noise. Then the local field of neuron
i can be expressed by
hi (t) = ξ
m (t) +N
(q − 1) ki
. (10)
Note that similar treatment of local field can also be found in [7].
Then the average state of neuron i was given formally by
〈si (t + 1)〉 =
dz (2π)
−z2/2
m (t) +
(q − 1) ki
, (11)
where 〈〉ξ1 stands for averaging over distribution of ξ
i , and P (ξ) = [δ(ξ + 1) + δ(ξ − 1)] /2.
When self-averaging is assumed, the average of neuron state in the next time
can be obtained by taking average over all N neurons,
〈s (t+ 1)〉 =
dz (2π)
−z2/2
m (t) +
(q − 1) ki
.(12)
Using the concept of degree distribution, we only need to take average over
the degree distributions as
〈s (t+ 1)〉 =
dkp (k)
dz (2π)
−z2/2
m (t) +
(q − 1) k
.(13)
The overlap parameters are obtained in the similar way,
m (t+ 1) =
dkp (k)
dz (2π)
−z2/2
m (t) +
(q − 1) k
.(14)
When focusing on the most interested case of zero temperature (β → ∞),
transfer function g (·) is replaced by sgn (·). From Eq. (14) one gets
m (t+ 1)=
dkp (k)
m(t)+
(q−1)k
dz (2π)
−z2/2
m(t)+
(q−1)k
dz (2π)
−z2/2
m(t)+
(q−1)k
dz (2π)
−z2/2
m(t)+
(q−1)k
dz (2π)
−z2/2
Then, the last equation can be further simplified to
m (t+ 1) =
dkp (k) erf
m (t)
(q − 1) /k
 , (16)
where
erf (u) =
−x2/2
dx. (17)
This finishes the Signal-to-Noise derivation of overlap parameter at zero tem-
perature. As long as the degree distribution of network is determined, using
Eq. (14-17), one can calculate temporal evolution of overlap parameters up to
an arbitrary time step.
Using auxiliary thermal fields γ (t) to express the stochastic dynamics [7], it
is easy to extend the method to arbitrary temperatures by averaging the zero
temperature results over the auxiliary fields.
s (t+ 1) = g (h (t) + γ (t) /β) , (18)
and the probability density of γ (t) is given by
p (γ (t)) =
1− tanh2 (γ (t))
. (19)
0 2 4 6 8 10
 in theory
 in simulation
Fig. 1. Time evolution of overlap parameters for Hopfield network with delta func-
tion degree distribution. Initial overlaps range from 1.0 to 0.1 (top to bottom). N
is 50000. Each neuron has 100 degrees and 20 patterns are stored in networks.
For illustrative examples, we apply our theory to networks with some specific
degree distributions and numerical simulations are performed to verify the
theoretical results. In all of our numerical simulations, we set N = 5 × 104
and the average degree k̄ = 100, varying only the arrangement of connections.
Each neuron is connected on average to 0.2% of the other neurons compared
to ∼ 0.1% in the mouse cortex [20].
The first numerical experiment is the delta function
p (k) = δ
k − k̄
, (20)
which means that every neuron has exactly k̄ connections. In practice, the con-
nection topology is generated by randomizing a regular lattice which average
degree is k̄. Time evolutions of overlap parameters from theory and numerical
simulations are plotted in Fig. 1.
The second degree distribution is binomial distribution which comes from a
Erdös-Renyi random graph [21] (see the left panel of Fig. 2)
p (k) = CkN
. (21)
The temporal evolution of overlap parameters are presented in the right panel
0 2 4 6 8 10
 in theory
 in simulation
0 50 100 150 200
Degree
Fig. 2. Left panel: the normalized binomial degree distribution of networks. Right
panel: the temporal evolution of overlap parameters for Hopfield network with de-
gree distribution shown in the left panel (Erdös-Renyi random graph). Initial over-
laps range from 1.0 to 0.1 (top to bottom). N = 50000, k̄ = 100, and 20 patterns
are stored in networks.
0 2 4 6 8 10
 in theory
 in simulation
54.59815 148.41316 403.42879
9.11882E-4
0.00248
0.00674
0.01832
0.04979
0.13534
0.36788
Degree
Fig. 3. Left panel: the normalized power-law degree distribution (log-log scale).
Right panel: the time evolution of overlap parameters for Hopfield network with
degree distribution plotted in the left panel. Initial overlaps range from 1.0 to 0.1
(top to bottom). N = 50000, k̄ = 100, and 20 patterns are stored in networks.
of Fig. 2.
The third one is power-law distribution (see the left panel of Fig. 3)
p (k) =
k̄2k−3, (22)
0 5 10 15 20
1.0  A
18.0 18.5 19.0
Fig. 4. Theoretical comparison of time evolutions of overlap parameters in networks
with the same average degree but different degree distributions. Degree distribution
of A is the delta function, B is binomial, and C is power-law. The inset shows
in detail that performance of network with delta function degree distribution is
slightly better than that with binomial distribution (Erdös-Renyi random graph).
N = 50000, k̄ = 100, and 20 patterns are stored in networks.
which is of great importance because it may comes from preferential attach-
ment in the growth process of neurons [22]. The right panel of Fig. 3 shows
the temporal evolutions of overlap parameters.
It is obvious that the theoretical results from our scheme are consistent with
the simulations for the above degree distributions. Emerging naturally from
the above statements, which form of degree distribution is the best one?
To investigate how degree distributions influence the performance of networks,
we theoretically compare time evolutions of overlap parameters with delta
function (A), binomial (B), and power-law degree distribution (C) in Fig. 4.
The inset shows the details near stationary states. It indicates that network
with delta degree distribution performs slightly better than that with binomial
distribution. The most rapid degradation in overlap occurs in network with
power-law distribution. This behavior can be interpreted as follows. Using
Eq. (16), it is easy to find that an individual neuron with fewer degrees suffers
more perturbations from crosstalk noise. In the case of power-law degree dis-
tribution, degrees are not uniformly distributed in networks and there are too
many neurons with small number of connections, which leads to negative per-
formance of the entire networks. However, note that despite the disadvantages
of power-law distribution, hubs (subset of networks which has higher degrees)
0 50 100 150 200
Degree
0 5 10 15 20
17 18 19
Fig. 5. Theoretical comparison of time evolutions of overlap parameters for Hopfield
networks for uniform degree distributions with different width. Degree distribution
of A is delta function. Left panel: the uniform degree distributions with various
width, 50 (B), 100 (C), 150 (D), and 200 (E). The inset in right panel shows in
detail that the performance of A is slightly better than that of B. N = 50000,
k̄ = 100, and 55 patterns are stored in networks.
in networks may be useful for partial storage [17].
In addition, for verifying the above statements, we construct the special cases
that the degree distributions of networks is uniform with various widthes, 50
(B), 100 (C), 150 (D), 200 (E), and the delta function A which width is 0
(see the left panel of Fig. 5). The temporal evolutions of overlaps are plotted
in the right panel of Fig. 5, and the inset shows the detailed information
near stationary states. It was found that the much more widespread degree
distribution tends to induce worse performance of networks.
In summary, the transient dynamics of sparsely connected Hopfield model with
arbitrary degree distributions is studied in this paper. It was found that the
delta function degree distribution is optimal in terms of network performance,
and there is a gradual improvement for network performance with increasing
sharpness of its degree distribution. We would like to emphasize that the model
investigated in this paper is a simple relaxation to real network topology,
by neglecting loops in it. But Ref. [23] suggested that the feedback loops
together with their correlations exist in networks and play important role in
network dynamics even in the case of sparsely connected systems. It would
be interesting to investigate Hopfield model with real complicated topology
influenced both by degree distributions and loops (feedbacks and correlations).
Acknowledgements
This work was supported by the National Natural Science Foundation of China
under Grant No. 10305005 and by the Special Fund for Doctor Programs at
Lanzhou University. One of us (PZ) thanks Dong Liu for useful suggestions.
References
[1] Hopfield, J. J. (1982). Neural Networks and Physical Systems with Emergent
Collective Computational Abilities. Proc. Nat. Acad. Sci. USA, 79(8), 2554-
2558.
[2] Amit, D. J., Gutfreund, H., & Sompolinsky, H. (1985). Spin-glass models of
neural networks. Phys. Rev. A, 32(2), 1007-1018.
[3] Amit, D. J., Gutfreund H., & Sompolinsky, H. (1985). Storing Infinite Numbers
of Patterns in a Spin-Glass Model of Neural Networks. Phys. Rev. Lett., 55,
1530-1533.
[4] Coolen, A. C. C. (2000). Statistical Mechanics of Recurrent Neural Networks
II. arXiv:cond-mat/0006011.
[5] Amari, S., & Maginu, K. (1988). Statistical neurodynamics of associative
memory. Neural Networks, 1, 63-73.
[6] Okada, M. (1995). A Hierarchy of Macrodynamical Equations for Associative
Memory. Neural Networks, 8, 833-838.
[7] Bolle, D., Blanco, J. B., & Verbeiren T. (2004). The signal-to-noise analysis of
the Little-Hopfield model revisited. J. Phys. A, 37, 1951-1969.
[8] Derrida, B., Gardner E., & Zippelius, A. (1987). An exactly solvable asymmetric
neural network model. Europhys. Lett., 4, 167-173.
[9] Patrick, A. E., & Zagrebnov, V. A. (1990). Parallel dynamics for an extremely
diluted neural network. J. Phys. A, 23, L1323-L1337.
[10] Theumann, W. K. (2003). Mean-field dynamics of sequence processing neural
networks with finite connectivity. Physica A, 328, 1-12.
[11] Zhang, P., & Chen, Y. (2007). Statistical neurodynamics for sequence processing
neural networks with finite dilution. Lect. Note Comput. Sci., 4491, 1144-1152.
[12] Wemmenhove, B., & Coolen, A. C. C. (2003). Finite connectivity attractor
neural networks. J. Phys. A, 36, 9617-9633.
[13] Castillo, I. P., & Skantzos, N. S. (2004). The Little-Hopfield model on a Random
Graph. J. Phys. A, 37, 9087-9099.
http://arxiv.org/abs/cond-mat/0006011
[14] Stephan, K.E., Kamper, L., Bozkurt, A., Burns, G.A.P.C., Young, M.P.,
& Koter, R. (2001). Advanced database methodology for the collation of
connectivity data on the Macaque brain (CoCoMac). Phil. Trans. R. Soc.
Lond. B Biol. Sci. 356, 1159-1186; Cherniak, C. (1994). Component placement
optimization in the brain. J. Neurosci. 14, 2418-2427; Scannell, J. W., Burns,
G. A. P. C., Hilgetag, C. C., ONeill, M. A., & Young, M. P. (1999). The
Connectional Organization of the Cortico-thalamic System of the Cat. Cerebral
Cortex 9, 277-299.
[15] Watts D. J., & Strogatz, S. H. (1998). Collective dynamics of ’small-world’
networks. Nature(London), 393, 440-442.
[16] Simard, D., Nadeau L., & Krögerar, H. (2005). Fastest learning in small
world neural networks. Phys. lett. A 336(11), 8-15; Li, C. & Chen, G. (2003).
Stability of a neural network model with small-world connections. Phys. Rev.
E 68, 052901-4; Davey, N. & Adams, R. (2004). High capacity associative
memories and connection constraints. Connection Science 16(1), 47-65; Davey
N., Christianson, B. & Adams, R. (2004). High capacity associative memories
and small world networks. Neural Networks Proceedings 1, 182; Stauffer, D.,
Aharony A., da, da Fontoura Costa L., & Adler, J. (2003). Efficient Hopfield
pattern recognition on a scale-free neural network. Euro. Phys. J. B 32, 395-399.
[17] McGraw, P. N., & Menzinger, M. (2003). Topology and computational
performance of attractor neural networks. Phys. Rev. E, 68, 047102-047105.
[18] Torres, J. J., Munoz, M. A., Marro, J., & Garrido, P. L. (2003). Influence of
topology on the performance of a neural network, arXiv:cond-mat/0310205.
[19] Kim, B. J. (2004). Performance of networks of artificial neurons: The role of
clustering. Phys. Rev. E, 69, 045101-045104.
[20] Braitenberg, V. & Schüz, A. (1998). Cortex: Statistics and Geometry of
Neruonal Connectivity. Springer-Verlag, Berlin.
[21] Erdös, P., & Renyi, A. (1959). On random graphs. Publ. Math. (debrecen) 6,
290-297.
[22] Barabási, A. L., & Albert, R. (1999). Emergence of scaling in random networks,
Science, 286, 509-512; Barabási, A. L., Albert, R., & Jeong, H. (1999). Mean-
field theory for scale-free random networks. Physica A, 272, 173-187.
[23] Zhang, P., & Chen Y. (2007). Topology and dynamics of attractor neural
networks: the role of loopiness, arxiv:cond-mat/0703405.
http://arxiv.org/abs/cond-mat/0310205
http://arxiv.org/abs/cond-mat/0703405
	References
