Physics
ComputerScience,1970-2008-Computer Science-1995.jsonl.gz.gz
Computer Science
2fca1318f6724bf0d87be46ab084515e416a6d8e
UNREALIZABLE LEARNING IN FEEDFORWARD NEURAL NETWORKS

Statistical mechanics is used to study unrealizable generalization in two large feed-forward neural networks with binary weights and output, a perceptron and a tree committee machine. The student is trained by a teacher being larger, i.e. having more units than the student. It is shown that this is the same as using training data corrupted by Gaussian noise. Each machine is considered in the high temperature limit and in the replica symmetric approximation as well as for one step of replica symmetry breaking. For the perceptron a phase transition is found for low noise. However the transition is not to optimal learning. If the noise is increased the transition disappears. In both cases $\epsilon _{g}$ will approach optimal performance with a $(\ln\alpha /\alpha)^k$ decay for large $\alpha$. For the tree committee machine noise in the input layer is studied, as well as noise in the hidden layer. If there is no noise in the input layer there is, in the case of one step of repl! ica symmetry breaking, a phase tra nsition to optimal learning at some finite $\alpha$ for all levels of noise in the hidden layer. When noise is added to the input layer the generalization behavior is similar to that of the perceptron. For one step of replica symmetry breaking, in the realizable limit, the values of the spinodal points found in this paper disagree with previously reported estimates \cite{seung1},\cite{schwarze1}. Here the value $\alpha _{sp} = 2.79$ is found for the tree committee machine and $\alpha _{sp} = 1.67$ for the perceptron.


Introduction
A Feed-forward neural network can be used to estimate an unknown rule from random examples [3] by adaption of its weights. Using methods from statistical mechanics of disordered systems [4] the performance of a student network trained on examples obtained from a teacher network of the same architecture has been studied (for a review see [5]). In this case the rule is said to be realizable since it is possible for the student to develop the same weights as the teacher.
One way to construct an unrealizable rule is to allow for a teacher that is larger (more units) than the student. This will be shown to be equivalent to adding Gaussian noise to the training set. The noisy data scenario has been investigated for networks with continuous weights [6], [7], [8]. In the limit where the teacher is infinitely larger than the student (large noise limit) the only thing the student can do is to learn each example by heart, and in this limit the problem reduces to that of storage capacity.
In this paper the generalization behavior of two different types of binary neural networks with binary weights is studied, a perceptron (section 2) and a tree committee machine (section 3), in the limit where the number of units is large. The rule is defined by a teacher of the same type but having more units than the student, making the task unrealizable.
The training of the student, having N units, is based on αN examples obtained by picking inputs ξ µ and assigning outputs τ µ as given by the teacher. With σ µ being the µth output of the student, a training energy E = µ θ(−σ µ τ µ ) is defined, which leads to a probability density with Boltzmann weight e −βE , where β = 1/T is the inverse temperature. First the high temperature limit is considered for each type of network, the perceptron in section (2.1) and the tree committee machine in section (3.1). Then, in sections (2.2) and (3.2), the replica trick is used, assuming replica symmetry (RS), to study the average over all training sets of the free energy, βf . In sections (2.3) and (3.3) the corrections given by one step of replica symmetry breaking (RSB) are discussed. Since, in the noiseless limit, the value of the spinodal point found in the RSB-case disagrees with previously reported estimates [1], [2], some time is spent on the saddle point equations in appendix A. Finally, in appendix B, the procedure for finding the asymptotic generalization behavior for large α is given.

A Large Binary Perceptron with Ising weights.
Let the student and the teacher have N and M input units respectively with N ≤ M.
Presented an input, s, the teacher evaluates, τ ( s) = sgn( v · s), while the student computes, σ( s 0 ) = sgn( w 0 · s 0 ), given the input s 0 . Here s and v are elements of R M , while vectors having a zero subscript are elements of R N . When the student is presented the same input vector, ξ, as the teacher, it only considers the N first components, ξ 0 . Thus the target rule will be, where v 0 is constructed from the first N components of v. Effectively this means that the student will be given the task τ ′ ( ξ 0 ) = sgn( v 0 · ξ 0 ) with noise on the input vector ζ 0 = ξ 0 + κ 0 and/or on the weight vector J 0 = v 0 + ω 0 . Since η is constructed from independent Gaussian random variables, v j and ξ j (j = N + 1, ..., M) with unit variance, η will also be Gaussian with variance, where γ = N/M . γ has the simple interpretation of the relative size of the student to the teacher. If γ = 1 the student and the teacher are of the same size, i.e. there is no noise. If γ = 0 the teacher is infinitely larger than the student, i.e. the data will be completely noisy.
The generalization error, ǫ g , obtained by taking the average of θ(−στ ) over normal distributed inputs, s j (j = 1, ..., M), is where R is the overlap between w 0 and v 0 . For R = 1 we obtain the optimal value, ǫ opt , of ǫ g .
First the high temperature limit is considered. Then by using the replica method, the RS approximation is studied, and finally the corrections given by one-step RSB are discussed.

High Temperature Limit
In previous work [1] the high temperature limit has proven to be interesting since it is both computationally easy and gives the general behavior of learning. It is defined so that both α and T approach infinity while αβ remains constant. The free energy is simply The qualitative behavior of the learning curves can be divided into two types depending on whether the noise level is above or below a particular value γ 0 . For γ 0 < γ < 1 there is, as in the realizable case, a range (αβ) sp1 < αβ < (αβ) sp2 , for which βf has two minima. In between (αβ) sp1 and (αβ) sp2 there is a transition point (αβ) tr at which the global properties of the minima change. In contrast to the noiseless case, (αβ) sp1 > 0 and thus for 0 ≤ αβ ≤ (αβ) sp1 there is only one minimum. The minimum persisting also for αβ ≥ (αβ) sp2 is close to R = 1 and approaches optimal performance as αβ increase. Note that in contrast to the realizable case there is no solution at R = 1. Typically (αβ) tr , (αβ) sp1 and (αβ) sp2 increase with decreasing γ and merge at γ = γ 0 . This is illustrated in figure (??).
The two minima of βf must be separated by a maximum, implying that ∂ 2 βf ∂R 2 = 0 at the spinodal points. Using the saddle point equation, to eliminate αβ from ∂ 2 βf ∂R 2 = 0, gives For γ = 1, (2.1.3) has one solution, R sp = 0.83 resulting in (αβ) sp = 2.08 in agreement with [1]. In the region γ 0 < γ < 1 (2.1.3) has two solutions giving (αβ) sp1 and (αβ) sp2 . At γ = γ 0 the two solutions merge and the two curves R and g(R, γ) are tangent to each other. Thus γ 0 can be found by solving, giving γ 0 = 0.965. For γ < γ 0 , βf has only one minimum (for all αβ) which moves towards R = 1 as αβ approaches infinity. Note that fairly small amounts of noise will change the qualitative behavior from phase transition to no transition.
In weight space this behavior can be understood as follows. In the noiseless case there are, for small α, two regions in weight space corresponding to the minima of βf , one with poor generalization and one with good. If α is small enough the "poor" region has the lowest free energy. As α increase the "poor" region moves towards the "good" and for α > α tr the "good" region has the lowest free energy. Since for α = α tr the "poor" and "good" regions are separated, there will be a phase transition.
If noise is added, the sizes of these regions will increase. For low α there is only one region in weight space corresponding to a minimum of βf . It will have poor generalization.
At α = α sp1 another region corresponding to a free energy minimum appears. This region gives better generalization. Again as α increase the "poor" region moves towards the "good" and for α > α tr the "good" region has the lowest free energy. Since for α = α tr the "poor" and "good" regions are separated there will be a phase transition. If the noise is increased the "poor" region is so large that when the "good" region is created it will overlap with the "poor". Thus there is only one region, moving towards better generalization and there is no phase transition.

Replica Symmetric Theory
Using the same methods as in [1] the RS approximation to the free energy is obtained, The saddle point equations generated by the extremal condition in (2.2.1) is given in appendix A. Here q is the typical overlap between two different w 0 . R, γ and α have the interpretation given above. Using the saddle point equations we can, given γ and β, eliminate the auxiliary variablesR andq, and find the dependence of R (and ǫ g ) on α.
First consider the zero temperature case. This corresponds to only allowing students that answers all questions correctly. If γ < 1 the training data is noisy and there is a maximum size of the training set α c N beyond which no student can perform optimally. α c (γ) and R c (γ) are plotted in figure (??). For γ = 0 the known result of Gardner [9] is reproduced.
Note that the curves do not give α c → ∞ as γ → 1. However this may not be expected since the curves only give correct predictions for states that are stationary points of βf RS and in the realizable case the state R = 1 is not stationary as was shown in [1]. For γ = 1 both the transition and the spinodal points agrees with the values found in [1]. A learning curve for At T > 0 the learning behavior is the same as for the high temperature limit but with a different γ 0 , depending on T , and with ǫ g and q having the asymptotic form, for large α. For details on how to compute the asymptotic form see appendix B. For some range of γ, γ A < γ < 1, there is a phase transition already at zero temperature while in a range γ B < γ < γ A there is no transition at low temperature. As the temperature is increased a transition develops which is illustrated in figure (??) for γ = 0.99. Finally when γ < γ B there seems to be no phase transition no matter how high the temperature.

Replica Symmetry Breaking
In the RS approximation the entropy will always turn negative at some finite α and therefore a region in αT -space for which the system exhibits replica symmetry breaking (RSB) is expected, see figure (??). Analogous to [1] one step of RSB gives, where the extremum is taken over R,R, q 0 ,q 0 , q 1 ,q 1 , and m. As in [1] the limit q 1 → 1, q 1 → ∞ is considered, implying that the stationary points of f RSB are given by the stationary points of f RS having zero entropy (see appendix A for details).
The learning behavior is analogous to the high temperature limit but with γ 0 = 0.995.
In appendix B the asymptotic form of ǫ g and q is computed, When 0.999 ≤ γ < 1, α tr occurs in between α sp1 and α sp2 while for 0.996 ≤ γ ≤ 0.998, α tr = α sp1 , i.e. the state with better generalization is stable as soon as it appears.
For the case γ = 0.05 the critical capacity, α c = 0.83 (q c = 0.56) is found which is compatible with the known results for γ = 0 [10]. Some values are given in table (??) and some typical learning curves are given in figure (??).
In the noiseless limit the result, α sp = 1.67, correcting a previous result by Seung et. al.
[1] (α sp = 1.63). The reason for this is given in appendix A.
It is also interesting to compare with some recently reported upper bounds for the Ising perceptron [11]. In this article the asymptotical behavior was found to be the same as (2.3.5). The authors found that the phase transition disappeared below γ = 0.998 thus not only predicting the correct qualitative behavior but also giving a tight quantitative bound on γ 0 . Also, at γ = 0.998, they found α tr = 2.6136 whereas α tr = 1.83 is obtained at γ 0 given above and using the replica method.

A Binary Committee Machine with Ising weights.
as the kth perceptron in the student is given the input s Analogous to the simple perceptron we find that this is equivalent to learning a noisy target rule, where η and η k are independent Gaussian random variables with variance, Using these parameters the generalization error is found, where the effective order parameter is given by R e = 2 π γ arcsin(δR) and R is the typical overlap between w k . Here, analogous to Schwarze and Hertz [2], it is assumed that R is independent of the hidden unit index k.
As for the perceptron case the high temperature limit is considered first. Then by using the replica method, the RS approximation is studied and finally the corrections given by one step of RSB are discussed.

High Temperature Limit.
Taking the limits T → ∞ and α → ∞ while keeping αβ fixed the free energy is found, If the noise level is low enough, there exists two spinodal points, α sp1 and α sp2 , with a phase transition in between. In contrast to the perceptron one find that if there is no input noise (δ = 1) there is a phase transition to optimal performance at a finite α for all values of γ.
Given a γ and that δ 0 (γ) < δ a transition to a state approaching optimal learning in the large α limit is found. For δ < δ 0 (γ) the transition vanishes and ǫ g approaches ǫ opt as α tends to infinity. Especially if δ > δ A = δ 0 (0) there is always a phase transition while for δ < δ B = δ 0 (1) there is no phase transition independent of the hidden noise. By the same procedure as in section (2.1) one find δ A = 0.965, δ B = 0.924 and δ 0 (γ) as shown in figure   (??). Also here α sp1 , α tr and α sp2 increase with increasing noise.

Replica Symmetric Theory.
Analogous to Schwarze and Hertz [2] the RS estimate to the free energy is found, where R e is given above and q e = 2 π arcsin q. The value of q is the typical correlation between two w (0) k which are assumed to be independent of the perceptron index k. The interpretations of R, γ, δ and α are as given above. By using the equations generated by the extremal condition in (3.2.1) to eliminateR andq we can find the dependence of R (and ǫ g ) on α given γ, δ and β.
For T = 0 one should, as for the perceptron, find a critical capacity, α c , beyond which the student can not perform optimally on the training set. However, this is not the case implying that the RS-approximation is bad. In the realizable case the values of both the transition and the spinodal point agree with [2] At T > 0 the behavior is much the same as in the high temperature limit with the exception that for δ = 1 the transition is not to an optimal state but to a state approaching optimal learning as α tends to infinity. The asymptotical form of ǫ g and q for large α can be found for δ = 1, γ < 1, and for δ < 1, γ ≤ 1, The asymptotic behavior can be found by the same method used in appendix B. As for the perceptron there is a range of noise levels for which there is no phase transition at low temperature but one is developed as the temperature is increased. One such example (γ = 1, δ = 0.99) is used in the phase diagram (??). If the noise is increased above some value there seems to be no phase transition no matter how high the temperature.

Replica Symmetric Breaking.
As was said in the previous section the RS-approximation fails in predicting a critical capacity. Also, the entropy will turn negative at some finite α and thus RSB is expected. In figure (??) a phase diagram for γ = 1, δ = 0.99 shows the RSB region. For one step of RSB, in the limit q 1 → 1,q 1 → ∞, the free energy is,

3.4)
For reasons analogous to those given in [1] for the perceptron the stationary points of f RSB are given by the stationary points of f RS having zero entropy.
In contrast to the RS-case, but analogous to the high temperature limit, a transition to optimal learning is found for all γ if δ = 1. Using the same notation as in section (3.1) the values of δ A and δ B are 0.9995 and 0.9847 respectively, and δ 0 (γ) is given in figure (??).
As α → ∞ the asymptotic forms of ǫ g and q are, Appendix B gives details of how to compute the asymptotic behavior, using the perceptron as an example. In the realizable limit (δ = γ = 1) the result α sp = 2.79, correcting the value found in [2] (α sp = 2.58). The reason for the correction is given in appendix A, where the perceptron is used as an example.

Summary.
In summary we have studied unrealizable learning in two large feed-forward neural network, a perceptron and a tree committee machine within the replica symmetric ansatz as well as for one step of replica symmetry breaking. The average generalization error has been calculated as a function of the load parameter α.
For the perceptron it was shown that using a noisy training set results in a generalization error approaching optimal learning with increasing α according to a power law of (ln α/α) k with k = 2 in the RSB-case. If the noise is low enough there is a phase transition at some finite α to a state which is close to R = 1. Increasing the noise makes the transition go away.
For the tree committee machine a similar generalization behavior was found, the main difference being that there is always a transition to optimal learning at some finite α if there is no noise in the input layer. Typically, noise in the input layer gives worse generalization behavior than noise in the hidden layer. For one step of RSB and with noise in the input layer as well as in the hidden layer the asymptotic form of ǫ g was found to be (ln α/α) k with k = 2/3.
In the realizable cases the values of α sp correct previously reported results [1], [2], for the RSB spinodal point in the two machines. Here α sp = 1.67 was found for the perceptron and α sp = 2.79 for the tree committee machine.
I thank J. Hertz for his valuable advice and direction and R. Urbanczik for many useful discussions. Also, I would like to thank H. Schwarze for sharing the code written in connection to ref. [2] which made it possible to sort out the disagreement on the spinodal points.

A. The Saddle Point Equations
In the limit q 1 → 1,q 1 → ∞ the one step RSB free energy (2.3.1), of the perceptron, is  At this point we could try to solve forq andR given α. However since ǫ g is a many-valued function of α it is more economical to eliminate α. This will give the equation, which can be solved forq givenR. α can be evaluated using (A.6) or (A.7). The advantage is that ǫ g is a single valued function ofR. In the RSB-case this will be helpful since more than one solution for each α has to be considered as we show below.
Once a stationary point has been found its second order properties has to be checked by computing the determinant of the Hessian matrix, H. Assume that the correct sign of det H, atR > 0, is given by the sign atR = 0. AsR is increased, the sign of det H will change first atR sp2 and then again atR sp1 . Note thatR sp2 <R sp1 whereas α sp1 < α sp2 . In the regimeR sp2 <R <R sp1 , βf has a stationary point but it has the incorrect curvature.
Even though the RSB-case is solved by using the RS-equations the determinant of the Hessian matrix of βf RSB , det H RSB , has to be used to determine the second order properties.
det H RSB consists of the second derivatives of βf RSB with respect to R, q,R,q and m whereas det H RS is computed from the second derivatives of βf RS with respect to R, q,R andq. Using det H RS = 0 as the criterion to determine the spinodal point (at γ = 1) would result in the values of α sp as given in [1] and [2]. Moreover, insisting on this RS-criterion will, for some γ, result in regions of α where no solution exist. Thus this procedure fails in a disastrous way.
However the correct condition, det H RSB = 0, will cure this problem and give α sp as given in this paper.
In the RSB-case det H RS will have the wrong sign close to α sp . This will correspond to points on ǫ g not considered in the RS-case since there exists another solution (with s > 0) at the same α but with det H RS having the correct sign. This is illustrated (for γ = 1) in and since 0 < m < 1, det H RSB has the same sign as F .

B. Asymptotics
For large α the saddle point equations (A.2) implies that q, R are close to 1 andq,R are large. From this the asymptotic form of the free energy (2.2.1) for non-zero temperatures can be found, βf RS = extr R,R,q,q G r (R, q, α, γ, β) + G s (R,R, q,q) , (B.1) The first two of these can be combined into, The asymptotic form of R, q and ǫ g is now found from (B.15) and (B.16) giving, For the tree committee machine the asymptotic forms of R, q and ǫ g can be found by the same procedure but using the asymptotic form of the free energy (3.2.1).