Neural network with multiplexed snyaptic processing
FIELD OF INVENTION 
     The invention pertains to the field of associative networks for emulating higher-order functions of the cortex of the human brain. More specifically, the invention pertains to multiplexed operation of neural cell resources resulting in a more economic implementation. 
     BACKGROUND OF THE INVENTION 
     FIG. 1 is a block diagram of a McCollough-Pitts (MP) type neuron widely in use today. The unit has a set of parallel input ports 11 for accepting an N-element input vector, [u 1 , u 2 , . . . , u N  ] T . Weighing elements 13 scale (or multiply) their corresponding input vector elements to form a set of product terms, {u j  w ij  }, each of which is applied to adder unit 15 to form the vector dot-product, [u·w]=Σu j  w ij . Also, an offset or bias term, θ i , applied to input 12 may be added to form the sum [u·w] i  +θ i . Each input branch consisting of an input data element 11 and a weighting element 13 corresponds to a neural synapse. The sum out of adder 15 is applied to a nonlinear element 19, usually having a sigmoidal transfer characteristic to form the neuron cell output signal. 
     One realization of the synapse portion of the MP type neuron shown in FIG. 1 may be found in U.S. Pat. No. 4,956,564, Sep. 11, 1990, by Holler et al. The patent discloses &#34;a synapse cell for providing a weighted connection between an input voltage line and an output summing line having an associated capacitance&#34;. In this case, the &#34;associated capacitance&#34; performs the summing function of adder 15. A single synaptic element of the above reference, shown in FIG. 2, is a four quadrant multiplier using floating gate transistors 40-43, differential input lines 50 and 51, and differential output lines 44 and 45. The drains of transistors 40 and 42 are coupled to current summing line 52 while the drains of transistors 41 and 43 are coupled to current summing line 53. Summing line 52 and 53 have at nodes 44 and 45 associated capacitances 46 and 47, respectively, for accumulating charge. Select devices 60 and 62 are n-channel transistors for selectively connecting the sources of floating gate transistors 40 and 42 to line 64; devices 61 and 62 selectively connect transistors 41 and 43 to line 65. Return lines 64 and 65 are also used for erasing and normal reading operations. 
     For positive inputs, a voltage pulse of t pp  seconds is applied to input line 50 while for negative inputs a voltage pulse of t pn  seconds is applied to input line 51. The duration of the pulse is made proportional to the magnitude of the input signal. The charge associated with the current flow through the floating-gate devices is representative of the product of the input pulse duration and the stored weight in the floating gate devices. The stored charge on capacitances 46 and 47, and hence voltage across 46 and 47, represent the four quadrant multiplier output values in differential form, i.e., the voltage difference between output lines 44 and 45. 
     For each input synapse of a given neuron, a structure similar to that shown in FIG. 2 is required. FIG. 3 shows a typical single layer neural network with N input channels and M neurons. Thus, N×M multiply units 13 are required, each similar to that shown in FIG. 2 or its functional equivalent. Line 21, the input line to accumulator 15, would be a differential input line pair, corresponding to lines 44 and 45 of FIG. 2, if that type of differential multiplier were to be used. Similarly, input line 11 would correspond to differential input lines 50 and 51. 
     Other forms of multiplier (weighting) elements may be used, based on the four quadrant transconductance multiplier cell shown in FIG. 4. (See J. Watson, &#34;Analog and Switching Circuit Design&#34;, Adam Hilger Ltd, Bristol, 1984, pp. 134-140). The product of inputs 102 and 77 are represented by the current difference, ΔI, which corresponds to the difference in collector currents flowing in leads 71 and 73. Thus, assuming matched transistors (81-86), ##EQU1## where q is the charge on an electron, 
     k is Boltzman&#39;s constant, 
     T is the absolute temperature. 
     For x small, tanh x≈x so that equation (1) may be approximately represented by ##EQU2## 
     One synapse implementation shown in FIG. 5, the commercially available Intel Model 80170NX electrically trainable analog neural network (manufactured by Intel Corporation, 3065 Bowers Avenue, Santa Clara, Calif., 95051), is based on a transconductance four quadrant multiplier using CMOS technology. Transistors 93 through 98 form a four quadrant multiplier unit similar to the type described above. The differential input vector component, ui, is applied by input lines 102 and the differential weights, w ij   +  and w ij   - , supplied by EEPROM cells 900, are applied to the gates of transistor 93 and 94, respectively. Select line 92 activates cells 900 by applying Vcc to the gates of transistors 91 and 99 that act as control switches for applying the drain voltage, Vd, and the control gate voltage, Vcg, to floating gate transistor 90 of each EEPROM cell 900. By selecting appropriate values of Vcg, Vd and source voltage, Vs, stored weights values (w ij   + , w ij   - ) in the floating gate transistors 90 may be modified or applied to transistors 93 and 94 of the four quadrant multiplier. The differential output current, ΔI j  =I j   +  -I j   - , available on output leads 21, is representative of the product u j  * w ij , where w ij  =w ij   +  -w ij   - . 
     As in the previously described synapse implementation by Holler et al., a single layer network of N inputs and M neurons requires an interconnected matrix of N×M synapse cells. 
     To change the weights stored in the differential pair of EEPROM cells 900, charge is added to one cell&#39;s floating gate and subtracted from the other using Fowler-Nordheim tunneling. Tunneling is commonly used in conventional EEPROM devices by applying high voltages across a thin dielectric causing electrons to &#34;tunnel&#34; directly through the dielectric. The Intel model 80170NX synapse multiply characteristics, shown in FIG. 6, are plot of differential output current, ΔI j , versus input voltage, u j , for various values of differential voltage stored in the floating gates of the EEPROM cell pair. 
     Because artificial neurons find application to problems requiring substantial parallel processing (many neural cells simultaneously active), the above implementations have the disadvantage of requiring N×M multipliers for each neural network layer of N input channels (input vector elements) and M neurons. This requirements limits the capacity (N×M) of any integrated circuit implementation. Also, accommodation of large input vectors and output vectors is difficult because of the large number of pins required. These two problems provide motivation for the present invention to be described. 
     BRIEF DESCRIPTION OF THE INVENTION 
     The present invention covers artificial neurons using a multiplexed synapse array based on the flash EEPROM floating-gate memory technology. The synapse array provides the input multi-channel weighting means to an artificial neuron cell by forming the vector dot-product of the multi-channel input data vector and a stored reference vector. Analog multiplication is accomplished by using a transconductance four-quadrant multiplier circuit. Significant savings in the number of transistors per synapse required is achieved by using a flash memory array. Each of its memory cells uses a floating-gate transistor to store an analog weight in the form of a charge on the floating gate. Pairs of transistors are used to store differential analog weights, w ij , representative of the value of a specific reference vector component by controlling the differential transconductance. When any such pairs of floating-gate transistors is used in a four-quadrant transconductance multiplier, the output signal is representative of the product u j  * w ij , where u j  is the input data value applied to the other differential input terminals of the multiplier. Thus, by selectively multiplexing floating gate transistor-pairs into a common transconductance four quadrant multiplier circuit a significant savings in component count per synapse structure is realized. 
     Another object of the present invention is to provide means for multiplexing input data channels in order to minimize the input data pin count. 
     Another object of the present invention is to provide pipeline processing of synaptic data thus providing more uniform input and output data rates, thus minimizing the output pin count and bandwidth requirements. 
     Another object of the present invention is to provide an efficient vector dot-product circuit for general signal processing requirements. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The novel features believed characteristic of the invention are set forth in the appended claims, The invention itself however, as well as other features and advantages thereof will be best understood by reference the description which follows, read in conjunction with the accompanying drawings, wherein: 
     FIG. 1 represents a typical prior art McCullough-Pitts (MP) artificial neuron. 
     FIG. 2 is a prior art differential synapse cell in accordance with U.S. Pat. No. 4,956,564. 
     FIG. 3 is a prior art, commercially available,. electrically trainable analog neural network. 
     FIG. 4 is a circuit diagram of a four-quadrant differential multiplier. 
     FIG. 5 is a simplified schematic of a prior art EEPROM synapse implementation using a four-quadrant multiplier. 
     FIG. 6 shows the multiply characteristics of the prior art EEPROM synapse of FIG. 5. 
     FIG. 7 is a simplified diagram of a flash (EEPROM) memory array. 
     FIG. 8 shows a flash memory array based neuron with multiplexed multiplier. 
     FIG. 9 shows an equivalent non-multiplexed single layer neural network. 
     FIG. 10 shows the structure of a pipelined and multiplexed neural network. 
     FIG. 11 shows a typical analog pipeline shift register element. 
     FIG. 12 is a timing diagram for the pipelined network of FIG. 10. 
     FIG. 13 is a block diagram of a pipelined and multiplexed neural network with parallel multipliers sharing a common integrator. 
     FIG. 14 shows the details of the multiplier units used in FIG. 13. 
     FIG. 15 is a block diagram of a parallel processing neuron cell with a multi-input integrator and segmented single vector input. 
    
    
     DETAILED DESCRIPTION 
     FIG. 7 shows a simplified diagram of an EPROM flash memory array. Each memory cell 101, comprises a single floating gate transistor connected in a two-dimensional addressable matrix. The control gates of a given row may be activated by selecting any one of the row-select lines 104 and applying an appropriate voltage level. Similarly, a specific transistor cell 101 along the selected row is chosen by selecting the appropriate column select line 105. Thus, selecting row i and column 2 j  would select the memory cell having w i ,j +  stored on its floating gate. 
     By selecting two columns, say 2j and 2j+1, a pair of floating gate transistors having weights w i ,j +  and w i ,j -  could be made available for use as transistor pairs corresponding, respectively, to transistors 93 and 94 of the transconductance multiplier shown in FIG. 5. 
     FIG. 8 shows how a flash memory array may be used to create a multiplexed neural cell. The neural cell comprises synaptic weight flash memory array 130, auxiliary synaptic unit 131, row selector 103, and output non-linearity 19. Pairs of columns of the flash memory array, say 2j and 2j+1, are connected to the sources of transistors 95, 96 and 97, 98 respectively. The drain of these transistor pairs are connected differentially to accumulator (integrator) 15 which accumulates the differential drain current flow, ΔI. The control gates of the transistor pairs 95, 96 and 97, 98 are differentially connected to a balanced input terminal pair 102. Row selector 103 provides output voltage levels that are selectively applied to gate control lines 11, thereby activating a single row of differential weights, 
     
         {w.sub.i,j =w.sub.i,j.sup.+ -w.sub.i,j.sup.- }. 
    
     Because the floating gate charge controls the drain to source conductance in proportion to the stored weights, the differential current flowing into accumulator 15 is representative of the product u i  * w ij . In order to form the complete dot-product,  i  Σu i  w ij , the input data vector values are sequentially selected (i.e., u 1 , u 2 , . . . , u N ) while row selector 103 sequentially selects the corresponding weights (w 1j , w 2j , . . . , w Nj ). In this manner, input data vector values, {u i  }, are multiplexed as well as the stored reference vector values, {w ij  }. The offset (threshold) value, i.e. θ j  of FIG. 3, may be introduced by having an input value, u k  =u, to assume a standard full scale or other standard value which is suitably scaled by the floating gate transistor of a prescribed row (k) of the memory array to form θ j  =u * w kj . 
     The output of accumulator 15, representative of the vector dot-product is applied to the output sigmoidal nonlinearity 19 to produce the j th  neural response, v j . For each available flash memory column cell pair (2 j , 2 j  +1), a similar arrangement permits parallel implementation of multiple neurons, each acting on the same elements of input vector, u, and simultaneously generating a multiplicity of unique vector dot-products. This mode of operation corresponds to a single layer multi-neuron network as shown in FIG. 9. The input to the single layer neural network of FIG. 9 is the N×1 input vector, u, on input lines 11 and is shown fully interconnected with M neuron cells 120 of the type shown in FIG. 1. 
     A direct comparison of the synapse implemented in FIG. 5 using EEPROM cell technology and that of FIG. 8 implemented using flash EPROM cell technology clearly shows the significant reduction in complexity achieved by the latter embodiment. Each EEPROM cell 900 requires 3 transistors (6 for each differential pair (w ij   + , w ij   - )) in addition to 6 transistors for implementing the four quadrant multiplier, a total of 12 transistors per synapse. The multiplexed flash EPROM of FIG. 8 each require 2 transistors for each differential weight pair in addition to 4 transistors (95-98) for completing the multiplexed four quadrant multiplier. Hence, for a single layer neural network, such as in FIG. 9, the ratio of the number of transistors required for the flash EPROM implementation to the number required for the EEPROM implementation is ##EQU3## or approximately a reduction of 6 to 1 for any reasonable value of N. This reduction reduces the silicon area required, as well as the power consumption. 
     In another preferred embodiment, the dynamic range of the transconductance multiplier units may be extended by varying the length of time over which the differential current, ΔI, is accumulated in integrator 15 of FIG. 8. Control line 17, when asserted, causes integrator 15 to operate as an integrator which accumulates the change due to the current flow of ΔI. When control line 17 is de-asserted, the integration stops. This action can be expressed mathematically as ##EQU4## so that the total contribution at output of integrator 15 of the j th  column is represented by the sum ##EQU5## where T ij  is the interval of time that control line 17 is asserted for the ith weighting pair of the j th  column (w ij   + , w ij   - ) Thus, the combined weight w ij  T ij  provides a means for extending linear dynamic range of weight w ij . 
     This method may also be used to extend the dynamic range of the input u i  by making all T ij  =T i . In other words, by applying the same integration time interval to all neurons at the same level, i. And similarly, extending both of the dynamic ranges of the weights, {w ij  }, and input values, {u i  } by making T ij  proportional to the product of a weighting scaling factor and an input scaling factor. 
     It should be recognized that many different realizations are possible: 
     (1) the flash array transistors may be a standard flash memory chip with the auxiliary synaptic circuitry, i.e. transistors 95-98 and units 15 and 19, off-chip; 
     (2) both flash memory and auxiliary synaptic circuitry may be consolidated on a single chip; 
     (3) multiple flash memory units may be connected to extend the column length of the input vector; 
     (4) complete synaptic chips, each including a full complement of auxiliary synaptic circuitry and flash memory, may have the individual synapse outputs jammed across the array of chips to extend the input vector length; and 
     (5) some level of multiplexing of the auxiliary synaptic circuitry across flash memory column pairs. 
     Other possible variations to those suggest will be evident to those skilled in the art. 
     Because the availability of output pins may be severely limited and because the basically parallel structure described in the above embodiment produces simultaneous parallel outputs from each column neuron it may be advantageous to organize the network as a pipeline process producing neuron outputs sequentially in time. This would reduce the output pin requirement by multiplexing a multiplicity of neuron outputs to a single output port. 
     FIG. 10 is a block diagram of a neural network with both input and output multiplexing. Memory array 130 comprising N×M memory 101 cells and row selector 103 are as previously described in FIG. 8. Auxiliary synaptic units 131, also described in FIG. 8, are connected to an associated memory cell set of j th  column pairs, {w i ,j + , w i ,j -  }. Each pair represented as memory cell unit 102 with a corresponding weight of w i ,j. The serially multiplexed input values, {u i  }, on input line 102 are supplied to auxiliary synaptic units (SYN) 131 from taps on all analog delay line comprising serially connected delay elements (SR) 150. In this manner, each synaptic unit (SYN 1  to SYN m ) receives a particular input vector element delayed with respect to its neighbor, i.e., SYN k  receives the value u i  one unit of time before SYN k+1 . Each SYN unit 131 forms a vector dot-product as in the previous embodiment except shifted in time by one unit relative to its neighbor. 
     The output of each SYN unit 131 is connected to analog selector means 133 accommodating M inputs. (The selector may be a set of M field effect transistors operating as analog switches.) The output of selector 133 is applied to output non-linearity 19 which provides the squashed output for the selected synaptic set. Alternatively, M nonlinear networks could be connected, on each at the output of each SYN 131 network, each output of nonlinear networks 19 being supplied to output select unit 133 which then delivers at its output the time division multiplexed neuron output values. 
     FIG. 11 is an example of an analog delay element (SR 150) employing two n-channel field effect transistors 151 and 152 whose gates are controlled by clock signals Ψ 1  and Ψ 2  respectively. The input signal is applied to capacitor 153 through transistor 151 by clock signal Ψ 1 , when asserted positive. During the next half period, Ψ 1 , is de-asserted and clock Ψ 2  asserted causing the voltage on capacitor 153 to be transferred through op-amp 155 and transistor 152 to capacitor 154 and appear at the output terminal of op-amp 156. On the next half cycle Ψ 1 , is again asserted and the process continues. In this manner the input voltage at transistor 151 at the beginning of a cycle is transferred to the output of SR 150 after one clock period delay. 
     FIG. 12 shows the timing relationship between serially multiplexed input vector elements, {u i   k  }, and the availability of neuron output values, {v j   k  }. Line (a) represents the time index. Lines (b)-(e) represent the arrival of input values at the M delay line taps. For example, at time t=0, SYN (1) has, u N   k , available (and w N1  available from memory 130 simultaneously) to form the product u N   k  ·w N1 . In this case the superscript, k, denotes that u N  is the N th  component of the k th  input vector, u k . Because this represents an on-going process, the complete vector dot-product, w i   T  ·u k  = i  Σu i   k  w it is available at the output of SYN(1). At the same time, sigmoid nonlinearity 19 produces the squashed output v 1   k  through output selector means 133 as shown on line (f) of FIG. 12. Similarly, at time (or clock) index 1, the output of nonlinearity 19 corresponds to the second neuronal output for the k th  input vector, v 2   k . At time index M-1, the M th  neuronal output, v M   k , is available. The output switching sequences is repeated N time index intervals later for input vector u k+1 , producing the set of outputs {v j   k+1  }, for j=1, 2, . . . , M. 
     This method of operation implies a staggered set of stored weights as indicated in memory array 130 of FIG. 10, where each square is labelled to indicate the contents of each differential memory cell 102, i.e. row one (i=1) contains the following ordered set: 
     
         {w.sub.1,1, w.sub.N,2, w.sub.N-1,3, w.sub.N-2,4, . . . , w.sub.N-M+2, M } 
    
     Each succeeding column is rotated downward by a single unit thereby introducing the required shift so that corresponding pairs of delayed input vector elements, u i   k , will be available at each of the M delay-line taps together with selected row weight contents, w ij  to form the proper product term. 
     In another embodiment, the parallel processing neuron cell shown in FIG. 13, weight memory array 130, row select unit 103, analog delay units (SR) 150 and output non-linearity 19 are as previously described and shown in FIG. 10. However, in this embodiment, a single, multi-input integrator 16 accepts the parallel output of a multiplicity of multiplier units 132. Each multiplier unit 132 comprises four field effect transistors 95, 96, 97, and 98 as shown in FIG. 14. Thus, by separating the integrator unit 15 from synaptic unit 131 of FIG. 8, it is possible to share a single integrator unit for parallel accumulation of product terms. If, as shown in FIG. 13, three multiplier networks 132 (MUL1, MUL2 and MUL3) were to share a single integrator 16, then input values {u i  } would be shifted by three input samples each time a new set of values were to be generated. In general, K multipliers units 132 feeding a single integrator 16 would require shifting input values by K time units for each output value. 
     In a further variation of the parallel processing neuron cell of FIG. 13, multiple parallel input vectors may be supplied from an external source obviating the need for a tapped delay line using SR memory elements 150. For example, inputs u i , u i-1 , and u i-2  would be supplied by three separate input lines. More generally, as shown in FIG. 15, the input vector, u, may be stored as non-overlapping segments in FIFO memory unit 200 which is organized as a multiplicity of parallel FIFO output units 201. For the example shown, a kM×1 input vector is stored in k parallel FIFO units 201 so that each 201 unit simultaneously supplies a set of k input vector components (u l , u u+l  . . . u.sub.(k-1)M+l) to one operand input of a corresponding auxiliary multiplier unit 132, causing the product terms 
     
         (u.sub.l ·w.sub.l, 1, u.sub.M+l ·w.sub.M+l, 2, . . . , u.sub.(k-1)M+l ·w.sub.(k-1)M+l, k) 
    
     to be simultaneously formed at the output of the set of k multiplier units 132.