Fuse and Mix: MACAM-Enabled Analog Activation for Energy-Efficient Neural Acceleration

Analog computing has been recognized as a promising low-power alternative to digital counterparts for neural network acceleration. However, conventional analog computing is mainly in a mixed-signal manner. Tedious analog/digital (A/D) conversion cost significantly limits the overall system's energy efficiency. In this work, we devise an efficient analog activation unit with magnetic tunnel junction (MTJ)-based analog content-addressable memory (MACAM), simultaneously realizing nonlinear activation and A/D conversion in a fused fashion. To compensate for the nascent and therefore currently limited representation capability of MACAM, we propose to mix our analog activation unit with digital activation dataflow. A fully differential framework, SuperMixer, is developed to search for an optimized activation workload assignment, adaptive to various activation energy constraints. The effectiveness of our proposed methods is evaluated on a silicon photonic accelerator. Compared to standard activation implementation, our mixed activation system with the searched assignment can achieve competitive accuracy with $>$60% energy saving on A/D conversion and activation.


INTRODUCTION
Deep neural networks (DNNs) have received an explosion of interest due to state-of-the-art inference accuracy in a myriad of artificial intelligence tasks. In parallel, the rapidly escalating model size and data volume raise a surging need for more efficient computing solutions. However, as Moore's law winds down, it becomes increasingly challenging for conventional digital counterparts to meet the computational demands of DNN workloads. A slew of new processor architectures employing analog techniques is keenly sought to reduce power dissipation and improve computational speed. Crossbarbased processing-in-memory (PIM) architectures [2,10,18,19,26] and integrated optical neural networks (ONNs) [6,8,9,20,21,28] are two prominent examples in this direction.
However, analog computing is mainly in a mixed-signal manner: digital inputs are transformed into analog signals for computation, and then computing results are converted back to the digital domain Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. ICCAD ' Figure 1: Power breakdown of (a) a crossbar-based CNN accelerator ISAAC [19] and (b) a silicon photonic accelerator Mars [17]. WDACs means DACs for weights, and VDACs means DACs for vector inputs. In Mars, it chooses a more costly DAC configuration than the ADC configuration, and most of the power consumption in digital part comes from data movement.
for the downstream operations, e.g., activation. Tedious digital/analog (D/A) and analog/digital (A/D) conversion overhead exists and hinders the overall system energy efficiency [7,22,27]. Figure 1 shows two case studies of energy breakdown in two representative analog convolutional NN (CNN) accelerators, where the A/D conversion achieved by costly analog-to-digital converters (ADCs) counts for a significant portion of overall power consumption. The A/D conversion overhead concern is further escalated in ONNs due to its high-speed ADC requirements.
Based on the above analysis, we observe a solid demand for seeking another efficient and low-latency alternative to costly ADCs to bring analog signals back to the digital domain. A recently proposed memristor-based analog content-addressable memory (ACAM) [13] shows up to be a promising candidate with picosecond-level latency and femtojoule-level energy consumption. BRAHMS [22] has explored a RRAM-based accelerator to use ACAM to implement nonlinear activation, pooling, and A/D conversion successively, thus eliminating the usage of ADCs. However, the separate implementation of the three operations requires routing the analog signals back to ACAM three times, raising a signal noise concern. Moreover, a severe accuracy drop has been seen in [22] when the precision of A/D conversion implemented by ACAM is insufficient. These obstacles need to be overcome before viable ACAM can be utilized in analog computing to reduce the A/D overhead.
In this work, we devise an efficient analog activation unit based on MTJ-based ACAM (MACAM), simultaneously implementing nonlinear activation and A/D conversion in a fused manner. To compensate for the limited representation capability of MACAM, we propose a mixed activation system that integrates the proposed low-energy analog activation and the conventional high-precision digital activation datapaths. With a given activation energy constraint, a SuperMixer training flow is proposed to automatically learn how to assign activation workloads on the mixed activation system, aiming at balancing the expressiveness and energy cost.
Our main contributions are as follows, • We propose a novel analog and mixed activation system for energy-efficient neural network acceleration. • We devise a fused analog activation unit based on MACAM that can simultaneously achieve nonlinear activation and A/D conversion with significant energy reduction. • We propose a mixed activation system that integrates both analog and digital activation dataflows to balance expressiveness and energy efficiency. • We develop a SuperMixer framework to automatically learn how to assign activation workloads on our mixed activation system adaptive to various energy constraints. • A learnable nonlinearity threshold is proposed with an enhanced training recipe to boost accuracy under limited MACAM resolution, allowing practical application of very nascent technologies to use their benefits and mitigate their present challenges. • We evaluate our methods on a photonic accelerator. Regarding the energy cost of A/D conversion and activation, experiment results show that using fully analog activation units gives ∼65% energy saving, and the searched assignment on the mixed activation system can achieve >60% energy reduction with comparable accuracy.

PRELIMINARIES 2.1 Analog Content-addressable Memory
Memristor-based analog content-addressable memory is recently proposed in [13]. Fig. 2(a) shows the single ACAM cell design that supports search in a continuous analog interval. The lower bound (LB) and upper bound (UB) of the match interval are represented by tuning the resistance of two memristor devices, M1 and M2. The ACAM cell can take an analog voltage as the input, which is applied to the data line (DL). DL is connected to the gate of two switching transistors, S1 and S2. When the input is smaller (larger) than the LB (UB), both S1 and S2 are OFF (ON), making T1 (T2) ON. Then the match line (ML) will be pulled down, leading to a mismatch. If the input is within the interval, the ML will not be pulled down, resulting in a match. The prior study [13,22] has proved the ACAM can achieve the search functionality with picosecond-level latency and femtojoule-level energy consumption. We can cascade multiple ACAM cells to form an ACAM array, where each cell represents one specific interval such that all intervals can consist of a large search interval. The maximum search range is decided by the minimum and maximum resistance of the memristor device. It is obvious that the number of implementable intervals is decided by the number of available resistances of the memristor device.
Regarding the choice of the memristor device, since the ACAM acts as a role of memory with potential frequent access, it inherently requires the chosen memristor to be long-endurance. The magnetic tunnel junction (MTJ) turns out to be a more suitable choice with excellent endurance (10 15 cycles) compared with PCM (10 7 cycles) and RRAM (10 5 cycles) [18]. Among MTJs, Figure 2 Figure 2: (a) Schematic of the ACAM cell [13]. (b) The adopted three-terminal domain wall-MTJ [12].
better endurance while also providing analog resistance levels with excellent stability of the resistance levels over numerous cycles [12]. Hence, in this paper, we choose the DW-MTJ as the memristor device of ACAM. But, this is a very new MTJ type, and currently the number of resistance levels implemented on DW-MTJ is rather limited, where 5 stable resistance levels are demonstrated in [12] as the state-of-the-art (SOTA) implementation.

Related Works and Motivation
Several prior works have explored reducing the A/D conversion overhead in PIM. In [16], building fully analog circuits is proposed such that the signals are transmitted between layers without A/D conversion. This fully analog manner has been proved to be accuracy unfriendly. PRIME [2] chooses to use sense amplifiers (SAs) to do A/D conversion instead of ADCs. But a SA can only convert one bit at a time, leading to a long latency to get the whole output. CASCADE [5] proposes to accumulate the partial sums in the analog domain by connecting the outputs of multiple crossbars via an additional buffer RRAM array. However, the last-mile A/D conversion is kept to convert the sum back to the digital domain for downstream tasks, e.g., activation. Typically, in analog computing, only matrix multiplications (MMs) are conducted in the analog domain, while nonlinear activation function and pooling are implemented in the digital domain. Thus, A/D conversion is necessary. BRAHMS [22] proposes to put all the MMs, activation, and pooling in the analog domain, where ACAMs implement the latter two in two stages. In this way, only those analog values which are kept after activation and pooling need to be converted to the digital domain. The A/D conversion is still done by ACAMs. In this case, costly A/D conversion can be reduced by using the efficient ACAM. However, it is prone to signal noise since the analog signals need to be routed through ACAM three times to do nonlinear projection, pooling, and AD conversion. Moreover, severe accuracy drops are observed when the precision of A/D conversion implemented by ACAM is insufficient. Therefore, in this paper, we devise a MACAM-based analog activation unit. It is capable of implementing nonlinear activation and A/D conversion at one time in a fused manner. It can replace the traditional digital activation path, i.e., costly ADCs and digital activation units. As a result, the overhead of A/D conversions can be reduced. Since choosing MTJ as the memristor device will raise the precision issue as well, a slew of efforts are dedicated to improving the expressiveness of this device type.

MACAM-ENABLED FUSED ANALOG ACTIVATION UNIT
Nonlinear activation functions in analog computing are normally implemented by digital logics or lookup tables in the digital domain.
Since the computation is conducted in the analog domain, analogto-digital conversion overhead exists to bring analog signals to the digital domain for activation. We denote this conventional activation implementation as electrical digital activation, which includes the needed ADCs and the following digital logics or lookup tables. Recently, BRAHMS [22] has explored the idea of using ReRAMbased ACAM to implement nonlinear activation functions and A/D conversions, benefiting from the high-speed possessing and low power consumption of ACAM. However, the nonlinear projection and A/D conversion are separate. The analog signals being nonlinearly projected need to be routed back to the ACAM to do the A/D conversion, which may suffer from signal noise. Moreover, severe accuracy drops are observed when the precision of ACAM-based A/D conversion is not sufficient.
In this paper, we propose to implement the nonlinear activation function using MACAM, where nonlinear activation and A/D conversion are simultaneously achieved by the same MACAM array in a fused fashion. The fundamental idea here is to utilize the analog search functionality of MACAM to digitize the analog input signals while introducing in-situ nonlinearity. We call this fused nonlinear activation unit an analog activation unit. In this way, activation energy cost can be largely eliminated, as most ADCs can be replaced by energy-efficient MACAM.
Considering the precision issue, instead of supporting general nonlinear functions as in [22], we choose to implement the ReLUfunction. ReLU-is widely used in quantized models that have limited bit-widths to represent weight and activation. The ReLUfunction with a clipping threshold works as follows, where X is the pre-activation feature map and X is the final output feature map. The clipping threshold can bound the output of the ReLU function, therefore, a small bit-width is capable of representing the bounded value range of feature maps.
For ReLU-, since the input values can be either positive or negative, we need two MACAM arrays to handle positive and negative inputs, respectively. The negative part of ReLU-is easy to implement by generating a constant digital result '0'. Fig. 3 illustrates the implementation for the positive part of ReLU-in a single MACAM array. In the -th row of the MACAM array, by tuning the resistance of the MTJ, one MACAM cell represents an acceptance search interval [ −1 , ) for a match. All the intervals of MACAM cells in the top rows consist of a searchable range of [ 0 , ). is decided by the number of implementable resistances of the MTJ. To achieve the search interval [ , ∞], we encode ∞ similar to [22] by using a resistor with considerably larger resistance than the resistance upper bound of the MTJ. It is sufficient to cover the possible maximum voltage output from the analog engine as it cannot be arbitrarily large. The match result of all match lines consists of a one-hot vector. After obtaining the one-hot match result, we then transfer it into a binarized index using a digital priority encoder such that it can be stored at a minimized memory storage cost. The digital priority encoder can also handle corner cases when input is on the bounds of interval. Each binarized index represents a group of inputs that falls into one specific interval. Similar to quantization, we set the binarized index to correspond to one shared value, in which we use the mid-point of the interval's lower bound and upper bound in our implementation. For interval [ , ∞], the corresponding value is set to . This mapping relationship can be defined as a codebook Q as shown in Fig. 3. Within this process, we actually digitize the analog input signals. For example, in Fig. 3, the input falls into [ 1 , 2 ), the one-hot match result corresponds to the binarized index '1', which is treated as 1 + 2 2 . Since the minimum voltage 0 cannot be 0, the input needs to be biased to support the search of inputs starting from 0. In the following, for simplification of illustration, we assume 0 = 0 and = .
Now we explain how the projection of the MACAM array can realize in-situ nonlinearity of ReLU-. Figure 4(a) show the transformation behavior of the MACAM array. The range [0, ] is divided into four intervals, represented by the MTJ with 5 resistance levels using the measured data from [12]. The inputs within each interval are treated as the same value after projection based on the Q. If we draw the red dashed line through the interval's midpoint, which is labeled by the star symbol, it implements = . Actually, we can treat this as a nonlinear quantized version of a ReLU-function.
The MACAM array can only support a fixed searchable range of [0, ]. In order to implement any arbitrary ReLU-, we need to first scale the X with to fit the search range, feed the scaled input into ACAM, and then scale it back to [0, ], as shown in Fig 4(b). The first scaling operations before the MACAM array can be done using OpAmps. We can use shared OpAmps at the output of the computation units without extra overhead [23]. The off-chip computers can do the second scaling operation.
All in all, the behavior of MACAM-based activation can be modeled as follows, where Π Q indicates the discretized projection function of the ACAM within [0, ] following the codebook Q. ⌊·, ⌉ denotes clipping to the range of [0, ].

MIXED ACTIVATION SYSTEM 4.1 Proposed Mixed Activation System
As the DW-MTJ can currently only demonstrate a small number of resistance levels, the number of implementable intervals in MACAM is restricted, further limiting the representation capability of the MACAM-based analog activation unit. Hence, we are motivated to provide a mixed activation system, which integrates both the fused analog activation and the traditional digital activation datapaths. The traditional digital activation datapath first uses high-precision ADCs to do A/D conversion, then performs nonlinearity in dedicated digital activation units. The output feature map can choose either of the two paths for nonlinear activation and A/D conversion. With the mixed activation system, we can jointly utilize the lowenergy analog activation datapath and the high-precision digital datapath to balance expressiveness and energy efficiency. Figure 5 shows the system architecture overview with the tensor computing system and the proposed mixed activation system. We represent the fundamental tensor computing unit as vector-dotproduct (VDP) unit, with each unit supporting the dot-product between two length-vectors. Multiple VDPs implement a largesize vector product due to the limited single VDP size, generating multiple partial sums (PS). Note that nonlinear activation is not applied to any intermediate PS but to the final computation results, thus applying our MACAM-based analog activation requires inplace partial sum accumulation in the analog domain, which is doable by partial sum summation units [5,22,24]. Then, the final results are assigned to either the analog activation or the digital activation datapath to do nonlinear activation and A/D conversion.
We elaborate a weight-stationary dataflow to meet the requirement of the overall system following [22]. Take a convolutional layer as an example, where the 2-D weights ∈ R ×( 2 ) and 2-D output feature map X ∈ R ×( ′ ′ ) . The size of vector-dotproduct in its computaion is 2 , which is distributed onto ⌈ 2 ⌉ VDPs. This is doable since the largest vector product in modern ResNet and VGG models is only (512 × 3 × 3). At each cycle, we obtain the convolution result of the whole dot-product, ensured by in-place partial sum summation units. Fig. 6 illustrates our dataflow on a simple filter with 2 input channels. The weights can be stationary in the VDP units to continually complete the convolutions of adjacent sliding windows. Instead of frequently fetching inputs from costly memory, inputs can be reused in the convolutions of adjacent sliding windows. After obtaining the whole result, it can be sent to the activation datapath to do A/D conversion and activation.

Fully Differentiable SuperMixer Training
Our mixed activation system provides a more costly but higher precision digital activation datapath to compensate for the expressiveness due to analog activation datapath. This intuitively raises the question, "how do we assign the activation tasks of output feature maps onto the two different activation datapaths to balance expressiveness and energy efficiency?" Problem Formulation. Considering the energy gap between the two activation datapaths, our target is to assign the activation task of each value of output feature map ∈ X to either of the two activation paths with high expressiveness under a given activation energy constraint. We define the assignment as A. In this way, the problem can be formulated as follows, The binary selection variable , decides that each output feature value ∈ X in the -th layer is passed through either 1 (·) (MACAM-based analog activation datapath) or 2 (·) (digital activation path). The selection variables consist of the assignment A, which is our primary search target with a activation energy cost (A) satisfying its constraints [ ]. Search Space Specification. Considering the on-chip routing issue and computation regularity, it is not realistic to tediously assign individual output feature map value to different activation units. Instead, in this work, we propose to assign the workloads in the filter level. Concretely, take a convolutional layer as an example. Suppose it contains filters and each filter has input channels and kernel size , where its 2-D input X −1 ∈ R ×( × ) and 2-D output feature X ∈ R ×( ′ × ′ ) . The entire -th output channel of the output feature map X is passed through either the analog activation path 1 (·) or the electrical activation path 2 (·), decided by a binary selection variable , . The behavior is given as follows, In this way, our search space is defined as the assignment A that assigns each channel of output feature maps to either the analog or the electrical activation path. For layer , the number of total combinations can be 2 . Thus, the total search space for a model with layers is extremely large, which is O ( 2 ). Fully Differentiable SuperMixer Training. Considering the enormous search space of A and its discreteness, we propose a differentiable SuperMixer training flow as shown in Fig. 7.
In the SuperMixer training, we need to optimize weight , clipping threshold in ReLU-, and assignment A. It is highly    ill-conditioned to jointly optimize all those continuous and discrete variables. Aware of this, in this work, we divide our SuperMixer flow into two phases. The first SuperMixer Warmup phase aims at stabilizing our following search stage, where only and are optimized to obtain a good initial point. The second SuperMixer Search phase aims at searching for the assignment A to boost the expressiveness. It optimizes (1) , and (2) A alternatively to avoid prohibitive co-optimization difficulty. We periodically enter the optimization of (1) and (2) with a ratio of 2:1. Moreover, during the SuperMixer Search phase, we need to satisfy the activation energy cost constraint, thus, an energy cost penalty L E is added besides the original loss L. After SuperMixer training, we fix the searched optimal assignment A ℎ and conduct variationaware training to improve the model's robustness regarding on-chip variations, e.g., the resistance variation of memristors in MACAM. Now we explain how to optimize assignment A in a differentiable way. As shown in Eq. (4), the selection variable , is a binary variable. Instead of searching A in such a discrete space, we relax the optimization problem by constructing a stochastic mixed activation unit. During the inference, the activation of the -th channel of feature in layer , X , is using either the analog activation path 1 (·) or the digital activation path 2 (·) with the sampling probability of Equivalently, the output of the stochastic mixed activation, X , can be expressed as, where , is a random variable in {0, 1} and is evaluated based on the sampling probability in (5). Therefore, through parameterizing the probability distribution of activation unit choices by the sampling coefficient , we can relax the problem as the optimization of probability of . However, we cannot propagate the gradient back through the discrete variable , to , . To sidestep this issue, Gumbel-Softmax (GS) trick is adopted to relax , to be a continuous varible as follows, , follows the Gumbel distribution Gumbel(0, 1) as a Gumbel noise. A temperature parameter is used to control the GS function. As is close to 0, the Gumbel-Softmax function approximates categorical samples based on (5). A larger introduces randomness to encourage exploitation of the assignment A. Therefore, during the SuperMixer Search phase, we gradually decay such that SuperMixer can automatically exploit the search space and learn the optimal assignment to augment A the models' expressiveness. Activation Energy-Constrained Optimization. The assignment A directly impacts the activation energy cost, as it defines the mixed way of using low-precision analog activation units and highprecision electrical activation units. Fully using analog activation datapath results in an ADC-free system to convert computation results to the digital domain, while fully using electrical activation datapath raised a serious A/D conversion cost. With the assignment A, we can get the activation energy cost (A) as follows, # and # denote the number of output feature maps being passed through the analog (anlg) activation datapath and the digital (digi) activation datapath, respectively. The and represent the energy cost of two datapaths. The former indicates the MACAM array's cost, and the latter contains ADC and digital activation unit costs. As the ADC cost is far larger [19] and the digital activation unit cost is dependent on the digital part's frequency, we use ADC cost to represent during the search phase. To honor the energy cost constraint [ , , , ], we add a probabilistic activation energy cost penalty term L E , We set the margin to 5% to tighten the constraint. Note that , in Eq. (8) is differentiable guaranteed by the GS trick. Precision-Adaptive Optimization. Implementing ReLU-can ease the issue of insufficient precision of MACAM. However, it is not enough to address this since the SOTA MTJ can only provide five stable resistance levels, as mentioned before. The precision of ReLU-implemented by ACAM is only around 2-bit. Thus, a solution is in great need to further tolerate the precision issue.
Instead of choosing a fixed in Eq. (1), inspired by PACT [3], we propose to adopt a learnable with an enhanced training recipe to accommodate the low resolution of MACAM for accuracy boost. This can be done by configuring the gain of the OpAmps [23]. In [3], during the learning process of , the gradient to is computed by X = Sign(X) when X ≥ . Activation X that is smaller than cannot contribute to the gradient, resulting in inaccurate gradient estimation to . Instead of updating in the same way of [3], we re-formulate the gradient to based on Eq.(2) as follows, where our scaling operation works as a reparameterization trick to preserve the gradient contribution from X ∈ [0, ). Thus, it can correct the inaccurate gradient estimation to , which is proved to get a significantly larger accuracy boost than [3] in our experiments.

CASE STUDY: PHOTONIC ACCELERATOR
We demonstrate our design principle on a photonic accelerator as a case study. We focus on a SOTA incoherent photonic design based on micro-ring (MR) resonators [24,25]. Other accelerators can also benefit from our methods. For example, we can simply replace the ACAM design in a RRAM-based accelerator [22] with ours but own better tolerance to signal noise and the precision of ACAM. Fig. 8 shows the adopted photonic vector-dot-product (VDP) engine [24,25] and photonic in-place partial sum (PS) summation unit [24] for our system in Fig. 5. The computation of the convolutional (CONV) layer and fully-connected (FC) layer are unfolded to matrix multiplication, where each vector dot product is implemented by the vector dot product (VDP) units based on micro-ring (MR) resonators. To support two length-vector multiplication within each VDP unit, the size vectors are decomposed into small-sized vector chunks. Each small-sized vector dot-product is performed using MRs in each arm of the VDP unit. In this way, a large × vector dot product can be achieved in one unit, e.g., 100 × 100 in [24]. Across VDP units, photonic partial summation units are used to accumulate the partial sums from multiple VDPs. The partial sums from multiple VDPs are converted from the analog domain to the photonic domain by VCSELs, multiplexed into one single waveguide, and summed via another photodetector. Energy Modeling of the A/D Conversion and Activation Cost. Here, we model the energy cost of the A/D conversion and activation, / + based on O-HAS [14]. Our mixed activation system provides two datapaths to do nonlinear activation and A/D conversion, with an extra photonic summation unit overhead. Consider one convolutional layer with filters and output feature map X ∈ R ×( ′ × ′ ) . Each filter has input channels and a kernel size of . The energy model of / + can be modeled as follows, denotes the energy of the analog activation datapath, , and , denote the A/D conversion and activation energy of the digital activation datapth. and denote the energy for VCSEL and photodetector in the photonic PS summation unit.
In conventional implementation, the A/D conversion is done right after obtaining partial sum and the digital activation datapath is fully used. The energy cost can be modeled as, 6 EXPERIMENTAL RESULTS

Experiment Setup
Models and Datasets. We evaluate our methods on two modern CNNs (VGG13 and ResNet18) and CIFAR100 [11], with 6-bit weight precision. In VGG13, we replace the last three FC layers with one to avoid over-fitting. In ResNet18, we move the residual path after activation such that no extra addition is needed before activation. Training Settings. In SuperMixer training flow, we train for 90 epochs using an SGD optimizer with an initial learning rate (lr) of 0.02, a momentum of 0.9, and a cosine lr scheduler. The Gumbelsoftmax temperature exponentially decreases from 5 to 0.5. is set to 0.6 for the activation energy cost penalty. The initial of the adaptive ReLU-is set to 8. We set 10 epochs for SuperMixer Warmup phase and 80 epochs for search phase. We sample an assignment A from the learned distribution , then enter the variation-aware training. We train all models for 200 epochs during variation-aware retraining using an SGD optimizer with an initial learning rate (lr) of 0.02, a momentum of 0.9, and a cosine lr scheduler. MACAM and ADC Designs. We consider two MACAM designs with two MTJ devices in this paper. MACAM-1 uses the DW-MTJ with 5 resistance levels with around 2-bit precision. MACAM-2 uses the DW-MTJ with 3 resistance levels with around 1-bit precision. We choose two 6-bit ADC designs shown in Table. 1. Noise Injection. We set the variation of MRs following a Gaussian distribution N (0, 0.05 2 ). For MACAM, we set the MTJ resistance device-to-device variation as N (0, 0.128 2 ) [15], run Monte Carlo simulation 10 4 times to capture the noisy distribution of MACAM's intervals, and equivalently add it to the input following [22]. A/D and Activation Energy Simulation. The performance and power dissipation of MACAM are evaluated with Cadence ADE and spectre simulations. We estimate the A/D conversion and activation energy by Eq. (11) based on O-HAS [14]. The size of VDP is set as 128. Since the final FC layer is the output layer without activation, for a fair comparison, we don't include its energy consumption.

Main Results
Evaluation of Our Precision-adaptive ReLU-. In Table 2, we validate the expressiveness of our proposed precision-adaptive ReLU-on VGG13. We compare it with the adaptive ReLU-in PACT [3] and the ReLU-with a fixed . For the ReLU-with fixed , the commonly used ReLU6 is adopted, while ReLU2 is adopted in the extremely low activation bit-width. Under 6-bit weight bitwidth, our implementation achieves the highest accuracy, especially on low activation bit-width cases. This attributes to our proposed precision-adaptive optimization scheme, which learns the to accommodate the low resolution. Compared to PACT [3], our enhanced training recipe can correct its inaccurate gradient, resulting in a better accuracy boost. Hence, our adaptive ReLU-can address the accuracy drop issue in [22] due to the low activation bit-width. It is essential to boost the expressiveness of the analog activation unit so as to achieve competitive model accuracy.
Evaluation of Our SuperMixer. We search the assignment A with the proposed SuperMixer flow on different MACAM designs, ADC designs, and activation energy constraints. We denote searched assignments as searched A 0 to searched A 3 . Table 3 and 4 show the test accuracy of searched assignments on VGG13 and ResNet18. Our searched assignment series show improved expressiveness in terms of accuracy with constrained activation energy cost. Especially, given a large activation energy cost budget on MACAM-1, our SuperMixer can find an assignment with comparable accuracy to the case fully using high-precision digital activation. In conclusion, our SuperMixer flow successfully utilizes the provided mixed activation system to boost the expressiveness. Energy Saving on AD Conversion and Activation. We simulate the energy cost of A/D conversion and activation cost on VGG13.
Configurations of ADC-1 and ACAM-1 are adopted. The conventional implementation of using ADCs for A/D conversion of partial sums and using digital activation datapath consumes 35.7 J. With photonic summation units to eliminate A/D conversion needs of partial sums, fully using digital activation datapath consumes 17.24 J with a 51.7% reduction, and fully using analog activation datapath uses 12.44 J with a 65.2% reduction. Our SuperMixer enables the mixed use of electrical and analog activation with a trade-off between energy cost and expressiveness, where the searched-A 3 consumes 14.2 J with 60.2% reduction but comparable accuracy. Searched Layer-wise Assignment A. We further validate the success of SuperMixer to learn a good assignment A for boosting expressiveness. Fig. 9(a) and Fig. 9(b) visualize the ratio of feature map channels being assigned to digital activation path in each layer of VGG13 and ResNet18. Under different activation energy cost constraints, SuperMixer learns a similar tendency to assign the workloads. In former layers, the size of each channel of the feature map is larger, thus fewer channels are assigned to   the electrical activation units to avoid violation of the energy constraint. In contrast, more channels are assigned to the electrical activation units in the latter layers to boost the model accuracy. This demonstrates our SuperMixer flow can automatically learn the optimized assignment with boosted expressiveness under energy constraints. Energy Penalty Curve. In Fig. 10(a), the activation energy (A) is visualized along the SuperMixer training process.
(A) is well-bounded in the given constraint. It continues the exploitation of the search space and converges with the temperate of the Gumbel Softmax approaching 0. Noise Robustness of Searched Models. In Fig. 10(b), we evaluate the variation-robustness between searched models and model using fully analog activation units. With the increasing variation on MACAM, our searched models show better noise robustness since of the involvement of electrical activation units.

CONCLUSION
In this work, we propose a novel analog and mixed activation system for energy-efficient neural network acceleration. We first devise a fused analog activation unit based on MACAM that is capable of achieving nonlinear ReLU-and A/D conversion simultaneously, with superior energy efficiency to conventional digital activation implementation. We further integrate both the analog and digital activation dataflows to create a mixed activation system. A SuperMixer training flow is developed to automatically learn how to assign activation workloads to the low-energy analog activation datapath and high-precision digital activation datapath, aiming at a balance of expressiveness and energy efficiency. Our proposed methods are evaluated in a silicon photonic accelerator case study. Compared to the standard activation implementation, our mixed activation system with the searched assignment can achieve competitive accuracy with >60% energy saving on the overall A/D conversion and activation energy cost. Our MACAM-enabled analog and mixed activation system is viable to break through the curse of A/D conversion overhead in analog computing.