EGRU: Event-based GRU for activity-sparse inference and learning
The scalability of recurrent neural networks (RNNs) is hindered by the
sequential dependence of each time step's computation on the previous time
step's output. Therefore, one way to speed up and scale RNNs is to reduce the
computation required at each time step independent of model size and task. In
this paper, we propose a model that reformulates Gated Recurrent Units (GRU) as
an event-based activity-sparse model that we call the Event-based GRU (EGRU),
where units compute updates only on receipt of input events (event-based) from
other units. When combined with having only a small fraction of the units
active at a time (activity-sparse), this model has the potential to be vastly
more compute efficient than current RNNs. Notably, activity-sparsity in our
model also translates into sparse parameter updates during gradient descent,
extending this compute efficiency to the training phase. We show that the EGRU
demonstrates competitive performance compared to state-of-the-art recurrent
network models in real-world tasks, including language modeling while
maintaining high activity sparsity naturally during inference and training.
This sets the stage for the next generation of recurrent networks that are
scalable and more suitable for novel neuromorphic hardware.