PRIORITY CLAIM

This U.S. patent application claims priority under 35 U.S.C. § 119 to: India Application No. 202021025784, filed on Jun. 18, 2020. The entire contents of the aforementioned application are incorporated herein by reference.

TECHNICAL FIELD

This disclosure relates generally to gesture recognition, and, more particularly, to system and method of gesture recognition using a reservoir based convolutional spiking neural network.

BACKGROUND

In an age of artificial intelligence, robots and drones are key enablers of task automation and they are being used in various domains such as manufacturing, healthcare, warehouses, disaster management etc. As a consequence, they often need to share work-space with and interact with human workers and thus evolving the area of research named Human Robot Interaction (HRI). Problems in this domain are mainly centered around learning and identifying of gestures/speech/intention of human coworkers along with classical problems of learning and identification of surrounding environment (and obstacles, objects etc. therein). All these essentially are needed to be done in a dynamic and noisy practical work environment. As of current state of the art vision based solutions using artificial neural networks (including deep neural networks) have high accuracy, however the models are not the most efficient solutions as learning methods and inference frameworks of the conventional deep neural networks require huge amount of training data and are typically compute and energy intensive. They are also bounded by one or more conventional architectures that leads to data transfer bottleneck between memory and processing units and related power consumption issues. Hence, this genre of solutions does not really help robots and drones to do their jobs as they are classically constrained by their battery life.

SUMMARY

Embodiments of the present disclosure present technological improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems. For example, in one aspect, a processor implemented method of identifying a gesture from a plurality of gestures using a reservoir based convolutional spiking neural network is provided. The processor implemented method includes at least one of: receiving, from a neuromorphic event camera, two-dimensional spike streams as an input; preprocessing, via one or more hardware processors, the address event representation (AER) record associated with at least one gestures from a plurality of gestures to obtain a plurality of spike frames; processing, by a multi layered convolutional spiking neural network, the plurality of spike frames to learn a plurality of spatial features from the at least one gesture; deactivating, via the one or more hardware processors, at least one filter block from the plurality of filter blocks corresponds to at least one gesture which are not currently being learnt; obtaining, via the one or more hardware processors, spatio-temporal features by allowing the spike activations from a CSNN layer to flow through the reservoir; and classifying, by a classifier, the at least one of spatial feature from the CSNN layer and the spatio-temporal features from the reservoir to obtain a set of prioritized gestures. In an embodiment, the two-dimensional spike streams are represented as an address event representation (AER) record. In an embodiment, each sliding convolutional window in the plurality of spike frames are connected to a neuron corresponding to a filter among plurality of filters corresponding to a filter block among plurality of filter blocks in each convolutional layer from plurality of convolutional layers. In an embodiment, the plurality of filter blocks are configured to concentrate a plurality of class-wise spatial features to the filter block for learning associated patterns based on a long-term lateral inhibition mechanism. In an embodiment, the CSNN layer is stacked to provide at least one of: (i) a low-level spatial features, (ii) a high-level spatial features, or combination thereof.

In an embodiment, the spike streams may be compressed per neuronal level by accumulating spikes at a sliding window of time, to obtain a plurality of output frames with reduced time granularity. In an embodiment, plurality of learned different spatially co-located features may be distributed on the plurality of filters from the plurality of filter blocks. In an embodiment, a special node between filters of the filter block may be configured to switch between different filters based on an associated decay constant to distribute learning of different spatially co-located features on the different filters. In an embodiment, a plurality of weights of a synapse between input and the CSNN layer may be learned using an unsupervised two trace STDP learning rule upon at least one spiking activity of the input layer. In an embodiment, the reservoir may include a sparse random cyclic connectivity which acts as a random projection of the input spikes to an expanded spatio-temporal embedding.

In another aspect, there is provided a system to identify a gesture from a plurality of gestures using a reservoir based convolutional spiking neural network. The system comprises a memory storing instructions; one or more communication interfaces; and one or more hardware processors coupled to the memory via the one or more communication interfaces. The one or more hardware processors are configured by the instructions to: receive, from a neuromorphic event camera, two-dimensional spike streams as an input; preprocess, the address event representation (AER) record associated with at least one gestures from a plurality of gestures to obtain a plurality of spike frames; process, by a multi layered convolutional spiking neural network, the plurality of spike frames to learn a plurality of spatial features from the at least one gesture; deactivate, at least one filter block from the plurality of filter blocks corresponds to at least one gesture which are not currently being learnt; obtain, spatiotemporal features by allowing the spike activations from a CSNN layer to flow through the reservoir; and classify, by a classifier, the at least one of spatial feature from the CSNN layer and the spatiotemporal features from the reservoir to obtain a set of prioritized gestures. In an embodiment, the two-dimensional spike streams is represented as an address event representation (AER) record. In an embodiment, each sliding convolutional window in the plurality of spike frames are connected to a neuron corresponding to a filter among plurality of filters corresponding to a filter block among plurality of filter blocks in each convolutional layer from plurality of convolutional layers. In an embodiment, the plurality of filter blocks are configured to concentrate a plurality of class-wise spatial features to the filter block for learning associated patterns based on a long-term lateral inhibition mechanism. In an embodiment, the CSNN layer is stacked to provide at least one of: (i) a low-level spatial features, (ii) a high-level spatial features, or combination thereof.

In an embodiment, the spike streams may be compressed per neuronal level by accumulating spikes at a sliding window of time, to obtain a plurality of output frames with reduced time granularity. In an embodiment, plurality of learned different spatially co-located features may be distributed on the plurality of filters from the plurality of filter blocks. In an embodiment, a special node between filters of the filter block may be configured to switch between different filters based on an associated decay constant to distribute learning of different spatially co-located features on the different filters. In an embodiment, a plurality of weights of a synapse between input and the CSNN layer may be learned using an unsupervised two trace STDP learning rule upon at least one spiking activity of the input layer. In an embodiment, the reservoir may include a sparse random cyclic connectivity which acts as a random projection of the input spikes to an expanded spatio-temporal embedding.

In yet another aspect, there are provided one or more non-transitory machine readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors causes at least one of: receiving, from a neuromorphic event camera, two-dimensional spike streams as an input; preprocessing, the address event representation (AER) record associated with at least one gestures from a plurality of gestures to obtain a plurality of spike frames; processing, by a multi layered convolutional spiking neural network, the plurality of spike frames to learn a plurality of spatial features from the at least one gesture; deactivating, at least one filter block from the plurality of filter blocks corresponds to at least one gesture which are not currently being learnt; obtaining, spatio-temporal features by allowing the spike activations from a CSNN layer to flow through the reservoir; and classifying, by a classifier, the at least one of spatial feature from the CSNN layer and the spatio-temporal features from the reservoir to obtain a set of prioritized gestures. In an embodiment, the two-dimensional spike streams are represented as an address event representation (AER) record. In an embodiment, each sliding convolutional window in the plurality of spike frames are connected to a neuron corresponding to a filter among plurality of filters corresponding to a filter block among plurality of filter blocks in each convolutional layer from plurality of convolutional layers. In an embodiment, the plurality of filter blocks are configured to concentrate a plurality of class-wise spatial features to the filter block for learning associated patterns based on a long-term lateral inhibition mechanism. In an embodiment, the CSNN layer is stacked to provide at least one of: (i) a low-level spatial features, (ii) a high-level spatial features, or combination thereof.

In an embodiment, the spike streams may be compressed per neuronal level by accumulating spikes at a sliding window of time, to obtain a plurality of output frames with reduced time granularity. In an embodiment, plurality of learned different spatially co-located features may be distributed on the plurality of filters from the plurality of filter blocks. In an embodiment, a special node between filters of the filter block may be configured to switch between different filters based on an associated decay constant to distribute learning of different spatially co-located features on the different filters. In an embodiment, a plurality of weights of a synapse between input and the CSNN layer may be learned using an unsupervised two trace STDP learning rule upon at least one spiking activity of the input layer. In an embodiment, the reservoir may include a sparse random cyclic connectivity which acts as a random projection of the input spikes to an expanded spatio-temporal embedding.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.

