Q:

Hand tracking, state of the art

I am looking for a framework for extracting hand region from the video feed from the webcam(analogous to Viola Jones HAAR detection of face). 
This region of interest will serve as input for a CNN for hand gesture recognition, which I have already coded using python and CAFFE.
There are way too many frameworks and I am confused on which one to use,
priority would be to minimise the processing time. I already tried HAAR cascades for hand detection but they are pretty unreliable.
It would be most helpful if implementation is readily available in python.
Thanks in advance

A:

The requirement for being able to detect the "profile" of hand gestures is not a straightforward one to satisfy. There are issues of form and rotation / scale independency which are not "easy".
There are two main problems here:

Segmenting the hands
Figuring out the gesture.

To segment the hands, let's assume that the hands are imaged over a constant single colour background. This means that the overall hand region can be isolated with a "simple" threshold.
The second problem is one of recognising form. In other words, what are the "shape" differences between "rock", "paper" and "scissors"?
Therefore, whatever is used should be "able" to discriminate between "holding two fingers extended" versus "all fingers retracted in the shape of a fist", etc.
The "easy" way out is to use template matching. Very "simply", take pictures of the hand gestures for each hand and for various orientations and use them to "search" for their existence in an image using cross correlation (or other similar concept in some feature space).
This however, done in the "simple" way is very costly (computationaly) and also scale dependent. So, if the hands are near the camera they are larger, if they are further away they are smaller and therefore very difficult to "catch" from static images that were acquired at some standard distance.
So, an alternative would be to use Morphological Operators and specifically Skeletonisation.
The main idea here is to apply skeletonisation on the extracted hand regions, quite a few times to end up with a set of "lines" describing the shape you are trying to recognise. In this way, in one pass you effect the skeletonisation and in another pass you collect the (very few, compared to the original) pixel locations that compose the gesture. To then figure out the gesture itself, you work with features extracted from the pixel locations.
The added benefit of this type of a solution is scale invariance. The relative positions of the "finger segments" of the "scissors" gesture are the same whether you hold it near the camera or far away from the camera (Two lines forming a V ending up in a big blob). With careful selection of the features and search process you can even use the same algorithm and training dataset for the left and right hands separately (for example, using graph search).
Hope this helps.

