M-Theory (learning framework)

In Machine Learning and Computer Vision, M-Theory is a learning framework inspired by feed-forward processing in the ventral stream of visual cortex and originally developed for recognition and classification of objects in visual scenes. M-Theory was later applied to other areas, such as speech recognition. On certain image recognition tasks, algorithms based on a specific instantiation of M-Theory, HMAX, achieved human-level performance.
The core principle of M-Theory is extracting representations invariant to various transformations of images (translation, scale, 2D and 3D rotation and others). In contrast with other approaches using invariant representations, in M-Theory they are not hardcoded into the algorithms, but learned. M-Theory also shares some principles with Compressed Sensing. The theory proposes multilayered hierarchical learning architecture, similar to that of visual cortex.
Intuition.
Invariant representations.
A great challenge in visual recognition tasks is that the same object can be seen in a variety of conditions. It can be seen from different distances, different viewpoints, under different lighting, partially occluded, etc. In addition, for particular classes objects, such as faces, highly complex specific transformations may be relevant, such as changing facial expressions. For learning to recognize images, it is greatly beneficial to factor out these variations. It results in much simpler classification problem and, consequently, in great reduction of sample complexity of the model.
A simple computational experiment illustrates this idea. Two instances of a classifier were trained to distinguish images of planes from those of cars. For training and testing of the first instance, images with arbitrary viewpoints were used. Another instance received only images seen from a particular viewpoint, which was equivalent to training and testing the system on invariant representation of the images. One can see that the second classifier performed quite well even after receiving a single example from each category, while performance of the first classifier was close to random guess even after seeing 20 examples.
Invariant representations has been incorporated into several learning architectures, such as neocognitrons. Most of these architectures, however, provided invariance through custom-designed features or properties of architecture itself. While it helps to take into account some sorts of transformations, such as translations, it is very nontrivial to accommodate for other sorts of transformations, such as 3D rotations and changing facial expressions. M-Theory provides a framework of how such transformations can be learned. In addition to higher flexibility, this theory also suggests how human brain may have similar capabilities.
Templates.
Another core idea of M-Theory is close in spirit to ideas from the field of compressed sensing. An implication from Johnson–Lindenstrauss lemma says that a particular number of images can be embedded into a low-dimensional feature space with the same distances between images by using random projections. This result suggests that dot product between the observed image and some other image stored in memory, called template, can be used as a feature helping to distinguish the image from other images. The template need not to be anyhow related to the image, it could be chosen randomly.
Combining templates and invariant representations.
The two ideas outlined in previous sections can be brought together to construct a framework for learning invariant representations. The key observation is how dot product between image formula_1 and a template formula_2 behaves when image is transformed (by such transformations as translations, rotations, scales, etc.). If transformation formula_3 is a member of a unitary group of transformations, then the following holds:
formula_4
In other words, the dot product of transformed image and a template is equal to the dot product of original image and inversely transformed template. For instance, for image rotated by 90 degrees, the inversely transformed template would be rotated by -90 degrees.
Consider the set of dot products of an image formula_1 to all possible transformations of template: formula_6. If one applies a transformation formula_3 to formula_1, the set would become formula_9. But because of the property (1), this is equal to formula_10. The set formula_11 is equal to just the set of all elements in formula_12. To see this, note that every formula_13 is in formula_12 due to the closure property of groups, and for every formula_15 in G there exist its prototype formula_16 such as formula_17 (namely, formula_18). Thus, formula_19. One can see that the set of dot products remains the same despite that a transformation was applied to the image! This set by itself may serve as a (very cumbersome) invariant representation of an image. More practical representations can be derived from it.
In the introductory section, it was claimed that M-Theory allows to learn invariant representations. This is because templates and their transformed versions can be learned from visual experience - by exposing the system to sequences of transformations of objects. It is plausible that similar visual experiences occur in early period of human life, for instance when infants twiddle toys in their hands. Because templates may be totally unrelated to images that the system later will try to classify, memories of these visual experiences may serve as a basis for recognizing many different kinds of objects in later life. However, as it is shown later, for some kinds of transformations, specific templates are needed.
Theoretical aspects.
From orbits to distribution measures.
To implement the ideas described in previous sections, one need to know how to derive a computationally efficient invariant representation of an image. Such unique representation for each image can be characterized as it appears by a set of one-dimensional probability distributions (empirical distributions of the dot-products between image and a set of templates stored during unsupervised learning). These probability distributions in their turn can be described by either histograms or a set of statistical moments of it, as it will be shown below.
Orbit formula_20 is a set of images formula_21 generated from a single image formula_1 under the action of the group formula_23.
In other words, images of an object and of its transformations correspond to an orbit formula_20. If two orbits have a point in common they are identical everywhere, i.e. an orbit is an invariant and unique representation of an image. So, two images are called equivalent when they belong to the same orbit: formula_25 if formula_26 such that formula_27. Conversely, two orbits are different if none of the images in one orbit coincide with any image in the other.
A natural question arises: how can one compare two orbits? There are several possible approaches. One of them employs the fact that intuitively two empirical orbits are the same irrespective of the ordering of their points. Thus, one can consider a probability distribution formula_28 induced by the group's action on images formula_1 (formula_21 can be seen as a realization of a random variable).
This probability distribution formula_28 can be almost uniquely characterized by formula_32 one-dimensional probability distributions formula_33 induced by the (one-dimensional) results of projections formula_34, where formula_35 are a set of templates (randomly chosen images) (based on the Cramer-Wold theorem and concentration of measures).
Consider formula_36 images formula_37. Let formula_38 , where formula_39 is a universal constant. Then
formula_40
with probability formula_41, for all formula_42 formula_43 formula_44.
This result (informally) says that an approximately invariant and unique representation of an image formula_1 can be obtained from the estimates of formula_32 1-D probability distributions formula_47 for formula_48. The number formula_32 of projections needed to discriminate formula_36 orbits, induced by formula_36 images, up to precision formula_52 (and with confidence formula_41) is formula_38, where formula_39 is a universal constant.
To classify an image, the following "recipe" can be used:
Estimates of such one-dimensional probability density functions (PDFs) formula_33 can be written in terms of histograms as formula_57, where formula_58 is a set of nonlinear functions. These 1-D probability distributions can be characterized with N-bin histograms or set of statistical moments. For example, HMAX represents an architecture in which pooling is done with a max operation.
Non-compact groups of transformations.
In the "recipe" for image classification, groups of transformations are approximated with finite number of transformations. Such approximation is possible only when the group is compact.
Such groups as all translations and all scalings of the image are not compact, as they allow arbitrarily big transformations. However, they are locally compact. For locally compact groups, invariance is achievable within certain range of transformations.
Assume that formula_59 is a subset of transformations from formula_12 for which the transformed patterns exist in memory. For an image formula_1 and template formula_62, assume that formula_63 is equal to zero everywhere except some subset of formula_59. This subset is called support of formula_63 and denoted as formula_66. It can be proven that if for a transformation formula_16, support set will also lie within formula_68, then signature of formula_1 is invariant with respect to formula_16. This theorem determines the range of transformations for which invariance is guaranteed to hold.
One can see that the smaller is formula_66, the larger is the range of transformations for which invariance is guaranteed to hold. It means that for a group that is only locally compact, not all templates would work equally well anymore. Preferable templates are those with a reasonably small formula_72 for a generic image. This property is called localization: templates are sensitive only to images within a small range of transformations. Note that although minimizing formula_72 is not absolutely necessary for the system to work, it improves approximation of invariance. Requiring localization simultaneously for translation and scale yields a very specific kind of templates: Gabor functions.
The desirability of custom templates for non-compact group is in conflict with the principle of learning invariant representations. However, for certain kinds of regularly encountered image transformations, templates might be the result of evolutionary adaptations. Neurobiological data suggests that there is Gabor-like tuning in the first layer of visual cortex. The optimality of Gabor templates for translations and scales is a possible explanation of this phenomenon.
Non-group transformations.
Many interesting transformations of images do not form groups. For instance, transformations of images associated with 3D rotation of corresponding 3D object do not form a group, because it is impossible to define an inverse transformation (two objects may looks the same from one angle but different from another angle). However, approximate invariance is still achievable even for non-group transformations, if localization condition for templates holds and transformation can be locally linearized.
As it was said in the previous section, for specific case of translations and scaling, localization condition can be satisfied by use of generic Gabor templates. However, for general case (non-group) transformation, localization condition can be satisfied only for specific class of objects. More specifically, in order to satisfy the condition, templates must be similar to the objects one would like to recognize. For instance, if one would like to build a system to recognize 3D rotated faces, one need to use other 3D rotated faces as templates. This may explain the existence of such specialized modules in the brain as one responsible for face recognition. Even with custom templates, a noise-like encoding of images and templates is necessary for localization. It can be naturally achieved if the non-group transformation is processed on any layer other than the first in hierarchical recognition architecture.
Hierarchical architectures.
The previous section suggests one motivation for hierarchical image recognition architectures. However, they have other benefits as well.
Firstly, hierarchical architectures best accomplish the goal of ‘parsing’ a complex visual scene with many objects consisting of many parts, whose relative position may greatly vary. In this case, different elements of the system must react to different objects and parts. In hierarchical architectures, representations of parts at different levels of embedding hierarchy can be stored at different layers of hierarchy.
Secondly, hierarchical architectures which have invariant representations for parts of objects may facilitate learning of complex compositional concepts. This facilitation may happen through reusing of learned representations of parts that were constructed before in process of learning of other concepts. As a result, sample complexity of learning compositional concepts may be greatly reduced.
Finally, hierarchical architectures have better tolerance to clutter. Clutter problem arises when the target object is in front of a non-uniform background, which functions as a distractor for the visual task. Hierarchical architecture provides signatures for parts of target objects, which do not include parts of background and are not affected by background variations.
In hierarchical architectures, one layer is not necessarily invariant to all transformations that are handled by the hierarchy as a whole. Some transformations may pass through that layer to upper layers, as in the case of non-group transformations described in the previous section. For other transformations, an element of the layer may produce invariant representations only within small range of transformations. For instance, elements of the lower layers in hierarchy have small visual field and thus can handle only a small range of translation. For such transformations, the layer should provide "covariant" rather than invariant, signatures. The property of covariance can be written as formula_74, where formula_75 is a layer, formula_76 is the signature of image on that layer, and formula_77 stands for "distribution of values of the expression for all formula_78".
Relation to biology.
M-theory is based on a quantitative theory of the ventral stream of visual cortex. Understanding how visual cortex works in object recognition is still a challenging task for neuroscience. Humans and primates are able to memorize and recognize objects after seeing just couple of examples unlike any state-of-the art machine vision systems that usually require a lot of data in order to recognize objects. Prior to the use of visual neuroscience in computer vision has been limited to early vision for deriving stereo algorithms (e.g.,) and to justify the use of DoG (derivative-of-Gaussian) filters and more recently of Gabor filters. No real attention has been given to biologically plausible features of higher complexity. While mainstream computer vision has always been inspired and challenged by human vision, it seems to have never advanced past the very first stages of processing in the simple cells in V1 and V2. Although some of the systems inspired - to various degrees - by neuroscience, have been tested on at least some natural images, neurobiological models of object recognition in cortex have not yet been extended to deal with real-world image databases.
M-theory learning framework employs a novel hypothesis about the main computational function of the ventral stream: the representation of new objects/images in terms of a signature, which is invariant to transformations learned during visual experience. This allows recognition from very few labeled examples - in the limit, just one.
Neuroscience suggests that natural functionals for a neuron to compute is a high-dimensional dot product between an "image patch" and another image patch (called template) 
which is stored in terms of synaptic weights (synapses per neuron). The standard computational model of a neuron is based on a dot product and a threshold. Another important feature of the visual cortex is that it consists of simple and complex cells. This idea was originally proposed by Hubel and Wiesel. M-theory employs this idea. Simple cells compute dot products of an image and transformations of templates formula_79 for formula_80 (formula_81 is a number of simple cells). Complex cells are responsible for pooling and computing empirical histograms or statistical moments of it. The following formula for constructing histogram can be computed by neurons:
formula_82
where formula_83 is a smooth version of step function, formula_84 is the width of a histogram bin, and formula_36 is the number of the bin.
Applications.
Applications to computer vision.
In authors applied M-theory to unconstrained face recognition in natural photographs. Unlike the DAR (detection, alignment, and recognition) method, which handles clutter by detecting objects and cropping closely around them so that very little background remains, this approach accomplishes detection and alignment implicitly by storing transformations of training images (templates) rather than explicitly detecting and aligning or cropping faces at test time. This system is built according to the principles of a recent theory of invariance in hierarchical networks and can evade the clutter problem generally problematic for feedforward systems. 
The resulting end-to-end system achieves a drastic improvement in the state of the art on this end-to-end task, reaching the same level of performance as the best systems operating on aligned, closely cropped images (no outside training data). It also performs well on two newer datasets, similar to LFW, but more difficult: significantly jittered (misaligned) version of LFW and SUFR-W (for example, the model's accuracy in the LFW "unaligned & no outside data used" category is 87.55±1.41% compared to state-of-the-art APEM (adaptive probabilistic elastic matching): 81.70±1.78%).
The theory was also applied to a range of recognition tasks: from invariant single object recognition in clutter to multiclass categorization problems on publicly available data sets (CalTech5, CalTech101, MIT-CBCL) and complex (street) scene understanding tasks that requires the recognition of both shape-based as well as texture-based objects (on StreetScenes data set). The approach performs really well: It has the capability of learning from only a few training examples and was shown to outperform several more complex state-of-the-art systems constellation models, the hierarchical SVM-based face-detection system. A key element in the approach is a new set of scale and position-tolerant feature detectors, which are biologically plausible and agree quantitatively with the tuning properties of cells along the ventral stream of visual cortex. These features are adaptive to the training set, though we also show that a universal feature set, learned from a set of natural images unrelated to any categorization task, likewise achieves good performance.
Applications to speech recognition.
This theory can also be extended for the speech recognition domain.
As an example, in an extension of a theory for unsupervised learning of invariant visual representations to the auditory domain and empirically evaluated its validity for voiced speech sound classification was proposed. Authors empirically demonstrated that a single-layer, phone-level representation, extracted from base speech features, improves segment classification accuracy and decreases the number of training examples in comparison with standard spectral and cepstral features for an acoustic classification task on TIMIT dataset.