Learning the semantic structure of objects from Web supervision

While recent research in image understanding has often focused on recognizing more types of objects, understanding more about the objects is just as important. Recognizing object parts and attributes has been extensively studied before, yet learning large space of such concepts remains elusive due to the high cost of providing detailed object annotations for supervision. The key contribution of this paper is an algorithm to learn the nameable parts of objects automatically, from images obtained by querying Web search engines. The key challenge is the high level of noise in the annotations; to address it, we propose a new unified embedding space where the appearance and geometry of objects and their semantic parts are represented uniformly. Geometric relationships are induced in a soft manner by a rich set of nonsemantic mid-level anchors, bridging the gap between semantic and non-semantic parts. We also show that the resulting embedding provides a visually-intuitive mechanism to navigate the learned concepts and their corresponding images.

Introduction
Modern deep learning methods have dramatically improved the performance of computer vision algorithms in selected tasks such as image classification [1] and object detection [2]. Parallel advances in tasks such as image captioning [3,4], activity recognition [5], and many others have ventured far beyond classification and detection in order to extract richer information from visual scenes. Even so, image understanding remains rather crude, oblivious to most of the nuances of real world images. Consider for example the notion of object category, which is a basic unit of understanding in computer vision. Modern benchmarks consider an increasingly large number of such categories, from thousands in the ILSVRC challenge [6] to hundred thousands in the full ImageNet [7]. However, there is only limited understanding of their internal semantic structure and geometry.
In this paper we aim at filling this gap by jointly learning about objects, their semantic parts, and their geometric relationship. Semantic nameable parts play a crucial role in visual understanding. However, learning them on a large scale using standard methods faces the difficulty of collecting vast quantities of arXiv:1607.01205v2 [cs.CV] 2 Dec 2021 Car images

Input
Learned concept "Car wheel" as an object "Car wheel" as a component Noisy Web results for "car wheel" Fig. 1. Our goal is to learn the semantic structure of objects automatically using Web supervision. For example, given noisy images obtained by querying an Internet search engine for "car wheel" and for "cars", we aim at learning the "car wheel" concept, and its dual nature: as an object in its own right, and as a component of another object.
corresponding annotated example images. Instead, scalable algorithms must be designed to discover this information, with minimal or no supervision.
As others have done for the problem of learning visual objects, in this paper we look at Web supervision to learn object parts from thousands of images obtained automatically by querying search engines (crf. fig. 1). However, this poses two significant challenges: identifying images of the parts in very noisy Web results (crf. fig. 2) while, at the same time, bridging the scale difference between parts seen in the context of the whole object or in isolation. The latter suggests in fact that parts have a dual nature: as components of an object as well as objects in their own right ( fig. 1 right), and models should be able to capture both. In order to address such challenges, we propose a new method to reason robustly about visual concepts and their geometric relationships.
Our first idea is to use the same representation for both objects and parts, thinking them as generic "semantic visual entities". Differently from methods such as Deformable Part Models (DPM) [8,9], our representation does not differentiate between objects and subordinate parts, promoting flexibility and robustness. Our second idea is to leverage non-semantic parts to learn about semantic ones; methods such as DPMs seek in fact visually stable parts, that are often non-semantic. While these are not very interesting for semantic abstraction, they may provide reliable geometric anchors to represent object deformations.
These two ideas come together in the two main contributions of the paper. The first contribution (section 2.1) is a novel embedding that captures appearance and geometry of all visual entities, either objects or semantic parts, in the same space. Geometry is expressed robustly against an object-centric reference frame implicitly captured by non-semantic anchor parts. The second contribution (section 2.2) is an effective method to learn these non-semantic anchors, which is an alternative to significantly more complex part discovery methods.
A byproduct of our method is a large collection of images annotated with objects, semantic parts, and their geometric relationships, that we refer to as a visual semantic atlas (section 4). This atlas allows to visually navigate images based on conceptual and geometric relations. It also emphasizes the dual nature of parts, as components of an object and as semantic categories, by naturally bridging images that zooms on a part or that contain the object as a whole. Top images retrieved from an Internet search engine for some example queries. Note that part results are more noisy than full object results (the remaining collected images get even noisier, not shown here).

Related work
Our work touches on several active research areas: localizing objects with weak supervision, learning with Web images, and discovering or learning mid-level features and object parts. Localizing objects with weak supervision. When training models to localize objects or parts, it is impractical to expect large quantities of bounding box annotations. Recent works have tackled the localization problem with only image-level annotations. Among them, weakly supervised object localization methods [10,11,12,13,14,15,16] assume for each image a list of every object type it contains. In the co-detection [17,18,19,20] and co-segmentation [21,22,23,24] problems, the algorithm is given a set of images that all contain at least one instance of a particular object. They differ in their output: co-detection predicts bounding boxes, while segmentation predicts pixel-level masks. Yet, co-detection, co-segmentation and weakly-supervised object localization (WSOL) are different flavors of the localization problem with weak supervision. For co-detection and WSOL, the task is nearly always formulated as a multiple instance learning (MIL) problem [10,11,25,19,26]. The formulation in [14,15] departs from MIL by leveraging the strong annotations for some categories to transfer knowledge to the remaining categories. A few approaches model images using topic models [13,20]. Recently, CNN architectures were also proved to work well in weakly supervised scenarios [27]. We will compare with [27] in the experiments section. None of these works have considered semantic parts. Closer to our work, the method of [28] proposes unsupervised discovery of dominant objects using part-based region matching. Because of its unsupervised process, this method is not suited to name the discovered objects or matched regions, and hence lack semantics. Yet we also compare with this approach in our experiments. Learning from Web supervision. Most previous works [29,30,31,32] that learn from noisy Web images have focused on image classification. Usually, they adopt an iterative approach that jointly learns models and finds clean examples of a target concept. Only few works have looked at the problem of localization. Some approaches [33,24] discover common segments within a large set of Web images, but they do not quantitatively evaluate localization. The recent method of [34] localizes objects with bounding boxes, and evaluate the learnt models, but as the previous two, it does not consider object parts. Closer to our work, [35] aims at discovering common sense knowledge relations between object cat- egories from Web images, some of which correspond to the "part-of" relation.
In the process of organizing the different appearance variations of Webly mined concepts, [36] uses a "vocabulary of variance" that may include part names, but those are not associated to any geometry. Unsupervised parts, mid-level features, and semantic parts. Objects are modeled using the notion of parts since the early work on pictorial structure [37], in the constellation [38] and ISM [39] models, and more recently the DPM [9].
Parts are most commonly defined as localized components with consistent appearance and geometry in an object. All these works have in common to discover object parts without naming them. In practice, only some of these parts have an actual semantic interpretation. Mid-level features [40,41,42,43,44,45] are discriminative [43,46] or rare [40] blocks, which are leveraged for object recognition. Again, these parts lack semantic. The non-semantic anchors that we use share similarities with [47] and [44], that we discuss in section 2.2. Semantic parts have triggered recent interest [48,49,50]. These works require strong annotations in the form of bounding boxes [48] or segmentation masks [49,50] at the part level.
Here we depart from existing work and aim at mining semantic nameable parts with as little supervision as possible.

Method
This section introduces our method to learn semantic parts using weak supervision from Web sources. The key challenge is that search engines, when queried for object parts, return many outliers containing other parts as well, the whole object, or entirely unrelated things ( fig. 2). In this setting, standard weaklysupervised detection approaches fail (section 3). Our solution is a novel, robust, and flexible representation of object parts (section 2.1) that uses the output of a simple but very effective non-semantic part discovery algorithm (section 2.2).

Learning semantic parts using non-semantic anchors
In this section, we first flesh out our method for weakly-supervised part learning and then dive into the theoretical justification of our choices. MIL: baseline, context, and geometry-aware. As standard in weaklysupervised object detection, our method starts from the Multiple Instance Learn-ing (MIL) [51] algorithm. Let x i be an image and let R(x i ) be a shortlist of image regions R that are likely to contain objects or parts, obtained for instance using selective search [52]. Each image x i can be either positive y i = +1 if it is deemed to contain a certain part or negative y i = −1 if not. MIL fits to this data a (linear) scoring function φ(x i |R), w , where w is a vector of parameters and φ(x i |R) ∈ R d is a descriptor of the region R of image x i , by minimizing: In practice, eq. (1) is optimized by alternatively selecting the maximum scoring region for each image (also known as "re-localization") and optimizing w for a fixed selection of the regions. In this manner, MIL should automatically discover regions that are most predictive of a given label, and which therefore should correspond to the sought visual entity (object or semantic part). However, this process may fail if descriptors are not sufficiently strong.
For baseline MIL the descriptor φ(x|R) = φ a (x|R) ∈ R da captures the region's appearance. A common improvement is to extend this descriptor with context information by appending a descriptor of a region Neither baseline or context-aware MIL leverage the fact that objects have a well-defined geometric structure, which significantly constrains the search space for parts. DPM uses such constraints, but as a fixed set of geometric relationships between part pairs that are difficult to learn when examples are extremely noisy. Furthermore, DPM-like approaches learn the most visually-stable parts, which often are not the semantic ones.
We propose here an alternative method that captures geometry indirectly, on top of a rich set of unsupervised mid-level non-semantic parts {p 1 , ..., p K }, which we call anchors ( fig. 3). Let us assume that, given an image x, we can locate the (selective search) regions R p k ,x containing each anchor p k . We define the following geometric embedding φ g of a region R with respect to the anchors: Here ρ is a measure such as intersection-over-union (IoU) that tells whether two regions overlap. By choosing a function ρ such as IoU which is invariant to scaling, rotation, and translation of the regions, so is the embedding φ g . Hence, as long as anchors stay attached to the object, φ g (x|R) encodes the location of R relative to an object-centric frame. This representation is robust because, even if some anchors are missing or misplaced, the vector φ g (x|R) is not greatly affected. The geometric encoding φ g (x|R) is combined with the appearance descriptor φ a (x|R) in a joint appearance-geometric embedding where ⊗ is the Kronecker product. After vectorization, this vector is used as a descriptor φ(x|R) = φ ag (x|R) of region R in geometry-aware MIL. The next few paragraphs discuss its properties.
Modeling multiple parts. Plugging φ ag of eq. (3) into eq. (1) of MIL results in the scoring function w, φ ag (x|R) = K k=1 w k , φ a (x|R) ρ(R, R p k ,x ) which interpolates between K appearance models based on how the region R is geometrically related to the anchors R p k ,x . In particular, by selecting different anchors this model may capture simultaneously the appearance of all parts of an object. In order to control the capacity of the model, the smoothness of the interpolator can be increased by replacing IoU with a softer version, which we do next.
Smoother overlap measure. The IoU measure is a special case of the following family of PD kernels (proof in the appendix): The IoU is obtained when R and Q are indicator functions of the respective regions (because R, Q = R(x, y)Q(x, y) dx dy = |R ∩ Q|). This suggests a simple modification to construct a Soft IoU (SIoU) version of the latter. For a is the Heaviside step function. SIoU is obtained by replacing the indicator by the smoother function H α (z) = exp(αz)/(1 + exp(αz)) instead. Note that SIoU is non-zero even when regions do not intersect.
Theorem 1 provides also an interpretation of the geometric embedding φ g of eq. (2) as a vector of region coordinates relative to the anchors. In fact, its entries can be written as ρ(R, R p k ,x ) = ψ SIoU (R), ψ SIoU (R p k ,x ) where ψ SIoU (R) ∈ H SIoU is the linear embedding (feature map) induced by the kernel ρ 1 . Modeling multiple aspects. So far, we have assumed that all parts are always visible; however, anchors also provide a mechanism to deal with the multiple aspects of 3D objects. As depicted in fig. 3.c, as the object rotates out of plane, anchors naturally appear and disappear, therefore activating and de-activating aspect-specific components in the model. In turn, this allows to model viewpointspecific parts or appearances. In practice, we extract the L highest scoring detections R l of the same anchor p k , and keep the one closest to R.
In order to allow anchors to turn off in the model, the geometric embedding is modified as follows. Let s k (R l |x) be the detection score of anchor k in correspondence of the region R l ; then If the anchor is never detected (s k (R l |x) ≤ 0 for all R l ) then ρ(R, R p k ,x ) = 0. Furthermore, this expression also disambiguates ambiguous anchor detections by picking the one closest to R. Note that in eq. (4) one can still interpret the factors SIoU(R, R l ) as projections ψ SIoU (R), ψ SIoU (R l ) .
Relation to DPM. DPM is also a MIL method using a joint embedding φ DPM (x|R 1 , . . . , R K ) that codes simultaneously for the appearance of K parts and their pairwise geometric relationships. Our Webly-supervised learning problem requires a representation that can bridge object-focused images (where several parts are visible together as components) and part-focused images (where parts are regarded as objects in their own right). This is afforded by our embedding φ ag (x|R) but not by the DPM one. Besides bridging parts as components and parts as objects, our embedding is very robust (important in order to deal with very noisy training labels), automatically codes for multiple object aspects, and bridges unsupervised non-semantic parts (the anchors) with semantic ones.

Anchors: weakly-supervised non-semantic parts
The geometric embedding in the previous section leverages the power of an intermediate representation: a collection of anchors {p k } K k=1 , learned automatically using weak supervision. While there are many methods to discover discriminative non-semantic mid-level parts from image collections (section 1.1), here we propose a simple alternative that, empirically, works better in our context.
We learn the anchors using a formulation similar to the MIL objective (eq. (1)): where [z] + = max{0, z}. Intuitively, anchors are learnt as discriminative midlevel parts using weak supervision. Anchor scores s k (R|x) = φ a (x|R), ω k are parametrized by vectors ω 1 , . . . , ω K ; the first term in eq. (5) is akin to the baseline MIL formulation of section 2.1 and encourages each anchor p k to score highly in images x i that contain the object (y i = +1) and to be inactive otherwise (y i = −1). The last term is very important and encourages the learned models {ω k } K k=1 to be mutually orthogonal, enforcing diversity. Note that anchors use the pure appearance-based region descriptor φ a (x) since the geometric-aware descriptor φ ag (x) can be computed only once anchors are available. Optimization uses stochastic gradient descent with momentum.
This formulation is similar to the MIL approach of [47] which, however, does not contain the orthogonality term. When this term is removed, we observed that the solution degenerates to detecting the most prominent object in an image. [41] uses instead a significantly more complex formulation inspired by mode seeking; in practice we opted for our approach due to its simplicity and effectiveness.

Incorporating strong annotations in MIL
While we are primarily interested in understanding whether semantic object parts can be learned from Web sources alone, in some cases the precise definition of the extent of a part is inherently ambiguous (e.g. what is the extent of a "human nose"?). Different benchmark datasets may use somewhat different definition of these concepts, making evaluation difficult. In order to remove or at least reduce this dataset-dependent ambiguity, we also explore the idea of using a single strongly annotated example to fix this degree of freedom.
Denote by (x a , R a ) the single strongly-annotated example of the target part. This is incorporated in the MIL formulation, eq. (1), by augmenting the score with a factor that compares the appearance of a region to that of R a : where C = avg i:yi=+1 exp β φ a (x i |R), φ a (x a |R a ) is a normalizing constant. In practice, this is used only during re-localization rounds of the training phase to guide spatial selection; at test time, bounding boxes are scored solely by the model of eq. (1) without the additional term. Other formulations, that may use a mixture of strongly and Webly supervised examples, are also possible. However, this is besides our focus, which is to see whether parts are learnable from the Web automatically, and the single supervision is only meant to reduce the ambiguity in the task for evaluation.

Experiments
This section thoroughly evaluates the proposed method. Our main evaluation is a comparison with existing state-of-the-art techniques on the task of Weblysupervised semantic part learning. In section 3.1 we show that our method is substantially more accurate than existing alternatives and, in some cases, close to fully-supervised part learning.
Having established that, we then evaluate the weakly-supervised mid-level part learning (section 2.2) that is an essential part of our approach. It compares favorably in terms of simplicity, scalability, and accuracy against existing alternatives for discriminability as well as spatial matching of object categories (section 3.2). Datasets. The Labeled Face Parts in the Wild (LFPW) dataset [53] contains about 1200 face images annotated with outlines for landmarks. Outlines are converted into bounding box annotations and images with missing annotations are removed from the test set. These test images are used to locate the following entities: face, eye, eyebrow, nose, and mouth.
The PascalParts dataset [49] augments the PASCAL VOC 2010 dataset with segmentation masks for object parts. Segmentation masks are converted into bounding boxes for evaluation. Parts of the same type (e.g. left and right wheels) are merged in a single entity (wheel ). Objects marked as truncated or difficult are not considered for evaluation. The evaluation focuses on the bus and car categories with 18 entity types overall: car, bus, and their door, front, headlight, mirror, rear, side, wheel, and window parts. This dataset is more challenging, as entities have large intra-class appearance and pose variations. The evaluation is performed on images from the validation set that contain at least one object instance. Furthermore, following [50], object occurrences are roughly localized before detecting the parts using their localization procedure. Finally, objects whose bounding box larger side is smaller than 80 pixels are removed as several parts are nearly invisible below that scale.
The training sets from both datasets are utilized solely for training the fully supervised baselines (section 3.1), and they are not used by MIL approaches. Experimental details. Regions are extracted using selective search [52], and described using 2 -normalized Decaf [54] fc6 features to compute the appearance embedding φ a (x|R). The context descriptor µ(R) is extracted from a region double the size of R. The joint appearance-geometric embedding φ ag (x|R) is obtained by first extracting the top L = 5 non-overlapping detections of each anchor and then applying eqs. (3) and (4). A separate mid-level anchor dictionary {p 1 , ..., p K } is learnt for each object class using the Web images for all the semantic parts for the target object (including images of the object as a whole) as positive images and the background clutter images of [55] as negative ones. Eq. (5) is optimized using stochastic gradient descend (SGD) with momentum for 40k iterations, alternating between positive and negative images. We train 150 anchor detectors per object class.
MIL semantic part detectors are trained solely on the Web images and the background class of [55] is used as negative bag for all the objects. The first five relocalization rounds are performed using the appearance only and the following five use the joint appearance-geometry descriptor (the joint embedding performs better with these two distinct steps). The MIL λ hyperparameter is set by performing leave-one-category-out cross-validation 2 .
Web images for parts are acquired by querying the BING image search engine. For car and bus parts, the query concatenates the object and the part names (e.g. "car door"). For face parts, we do not use the object name. We retrieve 500 images of the class corresponding to the object itself and 100 images of all other semantic part classes.

Webly supervised localization of objects and semantic parts
This section evaluates the detection performance of our approach. We gradually incorporate the proposed improvements, i.e. the context descriptor (C) and the geometrical embedding (G) to the basic MIL baseline (B) as defined in section 2.1 and monitor their impact.  We compare our method to the state-of-the-art co-localization algorithm of Cho et al. [28] and the state-of-the-art weakly supervised detection method from Bilen and Vedaldi [27]. To detect a given part with [28], we run their code on all images that contain that part (e.g. for co-localizing eyes we consider face and eye images). As reference, we also report a fully supervised detector, trained using bounding-boxes from the training set, for all objects and parts (F). For this, we use the R-CNN method of [2] on top of the same features used in MIL.
We mainly report the Average Precision (AP) per part/object class and its average (mAP) over all parts in each class. We also report the CorLoc (for correct localization) measure, as it is often used in the co-localization literature [56,17]. As most parts in both datasets are relatively small, following [49], the IoU threshold for correct detection is set to 0.4.
Results. Table 1 reports the average AP and CorLoc over all parts of a given object class for all these methods. First, we observe that even the MIL baseline (B) outperforms off-the-shelf methods such as [28] and [27]. For [27], we have observed that the part detectors degrade to detecting subparts of semantic parts, suggesting that [27] lacks robustness to drastic scale variations and to the large amount of noise present in our dataset. Second, we see that using the geometric embedding (+G) always improves the baseline results by 1 − 10 mAP points. On top of geometry, using context (+C) helps for face and car parts, but not for buses. Overall the unified embedding brings a large improvement for faces (+24.3 mAP) and for cars (+5.3 mAP) and more contained for buses (+0.6  Table 3. Part detection results using a single strong annotation (A): mAP and average CorLoc for the MIL baseline (B), our improved versions that use context (C), geometrical embedding (G). Mean and standard deviation over 25 random annotations. mAP). Importantly, these improvements significantly reduce the gap between using noisy Web supervision and the fully supervised R-CNN (F); overall, Webly supervision achieves respectively 84%, 67%, and 48% of the performance of (F).
Last but not least, we extended the fully supervised R-CNN method with the joint appearance-geometry embedding and the context descriptor (F+C+G), which improves part detections by +7.7, +9.1, +5.9 mAP points respectively. This suggests that our representation may be applicable well beyond weakly supervised learning. Table 2 shows per-part detection results for the car parts. We see that geometry helps for 6 parts out of 9. Out of the three remaining parts, two are cases for which the MIL baseline failed completely. In the less ambiguous fully-supervised scenario, the geometric embedding improves the performance in 8 out of 9 cases. Leveraging a single annotation. As noted in section 2.3, one issue with weakly supervised part learning is the inherent ambiguity in the part extent, that may differ from dataset to dataset. Here we address the ambiguity by adding a single strong annotation to the mix using the method described in section 2.3. We asked an annotator to select 25 representative part annotations per part class from the training sets of each dataset. We retrain every part detector for each of the annotations and report mean and standard deviation of mAP. As a baseline, we also consider an exemplar detector trained using the single annotated example (A).
Results are reported in table 3. Compared to pure Web supervision (B+C+G) in table 1, the single annotation (A+B+C+G) does not help for faces, for which the proposed method was already working very well, but there is a +2 mAP point improvement for cars and +6.8 mAP for buses, which are more challenging. We also note that the complete method (A+B+C+G) is substantially superior to the exemplar detector (A).

Validation of weakly-supervised mid-level anchors
This section validates the mid-level anchors (section 2.2) against alternatives in terms of discriminative information content and its ability of establishing meaningful matches between images, which is a key requirement in our application. Discriminative power of anchors. Since most of the existing methods for learning mid-level patches are evaluated in terms of discriminative content in a classification setting, we adopt the same protocol here. In particular, we evaluate the anchors as mid-level patches on the MIT Scene 67 indoor scene classification task [57]. The pipeline first learns 50 mid-level anchors for each of the 67 scene classes. Then, similar to [45], images are split into spatial grids (2x2 and 1x1) and described by concatenating the maximum scores attained by each anchor detector inside each bin of the grid. All the grid descriptors are then concatenated to form a global image descriptor which is 2 normalized. 67 one-vs-rest SVM classifiers are trained on top of these descriptors. To be comparable with other methods, we consider both Decaf fc6 and VGG-VD fc7 [58] descriptors. Table 4 contains the results of the classification experiment. Our weaklysupervised anchors clearly outperform other mid-level element approaches that are not based on CNN features [42,41,47,46]. Among CNN based approaches, our method outperforms the state-of-the-art mid-level feature based method from [45] on both VGG-VD and Decaf features. Remarkably, using our part detectors improves over the baseline which uses the global image CNN descriptor (FC) by 13.8 and 8.7 average accuracy points for Decaf and VGG-VD features respectively. Compared to other methods which are not based on detecting mid-level elements, our pipeline outperforms state-of-the-art FV-CNN for Decaf features and is inferior for VGG-VD.
Ability of anchors to establish semantic matches. The previous experiment assessed favorably the mid-level parts in terms of discriminative content; however, in the embedding φ g , these are used as geometric anchors. Hence, here we validate the ability of the mid-level anchors to induce good semantic matches between pairs of images (before learning the semantic part models).
To perform semantic matching between a source image x S and a target image x T , we consider each part annotation R S in the source and predict its best matcĥ R T in the target. The quality of the match is evaluated by measuring the IoU between the predictedR T and ground-truth R T part. When a part appears more than once (e.g. eyes often appear twice), we choose the most overlaping pair. Performance is reported by averaging the match IoU for all part occurrences and pairs of images in the test set, reporting the results for each object category.
Given a source part R S , the joint appearance-geometry embedding (anchorag) is extracted for the source part φ ag (x S |R S ) and the target regionR T that maximizes the inner product φ ag (x S |R S ), φ ag (x T |R T ) is returned as the predicted match. We also compare anchor-g that uses only the geometric embedding φ g (x|R) and the baseline a that uses only the appearance embedding φ a (x|R). We also compare two strong off-the-shelf baselines: DSP [60], state-of-the-art pairwise semantic matching method, and the method of [61], state-of-the-art for joint alignment. To perform box matching with [60] and [61] we fit an affine transformation to the disparity map contained inside a given source bounding box and apply this transform to move this box to the target image. Due to scalability issues, we were unable to apply [61] to the full dataset 3 , so we perform this comparison on a random subset of 50 images. Table 5 presents the results of our benchmark. On the small subset of 50 images the costly approach of [61] performs better than our embedding only on the LFPW faces, where the viewpoint variation is limited. On the car and bus categories our method outperforms [61] by 10% and 16% average IoU respectively. Our method is also consistently better than DSP [60] on both the small and full test set. We also note that the matching using geometric embeddings alone (anchor-g) achieves similar performance than the appearance-geometry matching (anchor-ag) which validates our intuition that the local geometry of an object is well-captured by the anchors.

An atlas for visual semantic
As a byproduct of Webly-supervised learning, our method annotates the Web images with semantic parts. By endowing an image dataset with such concepts, we show here that it is possible to browse these annotated images. All of this composes our visual semantic atlas (see a subset of the atlas in Figure 4) that allows to navigate from one image to another, even between an image of a full object and a zoomed-in image of one of its parts.

Conclusions
We have proposed a novel method for learning about objects, their semantic parts, and their geometric relationships, from noisy Web supervision. This is Fig. 4. Navigating the visual semantic atlas. Each pair of solid bounding boxes connected by an arrow denotes a preselected part bounding box (near the starting point of an arrow) as detected by our algorithm and the most similar semantic match (the endpoint of the arrow). The best matching bounding box is the detection with highest appearance-geometry descriptor similarity among all the detections in our database of web images. The dashed boxes denote anchors that contributed the most to the similarity. Please note that the matching gracefully occurs across scales.
achieved by first learning a weakly supervised dictionary of mid-level visual elements which define a robust object-centric coordinate frame. Such property theoretically motivates our approach. The geometric projections are then used in a novel appearance-geometry embedding that improves learning of semantic object parts from noisy Web data. We showed improved performance over co-localization [28], deep weakly supervised approach [27] and a MIL baseline on all benchmarked datasets. Extensive evaluation of our proposed mid-level elements shows comparable results to state-of-the-art in terms of their discriminative power and superior results in terms of the ability to establish semantic matches between images. Finally, our method also provides a visually intuitive way to navigate Web images and predicted annotations.
R, R − Q, Q ; if this factor is also a PD kernel, then the result holds as the product of PD kernels is PD. According to Lemma 3.2 of [62], −1/k is PD if, and only if,k is strictly negative (point-wise) and conditionally definite positive (CDP). The first condition is part of the assumptions. To show the second condition thatk is CDP pick n vectors R 1 , . . . , R n and real numbers c 1 , . . . , c n summing to zero c 1 + · · · + c n = 0; then where we used the fact that the terms R i , R i cancel out and the fact that R i , Q j is PD.


Contradiction Detection for Rumorous Claims

The utilization of social media material in journalistic workflows is increasing, demanding automated methods for the identification of mis- and disinformation. Since textual contradiction across social media posts can be a signal of rumorousness, we seek to model how claims in Twitter posts are being textually contradicted. We identify two different contexts in which contradiction emerges: its broader form can be observed across independently posted tweets and its more specific form in threaded conversations. We define how the two scenarios differ in terms of central elements of argumentation: claims and conversation structure. We design and evaluate models for the two scenarios uniformly as 3-way Recognizing Textual Entailment tasks in order to represent claims and conversation structure implicitly in a generic inference model, while previous studies used explicit or no representation of these properties. To address noisy text, our classifiers use simple similarity features derived from the string and part-of-speech level. Corpus statistics reveal distribution differences for these features in contradictory as opposed to non-contradictory tweet relations, and the classifiers yield state of the art performance.

Introduction and Task Definition
Assigning a veracity judgment to a claim appearing on social media requires complex procedures including reasoning on claims aggregated from multiple microposts, to establish claim veracity status (resolved or not) and veracity value (true or false). Until resolution, a claim circulating on social media platforms is regarded as a rumor (Mendoza et al., 2010). The detection of contradicting and disagreeing microposts supplies important cues to claim veracity processing procedures. These tasks are challenging to automatize not only due to the surface noisiness and conciseness of user generated content. One complicating factor is that claim denial or rejection is linguistically often not explicitly expressed, but appears without classical rejection markers or modality and speculation cues (Morante and Sporleder, 2012). Explicit and implicit contradictions furthermore arise in different contexts: in threaded discussions, but also across independently posted messages; both contexts are exemplified in Figure 1 on Twitter data.
Language technology has not yet solved the processing of contradiction-powering phenomena, such as negation (Morante and Blanco, 2012) and stance detection (Mohammad et al., 2016), where stance is defined to express speaker favorability towards an evaluation target, usually an entity or concept. In the veracity computation scenario we can speak of claim targets that are above the entity level: targets are entire rumors, such as '11 people died during the Charlie Hebdo attack'. Contradiction and stance detection have so far only marginally been addressed in the veracity context (de Marneffe et al., 2012;Ferreira and Vlachos, 2016;Lukasik et al., 2016).
We propose investigating the advantages of incorporating claim target and conversation context as premises in the Recognizing Textual Entailment (RTE) framework for contradiction detection in rumorous tweets. Our goals are manifold: (a) to offer richer context in contradiction modeling than what would be available on the level of individual tweets, the typical unit of analysis in previous studies; (b) to train and test supervised classifiers for contradiction detection in the RTE inference framework; (c) to address contradiction detection at the level of text similarity only, as opposed to semantic similarity (Xu et al., 2015); (d) to distinguish and focus on two different contradiction relationship types, each involving specific combinations of claim target mention, polarity, and contextual proximity, in particular: 1. Independent contradictions: Contradictory relation between independent posts, in which two tweets contain different information about the same claim target that cannot simultaneously hold. The two messages are independently posted, i.e., not occurring within a structured conversation. 2. Disagreeing replies: Contradictory relation between a claim-originating tweet and a direct reply to it, whereby the reply expresses disagreement with respect to the claim-introducing tweet.
Contradiction between independently posted tweets typically arises in a broad discourse setting, and may feature larger distance in terms of time, space, and source of information. The claim target is mentioned in both posts in the contradiction pair, since these posts are uninformed about each other or assume uninformedness of the reader, and thus do not or can not make coreference to their shared claim target. Due to the same reason, the polarity of both posts with respect to the claim can be identical. Texts paired in this type of contradiction resemble those of the recent Interpretable Semantic Similarity shared task (Agirre et al., 2016) that calls to identify five chunk level semantic relation types (equivalence, opposition, specificity, similarity or relatedness) between two texts that originate from headlines or captions.
Disagreeing replies are more specific instances of contradiction: contextual proximity is small and trivially identifiable by means of e.g. social media platform metadata, for example the property encoding the tweet ID to which the reply was sent, which in our setup is always a thread-initiating tweet. The claim target is by definition assumed to be contained in the thread-initiating tweet (sometimes termed as claim-or rumor-source tweet). It can be the case that the claim target is not contained in the reply, which can be explained by the proximity and thus shared context of the two posts. The polarity values in source and reply must by definition be different; we refer to this scenario as Disagreeing replies. Importantly, replies may not contain a (counter-)claim on their own but some other form to express disagreement and polarity -for example in terms of speculative language use, or the presence of extra-linguistic cues such as a URL pointing to an online article that holds contradictory content. Such cues are difficult to decode for a machine, and their representation for training automatic classifiers is largely unexplored. Note that we do not make assumptions or restrictions about how the claim target is encoded textually in any of the two scenarios.
In this study, we tackle both contradiction types using a single generic approach: we recast them as three-way RTE tasks on pairs of tweets. The findings of our previous study in which semantic inference systems with sophisticated, corpus-based or manually created syntactico-semantic features were applied to contradiction-labeled data indicate the lack of robust syntactic and semantic analysis for short and noisy texts; cf. Chapter 3 in (Lendvai et al., 2016b). This motivates our current simple text similarity metrics in search of alternative methods for the contradiction processing task.
In Section 2 we introduce related work and resources, in Sections 3 and 4 present and motivate the collections and the features used for modeling. After the description of method and scores in Section 5, findings are discussed in Section 6.

Related work and resources
Recognizing Textual Entailment (RTE) Processing semantic inference phenomena such as contradiction, entailment and stance between text pairs has been gaining momentum in language technology. Inference has been suggested to be conveniently formalized in the generic framework of RTE 1 (Dagan et al., 2006). As an improvement over the binary Entailment vs Non-entailment scenario, three-way RTE has appeared but is still scarcely investigated (Ferreira and Vlachos, 2016;Lendvai et al., 2016a). The Entailment relation between two text snippets holds if the claim present in snippet B can be concluded from snippet A. The Contradiction relation applies when the claim in A and the claim in B cannot be simultaneously true. The Unknown relation applies if A and B neither entail nor contradict each other.
The RTE-3 benchmark dataset is the first resource that labels paired text snippets in terms of 3-way RTE judgments (De Marneffe et al., 2008), but it is comprised of general newswire texts. Similarly, the new large annotated corpus used for deep models for entailment (Bowman et al., 2015) labeled text pairs as Contradiction are too broadly defined, i.e., expressing generic semantic incoherence rather than semantically motivated polarization and mismatch that we are after, which questions its utility in the rumor verification context.
As far as contradiction processing is concerned, accounting for negation in RTE is the focus of a recent study (Madhumita, 2016), but it is still set according to the binary RTE setup. A standalone contradiction detection system was implemented by (De Marneffe et al., 2008), using complex rulebased features. A specific RTE application, the Excitement Open Platform 2  has been developed to provide a generic platform for applied RTE. It integrates several entailment decision algorithms, while only the Maximum Entropy-based model (Wang and Neumann, 2007) is available for 3-way RTE classification. This model implements state-of-the-art linguistic preprocessing augmented with lexical resources (WordNet, VerbOcean), and uses the output of part-of-speech and dependency parsing in its structure-oriented, overlap-based approach for classification and was tested for both our tasks as explained in (Lendvai et al., 2016b).
Stance detection Stance classification and stance-labeled corpora are relevant for contradiction detection, because the relationship of two texts expressing opposite stance (positive and negative) can in some contexts be judged to be contradictory: this is exactly what our Disagreeing reply scenario covers. Stance classification for rumors was introduced by (Qazvinian et al., 2011) where the goal was to generate a binary (for or against) stance judgment. Stance is typically classified on the level of individual tweets: reported approaches predominantly utilize statistical models, involving supervised machine learning (de Marneffe et al., 2012) and RTE (Ferreira and Vlachos, 2016). Another relevant aspect of stance detection for our current study is the presence of the stance target in the text to be stance-labeled. A recent shared task on social media data defined separate challenges depending on whether target-specific training data is included in the task or not (Mohammad et al., 2016); the latter requires additional effort to encode information about the stance target, cf. e.g. (Augenstein et al., 2016). The PHEME project released a new stance-labeled social media dataset (Zubiaga et al., 2015) that we also utilize as described next.
the corpus was pre-annotated as explained in (Zubiaga et al., 2015) for several rumorous claims 9 -officially not yet confirmed statements lexicalized by a concise proposition, e.g. "Four cartoonists were killed in the Charlie Hebdo attack" and "French media outlets to be placed under police protection". The corpus collection method was based on a retweet threshold, therefore most tweets originate from authoritative sources using relatively well-formed language, whereas replying tweets often feature non-standard language use. Tweets are organized into threaded conversations in the corpus and are marked up with respect to stance, certainty, evidentiality, and other veracity-related properties; for full details on released data we refer to (Zubiaga et al., 2015). The dataset on which we run disagreeing reply detection (henceforth: Threads) was converted by us to RTE format based on the threaded conversations labeled in this corpus. We created the Threads RTE dataset drawing on manually pre-assigned Response Type labels by (Zubiaga et al., 2015) that were meant to characterize source tweet -replying tweet relations in terms of four categories. We mapped these four categories onto three RTE labels: a reply pre-labeled as Agreed with respect to its source tweet was mapped to Entailment, a reply pre-labeled as Disagreed was mapped to Contradiction, while replies pre-labeled as AppealforMoreInfo and Comment were mapped to Unknown. Only direct replies to source tweets relating to the same four events as in the independent posts RTE dataset were kept. There are 1,850 tweet pairs in this set; the proportion of contradiction instances amounts to 7%. The Threads dataset holds CON, ENT and UNK pairs as exemplified below. Conform the RTE format, pair elements are termed text and hypothesis -note that directionality between t and h is assumed as symmetric in our current context so t and h are assigned based on token-level length.
• CON <t>We understand there are two gunmen and up to a dozen hostages inside the cafe under siege at Sydney.. ISIS flags remain on display 7News</t> <h>not ISIS flags</h> • ENT <t>Report: Co-Pilot Locked Out Of Cockpit Before Fatal Plane Crash URL Germanwings URL</t> <h>This sounds like pilot suicide.</h> • UNK <t>BREAKING NEWS: At least 3 shots fired at Ottawa War Memorial. One soldier confirmed shot -URL URL</t> <h>All our domestic military should be armed, now.</h>. The independently posted tweets dataset (henceforth: iPosts) that we used for contradiction detection between independently emerging claim-initiating tweets is described in (Lendvai et al., 2016a). This collection is holds 5.4k RTE pairs generated from about 500 English tweets using semi-automatic 3-way RTE labeling, based on semantic or numeric mismatches between the rumorous claims annotated in the data. The proportion of contradictory pairs (CON) amounts to 25%. The two collections are quantified in Table 1. iPosts dataset examples are given below.
• CON <t>12 people now known to have died after gunmen stormed the Paris HQ of magazine CharlieHebdo URL URL</t> <h>Awful. 11 shot dead in an assault on a Paris magazine. URL CharlieHebdo URL</h> • ENT <t>SYDNEY ATTACK -Hostages at Sydney cafe -Up to 20 hostages -Up to 2 gunmen -Hostages seen holding ISIS flag DEVELOPING..</t> <h>Up to 20 held hostage in Sydney Lindt Cafe siege URL URL</h> • UNK <t>BREAKING: NSW police have confirmed the siege in Sydney's CBD is now over, a police officer is reportedly among the several injured.</t> <h>Update: Airspace over Sydney has been shut down. Live coverage: URL sydneysiege</h>. 9 Rumor, rumorous claim and claim are used interchangeably throughout the paper to refer to the same concept.

Text similarity features
Data preprocessing on both datasets included screen name and hashtag sign removal and URL masking. Then, for each tweet pair we extracted vocabulary overlap and local text alignment features. The tweets were part-of-speech-tagged using the Balloon toolkit (Reichel, 2012) (PENN tagset, (Marcus et al., 1999)), normalized to lowercase and stemmed using an adapted version of the Porter stemmer (Porter, 1980). Content words were defined to belong to the set of nouns, verbs, adjectives, adverbs, and numbers, and were identified by their part of speech labels. All punctuation was removed.

Vocabulary overlap
Vocabulary overlap was calculated for content word stem types in terms of the Cosine similarity and the F1 score. The Cosine similarity of two tweets is defined as C(X, Y ) = |X∩Y | √

|X|·|Y |
, where X and Y denote the sets of content word stems in the tweet pair. The F1 score is defined as the harmonic mean of precision and recall. Precision and recall here refer to covering the vocabulary X of one tweet by the vocabulary Y of another tweet (or vice versa). It is . Again the vocabularies X and Y consist of stemmed content words.
Just like the Cosine index, the F1 score is a symmetric similarity metric. These two metrics are additionally applied to the content word POS label inventories within the tweet pair, which gives the four features cosine, cosine pos, f score, and f score pos, respectively.

Local alignment
The amount of stemmed word token overlap was measured by applying local alignment of the token sequences using the Smith-Waterman algorithm (Smith and Waterman, 1981). We chose a score function rewarding zero substitutions by +1, and punishing insertions, deletions, and substitutions each by 0-reset. Having filled in the score matrix H, alignment was iteratively applied the following way: while max(H) ≥ t -trace back from the cell containing this maximum the path leading to it until a zero-cell is reached -add the substring collected on this way to the set of aligned substrings -set all traversed cells to 0.
The threshold t defines the required minimum length of aligned substrings. It is set to 1 in this study, thus it supports a complete alignment of any pair of permutations of x. The traversed cells are set to 0 after each iteration step to prevent that one substring would be related to more than one alignment pair. This approach would allow for two restrictions: to prevent cross alignment not just the traversed cells [i, j] but for each of these cells its entire row i and column j needs to be set to 0. Second, if only the longest common substring is of interest, then the iteration is trivially to be stopped after the first step. Since we did not make use of these restrictions, in our case the alignment supports cross-dependencies and can be regarded as an iterative application of a longest common substring match.
From the substring pairs in tweets x and y aligned this way, we extracted two text similarity measures: • laProp: the proportion of locally aligned tokens over both tweets m(x)+m(y) n(x)+n(y) • laPropS: the proportion of aligned tokens in the shorter tweet m(ẑ) n(ẑ) ,ẑ = arg min z∈{x,y} [n(z)], where n(z) denotes the number of all tokens and m(z) the number of aligned tokens in tweet z. Figures 2 and 3 show the distribution of the features introduced above each for a selected event in both datasets. Each figure half represents a dataset; each subplot shows the distribution of a feature in dependence of the three RTE classes for the selected event in that dataset.

Corpus statistics
The plots indicate a general trend over all events and datasets: the similarity features reach highest values for the ENT class, followed by CON and UNK. Kruskal-Wallis tests applied separately for all combinations of features, events and datasets confirmed these trends, revealing significant differences for all boxplot triplets (p < 0.001 after correction for type 1 errors in this high amount of comparisons using the false discovery rate method of (Benjamini and Yekutieli, 2001)). Dunnett post hoc tests however clarified that for 16 out of 72 comparisons (all POS similarity measures) only UNK but not ENT and CON differ significantly (α = 0.05). Both datasets contain the same amount of non-significant cases. Nevertheless, these trends are encouraging to test whether an RTE task can be addressed by string and POS-level similarity features alone, without syntactic or semantic level tweet comparison.

RTE classification experiments for Contradiction and Disagreeing Reply detection
In order to predict the RTE classes based on the features introduced above, we trained two classifiers: Nearest (shrunken) centroids (NC) (Tibshirani et al., 2003) and Random forest (RF) (Breiman, 2001;Liaw and Wiener, 2002), using the R wrapper package Caret (Kuhn, 2016) with the methods pam and rf, respectively. To derive the same number of instances for all classes, we applied separately for both datasets resampling without replacement, so that the total data amounts about 4,550 feature vectors equally distributed over the three classes, the majority of 4,130 belonging to the iPosts data set. Further, we centered and scaled the feature matrix. Within the Caret framework we optimized the tunable parameters of both classifiers by maximizing the F1 score. This way the NC shrinkage delta was set to 0, which means that the class reference centroids are not modified. For RF the number of variables randomly sampled as candidates at each split was set to 2. The remaining parameters were kept default.
The classifiers were tested on both datasets in a 4-fold event-based held-out setting, training on three events and testing on the remaining one (4-fold cross-validation, CV), quantifying how performance generalizes to new events with unseen claims and unseen targets. The CV scores are summarized in Tables 2  and 3. It turns out generally that classifying CON is more difficult than classifying ENT or UNK. We observe a dependency of the classifier performances on the two contradiction scenarios: for detecting CON, RF achieved higher classification values on Threads, whereas NC performed better on iPosts. General performance across all three classes was better in independent posts than in conversational threads.
Definitions of contradiction, the genre of texts and the features used are dependent on end applications, making performance comparison nontrivial (Lendvai et al., 2016b). On a different subset of the Threads data in terms of events, size of evidence, 4 stance classes and no resampling, (Lukasik et al., 2016) report .40 overall F-score using Gaussian processes, cosine similarity on text vector representation and temporal metadata. Our previous experiments were done using the Excitement Open Platform incorporating syntactico-semantic processing and 4-fold CV. For the non-resampled Threads data we reported .11 F1 on CON via training on iPosts (Lendvai et al., 2016b). On the non-resampled iPosts data we obtained .51 overall F1 score (Lendvai et al., 2016a), F1 on CON being .25 (Lendvai et al., 2016b).   We proposed to model two types of contradictions: in the first both tweets encode the claim target (iPosts), in the second typically only one of them (Threads). The Nearest Centroid algorithm performs poorly on the CON class in Threads where textual overlap is typically small especially for the CON and UNK classes, in part due to the absence of the claim target in replies. However, the Random Forest algorithm's performance is not affected by this factor. The advantage of RF on the Threads data can be explained by its property of training several weak classifiers on parts of the feature vectors only. By this boosting strategy a usually undesirable combination of relatively long feature vectors but few training observations can be tackled, holding for the Threads data that due to its extreme skewedness (cf. Table 1) shrunk down to only 420 datapoints after our class balancing technique of resampling without replacement. Results indicate the benefit of RF classifiers in such sparse data cases.
The good performance of NC on the much larger amount of data in iPosts is in line with the corpus statistics reported in section 4.3, implying a reasonably small amount of class overlap. The classes are thus relatively well represented by their centroids, which is exploited by the NC classifier. However, as illustrated in Figures 2 and 3, the majority of feature distributions are generally better separated for ENT and UNK, while CON in its mid position shows more overlap to both other classes and is thus overall a less distinct category.

Conclusions and Future Work
The detection of contradiction and disagreement in microposts supplies important cues to factuality and veracity assessment, and is a central task in computational journalism. We developed classifiers in a uniform, general inference framework that differentiates two tasks based on contextual proximity of the two posts to be assessed, and if the claim target may or may not be omitted in their content. We utilized simple text similarity metrics that proved to be a good basis for contradiction classification.
Text similarity was measured in terms of vocabulary and token sequence overlap. To derive the latter, local alignment turned out to be a valuable tool: as opposed to standard global alignment (Wagner and Fischer, 1974), it can account for crossing dependencies and thus for varying sequential order of information structure in entailing text pairs, e.g. in "the cat chased the mouse" and "the mouse was chased by the cat", which are differently structured into topic and comment (Halliday, 1967). We expect contradictory content to exhibit similar trends in variation with respect to content unit order -especially in the Threads scenario, where entailment inferred from a reply can become the topic of a subsequent replying tweet. Since local alignment can resolve such word order differences, it is able to preserve text similarity of entailing tweet pairs, which is reflected in the relative laProp boxplot heights in Figures 2 and 3.
We have run leave-one-event-out evaluation separately on the independent posts data and on the conversational threads data, which allowed us to compare performances on collections originating from the same genre and platform, but on content where claim targets in the test data are different from the targets in the training data. Our obtained generalization performance over unseen events turns out to be in line with previous reports. Via downsampling, we achieved a balanced performance on both tasks across the three RTE classes; however, in line with previous work, even in this setup the overall performance on contradiction is the lowest, whereas detecting the lack of contradiction can be achieved with much better performance in both contradiction scenarios.
Possible extensions to our approach include incorporating more informed text similarity metrics (Bär et al., 2012), formatting phenomena (Tolosi et al., 2016), and distributed contextual representations (Le and Mikolov, 2014), the utilization of knowledge-intensive resources , representation of alignment on various content levels , and formalization of contradiction scenarios in terms of additional layers of perspective (van Son et al., 2016).

Acknowledgments
P. Lendvai was supported by the PHEME FP7 project (grant nr. 611233), U. D. Reichel by an Alexander von Humboldt Society grant. We thank anonymous reviewers for their input.


Design of a cloud robotic system to support senior citizens: the KuBo experience

This paper describes a system based on a robot, called KuBo, which relies on cloud resources to extend its capabilities for human interaction and environmental sensing to provide services for independent living. The robot uses text-to-speech and speech recognition services as well as an electronic agenda and web resources to perform several tasks. Moreover it retrieves smart environmental data from a DataBase to be aware of the context. In this paper, the cloud robotics approach is used to increase the skills of a robot, endowing the system with abilities for human–robot interaction and environmental sensing. The robotic services have been defined with a focus group involving 19 elderly volunteers and the system has been tested in a real environment with a couple of elderly users for five days. The aim of the experiment was to test the technical feasibility of the proposed cloud services using quantitative tools. The technical results show a success rate of 86.2 % for the navigation task and more than 90 % for the speech capabilities. Furthermore, the robustness of the system was also confirmed by users’ qualitative feedback.

Introduction
Robot companions are becoming more common and familiar in our lives (Dario 2011). According to the International Federation of Robotics' statistics, in 2013, about 4 million service robots for personal and domestic use were sold, 28 % more than in 2012, increasing the value of sales to US$ 1.7 billion (IFR 2015). In particular, service robotics has received a lot of attention from industry and academia, in order to face societal and demographic challenges (Zaidi et al. 2006;Stula 2012).
According to ABI research (Solis and Carlaw 2013), service robots represent the second potential big market opportunity.
In this context, Ambient Assisted Living (AAL) solutions aim to meet users' and stakeholders' needs, providing ICT and robotic services able to assist the user during daily activities (van den Broek et al. 2010;Moschetti et al. 2014). The main benefits which can been achieved by service robots are: -support senior citizens during daily activities (e.g. participation in social events, reminders, surveillance) enhancing their independent living; -enhance the quality of life, compensating for motion and cognitive deficits; -improve the quality of health services, reducing the cost for society and public health system.
In general, domestic robots need to be friendly, interact with users, and autonomously move inside the house without revolutionizing the familiar environment. On the other hand, the solutions provided by robotics services should guarantee time continuity all day long. During the last years, two main robotics paradigms have been adopted. Stand-alone robots are designed according to a robot-centred approach, where the robot alone is in charge of the entire capabilities of sensing, planning and acting (Cesta et al. 2010;RP-Vita;Aldebaran). This approach has some limitations due to the extension of the robot sensing area, the robot payload, batteries and computational capabilities. Even though the robots increase their particular abilities, they remain insufficient for continuously supporting daily activities adequately. To better understand this turning point, just think of your daily life: you perform several activities in different contexts and rarely in one place. Now the question is: are stand-alone robots sufficient for continuously supporting a person during the day in a wide range of activities and in different environments? User requests are very different and depend on the specific needs of a specific moment. The networked robot paradigm (Sanfeliu et al. 2008) distributes sensors and computational capabilities over a smart environment and intelligent agents, like wearable and personal devices, extending the effective sensing range of stand-alone robots, improving their ability to plan and cooperate (Coradeschi et al. 2014;Volkhardt et al. 2011;Simonov et al. 2012). Therefore, the problem of the continuity of the service and computational limitations still remains (Kamei and Nishio 2012).
In this context, the cloud robotics paradigm is trying to overcome the limitations of the stand-alone and networked robotics paradigms, by integrating robots with cloud computing resources (Kuffner 2010). Recently, the cloud robotic paradigm has been defined as "any robot or automation system that relies on either data or code from a network to support its operation, where not all sensing, computation and memory is integrated into a single standalone system" (Kehoe et al. 2015). This new generation of low-cost robots (Kuffner 2010) can use wireless networking, big data, machine learning techniques, and the Internet of Things to improve the quality of their services of assistance (Lorencik and Sincak 2013). Robots with different capabilities have the possibility to share data, knowledge, and skills, exchanging information with other agents connected to the network, leading to the reduction of the overall costs (Tenorth et al. 2011).
Cloud robotics is not a completely new idea. Indeed, during the 90s, Prof. Inaba (Inaba 1997) conceptualized the remote brain paradigm, in which the hardware agents can access to a remote "intelligence" with high computational abilities. But only over the last few years with the rise of mobile technologies has the growth of internet resources and the global penetration of smartphones [Mobile Planet] made concrete the idea of cloud robotics.

Related research
Over the last few years, several research groups have focused their efforts on the challenges of cloud robotics, and some recent examples of its application in service robotics can be found in the literature. First of all, additional concepts related to the cloud robotic paradigm need to be cited. Kamei et al. (Kamei and Nishio 2012) expand the concept of networked robotics, proposing a new research field called Cloud Networked robotics. They describe "The Life Support Robot Technology", a Japanese project started in 2009, focused on the development of six robotic services with high safety, reliability, and adaptability. Furthermore, Chen et al. (Chen et al. 2010) introduced the concept of Robot as a Service (RaaS) which enforces the idea of a robot that uses services from a remote resources, "this all-in-one design gives the robot unit much more power and capacity, so that it can qualify as a fully self-contained cloud unit in the cloud computing environment." Bonaccorsi et al. (Bonaccorsi et al. 2015) expand the cloud robotics introducing the concept of Cloud Service Robotics as "The integration of different agents that allow an efficient, effective and robust cooperation between robots, smart environments and citizens." As stated by Kehoe (Kehoe et al. 2015), the cloud robotics field can be divided according to its benefits. Some research has been focused on the use of large datasets including video, images and vast sensor networks which are difficult to manage with the on-board capacity. In particular, the software ODUfinder [Odufinder] is able to perform object recognition exploiting external databases which contain over 3500 pictures, whereas in Kehoe et al. (2013) a robot uses the Google object recognition engine. RoboEarth was the first EU project in the cloud robotics paradigm [RoboEarth]. It allows robots to share and store information as well as manipulation strategies and object recognition models. Robots can use the cloud to offload computation and collaborate to achieve a common task. Other publications report the use of external cloud computing resources to speed up computationally intensive tasks such as SLAM (Benavidez et al. 2015;Riazuelo et al. 2014) algorithms, object recognition (Oliveira and Isler 2013) and video and image analysis (Nister and Stewenius 2006). In particular, Quintas et al. (Quintas et al. 2011) proposed a cloud robotics context aware approach for an automated system composed of mobile robots and a smart home. In order to enhance the level of scalability of the system, this approach relied on cloud computing services. Additional research has been focused on the sharing of knowledge and on the use of crowd-sourcing as a resource for the robot.
Among commercial solutions, Gostai [Gostai] has developed a cloud robotics infrastructure called GostaiNET. The robot intelligence is no longer embedded in the robot but executed in the cloud, allowing the remote execution of tasks such as voice recognition, face detection, and speech algorithms on any compatible robot. Engineers at Romotive [Romotive] have developed a companion robot which learns while you play. Thanks to the cloud, anyone can control Romo from anywhere in the world.
In this context, the aim of the present paper is to design and develop an innovative cloud-based robotic system with a user-centred approach and evaluate its technical feasibility for supporting senior citizens in daily activities at home. The system, called KuBo, is based on a mobile robot, shaped as a piece of domestic furniture, with low on-board abilities, which relies on cloud resources to extend its capabilities for interacting with humans and sensing the environment. A meticulous methodology was followed to, firstly, define the technical specifications to design and develop Kubo according to the end-users' requirements; and secondly, to assess the technical reliability and safety of the robot in performing navigation and speech capabilities in a real environment. In particular, a specific domestic use case was designed to test the system in an apartment, inhabited by a elderly couple. The goal of the use case was to demonstrate the technical feasibility of the system when used by users. Therefore the system was left for 5 consecutive days and the elderly, after appropriate training, were able to freely ask Kubo to provide the conceived services.
The rest of this paper is structured as follows. In Sect. 3, the authors detail the applied methodology used in this research. Section 4 describes the proposed system. Section 5 summarizes the results and Sect. 6 discusses the results. Section 7 concludes the paper.

Methodology
This section focuses on the methodology used to build and test the KuBo robotic system to support senior citizens. The methodology is based on four phases. Phase I has been dedicated to the definition of the services, starting from an analysis of the needs of the elderly. Phase II comprised the development and integration of the KuBo system. Then in Phase III, the experimental protocol was developed and defined. Lastly, during Phase IV, the system was tested in a real environment with real users for five days.

Phase I: KuBo service definition
The services of the system have been studied and designed applying the User Centred Design approach (Heerink et al. 2009) in order to identify a concept responding to usability and acceptability criteria. A focus group, involving 19 elderly volunteers, aged from 64 to 85 years (μ = 73.05, σ = 6.55), was organized in order to define the capabilities of KuBo according to the end-users' requirements, such as their needs and lifestyles. The outcome of the focus group has produced a set of services grouped into three main areas: the use of the robot to get help for some activities, the need to have information in appropriate situations, and remote user assistance (see Table 1). The first group includes the (I) Carrying Object service, which allows the user to call KuBo to get an object stored on its shelf, such as a tablet, book or prescription glasses, (II) Internet Access is used to access web resources, such as weather forecasts, by means of a speech interface. This service is also used autonomously by the robot in order to modulate its interaction with the user. For instance, if the user has to go outside for an appointment and the weather forecast states that it is going to rain, KuBo suggests taking an umbrella. The second group contains the (III) Reminder Service, which remembers commitments and appointments, and the (IV) Monitoring Service, which alerts the user when dangerous situations are recognized by the smart environment. With the (V) Telepresence service, a caregiver can use KuBo for remote assistance. Another outcome of the focus group is that the participants prefer to control the robot through a combination of GUI (tablet) and vocal commands.

Phase II: system architecture design and implementation
The hardware and software architecture of the entire KuBo system has been developed in order to meet the users' needs coming from Phase I. It is described in detail in Sect. 4. This phase led to the development of a small sized robot connected with cloud resources and environmental sensing abilities.

Phase III: definition of the experimental protocol
The KuBo system experimentation was conducted as a case study in which the trials took place in a real private house for 5 days. The case study was designed in order to allow elderly people to have interaction with the KuBo robot in a sequence of simple tasks according to the services defined in Phase I. During this period of testing, quantitative data was collected in order to evaluate the performance of two KuBo system modules in, e.g. navigation tasks and speech capabilities. The navigation module was chosen in order to understand whether on board functionality was a successful strategy, while the speech capabilities was selected as an example of a cloud service.

Technical performance evaluation tool
The evaluation metrics for assessing the technical performance are related to KuBo navigation tasks and speech recognition abilities. The navigation of the robot has three possible states: RUNNING, SUCCEEDED and FAILED. A navigation task begins when the robot goes to RUNNING (i.e. it receives a goal) and ends when it passes to SUCCEEDED or FAILED. The speech functions were evaluated considering each single sentence and the correctness of the elaboration. The following parameters have been employed to globally evaluate the system: Success rate This is the percentage of the total tasks that succeeded. It gives information about the reliability of the system.

Failure rate It is computed as
Failure rate(%) = 100 − Success rate(%) Effective robot velocity This is applied only to the navigation tasks: it represents the average velocity of KuBo. It is an important parameter which influences the robot's acceptability (Salvini et al. 2010). For the experiments, the velocity of KuBo was limited to 0.2 m/s. It is also related to the safety of the system. Confidence This is applied only to the speech recognition functions: it represents the probability of the correctness of the recognition.
In addition, at the end of the test period, a researcher asked the elderly volunteers to perform again the KuBo services applying the "think aloud" (TAL) method (Lewis 1982). With the TAL method, the user is encouraged to report aloud any action or thought while carrying out the tasks. All users' verbalizations were transcribed and then analyzed. In this study, a general inductive approach for the analysis of qualitative evaluation data (Thomas 2006) was applied. This approach consists of three phases: using raw data to create categories, establish the pertinence of the categories with the research objective, and the development of a theory. In this study, a model could not be developed due to the small sample size. However, the aim of this analysis is not to investigate the usability and acceptability because the sample is too small and not statistically significant. Therefore this analysis gives holistic information about the robustness of the system from an user point of view.

Phase IV: case study setup
The experiment was conducted as a case study in which a couple of Italian elderly volunteers tested the system in their home for 5 days. In this case study, the smart environment described in Sect. 4.2 is a simplified version comprising gas sensors, used as a proof of concept. During the 5 days, all the services were tested, including a simulated test of a gas leak. The developers provided their assistance in the house the first two days of experiments; for the remaining days, the system was accessed remotely and the users were left free to use the system. At the end of the experiment, the elderly users were more confident with the KuBo system and they performed again the services applying the TAL while a researcher transcribed their opinions.

System architecture
The architecture of the system is based on three components: the KuBo robot, the Smart Environments, and the Cloud Software as a Service (SaaS) (see Fig. 1). The high level software layer of the robot relies on cloud resources to endow KuBo with additional functions in a modular way. It is able to extend its sensing capabilities through using the smart environment, accessing web resources, and exploiting powerful voice and speech recognition services. The Smart Environment is composed of several devices providing the user's position within the house as well as sensors to monitor the temperature, human presence, water/gas leaks. For the sake of example, when a sensor triggers a gas leak event, the robot retrieves the user's position and moves towards him/her, warning the person.

KuBo robot
In this section, the design process and the software architecture of the prototype is detailed. The robot complies with several features required by a domestic robot.

Design
The role of user acceptability for a companion robot is crucial, therefore during the design phase of the prototype, two key points have been primarily considered: reduced dimensions to move easily in a domestic environment, and the use of a modern design style to improve the appearance and favour the friendliness of the platform. Hence, KuBo is based on the youBot [Kuka], commercialized by KUKA, a smallsized holonomic mobile base, and it is equipped with a laser scanner for navigation purposes, a depth camera, speakers, a microphone, and a tablet for human-robot interaction.
In order to favour the acceptance of the robot (Salvini et al. 2010), the original platform was modified with a design inspired by a typical "coffee table", a common piece of furniture in homes. Figure 2 shows the design process of the robot. The overall height extends by about 30 cm and a new cover, made of black opal methacrylate, is mounted to an internal frame. Some adhesive tape has been used to personalize the prototype. The final dimensions of the prototype are 40 × 40 × 60 cm.

Software
KuBo is conceived as a platform with low computational capabilities that has to exploit cloud resources to carry out its tasks. Figure 1 depicts the architecture of the system and the software layers of the KuBo robot. The Service Manager executes the tasks, using properly the robot functions. All the software modules are implemented in ROS (Quigley 2009) and the only on-board ability is that of autonomous indoor navigation, which relies on the ROS navigation stack and uses the Dynamic Window Approach (Fox et al. 1997) for local planning and Adaptive Monte Carlo (Thrun et al. 2005) for indoor localization. It uses a 2-D static map of the environment to navigate and a laser scanner for obstacle avoidance and self-localization abilities.
The Service Manager is able to use all the modules in order to accomplish a particular task. For example, when the Reminder Module triggers an event (e.g. a doctor appointment), the Manager retrieves the user's position through the Smart Environment Module, moves KuBo to him/her, notifies the user of the event using text-to-speech (TTS), and waits for user confirmation (Speech Recognition). If the user has to go outside, an appointment in this case, the Service Manager retrieves the weather forecast by means of the Internet Module and use the TTS to communicate the downloaded information. All these modules, with the exception of the Navigation Stack, rely on cloud resources.

Robot cloud modules
The system is implemented with five cloud modules:

Smart environment module This connects KuBo with the
DataBase Management Software (DBMS), allowing the retrieval of information from the database after an authentication phase. It runs two TCP clients: the first requests the user's position any time the robot needs to reach him/her, the second makes polling requests (1 Hz) to identify any changes in the database concerning environ-mental alarms. All the communications follow a proper string protocol based on JSON codification. Reminder module This is able to link the Google calendar service with KuBo. It is based on Google Calendar API v3 [Google Calendar] with JSON data object. Using this API, it is possible to search and retrieve calendar events, as well as create, edit, and delete events. The user can set appointments through a web browser (or the mobile App synchronized with the calendar) and this module is able to activate the reminder service at the proper time. Text-to-speech module This connects the robot with the Acapela Voice as a Service API [Acapela] using HTTP connections. This module stores locally the sentences already converted, to reduce the response time. It sends a text string and plays the audio files received from the service. Internet access module This is used to retrieve generic information from web sites. In this implementation, the authors have implemented a weather forecast service as an example. The robot retrieves an HTML file from a specific web site using the HTTP protocol. The file is then parsed to find the proper information to communicate. Speech recognition module This connects the robot with the Google Speech Recognition API through HTTP connections. This module also implements a dictionary of keywords to elaborate user requests, such as move to a particular room, request the weather forecast, or ask for the current time.

Smart environment
The Smart Environment is composed of two ZigBee-based wireless sensor networks (WSNs), one for user localization and the other for environmental monitoring. The user localization network is designed to locate multiple users at the same time, using received signal strength (RSS) (Esposito et al. 2015;Cavallo 2014). The WSN for environmental monitoring is composed of several sensors able to monitor the temperature, human presence, water/gas leaks, and control the lights. These sensors are distributed in the house in order to have a real-time measurement of the environment's conditions. The information is processed and stored in the Smart Environment DataBase (see Sect. 4.3). This system manages several alarm procedures, such as a door's opening during the night, a water or gas leak, and door/windows open when the user is outside. The performance and the accuracy of this kind of system was presented in (Bonaccorsi et al. 2015). As already explained in Sect. 3.4, during the 5-day experiment, a simplified version of the WSNs was used, in order to not be too intrusive in their daily life and compromise the overall experiment. A gas sensor was used as a proof of concept.

Cloud SaaS
The software modules described in Sect. 4.1.3 are connected with specific Cloud SaaS. In more detail, they comprise four modules: Smart environment database This stores the data collected from the WSN, while the DBMS administers entries and queries avoiding direct connection between the hardware agents (WSN and robot) and personal data. It is implemented as a relational database, based on MySQL, which has several tables: one for each sensor type containing their outputs, one with a list of the installed sensors (typology and unique identification number), one collecting environmental alarms, and another table recording the user's estimated position. The outputs from the physical agents and the estimated user position are sent to the DBMS and recorded in the DataBase. Acapela VaaS This takes as input the text string to translate, a language, and a voice type, and produces an MP3 audio file. This service allows using several languages and several voices (male, female). The robot uses this service only for unknown sentences, to reduce the response time during the interaction phase. Google services In this research, the developed system uses two Google Services. The first is the calendar API, while the second is the speech recognition API. Web resources The Web is full of information that can be retrieved by the robot to improve the user interaction experience. In this implementation, the authors have implemented the weather forecast as an example. The robot uses the HTTP protocol to retrieve the proper information from a dedicated web site.

Result
The authors provide here some technical results about the performance of the navigation and speech interactions. During the experiments, KuBo performed 94 navigation tasks. The success and the failure rates are computed analysing navigation log files that were updated every 4 s or on any change in the robot's state. The number of successes counts the transitions between the RUNNING state and the SUCCEEDED state, while a transition to a FAILURE state increments the total number of failed navigation tasks. Figure 3 shows the positions of KuBo during the navigation tasks that succeeded, for one day of the experimental sessions. The results, presented in Table 2, show that the success rate of the navigation task is 86.2 % while the failure rate is 13.8 %.
For safety reasons, the velocity of KuBo is limited to 0.2 m/s and the effective robot velocity, computed as the mean By means of speech interaction, performed in the Italian language, the user can activate four robot activities. He can move the robot through rooms saying "move" or "go" plus the name of the room. He can confirm a reminder event with the phrase "thank you" and can ask for information about the time, the day, and the weather forecast. The robot is also able to react to general greetings like "hello" or "what's your name". The Google Speech Recognition API has produced good results during the use case (see Table 3). It has a perfect recognition rate with words like "thanks" or sentences like "what time/day is it". Interesting considerations come up if we consider the failure cases. Since this service is intended to be used by smartphones and tablet applications, several utterances are translated with web entities, business company and geographical locations. The utterance "Ciao KuBo" (84 % success rate) has often produced the output "Yahoo", while asking for the weather forecast produced, on rare occasions, an URL. Some words, like "KuBo" or "Casa" (home), are turned into locations like Cuba, Cannes or Cagliari.
Taking into account these results, the development of a speech recognition module that uses such a cloud resource has to consider these cases to improve and provide a better interaction experience with the user.
In addition, some qualitative results come to light from the TAL method and the users' verbalizations have been transcribed and then analysed. The first step was to group the raw text data in order to identify some categories. The outcome of this analysis is the definition of five categories: AestheticsThe aesthetic attractiveness of the robot with respect to the user Regarding the KuBo's aesthetics, the general impression is positive because according to the elderly's answers, the colors are judged enjoyable and the size of the robot is small enough to be used in an indoor environment. Furthermore the shape of KuBo reminds older users of a coffee table, a piece of furniture suitable in interior design. Anxiety Negative emotional reactions evoked when the person uses it The participants reported being relaxed during the interaction with KuBo, because it is judged ease to use. Furthermore, the user had no anxious reaction, since they perceive themselves as having control over the robot, which looks like a small piece of furniture. Furthermore this outcome shows that the Effective Robot Velocity is adequate for domestic use by the elderly. Reliability The user's feeling about the robustness of the system The KuBo robot is judged reliable enough because it seems robust as confirmed by the high success rate. Moreover, according to the elderly people, the appearance of the robot is able to communicate its functions.   Ease of use The ease of the interaction modalities About the interaction modalities, the elderly perceive the interfaces based on vocal commands as easier to use than the graphical one. In fact, they have understood well which vocal command to use in order to interact with KuBo. The tablet introduces some difficulties, since they are not used to having it and, in addition, they need to wear prescription glasses. Perhaps the high confidence value of the speech capabilities positively influences the perceived ease of use. Utility Usefulness of the services in daily life The participants consider very useful the robotic capabilities for the improvement of their independence and safety in daily life, because, on getting old, physical and cognitive impairments can arise. Lastly, the participants claim to have fun using KuBo and they are well disposed to using it in the future because they think that the KuBo services could help overcome loneliness. The high number of acteivations of the robotic services shows the willingness of the users to use the system.

Discussion
Considering the performance in the navigation tasks, the quantitative results show a failure rate of 13.8 %. Although this value is not very high, the failures are more concentrated in certain areas, because they are strictly correlated with KuBo self-localization errors. Particularly, the robot positions during FAILURE states are more concentrated in the kitchen (41.67 %) and dining areas (25 %) that are contiguous to the kitchen (see Fig. 4). The experimental phase in the real environment highlights some computational limitation of the navigation performance. These high error values are due to the presence of The possibility of exploiting cloud solutions allowing the use of more computational power for this task will be investigated in future research. Such a solution is more advisable than replacing the hardware of the robot because the cloud resource is cheaper, shareable, and reliable.
Concerning the cloud functionalities, one of the main constraints depends on the delay sensitivity of the tasks (Hu et al. 2012). The choice between a stand-alone and a cloud architecture should rely on the maximum acceptable delay in the service delivery, which is also strictly correlated with the computational abilities and with the data-rate of the communication technology involved (i.e. the average value for LTE is 45 Mb/s whereas for home ADSL it is 10 Mb/s). So, a cloud robotics architecture should be designed taking into account the optimal trade-off between the distribution of the resources, the computational capabilities, and the performance of the tasks.
In this use-case, the users expressed neither positive nor negative comments about delays in the tasks. This suggests that the technical performance is acceptable from the user's point of view. Effectively, the speech capabilities have an high success rate, as reported in Sect. 5. Since the Google Speech Recognition is designed to be used mainly in mobile applications, the outputs are sometimes related to web resources, business company and geographical locations. The development of a recognition module for assisted living has to take into account this outcome to provide a better interaction experience with the user.
In addition, the KuBo services meet the users' needs because they were defined by 19 elderly people in order to develop a robotic system according to the final users' requirements; in fact the two participants reported that the KuBo system was really useful for them. Regarding the characteristics of the robot, the Aesthetics category is an important acceptability factor because the appearance of KuBo is aesthetically pleasing for the participants. Furthermore, according to the elderly persons' comments, the prototype could be easily integrated in a domestic environment since it is small and its shape reminds them of a coffee table. Moreover, the users' feelings about the robot's capabilities, defined as Reliability, was evaluated positively by older users: in fact they might not use a robot if its functionalities and capabilities are perceived as useless, dangerous, or not well performing (Klamer and Ben Allouch 2010). Concerning the Anxiety category, the participants reported not feeling any negative emotional reactions when using the KuBo system, and according to other studies (Heerink et al. 2009), an high degree of acceptability is correlated with a low level of anxiety. The KuBo system is judged easy to use and elderly participants say to be disposed to use in the future because they think that the service are useful to overcome loneliness.

Conclusion
In this paper, the authors described a robotic platform which provides cloud robotics services in a domestic environment.
From a technical point of view, the experiments demonstrate the technical feasibility of a cloud robotics solution to extend the interaction abilities of the robot.
In effect, the cloud robotics approach gives the possibility of increasing the skills of a robot in a modular way and endows the system with text-to-speech and speech recognition abilities for human interactions, smart environments for additional sensing, and access to internet resources.
Additionally, the experience with the couple of elderly users suggests a promising acceptance of the KuBo system. This will enourage us to perform future tests, promoting the use-case approach to a pilot site methodology, which will involve more users.
A video about the use case experiment is available at: https://youtu.be/uMjp8vN4MF8.  (2005)(2006); the course concluded with an internship in Piaggio Style Center, working on a concept motorcycle for Gilera. She is working at The BioRobotics Institute on several national and european projects and her main activity was focused on robots physical appearance with particular attention to the acceptability via an user-centered approach.

Paolo Dario received his Dr.
Eng. Degree in Mechanical Engineering from the University of Pisa, Italy, in 1977. He is currently a Professor of Biomedical Robotics at Scuola Superiore Sant'Anna (SSSA) in Pisa. He is and has been Visiting Professor at prestigious universities in Italy and abroad, like Brown University, Ecole Polytechnique Federale de Lausanne (EPFL), Waseda University, University of Tokyo, College de France, Zhejiang University. He was the founder and is currently the Coordinator of the BioRobotics Institute of Scuola Superiore Sant'Anna, where he supervises a team of about 120 researchers and Ph.D. students. He is the Director of Polo Sant'Anna Valdera, the research park of SSSA. His main research interests are in the fields of BioRobotics, medical robotics, micro/nanoengineering. He is the coordinator of many national and European projects, the editor of special issues and books on the subject of BioRobotics, and the author of more than 200 scientific papers.

Filippo Cavallo MScEE, Phd
in Bioengineering, is Assistant Professor at BioRobotics Institute, Scuola Superiore Sant'Anna, Pisa, Italy, focusing on cloud and social robotics, ambient assisted living, wireless and wearable sensor systems, biomedical processing, acceptability and AAL roadmapping. He participated in various National and European projects, being project manager of Robot-Era, AALIANCE2 and Parkinson Project, to name but a few. He was visiting researcher at the the EndoCAS Center of Excellence, Pisa; at the Takanishi Lab, Waseda University, Tokyo; at Tecnalia Research Center, Spain. He was granted from the International Symposium of Robotics Research Committee as Fellowship Winner for best PhD thesis in Robotics; from the Regional POR FSE 2007-2013 for a 3-years Research position at The BioRobotics Institute; from the ACCESS-IT 2009 for the Good Practice Label in Alzheimer Project; from the Well-Tech Award for Quality of Life with the Robot-Era Project . He is author of various papers on conferences and ICI journals.