---
abstract: 'This paper proposes an unsupervised representation model, based on rank-fusion graphs, for general applicability in multimodal tasks, either unsupervised or supervised prediction. Rank-fusion graphs encode information from multiple descriptors and retrieval models, thus being able to capture underlying relationships between modalities, samples, and the collection itself. By doing so, our method is able to promote a fusion model better than either early-fusion and late-fusion alternatives. The solution is based on the encoding of multiple ranks of a query, defined according to different criteria, into a graph. Later, we embed the generated graph into a feature space, creating fusion vectors. Those embeddings are employed to build an estimator that infers whether an input (even multimodal) object refers to a class (or event) or not. Performed experiments in the context of multiple multimodal and visual datasets, evaluated on several descriptors and retrieval models, demonstrate that our representation model is highly effective for different detection scenarios involving visual, textual, and multimodal features, yielding better detection results than state-of-the-art methods.'
address:
- 'Institute of Computing, University of Campinas (UNICAMP), Campinas, Brazil'
- 'Université de Lorraine-LORIA UMR 7503, Vandoeuvre-lès-Nancy, France'
- 'Department of ICT and Natural Sciences, Norwegian University of Science and Technology (NTNU), [Å]{}lesund, Norway'
author:
- Icaro Cavalcante Dourado
- Salvatore Tabbone
- Ricardo da Silva Torres
bibliography:
- 'bib.bib'
title: 'Multimodal Representation Model based on Graph-Based Rank Fusion'
---

multimodal fusion ,representation model ,graph-based rank fusion ,graph embedding ,rank aggregation ,prediction tasks

Introduction
============

Nowadays, data analysis involving multimedia and heterogeneous content is a hot topic that attracts a lot of attention from not only public and private sectors, but also from academia. The proliferation of digital content and social media is expanding substantially the volume and diversity of available digital content. For the most part, these data is unlabeled, heterogeneous, unstructured, and derived from multiple modalities.

Despite such challenges, such content is of great relevance to support the development of prediction and retrieval models. In particular, multimodal data analysis is required in several scenarios, such as , and of natural disasters such as flooding. Massive digital content has demanded the development of textual, visual, and multimedia descriptors for content-based data analysis. Despite the continuous advance on feature extractors and machine learning techniques, a single descriptor or a single modality is often insufficient to achieve effective prediction results in real case scenarios. Descriptors have specific pros and cons, because each one often focuses on a specific point of view of a single modality. For example, dedicated descriptors may be created to characterize scenes, textual descriptions, movement, symbols, signals, etc. For this reason, descriptors and retrieval models often provide complementary views, when adopted in combination.

Many research works have been proposed to combine heterogeneous data sources (remotely sensed information and social media) to promote multimodal analysis. In [@zhou:2017:CBIR], authors point out the benefit of exploring and fusing multimodal features with different models. Moreover, combining different kinds of features (local vs holistic) improves substantially the retrieval effectiveness [@zhang:2015:queryRankFusion], and most search fusion approaches are based on rank fusion [@dao:2018; @Dourado:2019:FG; @zhang:2015:queryRankFusion].

Scenarios involving heterogeneous data impose a challenge of selecting features or models to combine, which is performed by either supervised or unsupervised approaches. Unsupervised approaches are necessary in the absence of labeled data, which is prominent nowadays, or in scenarios involving lower computation capabilities or large amount of data.

Most previous initiatives for multimodal prediction are solely based on either CNN-based descriptors in isolation, feature concatenation [@dirsm17:BBischkeEtAl; @dirsm17:MSDaoEtAl; @dirsm17:XFuEtAl; @dirsm17:MHanifEtAl; @dirsm17:LLopezFuentesEtAl; @dirsm17:NTkachenkoEtAl], or graph-based feature-fusion [@dirsm17:KAvgerinakisEtAl; @werneck:2018:BocgBokg]. These approaches still ignore the correlation between modalities, as well as object correlation, and they are not consistently better than ranking models that do not rely on fusion.

This paper presents an unsupervised representation model, based on rank-fusion graphs, for general applicability in multimodal prediction tasks, such as classification and regression. We explore and extend the concept of a rank-fusion graph, that was originally proposed as part of a rank aggregation approach for retrieval tasks [@Dourado:2019:FG].

We present a fusion method based on the representation of multiple ranks, defined according to different criteria, into a graph. Graphs provide an efficient representation of arbitrary structures and inter-relationships among different elements of a model. We embed the generated graph into a feature space, creating fusion vectors. Next, an estimator is trained to predict if an input multimodal object refers to a target label (or event) or not, following their fusion vectors.

In [@Dourado:2019:FV], a methodology to apply rank-fusion graphs for efficient retrieval was presented, in the context of retrieval tasks. Here we explore those graph embedding approaches of rank-fusion graphs, now targeting a representation model for prediction tasks, either supervised or unsupervised. For this purpose, we propose specific components for the training and prediction phases, as well as a new application for the fusion vectors.

Our method has the advantage to be unsupervised, it also explores and captures relationships from the collection into the representation model, and it works on top of any descriptors for multimodal data such as visual or textual. By promoting a representation model solely based on base descriptors and unsupervised data analysis over the collection, we conjecture that our approach leads to a competitive multimodal representation model that explores and encodes information from multiple descriptors and underlying sample relationships automatically, while not requiring labeled data.

Experimental results over multiple multimodal and visual datasets demonstrate that the proposal is robust for different detection scenarios involving textual, visual, and multimodal features, yielding better detection results than state-of-the-art methods from both early-fusion and late-fusion approaches.

This paper extends the work presented in [@Dourado:2019:fusionGraphPrediction] where we introduced the notion of a representation model from rank-fusion graphs, and demonstrated its application for multimedia flood detection. Here we extend this formulation, proposing and discussing alternative approaches for the representation model. We also evaluate the method extensively over multiple multimodal and image scenarios, to analyze its applicability for prediction tasks in general. We also evaluate the method against additional baselines, either early-fusion and late-fusion approaches. Related work is presented and discussed in a more depth way as well. Finally, we evaluate the influence of hyper-parameters and provide guidelines for their selection.

Related Work
============

Representation learning models have been developed and advanced for the data modalities individually, such as image, text, video, and audio. However, their combined exploration for multimodal tasks is still an open issue. Multimodality may also impose even more challenges, depending on the scenario, such as translation between modalities, exploration of complementarity and redundancy, co-learning, and semantic alignment [@baltruvsaitis:2018:multimodal].

Some works have focused on multimodal events, which require modeling spatio-temporal characteristics of data [@tzelepis2016event]. @faria2016fusion proposed a time-series descriptor that generates recurrence plots for the series, coupled with a bio-inspired optimization of how to combine classifiers. In this paper, we focus on multimodal tasks that do not depend on temporal modeling as well as unsupervised models.

In order to achieve fusion capabilities, *early-fusion* approaches emphasize the generation of composite descriptions for samples, thus working at feature level. Conversely, *late-fusion* approaches perform a combination of techniques focused on a target problem, fusing at score or decision level. On a smaller scale, some papers propose hybrid solutions based on both approaches [@zadeh2017tensor; @lan2014multimedia]. As we propose a representation model, our solution can be seen as an early-fusion approach. Nevertheless, it is based on retrieval models, without the need to work directly on feature level. In this sense, we categorize the method as hybrid.

While early-fusion approaches are theoretically able to capture correlations between modalities, often a certain modality produces unsatisfactory performance and leads to biased or over-fitted models [@zadeh2017tensor]. Most early-fusion methods work in a two-step procedure, first extracting features from different modalities, then fusing them by strategies such as concatenation [@kiela2014learning], singular values decomposition [@bruni2014multimodal] or autoencoders [@silberer2014learning]. A few others focus on multimodal features jointly [@kottur2016visual; @hill2014learning], although generally restricted to a pre-defined textual attribute set. Concatenation is a straightforward yet widely used early-fusion approach, which merges vectors obtained by different descriptors. As a drawback, concatenation does not explore inherent correlations between modalities.

Supervised early-fusion optimizes a weighted feature combination, either during or after feature extraction. A common strategy is to build a neural architecture with multiple separate input layers, then including a final supervised layer such as a regressor [@nogueira2018exploiting]. Another approach is the design of a composite loss function, suited for the particular desired task [@park2016image]. Composite loss functions work well in practice, but, as they need both multimodal composition and supervision, they are tied to the domain of interest. Supervised early-fusion usually suffers from high memory and time consumption costs. Besides, they usually have difficulty in preserving feature-based similarities and semantic correlations [@lan2014multimedia].

Late-fusion approaches are particularly useful when the raw data from the objects are not available. Besides, they are less prone to over-fit. Mixture of experts (MoE) approaches focus on performing decision fusion, combining predictors to address a supervised learning problem [@yuksel2012twenty]. Majority voting of classifiers [@nogueira2018exploiting], rank aggregation functions [@Dourado:2019:FG] and matrix factorization [@dong2018late] are examples of late-fusion methods. Both fusions based on rank aggregation functions and matrix factorization are based on manifold learning, i.e. the exploration of dataset geometry.

Majority voting is a well-known approach to combine multiple estimators, being effective due to bias reduction. It is applied to scenarios involving an odd number of estimators, so that each predicted output is taken as the one most frequently predicted by the base estimators.

Rank aggregation functions combine results from different retrieval models. They target a good permutation of retrieved objects obtained from different input ranks, in order to promote more effective retrieval results [@Dourado:2019:FG].

In general, unsupervised learning needs more investigation, specially for multimodal representation models. We explore how unsupervised rank aggregation capabilities can be applied to prediction tasks. Despite their original intent regarding better retrieval accuracy, we claim that unsupervised rank aggregation functions can provide an effective dataset exploitation.

Considering previous works regarding fusion approaches for prediction tasks, a considerable amount of them are still based on classic visual descriptors [@dirsm17:MSDaoEtAl; @dirsm17:MHanifEtAl; @dirsm17:NTkachenkoEtAl; @werneck:2018:BocgBokg]. Most of them resorted to pre-trained CNN-based models for visual feature extraction [@dirsm17:KAhmadEtAl; @dirsm17:SAhmadEtAl; @dirsm17:KAvgerinakisEtAl; @dirsm17:BBischkeEtAl; @dirsm17:XFuEtAl; @dirsm17:LLopezFuentesEtAl; @dirsm17:KNogueiraEtAl], from which just a few fine-tuned their models [@dirsm17:LLopezFuentesEtAl; @dirsm17:KNogueiraEtAl]. When dealing with specific tasks, specially for competitions, some works use preprocessing steps, such as image cropping and filtering [@dirsm17:ZZhao], but this is beyond our general intent in this paper, regarding fusion methods and representation models.

In order to explore the textual modality, most initiatives used BoW, using either TF or TF-IDF weighting, while others presented more complex formulations, such as word embeddings [@dirsm17:BBischkeEtAl; @dirsm17:NTkachenkoEtAl], Long Short-Term Memory (LSTM) networks [@dirsm17:LLopezFuentesEtAl], or relation networks [@dirsm17:KNogueiraEtAl].

Regarding multimodal scenarios, most works rely on early-fusion approaches, such as a concatenation of visual and textual feature vectors [@dirsm17:BBischkeEtAl; @dirsm17:MSDaoEtAl; @dirsm17:XFuEtAl; @dirsm17:MHanifEtAl; @dirsm17:LLopezFuentesEtAl; @dirsm17:NTkachenkoEtAl] or graph-based attribute fusion [@dirsm17:KAvgerinakisEtAl; @werneck:2018:BocgBokg], while only a few others adopted late-fusion approaches [@dirsm17:KAhmadEtAl; @dirsm17:SAhmadEtAl].

Preliminaries {#secPrelim}
=============

Let a *sample* $s$ be any digital object, such as a document, an image, a video, or even a hybrid (multimodal) object. A sample is characterized by a *descriptor* ${\mathcal{D}}$, which relies on a particular point of view to describe $s$ as a vector, graph or another data structure $\epsilon(s)$. Descriptions allow samples to be compared to each other. Therefore, descriptors are the basis for retrieval and learning models.

A *comparator* ${\mathcal{C}}$, applied over a tuple $(\epsilon(s_i),\epsilon(s_j))$, produces a *score* ${\varsigma}\in \mathbb{R^+}$ (e.g., the Euclidean distance or the cosine similarity). Either similarity or dissimilarity functions can be used to implement ${\mathcal{C}}$. A query sample, or just *query* $q$, refers to a particular sample taken as an input object in the context of a search, whose purpose is to retrieve *response items* from a *response set* ($S$) according to relevance criteria. A response set ${S = \{ s_i\}}_{i=1}^n$ is a collection of $n$ *samples*, where $n$ is the collection size. In the context of prediction tasks, a query is called *test sample*, and a response set is called *train set*. These terms can be used interchangeably, but a train set is demanded to contain labeled samples.

A *ranker* is a tuple ${R}= ({\mathcal{D}},{\mathcal{C}})$, which is employed to compute a rank ${\tau}$ for $q$, denoted by ${{\tau}_{q}}$ to distinguish its corresponding query. A rank is a permutation of $S_L \subseteq S$, where $L \ll n$ in general, such that ${{\tau}_{q}}$ provides the most similar – or equivalently the least dissimilar – samples to $q$ from $S$, in order. $L$ is a cut-off parameter. A ranker establishes a ranking system, but different descriptors and comparators can compose rankers. Besides, descriptors are commonly complementary, as well as comparators. For $m$ rankers, $\{R_j\}_{j=1}^m$, used for query retrieval over a collection $S$, for every query $q$ we can obtain ${\mathcal{T}_{q}}= \{{{\tau}_{j}}\}_{j=1}^m$, from which a *rank aggregation* function $f$ produces a combined rank ${{\tau}_{q,f}}=f({\mathcal{T}_{q}})$, presumably more effective than the individual ranks ${{\tau}_{1}}$, ${{\tau}_{2}}$, $\dots$, ${{\tau}_{m}}$.

Representation and Prediction Based on Graph-Based Rank Fusion {#secMethod}
==============================================================

Figure \[fig:fusionVectorEstimation\] presents an overview of our method – a multimodal representation and estimator based on rank-fusion graphs. The solution is composed of three main generic components, briefly described here and detailed in the following sections. The multimodal representation is completely unsupervised, thus able to be adopted in any tasks in the absence of labeled data.

Two phases are defined. The [*training*]{} phase comprehends the modeling of the train set in terms of multiple rankers, as fusion graphs and then as fusion vectors. This step performs a graph embedding of fusion graphs and the training of a multimodal prediction model. The [*inference*]{} phase refers to the multimodal prediction preceded by a rank-based fusion approach for multimodal representation. The training phase is performed only once, while the inference phase is performed per prediction. The first two components – [*fusion graph extraction*]{} and [*graph embedding*]{} – are used in both phases.

![Proposed graph-based rank fusion for multimodal prediction.[]{data-label="fig:fusionVectorEstimation"}](fusionVectorEstimation){width="0.95\linewidth"}

The [*fusion graph extraction*]{} (component 1 in the figure) generates a fusion graph ${\mathcal{G}}$ for a given query sample $q$. ${\mathcal{G}}$ consists of an aggregated representation of multiple ranks for $q$, thus capturing and correlating information of multiple ranks. This formulation is presented in Section \[sec:fusionGraphExtraction\]. [*Graph Embedding*]{} (2) projects fusion graphs into a vector space model, producing a corresponding fusion vector ${\mathcal{V}}$ for ${\mathcal{G}}$. We propose an embedding formulation in Section \[sec:fusionGraphEmbedding\]. At the end, an [*event predictor*]{} (3) is built based on the response fusion vectors, in order to predict for queries (also modeled as fusion vectors). This component is detailed in Section \[sec:fusionVectorSearch\].

Rank-Fusion Graphs {#sec:fusionGraphExtraction}
------------------

This component produces a fusion graph ${\mathcal{G}}$ for a given query sample $q$, also in terms of $m$ rankers and $n$ response items. A fusion graph is a graph-based encoding of multiple ranks for $q$, that encapsulates and correlates ranks.

We follow the fusion graph formulation from [@Dourado:2019:FG], referred to as *FG*, that defines a mapping function $q \mapsto {\mathcal{G}}$, based on its ranks ${\tau}\in {\mathcal{T}_{q}}$ and ranks’ inter-relationships. The proposed formulation also includes a dissimilarity function for *FG*, and a retrieval model based on fusion graphs. Here, however, we focus on the definition of *FG*, extending its use as part of a rank-based late-fusion approach for representation model in prediction tasks, without these components.

The process is illustrated in Figure \[fig:graphFusionFromSample\]. Given a query $q$, $m$ rankers, and a training set of size $n$, $m$ ranks are generated. These ranks are then normalized to allow for producing the fusion graph ${\mathcal{G}}$ for $q$. In rank normalization, the scores in the ranks generated by dissimilarity-based comparators are converted to similarity-based scores. Besides, all ranks have their scores rescaled to the same interval. ${\mathcal{G}}$, for $q$, includes all response items from each ${{\tau}_{q}} \in {\mathcal{T}_{q}}$, as vertices. Vertices are connected by taking into account the degree of relationship between their corresponding response items, and the degree of their relationships to $q$.

![Extraction of a fusion graph (adapted from [@Dourado:2019:FG]).[]{data-label="fig:graphFusionFromSample"}](graphFusionFromSample2){width=".8\linewidth"}

Embedding of Rank-Fusion Graphs {#sec:fusionGraphEmbedding}
-------------------------------

Let ${\mathbb{G}}=\{{\mathcal{G}}_i\}_{i=1}^n$ be the fusion graph set related to the response set of a given collection. Based on ${\mathbb{G}}$, an embedding function ${\mathcal{E}}$ defines a vector space model in order to project a fusion graph ${\mathcal{G}}$ into that space as a fusion vector ${\mathcal{V}}$, i.e. ${\mathcal{V}}={\mathcal{E}}({\mathcal{G}})$ for any ${\mathcal{G}}$.

We investigate how ${\mathcal{V}}$ can be adopted as a representation model of multi-ranked objects. It encodes the use of multiple rankers and allows the fusion of multiple modalities, being therefore suitable for prediction tasks. ${\mathcal{E}}$ can be defined by unsupervised or supervised approaches. We explore three approaches in this paper, preliminarly presented in [@Dourado:2019:FV] in the context of retrieval tasks. Different from that paper, here we focus its use on prediction tasks involving supervised learning.

This first one, ${\mathcal{E}}_V$, derives the vector space based on vertex analysis. Let $w(v)$ be the weight of the vertex $v$, if $v \in {\mathcal{G}}$, otherwise 0. Similarly, let $w(e)$ be the weight of the edge $e$, if $e \in {\mathcal{G}}$, otherwise 0. Also, let $N$ be the dimensionality of the vector space model defined by ${\mathcal{E}}$, such that ${\mathcal{V}}\in {\rm I\!R}^N$. ${\mathcal{E}}_V$ derives ${\mathcal{V}}$ from the vertices of ${\mathcal{G}}$. There is one vector attribute relative to each response object, therefore $N=|{\mathbb{G}}|$. ${\mathcal{V}}$ is derived from ${\mathcal{G}}$ such that $|{\mathcal{V}}|=N$, ${\mathcal{V}}[i]$ is the importance value of the $i$-th attribute, and ${\mathcal{V}}[i]=w(v_i)$. Despite the vector space increases linearly to the collection size, the resulting fusion vectors are mainly sparse, i.e., composed of few non-zero entries, which makes this embedding formulation simple and efficient in practice.

${\mathcal{E}}_H$ is a hybrid embedding approach based on both vertex and edge analysis. ${\mathcal{E}}_H$ encodes more information into the vector space, at a cost of a higher dimensionality.

The third approach, ${\mathcal{E}}_K$, extends the archetype to embed graphs as a histogram of kernels, where the vectorial attributes are selected by unsupervised selection of common subgraph patterns. The kernels are obtained from the centroids of a graph clustering process. Then, a vector quantization process, consisting of assignment and pooling procedures, is adopted to embed an input graph to the vector space. We adopt SOFT assignment and AVG pooling, as suggested by the authors [@Dourado:2019:FV]. has been successfully applied in scenarios involving graph classification, textual representation and information retrieval [@Silva:2018:BoG; @Dourado:2019:BoTG; @Dourado:2019:FV]. To the best of our knowledge, this is the first extension of in the context of multimodal representation.

We refer to FV-V as the fusion vector generated by ${\mathcal{E}}_V$, while FV-H is generated by ${\mathcal{E}}_H$, and FV-K by ${\mathcal{E}}_K$.

Prediction based on Fusion Vectors {#sec:fusionVectorSearch}
----------------------------------

Fusion vectors allow the creation of predictors, such as classifiers or regressors, and also ad-hoc retrieval systems, depending on the underlying demanded task. In this work, we adopt them to build predictors, where training objects – associated with ground-truth information – are used to train an estimator for a certain input object be considered a label (or event) or not.

Let $S$ be a training corpus of size $n$. A predictor can be modeled as $f (X, \beta) \approx Y$, where $f$ is an approximation function, $X$ are the independent variables, $Y$ is the dependent variable (target), and $\beta$ are unknown parameters. A learning model explores $S$ to find a $f$ that minimizes a certain error metric. The training samples are generally labeled, so $Y$ may be categorical. Still, a regressor can be built, as $E(Y|X)=f(X, \beta)$, so that posterior probabilities are inferred in order to estimate a confidence of a sample to refer to a class of not. $X$, in our case, refers to the fusion vectors, acting as variables that describes the samples in terms of their multiple multimodal ranks.

Experimental Evaluation {#secExpEval}
=======================

We present, in this section, the experimental protocol used to evaluate the method, and the results achieved comparatively to state-of-the-art baselines. We evaluate the effectiveness of our method as a representation model in prediction tasks. The focus is on validating our fusion method comparatively to the individual use of descriptors with no fusion, as well as to compare it to early-fusion and late-fusion approaches.

Evaluation Scenarios
--------------------

We evaluate the proposed method on multiple datasets, comprised of heterogeneous multimodal data, in order to assess its general applicability. Our experimental evaluation comprises the following scenarios:

-   **ME17-DIRSM** dataset, the acronym for MediaEval 2017 Disaster Image Retrieval from Social Media [@bischke:2017:meSatellite], is a multimodal dataset of a competition whose goal is to infer whether images and/or texts refer to flood events or not. The samples contain images along with textual metadata, such as title, description, and tags, and they are labeled as either flood (1) or non-flood (0). The task predefines a development set (devset) of samples, and a test set of samples, as well as its own evaluation protocol.

-   **Brodatz** [@brodatz:1966] is a dataset of texture images, labeled across $111$ classes. There are $16$ samples per class, composing a total of samples.

-   **Soccer** [@van:2006:soccer] is an image dataset, labeled across 7 categories (soccer teams), containing $40$ images each.

-   **UW** [@Deselaers:2008:UW], also called University of Washington dataset, is a multimodal collection of images annotated by textual keywords. The images are pictures labeled across $22$ classes (locations). Pictures per class vary from 22 to 255. The number of keywords per picture vary from 1 to 22.

Evaluation Protocol
-------------------

For any dataset that does not explicitly define train and test sets, we initially split it in train and test sets, at a proportion of 80% and 20% respectively, in a stratified way so that the proportions per class remain equal. The same train and test sets per dataset are adopted to evaluate all methods under the same circumstances, as well as the evaluation metrics.

For each representation model, we fit a multiclass SVM classifier, with one-vs-all approach and linear kernel, as it is a good fit for general applicability. Hyper-parameters are selected by grid search on the train set, using an internal 5-fold cross validation.

We evaluate the effectiveness of each method by the balanced accuracy score, which is suitable to evaluate on either balanced and imbalanced datasets. The methods are compared by their balanced accuracy.

The descriptors compose rankers, which are employed to generate ranks in our late-fusion representation. Our method varies with respect to which rankers are used, whether visual or textual rankers, or even their combinations for the multimodal scenario are applied. Besides that, it also varies with respect to which embedding approach is adopted. We evaluate these aspects experimentally.

We model our solution as a rank-fusion approach, followed by an estimator based on rank-fusion vectors. This approach intends to validate our hypothesis that unsupervised graph-based rank-fusion approaches can lead to effective representation models for prediction tasks in general.

We adopt the same experimental evaluation for all datasets but ME17-DIRSM, that defines its own procedure. In this case, the task imposes three evaluation scenarios, as follows. In the first one, called “visual”, only visual data can be used. In the second scenario, called “textual”, only textual data are used. In the third scenario, called “multimodal”, both visual and textual data are expected to be used. The correctness is evaluated, over the test set, by the metric Average Precision at $K$ (AP@K) at various cutoffs (50, 100, 250, 480), and by their mean value (mAP).

Although the ME17-DIRSM task may be seen as a multimodal binary classification problem, the evaluation metrics require ranking-based solutions, or equivalently confidence-level regressors, so that the first positions are the most likely to refer to a flood event. For the estimator component in ME17-DIRSM, we adopt SVR, an L2-regularized logistic regression based on linear SVM in its dual form, with probabilistic output scores, and trained over the fusion vectors from devset. Probabilistic scores are used so that we can sort the test samples by confidence expectancy of being flood.

In ME17-DIRSM, our results are compared to those from state-of-the-art baselines. In Soccer, Brodatz, and UW, our results are compared to those from two major fusion approaches: concatenation, and majority vote. They cover baselines from both early-fusion and late-fusion families. For the concatenation procedure, besides the concatenation itself, we normalize the vectors to the $[0,1]$ interval for each attribute, in order to avoid disparities due to different descriptor attribute ranges. We apply majority voting in the scenarios involving an odd number of descriptors, so that each predicted class is taken as the one most frequently predicted by the estimators constructed for each descriptor.

Descriptors and Rankers
-----------------------

For ME17-DIRSM, we selected three visual descriptors and three textual descriptors, for individual analysis in the designed evaluation scenarios, and to evaluate different possibilities of rank-fusion aggregations. We adopt the following state-of-the-art visual descriptors:

-   *ResNet50IN*: 2048-dimensional average pooling of the last convolutional layer of ResNet50 [@he:2016:ResNet50], pre-trained on ImageNet [@Russakovsky:2015:ImageNet], a dataset of about 14M images labeled for object recognition;

-   *VGG16P365*: 512-dimensional average pooling of the last convolutional layer of VGG16 [@simonyan:2014:VGG16], pre-trained on Places365-Standard [@zhou:2017:Places], a dataset of about 10M images of labeled scenes;

-   *NASNetIN*: 2048-dimensional average pooling of the last convolutional layer of NASNet [@barret:2017:NASNet], pre-trained on ImageNet dataset.

Based on the textual metadata in ME17-DIRSM, we adopt the following descriptors:

-   *BoW*: Bag of Words (BoW) with Term Frequency (TF) weighting;

-   *2grams*: 2grams with TF weighting;

-   *doc2vecWiki*: 300-dimensional doc2vec [@le:2014:doc2vec] pre-trained on English Wikipedia dataset, of about 35M documents and dumped at 2015-12-01.

For the other datasets, we elected a number of heterogeneous descriptors:

-   Soccer: , , and .

-   Brodatz: , , and .

-   UW: , , , and , as visual descriptors, and *word2vecSum*, *word2vecAvg*, *doc2vecWiki*, and *doc2vecApnews*, as textual descriptors.

For the deep networks used for CNN-based visual feature extraction, as well as in the textual feature extraction with doc2vec, we take advantage of pre-trained models. This practice, known as transfer learning, has been effective in many scenarios [@kornblith:2018:imagenetModelTransfer], and it is also particularly beneficial for datasets that are not large enough to generalize the training of such large architectures, as in our case. Because the problem requires prediction of flood images, we prioritize, in the selection of visual descriptors, datasets for pre-training that focus on images of scenes, aiming at better generality to the target problem.

We perform the same preprocessing steps for every textual descriptor: lower case conversion, digit and punctuation removal, and English stop word removal. For *BoW* and *2grams*, we also apply Porter stemming.

The *word2vecSum* descriptor produces, for any input document, a vector corresponding to the sum of the word embedding vectors [@mikolov:2013:word2vec] related to each term within that document, while *word2vecAvg* computes the mean vector of them.

The doc2vec model promotes document-level embeddings for texts, and it is based on word embeddings [@mikolov:2013:word2vec], a preliminary work that assigns vector representations for words in order to capture their semantic relationships. *doc2vecApnews* stands for a 300-dimensional *doc2vec* [@le:2014:doc2vec] model, pre-trained over the Associated Press News textual dataset, of about 25M news articles from 2009 to 2015.

From the descriptors, rankers are defined as tuples of (descriptor, comparator), where the comparator corresponds to a dissimilarity function. We compose a ranker for each descriptor by choosing an appropriate comparator. Given that our method works on top of rankers, we have to define dissimilarity functions to be used along with those descriptors that is not explicitly associated with one. This is the case for the four textual descriptors adopted in UW, as well as the descriptors adopted in ME17-DIRSM. All remaining descriptors define their own comparators.

For the textual descriptors BoW and 2grams, we adopt the Weighted Jaccard distance, defined as $1 - J(\mathbf{u}, \mathbf{v})$, where $J$ is the Ruzicka similarity metric (Equation \[eq:Jaccard\]). Jaccard is a well-known and widely-used comparison metric for classic textual descriptors, specially for short texts, as in our case. For the remaining descriptors, we choose the Pearson correlation distance, defined as $1 - \rho(\mathbf{u}, \mathbf{v})$ (Equation \[eq:correlation\]), which is a general-purpose metric due to its suitability for highly dimensional data and scale invariance. $$\label{eq:Jaccard}
J(\mathbf{u},\mathbf{v}) = \frac{\sum_i \min(u_i,v_i)}{\sum_i \max(u_i,v_i)}$$ $$\label{eq:correlation}
\rho(\mathbf{u}, \mathbf{v}) = \frac{(\mathbf{u} - \bar{u}) \cdot (\mathbf{v} - \bar{v})}
{{\|(\mathbf{u} - \bar{u})\|}_2 {\|(\mathbf{v} - \bar{v})\|}_2}$$

The datasets, descriptors, and evaluation criteria, are summarized in Table \[tab:datasets\].

Fusion Setups
-------------

For both visual and textual scenarios in ME17-DIRSM, we analyze three variants of our method with respect to the input rankers for late-fusion. For the visual scenario, the combinations are ResNet50IN + NASNetIN, ResNet50IN + VGG16P365, and ResNet50IN + NASNetIN + VGG16P365. For the textual scenario, the combinations are BoW + 2grams, BoW + doc2vecWiki, and BoW + 2grams + doc2vecWiki.

As for the multimodal scenario, we investigate some combinations taking one ranker of each type, two of each, and three of each. Six multimodal combinations are evaluated: ResNet50IN + BoW, ResNet50IN + NASNetIN + BoW + 2grams, ResNet50IN + NASNetIN + BoW + doc2vecWiki, ResNet50IN + VGG16P365 + BoW + 2grams, ResNet50IN + VGG16P365 + BoW + doc2vecWiki, and ResNet50IN + NASNetIN + VGG16P365 + BoW + 2grams + doc2vecWiki.

We report three results for the adoption of *FV*, in its different embedding approaches, as a representation model for prediction tasks, in Soccer, Brodatz, and UW. We report the results for multiple descriptor combinations, in order to analyze: (i) the method against baselines, (ii) the embedding approaches, and (iii) the comparative effectiveness between the descriptor combinations. The descriptor combinations selected, although not exhaustive, are targeted for a large number of scenarios. In Soccer and Brodatz, all possible combinations were selected for evaluation. In UW, several visual combinations and multimodal combinations were selected. The descriptor combinations are:

-   In Soccer: ACC + BIC, BIC + GCH, ACC + GCH, and ACC + BIC + GCH.

-   In Brodatz: CCOM + FCTH, CCOM + JCD, FCTH + JCD, and CCOM + FCTH + JCD.

-   In UW: ACC + CEDD, ACC + JCD, CEDD + JAC, CEDD + JCD, ACC + CEDD + JCD, and CEDD + JAC + JCD, for visual fusion, and ACC + doc2vecApnews, JCD + doc2vecApnews, ACC + JCD + doc2vecApnews, ACC + JCD + doc2vecWiki, ACC + JCD + word2vecAvg, and ACC + JCD+word2vecSum, for multimodal fusion.

Results and Discussion
----------------------

### Base Results

Here we report results by the use of individual descriptors. They constitute an initial baseline for our method as well as for other fusion approaches.

We report, in Tables \[tab:resultsVisual\] and \[tab:resultsTextual\], the results for the visual and textual scenarios in ME17-DIRSM, achieved by the three visual and textual selected descriptors, along with a SVR regressor. As the task only mentioned AP@480 and mAP in their leaderboard, we focus our discussions on these two metrics. The correctness for the visual scenario is already high within these baselines, around $85\%$ in AP@480. In the textual scenario, AP@480 is around $65\%$, which suggests more room for improvement.

We report, in Tables \[tab:descriptorResultsSoccer\], \[tab:descriptorResultsBrodatz\], \[tab:descriptorResultsUWvisual\], and \[tab:descriptorResultsUWtextual\], the results obtained in Soccer, Brodatz, UW (visual), and UW (textual), respectively, by the use of the descriptors along with a SVM classifier.

[.49]{}

\[tab:resultsVisual\]

[.49]{}

\[tab:resultsTextual\]

[0.49]{}

  Descriptor     Balanced acc.
  ------------ ---------------
  BIC                    60.84
  GCH                    56.38
  ACC                    51.79

  : UW, textual[]{data-label="tab:descriptorResultsUWtextual"}

  Descriptor     Balanced acc.
  ------------ ---------------
  JCD                    78.65
  FCTH                   75.10
  CCOM                   74.65

  : UW, textual[]{data-label="tab:descriptorResultsUWtextual"}

[0.49]{}

  Descriptor     Balanced acc.
  ------------ ---------------
  JAC                    83.74
  ACC                    77.06
  JCD                    73.99
  CEDD                   72.46

  : UW, textual[]{data-label="tab:descriptorResultsUWtextual"}

### Parameter Analysis

The resulting size of ${\mathcal{G}}$ is affected by the input rank sizes, defined by the hyper-parameter *L*. For the same reason, larger *FG*’s either increase the vocabulary sizes of *FV* or the complexity to generate them. @Dourado:2019:FG showed that an increase in *L* leads to more discriminate graphs up to a saturation point. A practical upper bound for the choice of *L* tends to be the maximum rank size of users’ interest, indirectly expressed here by the evaluation metrics.

As ME17-DIRSM defines evaluation metrics for ranks up to 480, we start by empirically evaluating the influence of *L* in the mAP score, for values to up 480. Figure \[fig:parameterEvaluationL\] reports the influence of L for some of the elected fusion scenarios. The results were as expected: the effectiveness usually increased as *L* was larger. For the next evaluation scenarios in ME17-DIRSM, we adopt $L=480$. For the other datasets, we adopt $L=10$.

![Effect of the rank size limit (*L*) for the fusion graph extraction, in the mAP score, for different fusion scenarios in ME17-DIRSM.[]{data-label="fig:parameterEvaluationL"}](parameterAnalysisL){width="\linewidth"}

### Fusion Results in Event Detection

We present our results achieved for the three scenarios in ME17-DIRSM, using the combinations proposed, along with the results of the $11$ teams that participated in the competition. We also show the results achieved in [@werneck:2018:BocgBokg] in the visual and multimodal scenarios, which relied on early-fusion techniques. These results are presented in Tables \[tab:resultsRunVisual\], \[tab:resultsRunTextual\], and \[tab:resultsRunMultimodal\], respectively for the visual, textual, and multimodal scenarios. In ME17-DIRSM, we focused on the ${\mathcal{E}}_V$ embedding approach. For the other datasets, we evaluate all of them.

In the visual scenario, only [@dirsm17:SAhmadEtAl; @dirsm17:BBischkeEtAl] performed better, in terms of AP@480 and mAP, than our preliminary base setup, based on individual descriptors along with the SVR regressor. As for the textual scenario, only [@dirsm17:NTkachenkoEtAl] in 12 initiatives surpassed BoW + SVR in AP@480, and [@dirsm17:BBischkeEtAl; @dirsm17:KNogueiraEtAl; @dirsm17:ZZhao] in mAP. This indicates that descriptors properly selected to the target problem can overcome more complex models, also requiring less effort.

Our method was superior in the visual scenario to all baselines, for two of three proposed variants of ranker combinations. Compared to the strongest baselines considering this scenario, our method presents gains from around 1 to 2% in AP@480, and 1% in mAP. Compared to the visual base results, from the individual descriptors, 3 to 4% in AP@480, and 1 to 2% in mAP.

In the textual scenario, our gains were even more expressive. It was superior in the textual scenario to all related works, for all three proposed variants of ranker combinations. Compared to the strongest baselines, our method presents gains from 5 to 7% in AP@480, and 6 to 11% in mAP. Compared to the textual base results, from the individual descriptors, 6 to 8% in AP@480, and 14 to 16% in mAP. In the multimodal scenario, considered baselines were more competitive. Again, however, our method presents gains over them, of 0.5% in AP@480 and 0.23% in mAP.

### Fusion Results in Classification Tasks

We report in Tables \[tab:fusionResultsSoccer\], \[tab:fusionResultsBrodatz\], \[tab:fusionResultsUWvisual\], and \[tab:fusionResultsUWmultimodal\], the results obtained by our method variants, respectively in Soccer, Brodatz, UW for image data, and UW for multimodal data, besides the results obtained by the baselines.

Our method led to significant gains when compared to the best base result from the descriptors in each dataset: around 8 p.p. in Soccer, 13.2 p.p. in Brodatz, 2.2 p.p. in UW for visual fusion, and 5.9 p.p. in UW for multimodal fusion. The gains in UW were comparatively lower than others, yet consistent, because the base results were already higher, so that there were less room for improvement.

Our method, when compared to the best baseline in each descriptor combination in each dataset, had gains in all cases: up to 12.4 p.p. in Soccer, up to 2.9 p.p. in Brodatz, up to 7.1 p.p. in UW for visual fusion, and up to 3.9 p.p. in UW for multimodal fusion. Overall, all the FV approaches performed better than all baselines in all datasets. In only 6 of 20 descriptor combinations evaluated, any of the baselines surpassed any of the FV approaches.

The accuracy disparities, obtained by FV or any other fusion method, across the multiple descriptor combinations in each dataset, show that there is no obvious choice when dealing with which descriptors to be used together. As our representation model is meant to be unsupervised, this choice could only be guided by general heuristics, such as a selection of effective and low correlated descriptors, as discussed in [@Dourado:2019:FG]. We leave this exploration for future work.

FV-K usually performed better than FV-H and FV-V, and FV-H usually performed better than FV-V, although in both cases the gains were at most 3 p.p one over the other. On the opposite side, FV-V is the simplest among the three, and FV-K requires more computational steps. These two aspects combined impose that the practical choice among the three must take into account the trade-off between accuracy vs computational cost. In any case, our method is unsupervised and feasible for general applicability.

Conclusions {#secConc}
===========

In this paper, we presented an unsupervised graph-based rank-fusion approach as a representation model for multimodal prediction tasks. Our solution is based on encoding multiple ranks into a graph representation, which is later embedded into a vectorial representation. Next, an estimator is built to predict if an input multimodal object refers to a target event or not, given their graph-based fusion representations.

The proposed method extends the fusion graphs – first introduced in [@Dourado:2019:FG] – for supervised learning tasks. It also applies a graph embedding mechanism in order to obtain the fusions vectors, a late-fusion vector representation that encodes multiple ranks and their inter-relationships automatically.

Performed experiments in multiple prediction tasks, such as flood detection and multimodal classification, demonstrate that our solution leads to highly effective results overcoming state-of-the-art solutions from both early-fusion and late-fusion underlying approaches.

Future work will focus on investigating the impact of semi-supervised and supervised approaches for the fusion graph and fusion vector constructions. We also plan to investigate the use of our solution in other multimodal problems, such as recommendation and hierarchical clustering. Finally, we plan to evaluate the proposed approach for other multimedia data, such as audio and video.

References {#references .unnumbered}
==========
