Multi-modal data is becoming more common in big data background. Finding the semantically similar objects from different modality is one of the heart problems of multi-modal learning. Most of the current methods try to learn the inter-modal correlation with extrinsic supervised information, while intrinsic structural information of each modality is neglected. The performance of these methods heavily depends on the richness of training samples. However, obtaining the multi-modal training samples is still a labor and cost intensive work. In this paper, we bring a extrinsic correlation between the space structures of each modalities in coreference resolution. With this correlation, a semi-supervised learning model for multi-modal coreference resolution is proposed. We firstly extract high-level features of images and text, then compute the distances of each object from some reference points to build the space structure of each modality. With a shared reference point set, the space structures of each modality are correlated. We employ the correlation to build a commonly shared space that the semantic distance between multi-modal objects can be computed directly. The experiments on two multi-modal datasets show that our model performs better than the existing methods with insufficient training data.
Besides these methods above, there still are other models proposed for multi-modal problems. @cite_24 employed voxel-based multi-modal partial least square(PLS) to analyze the correlations between FDG PET glucose uptake-MRI gray matter volume scores and apolipoprotein E epsilon 4 gene dose in cognitively normal adults.
@cite_24
2004564253 2000292092 2005743500 2273249530
Abstract We previously introduced a voxel-based, multi-modal application of the partial least square algorithm (MMPLS) to characterize the linkage between patterns in a person's complementary complex datasets without the need to correct for multiple regional comparisons. Here we used it to demonstrate a strong correlation between MMPLS scores to characterize the linkage between the covarying patterns of fluorodeoxyglucose positron emission tomography (FDG PET) measurements of regional glucose metabolism and magnetic resonance imaging (MRI) measurements of regional gray matter associated with apolipoprotein E (APOE) e4 gene dose (i.e., three levels of genetic risk for late-onset Alzheimer's disease (AD)) in cognitively normal, late-middle-aged persons. Coregistered and spatially normalized FDG PET and MRI images from 70 of the subjects (27 e4 homozygotes, 36 e4 heterozygotes and 67 e4 non-carriers) were used in a hypothesis-generating MMPLS analysis to characterize the covarying pattern of regional gray matter volume and cerebral glucose metabolism most strongly correlated with APOE-e4 gene dose. Coregistered and spatially normalized FDG PET and MRI images from the remaining 30 of the subjects were used in a hypothesis-testing MMPLS analysis to generate FDG PET-MRI gray matter MMPLS scores blind to their APOE genotype and characterize their relationship to APOE-e4 gene dose. The hypothesis-generating analysis revealed covarying regional gray matter volume and cerebral glucose metabolism patterns that resembled those in traditional univariate analyses of AD and APOE-e4 gene dose and PET-MRI scores that were strongly correlated with APOE-e4 gene dose (p Many machine learning and pattern classification methods have been applied to the diagnosis of Alzheimer’s disease (AD) and its prodromal stage, i.e., mild cognitive impairment (MCI). Recently, rather than predicting categorical variables as in classification, several pattern regression methods have also been used to estimate continuous clinical variables from brain images. However, most existing regression methods focus on estimating multiple clinical variables separately and thus cannot utilize the intrinsic useful correlation information among different clinical variables. On the other hand, in those regression methods, only a single modality of data (usually only the structural MRI) is often used, without considering the complementary information that can be provided by different modalities. In this paper, we propose a general methodology, namely Multi-Modal Multi-Task (M3T) learning, to jointly predict multiple variables from multi-modal data. Here, the variables include not only the clinical variables used for regression but also the categorical variable used for classification, with different tasks corresponding to prediction of different variables. Specifically, our method contains two key components, i.e., (1) a multi-task feature selection which selects the common subset of relevant features for multiple variables from each modality, and (2) a multi-modal support vector machine which fuses the above-selected features from all modalities to predict multiple (regression and classification) variables. To validate our method, we perform two sets of experiments on ADNI baseline MRI, FDG-PET, and cerebrospinal fluid (CSF) data from 45 AD patients, 91 MCI patients, and 50 healthy controls (HC). In the first set of experiments, we estimate two clinical variables such as Mini Mental State Examination (MMSE) and Alzheimer’s Disease Assessment Scale - Cognitive Subscale (ADAS-Cog), as well as one categorical variable (with value of ‘AD’, ‘MCI’ or ‘HC’), from the baseline MRI, FDG-PET, and CSF data. In the second set of experiments, we predict the 2-year changes of MMSE and ADAS-Cog scores and also the conversion of MCI to AD from the baseline MRI, FDG-PET, and CSF data. The results on both sets of experiments demonstrate that our proposed M3T learning scheme can achieve better performance on both regression and classification tasks than the conventional learning methods. In Alzheimer's disease (AD) diagnosis process, functional brain image modalities such as Single-Photon Emission Computed Tomography (SPECT) and Positron Emission Tomography (PET) have been widely used to guide the clinicians. However, the current evaluation of these images entails a succession of manual reorientations and visual interpretation steps, which attach in some way subjectivity to the diagnostic. In this work, a complete computer aided diagnosis (CAD) system for an automatic evaluation of the neuroimages is presented. Principal component analysis (PCA)-based methods are proposed as feature extraction techniques, enhanced by other linear approaches such as linear discriminant analysis (LDA) or the measure of the Fisher discriminant ratio (FDR) for feature selection. The final features allow to face up the so-called small sample size problem and subsequently they are used for the study of neural networks (NN) and support vector machine (SVM) classifiers. The combination of the presented methods achieved accuracy results of up to 96.7 and 89.52 for SPECT and PET images, respectively, which means a significant improvement over the results obtained by the classical voxels-as-features (VAF) reference approach. Recent studies have demonstrated that biomarkers from multiple modalities contain complementary information for the diagnosis of Alzheimer's disease AD and its prodromal stage mild cognitive impairment MCI. In order to fuse data from multiple modalities, most previous approaches calculate a mixed kernel or a similarity matrix by linearly combining kernels or similarities from multiple modalities. However, the complementary information from multi-modal data are not necessarily linearly related. In addition, this linear combination is also sensitive to the weights assigned to each modality. In this paper, we propose a nonlinear graph fusion method to efficiently exploit the complementarity in the multi-modal data for the classification of AD. Specifically, a graph is first constructed for each modality individually. Afterwards, a single unified graph is obtained via a nonlinear combination of the graphs in an iterative cross diffusion process. Using the unified graphs, we achieved classification accuracies of 91.8 between AD subjects and normal controls NC, 79.5 between MCI subjects and NC and 60.2 in a three-way classification, which are competitive with state-of-the-art results.