We conduct extensive experiments which validate the superior accuracy, robustness, and reliability of our model thanks to the promising uncertainty estimation and multi-view integration strategy.
Related Work
Uncertainty-based Learning. Deep neural networks have achieved great success in various tasks. However since most deep models are essentially deterministic functions, the uncertainty of the model cannot be obtained. Bayesian neural networks (BNNs) [1]}, [2]}, [3]} endow deep models with uncertainty by replacing the deterministic weight parameters with distributions. Since BNNs struggle in performing inference and usually come with prohibitive computational costs, a more scalable and practical approach, MC-dropout [4]}, was proposed. In this model, the inference is completed by performing dropout sampling from the weight during training and testing. Ensemble based methods [5]} train and integrate multiple deep networks and also achieve promising performance.
Instead of indirectly modeling uncertainty through network weights, the algorithm [6]} introduces the subjective logic theory to directly model uncertainty without ensemble or Monte Carlo sampling. Building upon RBF networks, the distance between test samples and prototypes can be used as the agency for deterministic uncertainty [7]}. Benefiting from the learned weights of different tasks with homoscedastic uncertainty learning, [8]} achieves impressive performance in multi-task learning.
Multi-View Learning. Learning on data with multiple views has proven effective in a variety of tasks. CCA-based multi-view models [9]}, [10]}, [11]}, [12]}, [13]}, [14]} are representative ones that have been widely used in multi-view representation learning. These models essentially seek a common representation by maximizing the correlation between different views. Considering common and exclusive information, hierarchical multi-modal metric learning (HM3L) [15]} explicitly learns shared multi-view and view-specific metrics, while \(AE^2\) -Nets [16]} implicitly learn a complete (view-specific and shared multi-view) representation for classification. Recently, the methods [17]}, [18]}, [19]}, [20]} based on contrastive learning have also achieved good performance. Due to its effectiveness, multi-view learning has been widely used in various applications [21]}, [22]}, [23]}, [24]}.
Dempster-Shafer Evidence Theory (DST). DST, which is a theory on belief functions, was first proposed by Dempster [25]} and is a generalization of the Bayesian theory to subjective probabilities [26]}. Later, it was developed into a general framework to model epistemic uncertainty [27]}. In contrast to Bayesian neural networks, which indirectly obtain uncertainty through multiple stochastic samplings from weight parameters, DST directly models uncertainty. DST allows beliefs from different sources to be combined with various fusion operators to obtain a new belief that considers all available evidence [28]}, [29]}. When faced with beliefs from different sources, Dempster's rule of combination tries to fuse their shared parts, and ignores conflicting beliefs through normalization factors. A more specific implementation will be discussed later.
<FIGURE>
Trusted Multi-View Classification
It has been shown that using a softmax output as confidence for predictions often leads to high confidence values, even for erroneous predictions since the largest softmax output is used for the final prediction [30]}, [7]}. Therefore, we introduce an evidence-based uncertainty estimation technique which can provide more accurate uncertainty and allow us to flexibly integrate multiple views for trusted decision making.
Uncertainty and the Theory of Evidence
In this subsection, we elaborate on evidential deep learning to quantify the classification uncertainty for each of multiple views, which simultaneously models the probability of each class and overall uncertainty of the current prediction. In the context of multi-class classification, Subjective logic (SL) [32]} associates the parameters of the Dirichlet distribution (Definition REF  in the Appendix) with the belief distribution, where the Dirichlet distribution can be considered as the conjugate prior of the categorical distribution [33]}.
Accordingly, we need to determine the concentration parameters, which are closely related to the uncertainty. We elaborate on the Subjective logic [32]}, which defines a theoretical framework for obtaining the probabilities (belief masses) of different classes and overall uncertainty (uncertainty mass) of the multi-classification problem based on the evidence collected from data. Note that evidence refers to the metrics collected from the input to support the classification (step [baseline=(char.base)]
shape=circle,draw,inner sep=0.5pt] (char) 1; in Fig. REF ) and is closely related to the concentration parameters of Dirichlet distribution. Specifically, for the \(K\)  classification problems, subjective logic tries to assign a belief mass to each class label and an overall uncertainty mass to the whole frame based on the evidence. Accordingly, for the \(v^{th}\)  view, the \(K+1\)  mass values are all non-negative and their sum is one:
\(u^v+\sum _{k=1}^{K} b_{k}^{v}=1,\) 
where \(u^v \ge 0\)  and \(b_{k}^{v} \ge 0\)  indicate the overall uncertainty and the probability for the \(k^{th}\)  class, respectively.
For the \(v^{th}\)  view, subjective logic connects the evidence \(\mathbf {e}^v = [e^v_1, \cdots , e^v_K]\)  to the parameters of the Dirichlet distribution \(\alpha ^v=[\alpha ^v_1,\cdots ,\alpha ^v_K]\)  (step [baseline=(char.base)]
shape=circle,draw,inner sep=0.5pt] (char) 2; in Fig. REF ). Specifically, the parameter \(\alpha ^v_k\)  of the Dirichlet distribution is induced from \(e^v_k\) , i.e., \(\alpha _k^v=e^v_k+1\) .
Then, the belief mass \(b_k^v\)  and the uncertainty \(u^v\)  (step [baseline=(char.base)]
shape=circle,draw,inner sep=0.5pt] (char) 3; in Fig. REF ) are computed as
\(b_{k}^v=\frac{e_k^v}{S^v}=\frac{\alpha _{k}^v-1}{S^v}\quad \text{and} \quad u^v = \frac{K}{S^v},\) 
where \(S^v=\sum _{i=1}^{K}{(e^v_i+1)}=\sum _{i=1}^{K}{\alpha ^v_i}\)  is the Dirichlet strength. Eq. REF  actually describes the phenomenon where the more evidence observed for the \(k^{th}\)  category, the greater the probability assigned to the \(k^{th}\)  class. Correspondingly, the less total evidence observed, the greater the total uncertainty. The belief assignment can be considered as a subjective opinion. Given an opinion, the mean of the corresponding Dirichlet distribution \(\hat{\mathbf {p}}^v\)  for the class probability \(\hat{p}_{k}^v\)  is computed as \(\hat{p}_{k}^v=\frac{\alpha _{k}^v}{S^v}\)  [35]}.
Differences from traditional deep-neural-network classifiers. Firstly, the output of traditional neural network classifiers can be considered as a point on a simplex, while Dirichlet distribution parametrizes the density of each such probability assignment on a simplex. Therefore, with the Dirichlet distribution, SL models the second-order probability and uncertainty of the output. Secondly, the softmax function is widely used in the last layer of traditional neural network classifiers. However, using the softmax output as the confidence often leads to over-confidence. In our model, the introduced SL can avoid this problem by adding overall uncertainty mass. Existing methods [4]}, [5]} usually require additional computations during inference to output uncertainty. Since the uncertainty is obtained during the inference stage, it is difficult to seamlessly train a model with high accuracy, robustness and reasonable uncertainty in a unified framework. Accordingly, the limitations underlying existing algorithms (e.g., inability to directly obtain uncertainty) also limits their extension to trusted multi-view classification.
<FIGURE>For clarity, we provide typical examples under a triple classification task to illustrate the above formulation. Let us assume that \(\mathbf {e}=\langle 40, 1, 1\rangle \)  and accordingly we have \(\mathbf {\alpha }=\langle 41, 2, 2\rangle \) . The corresponding Dirichlet distribution, shown in Fig. REF (a), yields a sharp distribution centered on the top of the standard 2-simplex. This indicates that sufficient evidence has been observed to ensure accurate classification. In contrast, let us assume that we have the evidence \(\mathbf {e}=\langle 0.0001, 0.0001, 0.0001\rangle \) , which is little evidence for classification. Accordingly, we obtain the Dirichlet distribution parameter \(\mathbf {\alpha }=\langle 1.0001, 1.0001, 1.0001\rangle \)  and the uncertainty mass \(u\approx 1\) . As shown in Fig. REF (b), in this case, the evidence induces quite a flat distribution over the simplex. Finally, when \(\mathbf {e}=\langle 5, 5, 5\rangle \) , there is also a high uncertainty, as shown in Fig. REF (c), even though the overall uncertainty is reduced compared to the second case. As shown in Fig. REF (d), we can convert a Dirichlet distribution into a standard 3-simplex (a regular tetrahedron with vertices (1,0,0,0), (0,1,0,0), (0,0,1,0) and (0,0,0,1) in \(\mathbf {R}^4\) ) based on the subjective logic theory (Eq. REF  and Eq. REF ), where the point (\(\mathcal {M}\) ) in the simplex corresponding to \(\big \lbrace \lbrace b_k\rbrace _{k=1}^{3}, u\big \rbrace \)  indicates an opinion. Accordingly, the expectation value \(\hat{\mathbf {p}}\)  of the Dirichlet distribution is the projection of \(\mathcal {M}\)  on the bottom.

Dempster's Rule of Combination for Multi-View Classification
Having introduced evidence and uncertainty for the single-view case, we now focus on their adaptation to data with multiple views. The Dempster–Shafer theory of evidence allows evidence from different sources to be combined arriving at a degree of belief (represented by a mathematical object called the belief function) that takes into account all the available evidence (see Definition REF ). Specifically, we need to combine \(V\)  independent sets of probability mass assignments \(\lbrace \mathcal {M}^v\rbrace _1^V\) , where \(\mathcal {M}^v=\big \lbrace \lbrace b_k^v\rbrace _{k=1}^{K}, u^v\big \rbrace \) , to obtain a joint mass \(\mathcal {M}=\big \lbrace \lbrace b_k\rbrace _{k=1}^{K}, u\big \rbrace \)  (step [baseline=(char.base)]
shape=circle,draw,inner sep=0.5pt] (char) 4; in Fig. REF ).
Definition 3.1 (Dempster's combination rule for two independent sets of masses)
 The combination (called the joint mass) \(\mathcal {M}=\big \lbrace \lbrace b_k\rbrace _{k=1}^{K}, u\big \rbrace \)  is calculated from the two sets of masses \(\mathcal {M}^1= \big \lbrace \lbrace b_k^1\rbrace _{k=1}^{K}, u^1\big \rbrace \)  and \(\mathcal {M}^2=\big \lbrace \lbrace b_k^2\rbrace _{k=1}^{K}, u^2\big \rbrace \)  in the following manner:
\(\mathcal {M}=\mathcal {M}^1\oplus \mathcal {M}^2.\) 
The more specific calculation rule can be formulated as follows:
\(\begin{aligned}& b_{k}=\frac{1}{1-C}( b^1_kb^2_k + b^1_ku^2 + b^2_ku^1), u = \frac{1}{1-C}u^1u^2,\end{aligned}\) 
where \(C = \sum _{i\ne j}{b^1_i b^2_j}\)  is a measure of the amount of conflict between the two mass sets (the white blocks in Fig. REF ), and the scale factor \(\frac{1}{1-C}\)  is used for normalization.

The joint opinion \(\mathcal {M}\)  is formed based on the fusion of opinions \(\mathcal {M}^1\)  and \(\mathcal {M}^2\) . The joint belief mass of class \(k\)  (\(b_k\) ) and overall uncertainty (\(u\) ) correspond to the brown blocks in Fig. REF . Intuitively, the combination rule ensures: (1) when both views are of high uncertainty (large \(u^1\)  and \(u^2\) ), the final prediction must be of
low confidence (small \(b_k\) ); (2) when both views are of low uncertainty (small \(u^1\)  and \(u^2\) ), the final prediction may be of high confidence (large \(b_k\) ); (3) when only one view is of low uncertainty (only \(u^1\)  or \(u^2\)  is large), the final prediction only depends on the confident view.
Then, given data with \(V\)  different views, we can obtain the above-mentioned mass for each view. Afterwards, we can combine the beliefs from different views with Dempster's rule of combination. Specifically, we fuse the belief mass and uncertainty mass between different views with the following rule:
\(\begin{aligned}\mathcal {M}=\mathcal {M}^1\oplus \mathcal {M}^2 \oplus \cdots \mathcal {M}^V.\end{aligned}\) 
After obtaining the joint mass \(\mathcal {M}=\big \lbrace \lbrace b_k\rbrace _{k=1}^{K}, u\big \rbrace \) , according to Eq. REF , the corresponding joint evidence from multiple views and the parameters of the Dirichlet distribution are induced as
\(\begin{aligned}S = \frac{K}{u} , e_k = b_k \times S \text{ and } \alpha _k = e_k+1.\end{aligned}\) 
Based on the above combination rule, we can obtain the estimated multi-view joint evidence \(\mathbf {e}\)  and the corresponding parameters of joint Dirichlet distribution \(\mathbf {\alpha }\)  to produce the final probability of each class and the overall uncertainty.
Advantages of using subjective logic compared with softmax. Compared with softmax output, using subjective uncertainty is more suitable for the fusion of multiple decisions. Subjective logic provides an additional mass function (\(u\) ) that allows the model distinguish between a lack of evidence. In our model, subjective logic provides the degree of overall uncertainty of each view, which is important for trusted classification and interepretability to some extent.

Learning to Form Opinions
In this section, we will discuss how to train neural networks to obtain evidence for each view, which can then be used to obtain the corresponding masses \(\lbrace \mathcal {M}^v\rbrace _{v=1}^V\)  and \(\mathcal {M}\) . The neural networks can capture the evidence from input to induce a classification opinion [21]}, and the conventional neural-network-based classifier can be naturally transformed into the evidence-based classifier with minor changes. Specifically, the softmax layer of a conventional neural-network-based classifier is replaced with an activation function layer (i.e., RELU) to ensure that the network outputs non-negative values, which are considered as the evidence vector \(e\) . Accordingly, the parameters of the Dirichlet distribution can be obtained.
For conventional neural-network-based classifiers, the cross-entropy loss is usually employed:
\(\mathcal {L}_{c e} = -\sum _{j=1}^{K} y_{i j}\log (p_{i j}),\) 
where \(p_{i j}\)  is the predicted probability of the \(i\) th sample for class \(j\) . For our model, given the evidence of the \(i\) th sample obtained through the evidence network, we can get the parameter \(\alpha _i\)  (i.e., \(\alpha _i^v=\mathbf {e}^i_i+1\) ) of the Dirichlet distribution and form the multinomial opinions \(D(\mathbf {p}_i|\alpha _i)\) , where \(\mathbf {p}_i\)  is the class assignment probabilities on a simplex.
After a simple modification on Eq. REF , we have the adjusted cross-entropy loss:
\(\mathcal {L}_{ace}(\mathbf {\alpha }_{i})=\int \left[\sum _{j=1}^{K}-y_{i j} \log \left(p_{i j}\right)\right] \frac{1}{B\left(\alpha _{i}\right)} \prod _{j=1}^{K} p_{i j}^{\alpha _{i j}-1} d \mathbf {p}_{i}=\sum _{j=1}^{K} y_{i j}\left(\psi \left(S_{i}\right)-\psi \left(\alpha _{i j}\right)\right),\) 
where \(\psi (\cdot )\)  is the digamma function.
Eq. REF  is the integral of the cross-entropy loss function on the simplex determined by \(\mathbf {\alpha }_i\) . The above loss function ensures that the correct label of each sample generates more evidence than other classes, however, it cannot guarantee that less evidence will be generated for incorrect labels. That is to say, in our model, we expect the evidence for incorrect labels to shrink to 0. To this end, the following KL divergence term is introduced:
\(\begin{array}{l}K L\left[D\left(\mathbf {p}_{i} | \tilde{\alpha }_{i}\right) \Vert  D\left(\mathbf {p}_{i} | \mathbf {1}\right)\right] \\\quad =\log \left(\frac{\Gamma \left(\sum _{k=1}^{K} \tilde{\alpha }_{i k}\right)}{\Gamma (K) \prod _{k=1}^{K} \Gamma \left(\tilde{\alpha }_{i k}\right)}\right)+\sum _{k=1}^{K}\left(\tilde{\alpha }_{i k}-1\right)\left[\psi \left(\tilde{\alpha }_{i k}\right)-\psi \left(\sum _{j=1}^{K} \tilde{\alpha }_{i j}\right)\right],\end{array}\) 
where \(\tilde{\alpha }_i = \mathbf {y}_i+(1-\mathbf {y}_i)\odot \mathbf {\alpha }_i\)  is the adjusted parameter of the Dirichlet distribution which can avoid penalizing the evidence of the groundtruth class to 0, and \(\Gamma (\cdot )\)  is the gamma function.
Therefore, given parameter \(\alpha _i\)  of the Dirichlet distribution for each sample \(i\) , the sample-specific loss is
\(\mathcal {L}(\mathbf {\alpha }_i)= \mathcal {L}_{ace}(\mathbf {\alpha }_i)+\lambda _{t} K L\left[D\left(\mathbf {p}_{i} | \tilde{\alpha }_{i}\right) \Vert  D\left(\mathbf {p}_{i} |\mathbf {1}\right)\right],\) 
where \(\lambda _{t}>0\)  is the balance factor. In practice, we can gradually increase the value of \(\lambda _{t}\)  so as to prevent the network from paying too much attention to the KL divergence in the initial stage of training, which may result in a lack of good exploration of the parameter space and cause the network to output a flat uniform distribution.
To ensure that all views can simultaneously form reasonable opinions and thus improve the overall opinion, we use a multi-task strategy with following overall loss function:
\(\begin{aligned}\mathcal {L}_{overall} = \sum _{i=1}^{N}\left[\mathcal {L}(\mathbf {\alpha }_i) + \sum _{v=1}^{V}\mathcal {L}(\mathbf {\alpha }_i^v)\right].\end{aligned}\) 
The optimization process for the proposed model is summarized in Algorithm REF  (in the Appendix).

Experiments
Experimental Setup
In this section, we conduct experiments on six real-world datasets: Handwrittenhttps://archive.ics.uci.edu/ml/datasets/Multiple+Features, CUB [39]}, Caltech101 [40]}, PIEhttp://www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-Pie/Home.html, Scene15 [41]} and HMDB [42]}. We first compare our algorithm with single-view classifiers to validate the effectiveness of our algorithm in utilizing multiple views. Then, we apply existing classifiers to multi-view features and conduct experiments under different levels of noise to investigate their ability in identifying multi-view OOD samples. Details of these datasets and experimental setting can be found in the appendix.
<FIGURE>Compared methods. We compare the proposed method with following models: (a) MCDO (monte carlo dropout) [43]} casts dropout network training as approximate inference in a Bayesian neural network; (b) DE (deep ensemble) [5]} is a simple, non-Bayesian method which involves training multiple deep models; (c) UA (uncertainty-aware attention) [45]}
generates attention weights following a Gaussian distribution with a learned mean and variance, which allows heteroscedastic uncertainty to be captured and yields a more accurate calibration of prediction uncertainty; (d) EDL (evidential deep Learning) [6]} designs a predictive distribution for classification by placing a Dirichlet distribution on the class probabilities.

Experimental Results
Comparison with uncertainty-based algorithms using the best view. We first compare our algorithm with current uncertainty-based classification methods. The detailed experimental results are shown in Table REF . Since most existing uncertainty-based classification methods use single-view data, we report the results of each method with the best-performing view in terms of both accuracy and AUROC [47]} to comprehensively compare our method with others. As shown in Table REF , our model outperforms other methods on all datasets. Taking the results on PIE and Scene15 as examples, our method improves the accuracy by about 7.6% and 14.8% compared to the second-best models (EDL/MCDO) in terms of accuracy respectively. Although our model is clearly more effective than single-view uncertainty-based models, it is natural to further ask - what happens if all algorithms utilize multiple views?
<TABLE>Comparison with uncertainty-based algorithms using multiple views. To further validate the effectiveness of our model in integrating different various views, we concatenate the original features of multiple views for all comparison methods.
We add Gaussian noise with different levels of standard deviations (\(\sigma \) ) to half of the views. The comparison results are shown in Fig. REF . As can be observed that when the data is free of noise, our method can achieve competitive results. After introducing noise to the data, the accuracy of all the comparison methods significantly decreases. Fortunately, benefiting from the uncertainty-based fusion, the proposed method is aware of the view-specific noise and thus achieves impressive results on all datasets. Therefore, the effectiveness for both clean and noisy multi-view data is well validated. However, it will be more convincing to explicitly investigate the performance in uncertainty estimation.
<FIGURE>Uncertainty estimation. To evaluate the uncertainty estimation, we visualize the distribution of in-/out-of-distribution samples in terms of uncertainty. We consider the original samples as in-distribution data, while the samples with Gaussian noise are viewed as out-of-distribution data. Specifically, we add Gaussian noise with the fixed level of standard deviations (\(\sigma = 10\) ) to \(50\%\)  of the test samples. The experimental results are shown in Fig. REF . According to the results, the following observations are drawn: (1) Datasets with higher classification accuracy (e.g., Handwritten) are usually associated with lower uncertainty for the in-distribution samples. (2) In contrast, datasets with lower accuracy are usually associated with higher uncertainty for the in-distribution samples. (3) Much higher uncertainties are usually estimated for out-of-distribution samples on all datasets. These observations imply the reasonability of our model in estimating uncertainty, since it can facilitate discrimination between these classes. Fig. REF  shows that our algorithm provides much more accurate predictions as the prediction uncertainty decreases. This implies that trusted decisions are supported based on the output (classification and its corresponding uncertainty) of our model.
<FIGURE>
Conclusion
In this work, we propose a novel trusted multi-view classification (TMC) algorithm which, based on the Dempster-Shafer evidence theory, can produce trusted classification decisions on multi-view data. Our algorithm focuses on decision-making by fusing the uncertainty of multiple views, which is essential for making trusted decisions. The TMC model can accurately identify the views which are risky for decision making, and exploits informative views in the final decision. Furthermore, our model can produce the uncertainty of a current decision while making the final classification, providing intepretability. The empirical results validate the effectiveness of the proposed algorithm in classification accuracy and out-of-distribution identification.
Acknowledgments
This work was supported in part by National Natural Science Foundation of China (No. 61976151, No. 61732011), and the Natural Science Foundation of Tianjin of China (No. 19JCYBJC15200).

Appendix
Definition of Dirichlet distribution
Definition A.1 (Dirichlet distribution)
The Dirichlet distribution is parameterized by its concentration \(K\)  parameters \(\mathbf {\alpha }=[\alpha _1, \ldots , \alpha _K]\) . The probability density function of the Dirichlet distribution is given by
\(D(\mathbf {p} \mid \alpha )=\left\lbrace \begin{array}{ll}\frac{1}{B(\alpha )} \prod _{i=1}^{K} p_{i}^{\alpha _{i}-1} & \text{ for } \mathbf {p} \in \mathcal {S}_{K} ,\\0 & \text{ otherwise ,}\end{array}\right.\) 
where \(\mathcal {S}_{K}\)  is the \(K\) -dimensional unit simplex, defined as
\(\mathcal {S}_{K}=\left\lbrace \mathbf {p} \mid \sum _{i=1}^{K} p_{i}=1 \text{ and } 0 \le p_{1}, \ldots , p_{K} \le 1\right\rbrace ,\) 
and \(B(\alpha )\)  is the K-dimensional multinomial beta function.


Algorithm for trust Multi-View Classification
[h]
Algorithm for Trusted Multi-View Classification
/*Training*/
Multi-view dataset: \(\mathcal {D} = \lbrace \lbrace \mathbf {X}_n^v\rbrace _{v=1}^{V}, y_n\rbrace _{n=1}^N\) .
Initialize: Initialize the parameters of the neural network.
not converged
\(v=1:V\)  

\(\mathbf {e}^v\leftarrow \)  evidential network output;
\(\mathbf {\alpha }^v\leftarrow \mathbf {e}^v + 1\)  ;
Obtain opinion \(\mathcal {M}^v\)  with Eq. REF ;

Obtain joint opinion \(\mathcal {M}\)  with Eq. REF ;
Obtain \(\alpha \)  with Eq. REF ;
Obtain the overall loss by updating \(\alpha \)  and \(\lbrace \mathbf {\alpha }^v\rbrace _{v=1}^{V}\)  in Eq. REF ;
Update the networks with gradient descent according to Eq. REF ;

networks parameters./*Test*/
Calculate the joint belief mass and the uncertainty mass.

Details of the datasets
To better evaluate our model, we conduct experiments on the following six real-world datasets. Details of these datasets are as follows:
1) Handwritten consists of 2000 samples of 10 classes from digit `0' to `9' with 200 samples per class, where six different types of descriptors are used as multiple views.

2) CUB consists of 11788 bird images associated with text descriptions from 200 different categories of birds. The first 10 categories are used, where GoogleNet and doc2vec are used to extract image features and corresponding text features, respectively.

3) PIE consists of 680 facial images of 68 subjects. Three types of features including intensity, LBP and Gabor are extracted.

4) HMDB is one of the largest human action recognition dataset. There are 6718 samples of 51 categories of actions, where HOG and MBH features are extracted.

5) Caltech101 consists of 8677 images from 101 classes. We extract two types of deep features with DECAF and VGG19 respectively.

6) Scene15 consists of 4485 images from 15 indoor and outdoor scene categories. Three types features (GIST, PHOG and LBP) are extracted as multiple views.
Comparison with CCA-based algorithms
We compared our method with the CCA-based algorithms. Specifically, we employ CCA-based methods to obtain the latent representations and then the linear SVM classifier is used for classification. The following CCA-based algorithms are used as baselines. (1) CCA [9]} is a classical algorithm for seeking the correlations between multiple types of features. (2) DCCA [12]} obtains the correlations through deep neural networks. (3) DCCAE [13]} employs autoencoders for seeking the common representations. (4) BCCA [11]} presents a Bayesian model selection algorithm for CCA based on a probabilistic interpretation. Due to randomness (e.g., training/testing partition and optimization) involved, for each method, we run 30 times and report its mean and standard deviation in terms of classification accuracy. The experimental results are shown in Table REF . On all datasets, our method consistently achieves better performance compared with these CCA-based algorithms. Note that our method is quite different from the CCA-based methods. Ours is a classification model while most CCA-based methods are unsupervised representation learning models. Meanwhile, to the best of our knowledge, existing CCA-based algorithms are unable to provide trusted decisions.
<TABLE>
Experimental analysis of removing view
<TABLE>We conduct experiments by removing one view manually on two datasets (i.e., PIE and Scene15) associated with three views. We run 30 times and report the mean and standard deviation in terms of classification accuracy. The experimental results are shown in Table REF . It is observed that by using more information, the performance of our model tends to be improved.

Details of the implementation
For our trusted multi-view classification (TMC) algorithm, we employ the fully connected networks for all datasets, and \(l_2\) -norm regularization is used with the value of the trade-off parameter being 0.0001. The Adam [52]} optimizer is used to train the network. 5-fold cross-validation is employed to select the learning rate from \(\lbrace 3e^{-3}, 1e^{-3}, 3e^{-4}, 1e^{-4}\rbrace \) . For all datasets, \(20\%\)  of samples are used as test
set. The hyperparameter \(\lambda _t\)  in
Eq. REF  slowly increases from 0 to 1. For all methods, we run 30 times to report the mean values and standard deviations.

End-to-end experiment
Although the effectiveness has been well validated on datasets with multiple types of features, we further conduct experiments in an end-to-end manner on multi-modal datasets. The UMPC-FOOD101 dataset consists of 86,796 samples scraped from web pages, each sample is described with both image and text [53]}. Similarly to existing work [23]}, there are 60,101 samples used as the training set, 5,000 samples used as the validation set, and the remaining 21,695 samples are used as the test set. The samples of NYUD (RGB-D) dataset are collected by cropping out tight bounding boxes around instances of 19 object classes in the NYUD [55]} dataset. There are 4,587 samples, and each sample is composed of both RGB and depth modalities. We use 2,186, 1,200, and 1,201 samples as the training set, validation set, and test set respectively. In this experiment, we used the original datasets which contain noise due to the collection process.
For UMPC-FOOD101, we employ ResNet-152 [56]} pre-trained with ImageNet and BERT [57]} as the backbone network for image and text respectively. For NYUD (RGB-D), ResNet-50 [56]} pre-trained with ImageNet is used as the backbone network for depth and RGB images. For comparison algorithms, we report the results of the best performing views, and furthermore, we concatenate the outputs of the two backbone networks as the input of the classifier as shown in Fig. REF . For all algorithms, due to the randomness involved, we run 10 times to report the mean accuracy and standard deviation. The Adam [52]} optimizer is employed for all methods with learning rate decay.
Quantitative experimental results. The experimental results on the above datasets are shown in Table REF . We report the accuracy of each compared algorithm with the best-performing view (`best view' in Table REF ) and combined views (`feature fusion' in Table REF ). According to Table REF , our algorithm outperforms other methods on all datasets. For example, on the Food101 dataset, our method achieved \(91.3\%\)  in terms of accuracy, compared with \(90.5\%\)  from the second performer. The performance of using the best single view is relatively low.
<FIGURE><TABLE>Qualitative experimental results.
We also present typical examples (from UMPC-Food101) with the highest and lowest uncertainty in prediction in Table REF  and Table REF , respectively. Since the UMPC-Food101 dataset is scraped from web pages, for a few samples the corresponding text descriptions are meaningless (e.g., the 1st one in Table REF ). At the same time, some images are also quite challenging to classify (e.g., the 2nd and the 4th in Table REF ). For these samples that can be considered as out-of-distribution ones, it is difficult to obtain evidence related to classification. As expected, these samples are assigned with high uncertainty (top 5) by our algorithm. In contrast, Table REF  presents the top 5 samples with lowest uncertainty. We can find that for these samples the labels are usually explicitly mentioned in the text and the images also clearly show the characteristics of `macarons'.
<TABLE>