Virtual Sample Generation and Ensemble Learning Based Image Source Identification With Small Training Samples

Nowadays, source camera identification, which aims to identify the source camera of images, is quite important in the field of forensics. There is a problem that cannot be ignored that the existing methods are unreliable and even out of work in the case of the small training sample. To solve this problem, a virtual sample generation-based method is proposed in this paper, combined with the ensemble learning. In this paper, after constructing sub-sets of LBP features, the authors generate a virtual sample-based on the mega-trend-diffusion (MTD) method, which calculates the diffusion range of samples according to the trend diffusion theory, and then randomly generates virtual sample according to uniform distribution within this range. In the aspect of the classifier, an ensemble learning scheme is proposed to train multiple SVM-based classifiers to improve the accuracy of image source identification. The experimental results demonstrate that the proposed method achieves higher average accuracy than the state-of-the-art, which uses a small number of samples as the training sample set.


INTRoDUCTIoN
Nowadays, the digital images generation is popular and easier, which makes it possible for some individuals to upload unsuitable images for their interests or to steal images of others for commercial purposes. Therefore, image source identification is very important in the judicial field, which can offer help to bring evil men to justice. The issue of image source identification is usually modeled as a classification problem, which means decent results are expectant with enough training samples. However, it is well known that obtaining a large number of sufficient training samples may be very difficult, and the classifiers perform very poorly in this scenario of small training samples. Therefore, it is always a big challenge when there are only a small set of labeled images used as references in the practical forensic application.
In recent years, many methods are proposed for the small training sample problem, which are mainly divided into three categories. The first category is active learning and semi-supervised learning based methods, but they usually require a large number of unlabeled samples as auxiliary information, and it is sometimes unrealistic in practical forensic applications; the second category is the methods based on gray prediction model, such as BGM (Chang, Li, Huang, & Chen, 2015), GBM (Wang, Wang, Sun, & Zhang, 2014), ANGM (Chang, Li, & Chen, 2014), which is used to deal with raw samples. However, these methods usually ignore the internal mechanism, and then make the generated virtual samples unsuitable; the third category is consist of the methods based on virtual samples generation, which is proposed by Poggio andVetter in 1992 (Poggio &Vetter, 1992). Considering the insufficient training samples, the appropriate virtual samples are generated under the condition of the training samples' prior information to increase the number of training samples. By obtaining the virtual samples, the training set is supposed to be expanded to effectively improve the generalization ability of the classifier.
In recent years, there are many kinds of researches respect to virtual samples generation. In order to improve the energy prediction accuracy of small training samples problem, He et al. (He, Wang, Zhang, Zhu, & Xu, 2018) propose nonlinear interpolation virtual samples generation method based on the highly nonlinear characteristics of input data and output data. After the virtual samples generation, the images are classified by the extreme learning machine (ELM) (Huang, Zhu, &Siew, 2004) and the experimental results are promising. Li et al. (Li & Fang, 2009) propose a nonlinear virtual sample generation technique (NVSG) and receive an average classification accuracy of 76% for camera models in the Iris data set. The methods of virtual sample generation based on the original samples' distribution are also widely used. Yang et al. (Yang, Yu, Xie, & Zhang, 2011) assume that the samples obey the Gaussian distribution and calculates the mean and variance of the Gaussian distribution from the original training set. Experiments on the Iris data set show that the classification accuracy increases 18%.
In this paper, a MTD based virtual sample generation method is introduced to identify the image source when the training samples are small. By box plot based MTD and sample attributes correlation based method, a reasonable virtual samples generation range is obtained and the virtual samples are generated based on average distribution. Considering the randomness of virtual sample generation, multiple groups of samples are obtained and combined with the original training samples. Multiple weak classifiers based on SVM are trained and integrated to obtain the classifier.
The rest of this paper is organized as follows: Section 2 describes the related work: LBP features and virtual sample generation method; the virtual sample generation and ensemble learning based method are proposed in Section 3; Section 4 demonstrates the experimental design and the discussion of the results and finally, the paper is concluded in Section 5.

LBP
LBP is a local operator describing the texture features of an image. The uniform gray-scale invariant LBP (Tan, Wang, Li, Guo, Kong, & Shi, 2015) operator is donated by LBP P R , and defined as follows: where P represents the number of pixels in the neighborhood and R represents its radius. g c is the center pixel value, and the values of the neighborhood pixels in a circle with a radius R is g p . We set P = 8 , R = 1 . The threshold function s x ( ) is defined as: Refer to (1), (2), the difference between the gray-level value of the center pixel and its neighborhood pixels can be calculated, and there are 256 different patterns of combination. As what is defined in (Tan, Wang, Li, Guo, Kong, & Shi, 2015), ''uniform'' local binary patterns are more likely to appear than other patterns which are called ''non-uniform'' local binary pattern, and the authors integrate all ''non-uniform'' local binary patterns into one pattern. Thus the dimension of features is reduced from 256 to 59.
The whole procedure of LBP feature extraction is shown in Figure 1. The original image is estimated by the prediction function to obtain the predicted image, and the prediction-error image is obtained by subtracting a predicted image from the original one. We finally extract LBP features from the red and green channels of the original image, the 1st-level diagonal wavelet sub-band, and prediction-error 2D array mentioned in (gardenfors, 2004). Thus a total dimension of 59 3 2 354 × × = improved LBP features are obtained.

MTD
The MTD method proposed by Li et al. (Li, Wu, Tsai, & Lina, 2007) is based on the technique of global fuzzification and information diffusion, which is used to fill the blank caused by incomplete samples. It can augment the training set by generating virtual samples based on the distribution of the existing small training samples. Regarding the training samples as a whole, the upper and lower limits of the specific boundary for virtual samples generation are determined as: where L and U are the lower and upper bounds of the virtual samples diffusion, represents the variance of the real samples, N L and N U represent the actual number of real samples whose value are less than and greater than the center value, respectively.
x i and x represent the real samples and their mean, ˆ/ s N x L 2 represents the real samples' diffusion coefficient, n represents the number of the real samples, u min max set = + ( ) / 2 represents the real samples' center value, min and max are the real samples' minimum and maximum value, respectively.
In order to estimate the trend of the real samples while extending the real samples' range, they asymmetrically spread the real samples and consider the skewness of the real samples, the equations are: Where Skew L and Skew U are the left and right skewness of the real samples point relative to the center of the real samples. Set ϕ ϕ ( ) ( ) a b = = − 10 20 in Equation (1) and (2), the upper and lower boundaries after diffusing the final sample set are: then the virtual samples are generated on the uniform distribution within this range.

Box Plot Based MTD
Box plot, also known as boxer map, is widely used to describe the sporadic degree of the data by five statistics: minimum, first quartile, median, third quartile and maximum. It can be unaffected by the outliers when measuring the characteristics of the given data, which is the biggest advantage of the box plot. The core of the box plot based MTD virtual samples generation method (Li, Chen, & Chang, 2012) is to reasonably spread the real samples' range. It is quite essential to accurately estimate the boundary and the skewness of the data set and box plot is proficient at that. The box plot is established as shown in Figure 2, and the data situation can be judged according to its position in the box plot.
After establishing the box plot, we can get the upper and lower limits of the extended data based on the real samples. Different from the MTD method, the diffused upper and lower limits of the training samples are obtained according to the real samples' box plot (Lin, Li, & Pan, 2016), as shown in the following Equation (7) and (8). Therefore, the final determined virtual samples' range is [ L ,U ].
Considering that the randomness of generating virtual samples based on the uniform distribution within the range [ L ,U ], a triangular membership function is utilized to estimate can the occurrence possibility of the generated virtual samples. According to the fuzzy theory, they draw an asymmetric triangle membership function ( MF ) (Li, Lin, Chen, Chen, & Lin, 2018), as shown in Figure 3.

Figure 2. A right-skewed distribution drawn in a box-and-whisker plot
In Figure 4, MF represents the distribution of the training samples, and tv is the virtual samples' value which is randomly generated in the uniform distribution within [ L ,U ]. Based on the generated tv value, the corresponding MF value is calculated according to the following Equation (9), and the MF value is used as the occurrence probability to evaluate whether the randomly generated virtual sample tv is the most suitable. The class label for the generated virtual sample is the same as the original sample. Through the above steps, we can generate sufficient virtual samples to expand our training set for images source identification when the number of training samples are small.

Sample Attributes Correlation Based MTD
Compared with the box plot based MTD method in Section 2.2.3, sample attributes correlation based MTD method considers the correlation between sample attributes when determining the diffusion range of the real samples. Li et al. consider the correlation of sample attributes when calculating the range of generated virtual samples (Gole & Bhme, 2010). Taking the correlation of sample attributes into account, the real samples' data center is calculated by Equation (10), which is used to estimate the samples' distribution.
As shown in the Equation (10), x nth represents the real sample. Considering the mean is susceptible to outliers, Li et al. decide to use the real samples' median instead of the mean to calculate the correlation of sample attributes, because the median can better reflect the deviation of the data center than the mean. The correlation is replaced with the TSA (trend similarity between attributes): where g h i j ( ) , represents degree of similarity between the two attributes' dimensions of real samples in the same class. If g h i j ( ) , = 1 , the similarity trend between the two attributes' dimensions of m and n is extremely high, that is, both are on the same side of the center point CL. S i j , represents the similarity between different attributes' dimensions of the real samples, and k represents the number of samples in the same class. In order to avoid using mean which is susceptible to outliers, we use Euclidean distance instead of sample standard deviation to measure the dispersion as shown in (12). Incorporating the correlation of sample attributes, the ultimate boundary for virtual sample generation after diffusion is calculated as:

PRoPoSED METHoD
According to the above analysis, virtual samples generation is a random procedure and the virtual samples generated by MTDBOX and MTDRELATION are not stable enough to obtain a promising identification classifier, which will drag down the identification accuracy. But on the other hand, instability also means diversity and that is quite suitable for ensemble learning. So in this paper, we incorporate virtual samples generation with ensemble learning to identify the image source when the training samples are small, the framework is shown in Figure 5. In our proposed method, we follow the basic idea of the Bagging in ensemble learning rather than use a specific ensemble learning method. Different from the previous ensemble learning methods, we don't directly operate on the training samples or the features of training samples, instead we generate virtual samples by MTDBOX and MTDRELATION and add these virtual samples to the image set of real samples in order to increase the number of training samples. And then train the base classifier by SVM on the expanded training set.
Ensemble learning is a form of "expanding others" by constructing multiple weak classifiers and combining them into a powerful classifier to effectively accomplish their tasks, it comprehensively determines the learning results by combine multiple learners obtained through training. A weak classifier has a slightly better classification effect than a random guess. Throughout the experiment, the weak classifiers obtained through the initial training are assembled, because the classification results of a single base classifier usually can't reach the ideal standard. By learning and complementing each other, the classifiers can ultimately improve the accuracy of the classification and achieve the desired experimental results.
There are many strategies for integrating classifiers, such as averaging, voting, and learning. For the prediction of classification problems, the voting method is popular, so we use it in this paper. The voting method is also "Majority rule", which means if there are n classifiers, the class obtained by a large number of classifiers is the final class of the image.
In this paper, we firstly extract small training samples' LBP features, and then use the MTD based method to generate virtual samples. The generated virtual samples share a class label with the corresponding original samples. Because the virtual samples generated by the MTD based method are random and the number of generated virtual samples is variable, the MTD based method can be used to train multiple base classifiers. Finally, the results of multiple base classifiers are operated by ensemble learning -the relative majority voting method, and the final sample's label is obtained.

EXPERIMENTS Experiments Set Up
In order to evaluate the proposed algorithm in this paper, the well-known "Dresden Image Database"(Gole & Bhme, 2010) is used. The 16 different camera models from 11 camera brands in the experiments are listed in Table 1. The training samples range from 5 to 20 and there are 180 test samples for each camera.
In our experiments, the box plot based and sample attributes correlation based MTD methods are used to generate virtual samples respectively. Combining the two virtual sample generation methods with SVM classifier directly, we get two baseline algorithms named as MTDBOX and MTDRELATION for comparison. The proposed method using ensemble learning is called MTDEM in the following. In our experiment, we randomly extract 5, 10, 15, and 20 samples from each camera (a total of 16 cameras) as training samples. For each set of samples, 10 virtual samples are supposed to be generated. In order to reduce the randomness of sampling and generating virtual samples, we repeat the experiment 10 times for each class of samples and take the average accuracy as the results of the image source identification.

RESULTS AND DISCUSSIoN
The experimental results are all shown in Table 2.
Without the virtual samples, the accuracy of 39.64%, 63.19%, 70.94%, and 76.32% is achieved based on the image source identification model trained with the raw database of 5, 10, 15, and 20 training samples, respectively. Combing the 10 virtual samples generated by box plot based MTD, the accuracy of image source identification comes to 47.51%, 64.25%, 71.86%, and 77.03%. And also, the virtual samples from sample attributes relation based MTD help the SVM classifier to improve 2.32%, 0.76%, 0.87% and 1.25% in the case of 5, 10, 15 and 20 raw training samples. From the results without virtual sample, MTDBOX and MTDRELATION based virtual sample, an important observation is that the high quality virtual samples do help the classification model improve the performance, while the improvement seems limited. In another word, the virtual samples provide benefits for constructing weak classifiers.
For the proposed method, we obtain a decent accuracy of 53.93% in the case of 5 raw training samples, and high accuracy of 83.28% when the number of training images increases to 20. The highest identification accuracy identify the effectiveness of the ensemble learning used in the proposed method.

CoNCLUSIoN
In this paper, virtual sample generation based algorithms combined with ensemble learning is proposed to identify the image source in the scenario of limited training samples. Two different virtual sample generation methods, MTDBOX and MTDRELATION are used to obtain virtual samples as the training sample. Considering the weak classifiers based on the two virtual sample generation methods, the MTDEM algorithm, which introduces the ensemble learning to improve the performance of the classifier, is proposed in this paper. The experimental results demonstrate that the effectiveness of the proposed method and show that the performance of MTDEM is superior to existing methods when identifying the image source with small training samples.

ACKNoWLEDGMENT
This work was supported by the National Natural Science Foundation of China (No. U1936117, No. U1736119, and No. 61772111).