the support vector machine ( svm ) algorithm is 
 among the most widely used supervised machine learning methods in 
 chemoinformatics and computer - aided drug discovery . the popularity 
 of svm modeling primarily stems from generally high predictive performance 
 in compound classification and virtual screening . although svms have been applied to investigate a variety 
 of class label prediction and also regression tasks in chemoinformatics 
 and drug discovery research , so far only very few 
 studies have addressed the issue of training set composition and size 
 for svm modeling and other machine learning 
 methods . especially the choice of negative training 
 examples is often little considered in machine learning . typically , 
 to train models for compound classification , a subjectively chosen 
 number of molecules are randomly selected from chemical databases 
 to serve as negative training instances , without further analysis . two previous studies have investigated the choice of negative training 
 examples in greater detail . for svm modeling , the 
 use of experimentally confirmed negative training compounds from screening 
 assays and randomly chosen compounds from the zinc database was compared in the prediction of active compounds . it was shown that the source of negative training 
 instances affected the performance of svm classification . perhaps 
 surprisingly , randomly selected zinc compounds often resulted in better 
 models than screening compounds that were confirmed to be inactive 
 against a target for which active compounds were predicted . negative training sets were assembled from different 
 databases for compound classification using different machine learning 
 approaches . these calculations revealed 
 a notable influence of negative training examples on the predictions 
 and a preference for randomly selected zinc compounds over compounds 
 from other sources . in this case , the 
 size of negative training sets was varied when building models using 
 different machine learning methods including svms with polynomial 
 kernels . performance relationships 
 for varying numbers of negative and positive training examples were 
 not investigated . in other studies , positive and negative training 
 examples were balanced to improve the performance of machine learning 
 models , addressing the issue of data imbalance in 
 machine learning . herein , we report an analysis 
 of the influence of training set 
 composition and size on svm classification and ranking by systematically 
 varying the number of negative and positive training examples and 
 determining how these variations affect the prediction of active compounds 
 and stability of the calculations . for svm classification , training compounds 
 are defined by a feature vector x    and a class 
 label   { 1 , 
 1 } and projected into the reference space . svms 
 solve a convex quadratic optimization 
 problem to find a hyperplane h = { x|w , x + b = 0 } that separates 
 the positive and negative class . the hyperplane h is defined by a normal vector w and a bias b and maximizes the margin between the two classes . to achieve 
 model generalization , non - negative slack variables i in addition , the cost hyperparameter c controls the 
 trade - off between margin maximization and permitted training errors , 
 and its value can be optimized by cross - validation . once the decision boundary is defined , test instances 
 are projected into the feature space . new compounds of unknown class 
 label are classified according to the side of the hyperplane on which 
 they fall or , alternatively , ranked according to the value of g(x ) = w , x. the latter strategy is equivalent 
 to changing the bias of the hyperplane , sliding it from the most distant 
 points on the positive side toward the negative side , and ranking 
 compounds in the order they pass through the plane . in the case 
 of nonlinearly separable training data in a given reference 
 space , the scalar product <  ,  > can be replaced by 
 a 
 kernel function k( ,  ) , which is known as the kernel 
 trick.(14 ) using kernel functions , 
 the scalar product of two feature vectors can be computed in a higher 
 dimensional space   where the 
 data may be linearly separable 
 without the need to explicitly compute the mapping of   into . in svm - based 
 compound classification , the tanimoto kernel is one of the most frequently used kernel functions 
 for binary fingerprints . for imbalanced 
 data sets , different class weights can be assigned 
 to put relative weights on misclassification of positive and negative 
 training instances and avoid orienting the hyperplane toward the minority 
 class . accordingly , c+ and c balance 
 the weight on slack variables for the positive and negative class , 
 respectively . ten sets with 
 at least 600 active compounds ( positive instances ) were obtained from 
 chembl version 22 . only compounds with 
 numerically specified equilibrium constants ( ki values ) for single human proteins were selected , while omitting 
 borderline active compounds ( pki < 
 5 ) that might often represent artifacts . table 1 reports the accession number , target name , 
 number of compounds and mean pki values 
 for these 10 compound data sets . as background set ( pool of negative 
 instances ) , random subsets of these compounds were used as 
 negative training and test examples . for model building , all active 
 and inactive compounds were represented as standard maccs fingerprints consisting of 166 bits monitoring the presence 
 ( bit set on ) or absence ( set off ) of predefined structural fragments 
 or patterns . although we deliberately selected the simplistic and 
 easy to rationalize maccs fingerprint for our proof - of - concept investigation , 
 control calculations were also carried out using the folded version 
 of the extended connectivity fingerprint with bond diameter 4 ( ecfp4 ) . ten compound 
 data sets were selected 
 from chembl and used for svm modeling . for each activity class , the 
 chembl accession no . , target name , number of compounds , and mean pki value are reported . ( 1)each activity class was randomly divided 
 into training and test ( prediction ) sets . training set size was varied 
 across values # i = { 10 , 50 , 100 , 500 , 1000 } for the negative ( inactive ) 
 class and # a = { 10 , 50 , 100 , 250 , 500 } for the positive ( active ) class . test sets always consisted of 10  000 inactive and 100 active 
 compounds.(2)preprocessing 
 of the fingerprints 
 of the training and test data was carried out by removing zero - variance 
 features and applying centering and unit variance scaling to all features 
 on the basis of the training set for each trial.(3)for each of the 25 training set combinations , 
 svm models were built using the linear and tanimoto kernel with class 
 weights c+ and c. in addition , cost factors c controlling 
 the influence of individual support vectors were optimized using values 
 of 0.01 , 0.1 , 1 , and 10 . for cost factor optimization , 10-fold cross - validation 
 was carried out with training data splits of 60% ( model derivation ) 
 and 40% ( testing , internal validation ) . models with best cost factors 
 were selected on the basis of largest area under the roc curve ( auc).(4)the optimized svm model 
 was used to 
 rank test set compounds in the order of decreasing probability of 
 activity based upon the signed distance from the hyperplane ( positive 
 to negative side ) . model performance was assessed by determining the 
 recall rate of active compounds within the top 1% of ranked test compounds . in addition , balanced accuracy ( ba ) was calculated , defined as(tp , true positives ; tn , true negatives ; fp , 
 false positives ; fn , false negatives).(5)for each activity class and combination 
 of a kernel function and training set size , the modeling process was 
 carried out 50 times to obtain a distribution of recall rates.(6)the results were compared 
 using hypothesis 
 testing . the nonparametric kolmogorov smirnov test was employed to account for differences between 
 cumulative recall distributions and the levene test to compare the variance of these distributions . in addition , 
 the bonferroni correction was introduced 
 for multiple testing.the calculation protocol 
 was implemented in r , and the kernlab package was used for 
 svm modeling . each activity class was randomly divided 
 into training and test ( prediction ) sets . training set size was varied 
 across values # i = { 10 , 50 , 100 , 500 , 1000 } for the negative ( inactive ) 
 class and # a = { 10 , 50 , 100 , 250 , 500 } for the positive ( active ) class . preprocessing 
 of the fingerprints 
 of the training and test data was carried out by removing zero - variance 
 features and applying centering and unit variance scaling to all features 
 on the basis of the training set for each trial . for each of the 25 training set combinations , 
 svm models were built using the linear and tanimoto kernel with class 
 weights c+ and c. in addition , cost factors c controlling 
 the influence of individual support vectors were optimized using values 
 of 0.01 , 0.1 , 1 , and 10 . for cost factor optimization , 10-fold cross - validation 
 was carried out with training data splits of 60% ( model derivation ) 
 and 40% ( testing , internal validation ) . models with best cost factors 
 were selected on the basis of largest area under the roc curve ( auc ) . the optimized svm model 
 was used to 
 rank test set compounds in the order of decreasing probability of 
 activity based upon the signed distance from the hyperplane ( positive 
 to negative side ) . model performance was assessed by determining the 
 recall rate of active compounds within the top 1% of ranked test compounds . in addition , balanced accuracy ( ba ) was calculated , defined as(tp , true positives ; tn , true negatives ; fp , 
 false positives ; fn , false negatives ) . for each activity class and combination 
 of a kernel function and training set size , the modeling process was 
 carried out 50 times to obtain a distribution of recall rates . smirnov test was employed to account for differences between 
 cumulative recall distributions and the levene test to compare the variance of these distributions . for different 
 activity classes , svm classification and ranking 
 models were built under systematic variation of training set composition 
 and size and active compounds were predicted . specifically , the number 
 of negative and positive training examples was varied in the ranges 
 of 101000 and 10500 , respectively , and all possible 
 combinations were explored . in addition , cost factors were optimized 
 by cross - validation and class - specific weights were used to account 
 for data imbalance in the training set . figure 1 compares 
 balanced accuracy of the predictions in the 
 presence or absence of class weights for two representative activity 
 classes . consideration of class - specific weights consistently improved 
 the accuracy of the predictions for imbalanced training sets , except 
 for three cases of large training sets with at least 250 actives and 
 500 inactives for which the performance was comparable . hence , the 
 explicit consideration of different class weights for positive and 
 negative training instances produced more accurate classification 
 models . under these conditions , the derived hyperplane was not skewed 
 toward the minority class , resulting in improved model generalization , 
 especially in the presence of large training data imbalance . these 
 effects were outweighed only for the largest and least imbalanced 
 training sets . given the demonstrated relevance of class weights for 
 prediction accuracy , a factor that is not always considered in svm 
 modeling , results reported in the following included class weight 
 settings heat map representations 
 show balanced accuracy over 50 independent trials ( using a two - color 
 gradient ) for training sets of varying composition and size : ( top ) 
 melanocortin receptor 4 ( mc4r ) ligands , ( bottom ) thrombin inhibitors . in addition , optimization of cost 
 factors was carried out using 
 cross validation . the best cost factors often varied depending on 
 training set composition , but for well - performing training sets ( i.e. , 
 those with large numbers of actives and inactives ) , there was an overall 
 preference for c values of 0.01 for both the linear 
 and tanimoto kernels . for highly imbalanced data sets , larger cost 
 factors were frequently selected , indicating that adjusting margin 
 softness ( stability ) also contributed to model generalization . it 
 is noteworthy that for different training set compositions and regardless 
 of the cost factor chosen the hyperplanes generated by the svms were 
 very frequently able to separate the training data without error and 
 thus resulted in a hard margin classifier . figure 2 reports 
 compound recall for alternative 
 kernel functions under systematic variation of inactive and active 
 training instances for two representative activity classes . figure 3 shows corresponding 
 density plots for recall rate distributions over multiple trials . the results in figure 2 and 3 reveal generally higher recall performance for the tanimoto 
 than the linear kernel , frequently reaching a recall level of 0.9 . however , even for the linear kernel , satisfactory recall was observed , 
 often approaching a recall level of 0.75 . differences in recall performance 
 between the linear and tanimoto kernel were quantitatively assessed 
 for all activity classes and statistically compared using the two - sided 
 and paired kolmogorov the results confirmed that 
 the tanimoto kernel generally performed significantly better than 
 the linear kernel for training instances of # a = { 100 , 250 , 500 } and 
 # i = { 100 , 500 , 1000}. however , there was no significant difference 
 in the cases of # a = { 10 } and # i = { 50 , 100 , 500 , 1000 } where prediction 
 accuracy was limited . furthermore , as shown in figure 3 , svm models derived using the tanimoto kernel 
 were generally more robust , i.e. , corresponding recall rate distributions 
 were sharper for the tanimoto than for linear kernel . the presence 
 of narrow distributions indicated that models derived from different 
 training sets had comparable prediction accuracy for alternative test 
 instances . as a control , svm calculations were also repeated using 
 the radial basis function ( rbf ) kernel , another popular 
 kernel function , with a sigma setting , corresponding to the inverse 
 kernel width , of 0.01 . the results obtained 
 using the rbf kernel were , on average , nearly indistinguishable from 
 those obtained using the tanimoto kernel discussed in the following . as an additional control , the calculations were also carried out using 
 ecfp4 instead of maccs to compare the trends observed for training 
 set variation . with both fingerprints , the same trends were observed 
 ( with the typical slightly better recall performance of ecfp4 relative 
 to maccs ) . recall performance . the median value and interquartile range of 
 the recall rate of active compounds among the top 1% of the ranking 
 is reported for 50 trials with the linear ( blue dashed line ) or tanimoto 
 ( red solid line ) kernel . results monitor the evolution of recall for 
 a constant number of inactives ( or actives ) and increasing number 
 of actives ( or inactives ) in the training set : ( a ) melanocortin receptor 
 4 ligands , ( b ) thrombin inhibitors . the distribution of recall rates over 50 trials 
 is given for 100 ( top ) and 1000 ( bottom ) inactive and increasing numbers 
 of active training compounds : ( a ) melanocortin receptor 4 ligands , 
 ( b ) thrombin inhibitors . the results 
 in figures 2 , 3 , and 4 revealed two key 
 findings ; ( i ) recall performance and model generalization consistently 
 improved with increasing size of training sets and ( ii ) the ratio 
 of active vs inactive training examples significantly influenced prediction 
 accuracy . the increases in recall performance observed in figure 2 were detected for 
 all activity classes . when the number of active training instances 
 was kept constant , recall rates increased with increasing numbers 
 of inactive instances , except in the case of 10 actives , where prediction 
 accuracy was generally low even over the range of 1001000 
 negative instances . thus , a minimum number of active training compounds 
 was required for training sets of increasing size . similar observations 
 were made when the number of inactive training compounds was kept 
 constant and the number of active examples was increased . ten negative 
 examples were consistently insufficient for building effective models 
 and 50 negative training instances were often insufficient ( figure 2 ) . however , in the 
 presence of at least 100 negative training instances , high prediction 
 accuracy was consistently achieved when the number of active examples 
 was increased ( figure 3 ) . density estimates obtained from the distribution of recall rates over 
 50 trials are presented for training sets of varying size and composition . 
 for a constant number of 10500 active training compounds , recall distributions are shown for 10 ( pink ) , 100 ( green ) , and 1000 
 ( blue ) inactive training compounds : ( a ) melanocortin receptor 4 ligands , 
 ( b ) thrombin inhibitors . for all compound classes , incremental increase in the number 
 of 
 negative ( positive ) training instances led to systematic performance 
 enhancements when at least 50 positive ( 100 negative ) training compounds 
 were used , as confirmed by the one - sided kolmogorov while overall highest prediction accuracy was achieved for training 
 sets consisting of 500 active and 1000 inactive examples , similar 
 accuracy was already observed for 100 active and 500 inactive training 
 compounds . furthermore , recall generally began to reach a plateau 
 when at least 100 active and 500 inactive training instances were 
 used ( figure 2 ) . however , 
 with further increasing training set size , recall rate distributions 
 became narrower , as illustrated in figure 3 and 4 , which was 
 indicative of models with consistent prediction accuracy despite training 
 set variations , as mentioned above . table 2 compares 
 the recall performance over all activity classes for one of the worst 
 and the best performing training set compositions of 10 actives/100 
 inactives and 500 actives/1000 inactives , respectively . in the bad 
 case scenario , recall rates of compounds were  with one exception lower 
 than 50% with large standard deviations and balanced accuracy was 
 around the 80% level . by contrast , for the best performing large training 
 sets , recall rates were consistently high , with a mean of 87% , and 
 balanced accuracy was approaching 100% with very low standard deviations 
 ( table 2 ) . interestingly , 
 training set imbalance only limited the accuracy of predictions in 
 the case of small but not large training sets , as illustrated in figure 4 , an effect that 
 can be ascribed to the use of class weights for svm models , as detailed 
 above . for example , while an inactive vs active ratio of 10:1 produced 
 inaccurate predictions for training sets comprising 100 inactive and 
 10 active training examples , prediction accuracy was high when 1000 
 inactive and 100 active training compounds were used . are the mean (  ) 
 and standard deviation (  ) of recall of active compounds and 
 balanced accuracy after 50 independent trials for differently composed 
 training sets :  10 active and 100 inactive compounds  
 ( low performance ) and  500 active and 1000 inactive compounds  
 ( high performance ) . results are shown for 10 compound classes , referred 
 by accession no . , according to table 1 . taken together , the results in figure 3 and 4 clearly indicate 
 that the predictions became stable with increasing 
 size of training sets , another key finding . figure 5 reports the variance of recall rates over 
 independent predictions using training sets of increasing size and 
 provides confirmatory evidence . furthermore , levene tests for all 
 activity classes confirmed that the variance of recall distributions 
 significantly differed in 38 of 40 cases ( resulting from 10 compound 
 classes and four training set conditions ) when training sets with 
 at least 50 active and 10 or 1000 inactive examples were used . by 
 contrast , no statistically significant differences in variance of 
 recall rate distributions were detected when the svm models were trained 
 with 100 or 1000 inactive examples , regardless of the number of actives . heat map representations show variance of recall rates over 50 independent 
 trials ( using a two - color gradient ) for training sets of varying composition 
 and size : ( left ) melanocortin receptor 4 ( mc4r ) ligands , ( right ) thrombin 
 inhibitors . herein , 
 we have systematically analyzed the influence of training 
 set composition and size on the prediction accuracy of svm classification 
 models . different from earlier studies , our calculations have stressed 
 the importance of considering class weights and optimizing cost factors 
 when imbalanced training sets are used . furthermore , the ratio of 
 active vs inactive training examples substantially affected the ability 
 of svm models to correctly predict active compounds . however , recall 
 rates and balanced accuracy consistently improved for training sets 
 of increasing size for all compound classes under study . increasing 
 size of training sets also compensated for inherent data imbalance . moreover , large training sets led to robust predictions and the accuracy 
 was essentially constant when different training sets of the same 
 size were used . taken together , our findings have implications for 
 practical applications of svm classifiers . best performing svm models were obtained when 250500 
 active and 5001000 randomly selected inactive training instances 
 were used . moreover , as long as 50 known active compounds 
 are available for training , increasing numbers of 5001000 
 randomly selected negative training examples improve and stabilize 
 model performance when class weights are taken into consideration , 
 which provides a clear guideline for virtual compound screening . finally , we note that large numbers of active compounds may not 
 always be available for training . however , since svm classification 
 and ranking models do not take compound potency as a parameter into 
 account , in contrast to support vector regression , large numbers of 
 hits often obtained from confirmatory screening assays might be readily 
 used for svm model building .