advances and accessibility to technology for capturing and storing large amounts of data has transformed many scientific fields . yet , for many important scientific questions very large data sets are not available . in these cases , when learning algorithms are used , resampling techniques allow the estimation of the generalization error , an important aspect of algorithmic performance .  the generalization error is defined as the error an algorithm makes on cases that the algorithm has never seen before . the generalization error is important because it relates to the algorithm s prediction capabilities on independent data .the literature includes both , theoretical investigations of risk performance of machine learning algorithms as well as numerical comparisons .estimation of the generalization error can be achieved via the use of resampling techniques .the process consists of splitting the available data into a learning or training set and a test set a large number of times and averaging over these repetitions .a very popular resampling technique is cross validation .a detailed overview of the vast literature on cross - validation is outside the scope of this paper .the interested reader is referred to for foundational aspects of cross validation , ( * ? ? ?* ch.s 3,8 ) , , and to for a comprehensive survey .here , it is sufficient to notice that although cross validation procedures are extensively used in practice , their performance is studied by very few theoretical papers .an important question in constructing cross validation estimators of the generalization error is the selection of the size of the training ( and hence test ) set for `` optimal '' estimation of the generalization error . in this paper, we address the question of training sample size selection and refer to work by in identifying the rules for `` optimal '' selection , where `` optimality '' is defined via minimization of the variance of the cross validation estimators .  estimating the generalization error of a learning method is important not only for understanding its future prediction accuracy , but also for choosing a classifier from a given set , or for combining classifiers .furthermore , one may be interested in testing hypotheses about equality of generalization errors between learning algorithms . in , in the field of biostatistics , an inference framework is developed for the difference in errors between two prediction procedures . in that paper , for each splitting in repeated sub - sampling the predictions of the two classifiers are compared by a wilcoxon test , and the resulting -values are combined . state that `` little attention is paid to inference for comparing prediction accuracies of two models or , more generally , prediction procedures '' . carrying out inference on equality of generalization errors requires insight into the variance of the estimator of the generalization error . but providing a variance of the generalization error estimator is a more difficult problem because of the complexity of existing estimators and the many dependencies they exhibit on , for example the loss function , the resampling method used , the ratio of the training to test sample size and other factors . were the first to provide estimators for the variance of the random cross validation estimator of the generalization error .furthermore , proposed a moment approximation - based estimator of the same cross validation estimator of the generalization error , and compared this estimator with those provided by . other relevant work on variance estimation includes .  selecting the size of the training set and understanding the effect of this selection on the generalization error , its bias and its variance , is of interest to many areas of scientific investigation .examples include pattern recognition and machine learning , statistics , remote sensing , biostatistics and bioinformatics among others . in this paper , we offer an analysis of the problem of training sample size selection when the total sample size is fixed and interest centers on carrying out inference on the generalization error .the analysis is complex , and for this reason we consider three kinds of problems to cover a good range of possible applications .these include the case where the decision rule is the sample mean , the regression case and classification via logistic regression .the strategy we follow allows us to draw general conclusions about `` optimal '' selection of the training ( and hance test ) set sample size , and to recommend simple rules for practical use despite the complexity of the problem .we analyze training sample size selection for cross validation estimators of the generalization error .specifically , we study random cross validation estimators and -fold estimators based on general loss functions .we first discuss two novel approaches for obtaining the resampling size of the random cross validation estimator of the generalization error .* , sect .5 ) state `` we also want to make recommendations on the value of to use for those methods that involve '' .we offer methods for obtaining that are based on the .furthermore , we establish rules for the `` optimal '' selection of the training set sample size for cross validation estimators of the generalization error .we show that when the decision rule is the sample average and the loss function is sufficiently smooth and belongs in s -class ( ) , the optimal value of the training sample size is , independent of the data distribution , where indicates the upper integer part of a real number .when the decision rule is regression and the loss function is squared error loss , or the task is classification via logistic regression , the optimal value of the training sample size is again , independent of the data distribution . in general, the optimal value of the training sample size depends primarily on the loss function and secondly on the data distribution .we offer a construction of a new loss function that exemplifies this dependence .additionally , we provide a rule for selecting the `` optimal '' number of folds , in a -fold cross validation experiment .we follow with a discussion of the merits and limitations of different forms of cross validation , and offer recommendations for use in practice .this paper is organized as follows .section [ sec : relevant ] presents relevant previous work and section [ sec : j ] discusses two approaches to obtain the resampling size .section [ sec : training / test ] presents the analysis for optimally choosing the training sample size , while section [ sec : simu ] presents simulation results .section [ sec : real ] applies the methods to real data sets , while section [ sec : conclusions ] offers discussions and recommendations .finally , proofs of the obtained results are presented in the appendix , and also in the online supplement .in this section we review briefly related work and set up the notation we will use in this paper .interest in the problem of `` optimal '' partitioning of a fixed sample in two sets , training and testing , dates back to 1962 . studied this problem in the context of designing a pattern recognition machine and defined the `` optimum '' partitioning of the total sample as that partitioning which minimizes the variance of an unbiased estimator of the error probability of the pattern recognition machine . states that the same rule , that is the minimization of variance of the error of a classification rule , can be used with estimators of the error that can be corrected for bias .similar work has been carried out by .these authors address the problem of splitting a fixed sample into a training , a testing and a validation set using as a criterion the mean squared error .only the squared error loss is studied in the simple case of location parameter estimation .the cross validation procedures they study are hold - out cross validation , -fold and randomized permutation cross validation .the last cross validation procedure is defined by resampling the test set times , , is the total sample size and is the test set sample size .the same problem appears also in the biostatistics literature and in particular in the area of bioinformatics . consider the problem of developing genomic classifiers , empirically address the question of what proportion of the samples should be devoted to the training set , and study the impact that this proportion has on the mean squared error of the prediction accuracy estimate .the context within which address this question is the one where the number of predictors is considerably larger than the number of samples , i.e. it is the high - dimensional , small sample size case . evaluated two approaches to data split .the first consists of using results of simulations that were designed to understand qualitatively the relationships among data set characteristics and optimal split proportions to evaluate commonly used rules of thumb , such as allocating or of the data set to training and the remaining to the test sets .the second approach consists of developing a nonparametric procedure that is based on decomposing the mean squared error into three components , two corresponding to appropriately defined variance terms and one corresponding to a squared bias term .there are fundamental differences between the work of these authors and our work .first , their setting of small , large is different than ours and secondly , even in this setting , they do not address the case of cross validation estimators .  in this paper, we do not address the high - dimensional , small sample size case .we address the allocation of cases to training / test set when the total sample size is fixed and known , reflecting a limited or medium size data with considerably smaller than . in this setting , we study the rules for optimal split of the data for the random cross validation and the -fold cv estimators of the generalization error , which we describe in detail below . but before we do this , we briefly describe two very relevant to our work papers that we use in setting the rules for selecting `` optimally '' the size of training , and hence test , set .  in a series of impressive papers studied the performance of -fold cross validation and of the repeated learning - testing method , which is another name to describe random cross validation . in what follows , we will succinctly describe burman s results .reference to the -fold cross validation ( which burman calls -fold cv ) the following , relevant to our paper , points are made .1 ) in the case of -fold cv with a small number of folds , the uncorrected cv estimate may have large bias ; thus , a corrected estimate is proposed , and it is denoted by ; 2 ) the bias of the corrected -fold cv estimator of the generalization error is _ always smaller _ than the bias of the uncorrected estimator ; 3 ) however , when and ; here , is defined as , is a loss function , is the empirical cumulative distribution function of the sample , and represents the underlying true and unknown distribution ; 4 ) the bias and variance of the -fold cv estimator of the generalization error decrease as the number of folds increases .the last two results indicate that for the case of -fold cv estimator of the generalization error it is sufficient to study only the variance of the uncorrected estimator as opposed to the mean squared error of the uncorrected estimator .this is because the variance of the corrected , almost unbiased estimator , is approximately the same with the variance of the uncorrected estimator .this observation greatly simplifies the analysis of obtaining `` optimal '' splits of the sample .  for the case of repeated learning - testing method , or in our terminology random cross validation , the following points are relevant to the work presented here .1 ) a bias correction term is needed to obtain an almost unbiased estimate of the generalization error .this term is provided in ; 2 ) the variances of the corrected and uncorrected estimators decrease as the number of repeats increases . presents a simulation study where the two variances are approximately equal when the number of repetitions and the total sample size is for all values of the proportion of samples allocated to the test set ; 3 ) the bias does not depend on the number of repeats ; 4 ) the only way to reduce variance is by increasing the number of repeats .we provide rules that allow `` optimal '' selection of the number of repeats in the random cross validation case .the relevance of these results for our work is that again we can use the variance of the uncorrected random cv estimate of the generalization error , instead of the more complicated mean squared error , to set `` optimal '' rules for data splitting into training and test sets .this is because the variance of the bias - corrected cross validation estimator is approximately the same with the variance of the uncorrected cross validation estimator .next we set the notation we use in this paper and explicitly discuss the two estimators of the generalization error we work with .fix a positive integer and consider the set .let , be data collected such that the data universe , , is a set of independent , identically distributed observations that follow an unknown distribution .let be a subset of size , , taken from , , the complement of with respect to .the subset of observations is called a training set , used for constructing a learning rule .the test set contains all data that do not belong in ; it is defined as , the complement of with respect to the data universe .let denote the number of elements in the test set , , .now let be a function , assume that is a target variable , and is a decision rule .the function that measures the error between the target variable and the decision rule is called a loss function .examples of loss functions widely used in the literature , and in the present paper , include the squared error loss function , absolute error loss , and 0/1 loss function used in classification .the _ generalization error _ of an algorithm is defined as ,\ ] ] where is an independent copy of the data and the expectation is taken over everything that is random .that is , we take into account the variability in both , training and test set .furthermore , let , , be random sets such that where ; , , are _ uniformly distributed _ on , such that each is independently sampled , with corresponding complement set ( with respect ) .the number is called the _ resampling size_. if is defined similarly as above for all , the usual _ average test set error _ is and is a function of both the training set and the test set . defined the _random cross validation estimator of the generalization error _ as by a straightforward calculation , it follows that   in the cases where the prediction rule takes real values , presented a wide class of loss functions , called -class . specifically , if is an absolutely continuous and concave real function , the corresponding -class loss function is where is the almost sure derivative of the _ generator _ .commonly used loss functions , such as squared error loss and 0/1 loss belong to this class . in section [ sec: training / test ] we use the -class of loss functions to elucidate the training sample size selection rules .we now provide guidance for the selection of the resampling size entering the construction of the random cross validation estimator of the generalization error . in subsection [ ssec : res - eff ]we discuss two novel methods for selecting the resampling size , the -effectiveness and the -reduction methods . here , we note that the -effectiveness and -reduction , and hence the selection of , are affected by the training set sample size .we illustrate these relationships in section [ sec : simu ] .we first improve upon the following result given in . under the assumption that `` the distribution of does not depend on the particular realization of , '' that the quantities and in do not depend on the indices and ( see , also , * ? ? ?* ) . in what followswe first prove that the aforementioned statement is generally true , that is this assumption is not needed for the result to hold .this is shown in proposition [ prop.v , c ] of this section .[ prop.v , c ] let and be fixed , and let be a loss function such that  ] gives <\infty ] , and observe that the random vector is uniformly distributed on .using the same arguments as above the proof is completed .  proposition [ prop.v , c ] allows one to write , , effectively obtaining that is , the variance of the random cross validation estimator of the generalization error is a function of the variance of the average test set error and the covariance between two different average test set errors , which are constants with respect to , .figure [ fig.j , var(mu_j ) ] shows the behavior of as a function of . by cauchy - schwarz inequalitywe obtain that , where the equality characterizes the trivial cases in which is a constant with probability 1 , e.g. , if is degenerate .so ( see figure [ fig.j , var(mu_j ) ] ) , and again the equality characterizes the same trivial cases .  from andwe obtain the following important result .[ theo.var_bound ] the variance of given by satisfies the following double inequality : where and .  using cauchy schwarz inequality and we get , where . also , gives .finally , for each fixed , the function , , is increasing , and in view of the proof is completed .  the importance of the theorem will become obvious when we define appropriate rules for selecting the sample size of the training set . since the variance of the cv estimator of the generalization error is bounded above by the variance of the test error , we can use to create an `` optimal '' rule for selecting the training ( and hence the test ) sample size .we discuss two novel methods for selecting the resampling size .the quantities and in depend on the sizes of the training and test set , and , as well as on and . recall that an experimentalist selects that value of for which there is no appreciable reduction in the variance of cv estimator of the generalization error .taking into account the behavior of we give the following definition .[ defi.re,rr ] we define :  \(a ) the _ resampling effectiveness _ of by .\(b ) the _ reduction ratio _ of by , where denote the backward difference with respect to .  notice that while provides a comparison of the variance of for a given with the limiting variance , the provides a comparison between the two variances for given and . in this sense is a local measure of change in variance that is used to obtain .this is what experimentalists are observing in order to set the value of ..48  for and ; ( b ) the reduction ratio of for and .,title="fig : " ]  .48  for and ; ( b ) the reduction ratio of for and .,title="fig : " ]  now we are in a position to give a minimum acceptable resampling size , either via the resampling effectiveness of or via its reduction ratio .observe that in non - trivial cases the resampling effectiveness of takes a value in the interval ; for the trivial cases this is assumed to be 1 . fromit follows immediately that where .figure [ sfig.reseff ] shows the behavior of the resampling effectiveness of for various -values .notice that when the correlation is low a larger resampling size is required in order to achieve high resampling effectiveness . as the correlation value increases the resampling size decreases ; for example , when , is sufficient to obtain a resampling effectiveness of 90% , while when , obtains the same resampling effectiveness .if is the desired resampling effectiveness rate , then , and the minimum value of the resampling number is where denotes the upper integer part of .we call the number as the _ -effectiveness minimum resampling size_.  note that , we define the resampling effectiveness ( and thus select that value of ) as a number indicating percentage of the ratio , which is between 0 and 1 .the bigger the ratio , the closer the is to the asymptotic value of the variance of the cv estimator of the generalization error .that indicates that the value of should be close to 1 , i.e. .8 , .9 , .95 etc .an alternative way to obtain is as follows . fromwe obtain we then fix the desired reduction ratio to a value and require that the reduction ratio of should not exceed this value , that is , .the minimum positive integer satisfying the preceding inequality is given by which we call the _-reduction ratio minimum resampling size_. we define the reduction ratio ( and thus select that value of ) as a number indicating the relative reduction of the variance of the cv estimator , which is a positive number .the smaller this ratio , the closer is to the asymptotic value of the variance of the cv estimator of the generalization error .that indicates that the value of should be close to 0 , i.e. .1 , .05 , .025 , .01 etc .figure [ sfig.redratio ] shows the reduction ratio curves as a function of the correlation .again , the smaller the value of the correlation between , , the larger the value of to obtain a desired variance reduction ratio .table [ table.jre,jrr ] contains the specific values of and for various values of , and .notice that large values of are required when the correlation is small and the desired resampling effectiveness and reduction ratio are respectively large or small .furthermore , the smaller the reduction ratio the larger the value of , for any given correlation . that is , when and the desired , one obtains .this selection corresponds to resampling effectiveness of approximately .[ rem.est-v,c ] equations , depend on that is generally unknown . this parameter needs to be estimated from the data . propose an approximation of the correlation coefficient given by , where is the cardinality of the test set .this estimator , however , may overestimate or underestimate the true correlation . suggested moment approximation estimators of and in certain cases that may be used to estimate the correlation , and hence obtain the values and .additionally , if the loss function belongs in efron s -class of loss functions , then the parameters can be easily estimated .see sub - subsection [ sssec : q - class ] for details .a point that relates to the design of the random cross validation estimator of the generalization error is the selection of the training set .practitioners select the training sample size to be 5 or even 10 times larger than the test size .our results indicate that when `` optimality '' is quantified via minimization of the variance of the average test set error , the training sample size , for fixed value of , equals the test set sample size .in what follows , we present and justify our choice of the optimality rule and apply it in subsections [ ssec : mean ] and [ ssec : regression ] .relation leads to the following two potential rules .the first rule , suggesting minimization of the limiting variance , is not useful because , as we will see in section [ sec : simu ] , is a constant .the second rule suggests that minimization of the is equivalent to minimizing the variance of ( this is equivalent to minimizing ) . in light of theorem [ theo.var_bound ], the variance of is bounded above by and places a small upper bound on the limiting variance .these observations lead us to select quantifying `` optimality '' via this rule . therefore , for fixed , we define the _ optimal value of the training set size _ by   the situation is different for the -fold cv . because in -fold cv there is dependence between , and , for fixed , the aforementioned optimization rules are not useful .we propose selecting the optimal value of as   to be able to work with the optimization rules given in , we need either the exact or an approximate form , of the variances involved . in what followswe consider three kinds of problems to cover a good range of possible applications .these correspond to using the sample mean or linear regression ( with and without normality of the errors ) as decision rules and classification via logistic regression .we begin by studying the simple case where the decision rule is the sample mean .the variance formulas we use in our rules , and the conditions under which they are obtained , are given in . here , for reasons of completeness, we briefly summarize the results we use .let and .3.1 ) gave the following moment approximations of , +o(\sfrac{1}{n_1 ^ 2}),\ ] ] where , \qquad\qquad ~\beta\doteq{\operatorname{\mathsf{var}}}[l_\mu(x)],\\ \gamma&\doteq\sigma^2{\operatorname{\mathsf{var}}}[l'_\mu(x ) ] , \qquad\qquad \delta\doteq\sigma^2{\operatorname{\mathsf{cov}}}[l_\mu(x),l''_\mu(x ) ] , \end{split}\ ] ] with , .  note that , since , the notation has the same meaning with the notation , which will be used in the sequel .in random cv each of and follows a hypergeometric distribution , see ( * ? ? ?* lemmas 3.1 , 3.2 ) .specifically , and ; thus , the approximated covariance , , in this case is in -fold case and are constants ; specifically ,    to solve the optimization problem we consider the expression for the variance of given by .when is sufficiently large , the term is a negligible quantity , and using the fact that , and the identity =1/n_1 + 1/(n - n_1) ] .table [ table.re.sq.norm.k-fold ] shows the specific values of this relative efficiency for various values of and , indicating that the relative efficiency is a function of both , sample size and number of folds .it approaches 1 when and .the results of sections [ sssec : mean.usual ] and [ sssec : mean.k - fold ] apply to the -class of loss functions .we have where is differentiable having at least up to five derivatives , and such that the fourth derivative of is bounded .the main gain is that the quantities , , and ( therefore and too ) can be computed easily .observe that hence , ^ 2\sigma^2+{\operatorname{\mathsf{var}}}[q(x)]-2q'(\mu){\operatorname{\mathsf{cov}}}[x , q(x)],\\ \gamma=[q''(\mu)]^2\sigma^4 , & \delta = q'(\mu)q'''(\mu)\sigma^4-q'''(\mu){\operatorname{\mathsf{cov}}}[x , q(x)]\sigma^2 . \end{array}\ ] ]  [ rem.est-a,b,c,d-q ] the function is a known function , but the parameters are unknown ( unless the distribution is known ) .we estimate these parameters from the sample values by replacing the parameters , ,  ] by , , and  ] , and is the -th diagonal element of the hat matrix .since }{(n-1)n_1 ^ 2n_2}=o(\sfrac{1}{n^2}) ] which implies that .\(c ) using the same arguments as in proof of theorem [ theo.opt-n_1 ] the optimal value of in follows .  [ rem.opt ] similarly to the case where the desision rule was , for the optimal value of , and , hence , .therefore , the resampling size can be chosen either via specification of the resampling effectiveness or specification of the reduction ratio of , for a given or , by the following relations [ see , ] , cf .table [ table.jre,jrr ] ,   [ rem.regr_general ] the above results can be extended to a subclass of the -loss functions class ; this subclass contains differentiable functions that can be expressed as functions of the errors .   to illustrate the difficulty of obtaining a close form solution for the training set sample size in general , when other than the squared error loss functions are used , we discuss the case of classification via logistic regression . formalize logistic regression as a linear classification method , where is the label of the data and is a feature vector .the loss function here is the logistic loss given as , where is a label and is the algorithm prediction , such that   in this case the minimization problem does not have a closed - form solution for obtaining the optimal value of , and the problem is reduced to numerical optimization .we offer the following algorithm in order to obtain , numerically , the optimal value of the training set sample size .  in view of ,first we show the following general result .[ theo.var(mu_j ) ] let be a loss function , an index set of size ( ) and .if the asymptotic values of  ] do not depend on the particular realization of , say and respectively , then , the asymptotic value of and are \right\}. \end{split}\ ] ]  see online appendix .  theorem [ theo.var(mu_j ) ] provides expressions for the expected value and variance of that enter the specification of the optimality rule .compute the matrix and via logistic regression estimate the parameters and , say and respectively , using the entire sample .compute the following probabilities and quantities   for   1 .compute the quantities , and of theorem [ theo.var(mu_j ) ] 2 .  using theorem [ theo.var(mu_j ) ] ,compute .set .given and standardize the components of in lemma [ lem.w ] as , and consider the random vector .then , also , define   the decision rule for classification is then given as follows . for a training set , from the model we estimate the probability for each ,say its estimator .after , we estimate the value as the loss function is ( see * ? ? ? * ) .we compute the values ={\operatorname{\mathds{e}}}[l^2_{0/1}({\widehat{y}}_{s_j , i},y_i)|s_j , i] ] , see in online appendix , which are given as where is the cumulative distribution function of standard normal distribution and is the cumulative distribution function of .theorem [ theo.var(mu_j ) ] gives a formula to calculate the variance of .but , the minimization process of does not give a close form for the optimal value of , it is reduced to a numerical minimization process .computing the quantities , and , via theorem [ theo.var(mu_j ) ] and , we calculate the variance of for . the value of which gives the minimum value of the variance is the optimal choice of .  the parameters , are unknown and we estimate those before the minimization process begins using the entire data set .we also compute as .in this section we present simulation results using a variety of distributions and loss functions with the goal of illustrating empirically our theoretical results .we discuss the empirical performance of our rules organizing the presentation according to the cases studied above , and we note that general simple recommendations about optimal selection of training sample size are possible .   using the version of , _ri386 3.1.2 _ on a _ dell latitude e7240 _ pc we simulated samples of size from distributions that can be categorized into symmetric with a variety of tail behaviors ( normal , , , ) and asymmetric ( , log - normal , pareto , pareto ) .the normal distribution is central in statistics , while the two -distributions exhibit heavier than the normal , tail behavior .the log - normal distribution is used in biostatistics in biomarker studies , while the pareto distribution is a power law distribution used in the description of social , geophysical , scientific , actuarial and many other observable phenomena .the selection of and pareto(6 ) distributions is not arbitrary .both the and pareto(6 ) distributions possess less than six moments , and for our theory to apply we require the existence of up to six moments .thus , these selections reflect the performance of the methods in limit cases . to illustrate the effect of the choice of loss function has on the size of training set we use the loss functions presented in table [ table.loss ] .our results indicate that for the -class of loss functions the optimal training sample size is , independent of the data distribution .however , one can construct loss functions , such as the modified squared error loss given in table [ table.loss ] , for which the optimal training sample size is not .notice that , the modified squared error loss functions does not belong in the -class .tables [ table.sq][table.dousq ] ( and table [ table.abs ] included in the online appendix ) present the estimated and theoretical values of , and when the decision rule is the sample mean for aforementioned distributions using the loss functions presented in table [ table.loss ] .the estimated values of , and hence the estimated proportion , are obtained by estimating the values of the parameters , of theorem [ theo.opt-n_1 ] , using the data .similarly , to estimate the value of , and thus , we estimate , using the data , the values of , of proposition [ prop.opt-k ] ( see also ) . because and are functions of the total sample size , tables [ table.sq][table.dousq ] present these estimated proportions along with their mean squared error . therefore , if then , and if then , a value that corresponds to loocv .table [ table.sq ] presents the estimated and , in parenthesis , the theoretical value of , the estimated value of , and the estimated from the data value of .the second line , for each sample size , reports the mean squared error ( mse ) .the loss function here is squared error loss , which belongs in the -class of loss functions .the results of table [ table.sq ] indicate that , and corresponds to loocv for all distributions .furthermore , the estimator of the correlation coefficient is highly accurate .similarly , tables [ table.efron ] and [ table.abs ] ( see online appendix ) present simulation results when the loss functions are a -loss with generator and the approximated absolute error loss . the results of these tables indicate that and corresponds again to loocv .the approximated absolute error loss does not belong in the -class of loss functions ; we take .we now contrast the above results with those presented in tables [ table.modsq ] and [ table.dousq ] . tables [ table.modsq ] , [ table.dousq ] present simulation results using the sample mean as the decision rule , for a variety of distributions and sample sizes indicated in the tables .note that the minimum divisor of sample sizes 60 , 100 , 750 and 5000 is 2 , while the minimum divisor for sample size 301 is 7 and of sample size 1501 is 19 .tables [ table.modsq ] , [ table.dousq ] exemplify clearly the interaction between the loss function and the data distributions , with the selection of the optimal training sample size and the selection of the optimal number of folds in -fold cv .table [ table.modsq ] presents the proportion of the sample size allocated to the training set . while distributions such as or log - normal select , other distributions such as or require to be approximately 80% of the total sample size . on the other hand , in all cases the modified squared error loss function ,offers , the loocv .table [ table.dousq ] presents analogous results for the double squared error loss .the results indicate the impact of the loss function and data distribution on the selection of .note that , our limit cases here are represented by and pareto(9 ) distributions ( recall that we require the existence of at least eight moments ) .note that all distributions , with the exception of the pareto(15 ) and pareto(9 ) select as , for all sample sizes used .the pareto distributions select , indicating optimality for the loocv .the double squared error loss does not belong in the -class of loss functions .the comparison with the results presented , for example in tables [ table.sq ] and [ table.efron ] , where the loss functions were members of the -class , clearly indicates the impact of the loss function on the optimal sample size selection for the training and hence the test set .when cross validation estimators of the generalization error are used the usual recommendation made is to use 70% - 80% of the data for training and the remaining for testing .figures [ fig.sq ] and [ fig.modsq ] plot the variance of the test set error as a function of the sample splitting for squared error and modified squared error losses .the graphs show that the test set variance is minimized for the optimal value of .on the other hand , figure [ fig.n/2 ] plots the relative efficiency of against the ratio , for ( it corresponds to ) , and .the relative efficiency of is defined as the ratio of the variance of for any given data splitting over the variance of at the optimal data splitting .figure [ fig.n/2 ] clearly shows that , for all sample sizes , all loss functions with , and all distributions , the popular recommendations in terms of splitting exhibit substantial increase in the variance of .for example , when 75% of the total sample size is used for training , leaving 25% for testing , the increase in the variance of when squared error loss is used and is 17.3% , while when 80% ( or 90% ) of the total sample is used for training this increase is 24.4% ( or 88.6% ) . andif , then the increase in variance is 11.86% when the training set is 75% of the sample ; it is 16.8% ( or 61.2% ) when 80% ( or 90% ) of the sample is used for training .notice that is almost unaffected by the splitting mechanism across all loss functions , independently of data distribution .this provides further justification of our selection of the optimality rule .this indicates that is equivalent to minimizing and to minimizing for all ( see also relation ) .furthermore , when ( both theoretically and empirically ) , so that , and the limited variance is almost unaffected by the splitting . in view of figure [ fig.n/2 ] , , while , and , where denotes the variance of the test set error for training sample size .the notations and have similar interpretation .thus , , , , , and .the relationship of these results to the resampling effectiveness and reduction ratio of are shown in table [ table.re,rr.for_n/2 ] for , and when the training sample size is ( ) , , , and .these results show that when the training sample size in not optimal then we need to increase the resampling size to obtain acceptable levels of reduction ratio and/or resampling effectiveness , thereby increasing the computational burden of the procedure .  1  1  1  1  1  1  1  1  1   we empirically now study the variance of generalization error in the cases of linear regression and classification via logistic regression .date were generated as , , where the sample size , , and and the error vector follows distribution .the four covariates were generated only once as follows .the first covariate is a binary variable generated from distribution , the second variable is generated from distribution , the third variable is generated from the distribution and variable is generated from the distribution .the number of monte carlo repetitions equals 5000 .table [ table.sqregr ] and figure [ fig.sqreg ] present the results ; notice that , both the table and figure , indicate an increase in the variance of the test set error , , as well as in with , when the training sample size moves away from the optimal value of .this is true for all sample sizes .table [ table.zeroone ] presents relevant results for classification via logistic regression .the model contains three covariates , is generated once as , is and is generated from discrete uniform on the set .the vector of errors is again , and we generate 50 monte carlo samples . table [table.zeroone ] presents estimates of the for two values of the sample size , 60 and 100 , and for different values of the training sample size .again , increases if the popular choices of , or , are used , and is minimized when , and this holds for all sample sizes and error distributions .  1  1  1in what follows , we briefly describe the two data sets we use to illustrate our results .one is a data set on birth weight and the second is a part of the data set from the _ national health and nutrition examination survey _ ( nhanes iii ) conducted by the _ national center for health statistics _ ( nchs ) between 1988 and 1994 ( see * ? ? ?low birth weight , defined as a birth weight less than 2500 grams , is an outcome that has been of concern to physicians for years , because of its association between high infant mortality and birth defect rates .data were collected as part of a large study at baystate medical center in springfield , ma .the outcome variable was a binary variable taking values 0 if the weight of a baby at birth is greater than or equal to 2500 gr and 1 if it is less than 2500 gr , i.e. it is actual low birth weight .measurements on 10 , additional to the outcome , variables were collected on 189 subjects .out of 189 subjects 59 were observed as low birth weight and 130 as normal birth weight , so the class proportions were 31.22% low birth weight and 68.78% normal birth weight . because the race variable has three categories , white , black and other , it was coded using two design variables defined as follows .the first design variable takes the value 1 when the race is `` black '' and 0 otherwise , while the second design variable takes the value 1 when the race is `` other '' and 0 when it is `` white '' or `` black '' .the other variable we use was mother s weight at the last menstrual period , and the logistic regression model we use contains a constant term ( see * ? ? ? * ) .the second data set has a binary outcome variable and five covariates .the binary outcome variable takes values 0 if the average systolic blood pressure is less than or equal 140 and is 1 if it is greater than 140 .data were collected via physical examinations and clinical and laboratory tests , and only adults defined as 20 years of age and older were included in the survey .we selected a sample of size 1,000 with complete observations on the covariates representing age ( in years ) , sex ( male or female ) , race ( white , black , other ) , body weight ( in pounds ) and standing height ( in inches ) .because the race variable has three categories , it was coded using the same design variables , as in the previous example . applying our proposed algorithm [ algorithm ] ( for classification via logistic regression ) , we obtain that as the size of the training sample increases so does the variance of the test set error .we obtain minimum variance when .in this paper we address the problem of `` optimal '' selection of the size of training sets when interest centers in making inferences about the generalization error of prediction algorithms .we study two types of cross validation estimators of the generalization error .these are random cross validation ( or repeated learning - testing method ) and -fold cross validation estimators . the statistical rule that defines `` optimality '' , in light in proposition [ prop.v , c ] and theorem [ theo.var_bound ] ,calls for the minimization of the variance of the test set error .our results indicate that the optimal training sample size selection is a complex problem and it depends primarily on the loss function that is used , as well as on the data distribution .describe the complexity of the analysis , simple general rules for practical use can be drawn .when the loss function belongs to the -class then , for the case of random cross validation , ( for both , sample mean and regression decision rules ) .when the loss function does not belong in the -class then the value of may or may not equal , indicating that -class membership is a sufficient condition for , but not a necessary condition .we present cases where the loss function does not belong in the -class , yet ( see , for example , table [ table.abs ] ) .furthermore , we illustrate the effects of complex interactions between the loss function a researcher chooses and the data distribution on the optimal training sample size selection in tables [ table.modsq ] .  furthermore , we studied the effect that popular training set sample size selection , such as or , has on the and , .we found out that as the training sample size increases away from its optimal value the aforementioned variances increase substantially . to decrease those we need to increase the resampling size from 15 to a value that achieves acceptable reduction ratio and/or resampling effectiveness , thereby increasing the computational cost .the selection of the resampling size of a random cross validation estimator of the generalization error is important , as it contributes to the variance reduction of this estimator .we propose two methods of selecting and exemplify their use .our analysis indicates that , when the correlation between two different test sets is moderate , i.e. in range of  ] , we have that   [ [ normal - error - distribution ] ] normal error distribution + + + + + + + + + + + + + + + + + + + + + + + + +  assume the errors in the linear regression model are normally distributed , that is .let , be the ordinary least square ( ols ) estimators computed on the training sets , .it is well known that ( and similarly for the index ) , with and ( and similarly for the index ) are defined above as the product of and .this allows one to write and .let the bivariate random vector then , as becomes large and the moments of are approximated by the associate moments of the approximated distribution .we offer a lemma that is fundamental in proving the results presented in this paper .[ lem.indices ] let and be two index sets of size ( and as in random cv ; and let us consider the fixed indices . then ,  \(a ) .\(b ) and .\(c ) .\(d ) and .\(e ) .\(f ) .\(g ) .\(h ) .\(i ) , .\(j ) }{n^2(n-1)^2(n-2)} ] , =\frac{1}{nn_2}\big\{\sum_{i=1}^n{\mathsf{e}}_{i , i}+\frac{2(n_2 - 1)}{n-1}\mathop{\sum\sum}_{1\le i < i'\le n}{\mathsf{e}}_{i , i'}\big\} ] ,  ] . applying theorem [ theo.var(mu_j ) ] , .for the variance of we write ; and so , also , the quantity  ] is ; and using ,   now observe that and .thus , noting that , since and are given , the indicator functions are constants . therefore , in view of and using theorem [ theo.isserlis ] , we obtain the following . if , since and , and if , , and write as , where , and . using lemma [ lem.indices ] , & = \sigma^4\sum_{r=0}^{2}a_{r;i}{\operatorname{\mathds{e}}}[{\operatorname{card}}^{r}(s_j\cap s_{j'}){\mathds{1}_{\{i\in s_j^c\cap s_{j'}^c\}}}]\\ & = \sigma^4\left\{\frac{n_2 ^ 2}{n^2}a_{0;i}+\frac{n_1 ^ 2n_2 ^ 2}{n^2(n-1)}a_{1;i}+\frac{n_1 ^ 2n_2 ^ 2(n_1(n_1 - 1)+n_2 - 1)}{n^2(n-1)(n-2)}a_{2;i}\right\}. \end{split}\ ] ] since , and , after some algebra , \\ = & ~\sigma^4\left\{\frac{3n_2 ^ 2}{n}+\frac{2n_2 ^ 2(n+2n_1 - 1)}{n^2(n-1)n_1}c_1+\frac{n_2 ^ 2[n(n-1)+2n_1(n_1 - 2)]}{n^2(n-1)(n-2)n_1 ^ 2}c_2\right\}. \end{split}\ ] ] now write , where and .denote and observe that \\  &  + \frac{2}{n_1 ^ 2}{\mathds{1}_{\{i\in s_j^c , i'\in s_{j'}^c , i'\in s_{j},i\in s_{j'}\ } } }  + \frac{1}{n_1 ^ 2}[{\mathds{1}_{\{i\in s_j^c , i'\in s_{j'}^c , i'\in s_{j}\}}}+{\mathds{1}_{\{i\in s_j^c , i'\in s_{j'}^c , i\in s_{j'}\ } } } ] .\end{split}\ ] ] from lemma [ lem.indices ] , after some algebra , we get }{n^2(n-1)^2(n-2)n_1 ^ 2}.\ ] ] hence ,  = \sigma^4\left\{b_{0;i , i'}{\operatorname{\mathds{e}}}({\mathds{1}_{\{i\in s_j^c , i'\in s_{j'}^c\}}})+b_{1;i , i'}{\operatorname{\mathds{e}}}(\vargamma_{s_j , s_{j'};i , i'})\right\}\\ & = \sigma^4\left\{\frac{n_2 ^ 2}{n^2 } b_{0;i , i ' }          + \frac{n_2 ^ 2[(n-2)(n+n_1 ^ 2 + 2n_1n_2 - 1)-(n_1 - 1)^2]}{n^2(n-1)^2(n-2)n_1 ^ 2}b_{1;i , i'}\right\}. \end{split}\ ] ] since and , = \sigma^4 \left\ { \frac{(n-1)n_2 ^ 2}{n } + \frac{2(n-1)n_2 ^ 2}{n^2n_1}c_1\right.\qquad\qquad\\ \left.+\frac{n_2 ^ 2[(n-2)(n+n_1 ^ 2 + 2n_1n_2 - 1)-(n_1 - 1)^2]}{n^2(n-1)^2(n-2)n_1 ^ 4}c_3 + \frac{n_2 ^ 2}{n^2n_1 ^ 2}c_4 \right\}. \end{split}\ ] ] thus , by  ] ; and so ,  ] . by the fact that the asymptotic expected values are , , , and , we get .\ ] ] applying theorem [ theo.var(mu_j ) ] and using , the form of the variance of follows .note that .proof of using the asymptotic distribution of , the expected values are : \\ = & { \operatorname{\mathds{p}}}(l_{0/1}({\widehat{y}}_{s_j , i},y_i)=1|s_j , i )  = { \operatorname{\mathds{p}}}({\widehat{y}}_{s_j , i}=0,y_i=1|s_j , i)+{\operatorname{\mathds{p}}}({\widehat{y}}_{s_j , i}=1,y_i=0|s_j , i)\\ = & { \operatorname{\mathds{p}}}(\varpsi_i<-\sqrt{n_1}\zeta_i){\operatorname{\mathds{p}}}(y_i=1 )  + { \operatorname{\mathds{p}}}(\varpsi_i\ge-\sqrt{n_1}\zeta_i){\operatorname{\mathds{p}}}(y_i=0)\\ = & \varphi(-\sqrt{n_1}\zeta_i)p_i+\varphi(\sqrt{n_1}\zeta_i)(1-p_i ) , \end{split}\ ] ] \\ = & { \operatorname{\mathds{p}}}(l_{0/1}({\widehat{y}}_{s_j , i},y_i)l_{0/1}({\widehat{y}}_{s_j,{i'}},y_{i'})=1|s_j , i , i')\\ = & { \operatorname{\mathds{p}}}({\widehat{y}}_{s_j , i}=0,y_i=1,{\widehat{y}}_{s_j , i'}=0,y_{i'}=1|s_j , i , i ' ) \\  & + { \operatorname{\mathds{p}}}({\widehat{y}}_{s_j , i}=0,y_i=1,{\widehat{y}}_{s_j , i'}=1,y_{i'}=0|s_j , i , i')\\  & + { \operatorname{\mathds{p}}}({\widehat{y}}_{s_j , i}=1,y_i=0,{\widehat{y}}_{s_j , i'}=0,y_{i'}=1|s_j , i , i')\\  & + { \operatorname{\mathds{p}}}({\widehat{y}}_{s_j , i}=1,y_i=0,{\widehat{y}}_{s_j , i'}=1,y_{i'}=0|s_j , i , i')\\ = & { \operatorname{\mathds{p}}}(\varpsi_i<-\sqrt{n_1}\zeta_i,\varpsi_{i'}<-\sqrt{n_1}\zeta_{i'}){\operatorname{\mathds{p}}}(y_i=1){\operatorname{\mathds{p}}}(y_{i'}=1)\\  & + { \operatorname{\mathds{p}}}(\varpsi_i<-\sqrt{n_1}\zeta_i,\varpsi_{i'}\ge-\sqrt{n_1}\zeta_{i'}){\operatorname{\mathds{p}}}(y_i=1){\operatorname{\mathds{p}}}(y_{i'}=0)\\  & + { \operatorname{\mathds{p}}}(\varpsi_i\ge-\sqrt{n_1}\zeta_i,\varpsi_{i'}<-\sqrt{n_1}\zeta_{i'}){\operatorname{\mathds{p}}}(y_i=0){\operatorname{\mathds{p}}}(y_{i'}=1)\\  & + { \operatorname{\mathds{p}}}(\varpsi_i\ge-\sqrt{n_1}\zeta_i,\varpsi_{i'}\ge-\sqrt{n_1}\zeta_{i'}){\operatorname{\mathds{p}}}(y_i=0){\operatorname{\mathds{p}}}(y_{i'}=0)\\ = & { \operatorname{\mathds{p}}}(\varpsi_i<-\sqrt{n_1}\zeta_i,\varpsi_{i'}<-\sqrt{n_1}\zeta_{i'}){\operatorname{\mathds{p}}}(y_i=1){\operatorname{\mathds{p}}}(y_{i'}=1)\\  & + { \operatorname{\mathds{p}}}(\varpsi_i<-\sqrt{n_1}\zeta_i,-\varpsi_{i'}\le\sqrt{n_1}\zeta_{i'}){\operatorname{\mathds{p}}}(y_i=1){\operatorname{\mathds{p}}}(y_{i'}=0)\\  & + { \operatorname{\mathds{p}}}(-\varpsi_i\le\sqrt{n_1}\zeta_i,\varpsi_{i'}<-\sqrt{n_1}\zeta_{i'}){\operatorname{\mathds{p}}}(y_i=0){\operatorname{\mathds{p}}}(y_{i'}=1)\\  & + { \operatorname{\mathds{p}}}(-\varpsi_i\le\sqrt{n_1}\zeta_i,-\varpsi_{i'}\le\sqrt{n_1}\zeta_{i'}){\operatorname{\mathds{p}}}(y_i=0){\operatorname{\mathds{p}}}(y_{i'}=0)\\ = & ~f_{\varpsi_i,\varpsi_{i'}}(-\sqrt{n_1}\zeta_i,-\sqrt{n_1}\zeta_{i'})p_ip_{i ' }  + f_{\varpsi_i,-\varpsi_{i'}}(-\sqrt{n_1}\zeta_i,\sqrt{n_1}\zeta_{i'})p_i(1-p_{i'})\\ & + f_{-\varpsi_i,\varpsi_{i'}}(\sqrt{n_1}\zeta_i,-\sqrt{n_1}\zeta_{i'})(1-p_i)p_{i ' }  + f_{-\varpsi_i,-\varpsi_{i'}}(\sqrt{n_1}\zeta_i,\sqrt{n_1}\zeta_{i'})(1-p_i)(1-p_{i ' } ) , \end{split}\ ] ] completing the proof .

Abstract: an important question in constructing cross validation ( cv ) estimators of the generalization error is whether rules can be established that allow `` optimal '' selection of the size of the training set , for fixed sample size . we define the _ resampling effectiveness _ of random cv estimators of the generalization error as the ratio of the limiting value of the variance of the cv estimator over the estimated from the data variance . the variance and the covariance of different average test set errors are independent of their indices , thus , the resampling effectiveness depends on the correlation and the number of repetitions used in the random cv estimator . we discuss statistical rules to define optimality and obtain the `` optimal '' training sample size as the solution of an appropriately formulated optimization problem . we show that in a broad class of loss functions the optimal training size equals half of the total sample size , independently of the data distribution . we optimally select the number of folds in -fold cross validation and offer a computational procedure for obtaining the optimal splitting in the case of classification ( via logistic regression ) . we substantiate our claims both , theoretically and empirically .   example.eps gsave newpath 20 20 moveto 20 220 lineto 220 220 lineto 220 20 lineto closepath 2 setlinewidth gsave .4 setgray fill grestore stroke grestore


complex systems and phenomena are dealt with in many scientific domains . according to the domain of interest , different definitions of complexity and of the way of measuring ithave been proposed and are continuously being proposed since the science of complexity is still fast growing .the literature on this topic is so vast that any attempt at providing an exaustive bibliography would be here out of place and a very hard task . as a consequence , instead of trying to list them all ,let us notice that the many ways of measuring complexity belong to a restricted number of categories .in particular , the attempts at quantifying the degree of organization of a complex system often resort to some definition of an entropy function stemming from the archetype " represented by shannon s information entropy .the latter has its precursor - at least from the point of view of physics - in boltzmann s entropy of kinetic theory .in fact , shannon s information entropy is equivalent to negative boltzmann entropy , as it was proved by l. brillouin .  among the different statistical - mechanical approaches to networks hitherto proposed ,one of these is the class of models with hidden variables ; here , the approach starts with a set of independent nodes and a general hidden variable ; then an undirected network is generated by : ( i ) assigning to each node a variable , independently drawn from the probability ; ( ii ) creating for each pair of vertices and , with respective hidden variables and , an undirected link with probability .so , given the independent assignment of hidden variables and links among nodes , correlated random networks are generated without neither loops nor multiple links , where the degree distribution and the correlation properties of the network are encoded in the two functions and . in the present work ,we consider random variables as arbitrary hidden variables sitting on the nodes , and their correlations are seen as weighted links among the nodes , again .the difference from the previous approach consists of focusing the attention on the knowledge of some parameters characterizing the hidden variables .all the informations about the system are retained in these parameters . in particular ,given the information on the variances and covariances of the multiple hidden variables , a multivariate gaussian probability distribution can be derived to describe the whole given network , by means of the maximum entropy principle .thus a parameter - space is associated with any given network .this space encodes all the information about the structure of the associated network .notice that a similar way of associating a probability distribution to a network , is that of probabilistic graphs models .actually gaussian networks are extensively used in many applications ranging from neural networks , to wireless communication , from proteins to electronic circuits , and so on .then , by resorting to information geometry , the space of the accessible values of the parameters of a given network can be endowed with the fisher - rao metric , so defining a riemannian manifold . in analogy with statistical mechanics , this manifold is the space of all the possible states of the associated network , that is , the analogous of the phase space of a physical system . by exploiting this analogy, we may define an entropy function as the logarithm of the riemannian volume of the manifold associated to the given network .a first step in this direction was put forward in ; in this paper , we have found that the geometric entropy associated with the fisher - rao metric reflects the topological features of the network : it is an increasing function of the simplices dimension .however , as it will be discussed in the following , this approach can not be constructively applied to networks having more than a few nodes .then , in a new metric - obtained by a suitable deformation " of the standard fisher - rao metric of information geometry - was proposed which allows to constructively lift the properties of any given network to the geometric structure of a manifold . in this way , we associated a differentiable system ( riemannian manifold ) to a discrete system ( network ) through the description of network by a set of probability distribution functions . among the wide use of probabilistic methods in literatureit is worth mentioning the random walk method . herethe green function , meaning the transition amplitude from one vertex to another by accounting for all possible walks , gives rise to a metric , thus allowing as well for a geometric approach .however the main difference is that by considering random walk the ( transition ) probability is given through the adjacency matrix , while in our case probabilities are given through gaussian joint distribution of random variables sitting on nodes of the network .in addition , as it is clearly shown in , such a geometric entropy is able to detect the classical transition in random graphs predicted by the erds - rnyi theorem .here we want to propose that such an entropy qualifies as a networks complexity measure .to this end we deepen the study about the random graphs model of and then validate our measure on complex networks .the layout of the paper is as follows .for the sake of self consistency and readability of the paper , the mathematical framework is reported in sections ii and iii . in particular , in section ii we briefly recall the relation between gaussian statistical model and underlying network putting forward the metric structure of the associated manifold . while in section iii we present the geometric measure of complexity as the logarithm of the riemannian volume of the manifold . to the end of validating this complexity measure, we check it against both theoretical and empirical networks , spreading from random graphs to real networks by stepwise increasing `` the degree of complexity '' . in sectioniv we report a strong validation point of our complexity measure of networks , a first account of which was given in .this first validation was obtained by computing our geometric entropy for the erds - rnyi phase transition in random graphs , that is by checking how it performs against a rigorous analytical result on networks . in sectionv we extend our investigation on different network models .we start considering theoretical models of exponential random graphs and configuration models which are well - known and studied in the literature .next , we show that our geometric entropy is also able to detect the emergence of a giant component as predicted by the molloy and reed criterium for scale free networks . then , we apply our complexity measure to real networks and compare the outcomes so obtained to the results that have been already reported in the literature on these same systems using other complexity measures .section vi is devoted to possible future developments concerning the possibility to predict the stability of a network system .conclusions are drawn in section vii .usually in mathematics in order to get information on a geometric object one endows it with a superstructure ( e.g. bundles over manifolds , coverings over topological spaces , and so on ) .likewise we endow a network with a statistical riemannian manifold .this can be obtained basically via two steps ; first by understanding a network as an undirected graph without loops on the nodes , and account for links ( weighted edges ) between nodes expressed by the adjacency matrix by means of correlations .then , by associating some random variables with the vertices of a network , one can resort to the methods of information geometry to associate a statistical riemannian manifold with the network .so , let us consider a set of real - valued random variables distributed according to a multivariate gaussian probability distribution ( assumed for the sake of simplicity of zero mean ) , \label{pxt}\ ] ] where with denoting the transposition .furthermore , are the real valued parameters characterizing the above probability distribution function , namely the entries of the covariance matrix . as a consequence .next consider the family of such probability distributions where . upon requiring the mapping to be injective, becomes an -dimensional statistical model on .the open set results defined as follows and we refer to it as the parameter space of the statistical model .since any element is univocally characterized by the parameter vector , it follows that the mapping defined by is a coordinate chart .so ,  ] , whose entries are defined by with . the matrix results symmetric , positive definite and provides a riemannian metric for the parameter space  .for our case the integral in eq .is gaussian and can be computed as \nonumber\\ & & = \exp\left[\frac 1 2 \sum_{i , j=1}^n  c_{ij}\frac{\partial}{\partial x_i}\frac{\partial}{\partial x_j}\right]f_{\mu\nu } |_{x=0 } , \label{gint}\end{aligned}\ ] ] where \ \partial_\nu \log[p(x;\theta)],\ ] ] and the exponential stands for a power series expansion over its argument ( the differential operator ) .the derivative of the logarithm reads =-\frac 1 2\bigg[\frac{\partial_\mu(\det c)}{\det c } + \sum_{\alpha,\beta=1}^n \partial_\mu(c_{\alpha\beta}^{-1})x_\alpha x_\beta\bigg],\nonumber\\\end{aligned}\ ] ] where denotes the entries of the inverse of the covariance matrix .  the computational complexity of the metric components in eq. can be readily estimate .indeed , the well - known formulae require the calculation of derivatives , with respect to the variables , in order to work out the derivative of the logarithm in . then , to obtain the function in , we have to calculate derivatives . with growing becomes a daunting task , even when afforded numerically .   in order to overcome the difficulty of computing the components of the fisher - rao metric, we follow and define a ( pseudo)-riemannian metric on the parameter space which account as well for the network structure given by the adjacency matrix .  to this endwe consider first a trivial network with null adjacency matrix that is associated with a set of independent gaussian random variables . notice that in this particular case , the covariance matrix in is a diagonal matrix with entries given by .let us denote this matrix as .so , employing eqs . and, a statistical riemannian manifold , with is associated to the bare network .let us remark that the entries of the metric in , worked out in , depend on the entries of the matrix .in fact , the entries of the inverse matrix of are given by .then , from it is evident that .inspired by this functional form of , we associate a ( pseudo)-riemannian manifold to any network with non vanishing adjacency matrix by deforming " the manifold in via the map defined by by we denote the set of the symmetric matrices over with vanishing diagonal elements that can represent any simple undirected graph .therefore , the manifold associated to a network , with adjacency matrix , is .here it is and with components where are the entries of the inverse of the matrix .we now define a statistical measure of the complexity of a network with adjacency matrix and associated manifold as where is the volume of evaluated from the element notice , however , that in such a way results ill - defined .in fact , the set in eq . is not compact because the variables are unbound from above .furthermore , from eq . , diverges since approaches zero for some .thus , as it commonly happens , we regularize the volume as follows where is any suitable `` infrared '' and `` ultraviolet '' regularizing function , i.e. providing a kind of compactification of the parameter space and excluding the contributions of making divergent .theoretically , a regularizing function might be devised by taking into account particular structures of the integration set and the functional relation . in practice, a very suitable function has been built in ref. ; it reads as , \ e^{-\mbox{tr}c(\theta)},\end{aligned}\ ] ] when the covariance matrix has only or as off diagonal entries . here, the logarithm hales contributions of that make diverge at the lower bound of the parameter space ; while , the exponential fixes the problem of non - compact integration space . in this paper , we tackle networks with weighted links among the nodes .still the functional type of the regularizing function is like in .the definition is inspired by the microcanonical definition of entropy in statistical mechanics that is proportional to the logarithm of the volume of the riemannian manifold associated with the underlying dynamics .of course we need to validate the proposed measure of network complexity defined in eq .. though in principle any measure of complexity is admissible , we may wonder how to assess its effectiveness .a first step is to check a complexity measure against a system which makes a clear jump of complexity as some parameter is varied . in physicsa paradigmatic situation is offered by phase transitions ( a snowflake is intuitively more complex than a drop of water ) .applied to networks this leads us to consider the classical erds - rnyi phase transition in random graphs .then , moving on from random graphs , more complex networks can be considered and the proposed measure of complexity compared with other known measures. these will be the subjects of the following sections .one of the basic models of random graphs is the _ uniform random graph _ .this is devised by choosing with uniform probability a graph from the set of all the graphs having vertices and edges , with a non negative integer .we can think of as a process evolving by adding the edges one at a time . when has the same order of magnitude of , the evolution of from to yields , according to erds - rnyi theorem , a _ phase transition _, revealing itself in a rapid growth with of the size of the largest component ( number of vertices fully connected by edges ) .specifically , the structure of when the expected degree of each of its vertices is close to , i.e. , shows a jump : the order of magnitude of the size of the largest component of rapidly grows , asymptotically almost surely ( a.a.s . ) , from to , if has the same order of magnitude of .in fact , if , as the process evolves , the components of [ the largest of them being a.a.s . of size mainly by attaching small trees ; thus they grow slowly and quite smoothly .nonetheless , at the same point of the process , the largest components become so large that it is likely for a new edge to connect two of them .thus , fairly quickly , all the largest components of merge into one giant component , much larger than any of the remaining ones .it is worth noticing that this process represents the mean - field case of percolation .we numerically compute , the geometric entropy in eq . vs for a fixed , in order to investigate its sensitivity to the appearance of the giant component during the evolution of the random graph model .it is worth mentioning that a gibbs entropy has been defined for the statistical set of random graphs as    following up , a research line has been pursued to characterize other classes of random graphs , like scale - free or fixed - degree sequence .however , the entropy [ entrobianco ] ( as function of ) is not able to detect the erdos - rnyi phase transition .  in practicewe have considered four different numbers of vertices : .notice that the magnitude of is not important , what matters is the -dependence , the so - called finite size scaling , of the relevant observables .the magnitude of simply determines the dimension of the manifold . for any fixed we have considered the number of links , to be .then , for a any pair we have randomly generated a set of entries , with , of the non - vanishing adjacency matrix elements .  in this way , since the covariance matrix is functionally assigned , we have gotten of eq . and finally the metric of eq .. next , having determined , we computed the volume in eq . andthe entropy of eq .. the volume regularization is performed in two steps .first by restricting the manifold support to an hypercube . inside it we generated a markov chain to perform a monte carlo estimation of the average .the number of considered random configurations ranges between and . as second step of the regularizationwe have excluded those points where the value of exceeds ( the numerical overflow limit of the computers used ) .then , for any given pair this computational procedure is repeated times , each time considering a different randomly generated realization of the adjacency matrix .thus , the final values of the entropy are obtained as averages over different manifolds , namely where is the metric corresponding to the null adjacency matrix .  in figure[ bifurcation ] we report the behavior of of the case of equal weights for all the non - vanishing links .this is what in the context of statistical mechanics is known as a _ collapse plot _ of the results obtained at different -values .it shows a typical phenomenon arising in numerical investigations of second order phase transitions : likewise finite - size effects observed for the order parameter , what asymptotically would be a sharp bifurcation is rounded at finite .however , the larger is , the more pronounced the `` knee '' of becomes .this is agreement with an -asymptotic bifurcation at ( black solid line ) where the erds - rnyi phase transition takes place .   of ( magenta points ) , ( black points ) , ( red points ) and ( blue points ) networks as a function of the number of randomly chosen links of weights equal to . the black solid line is a guide to the eye coming from a linear fitting of a linear - logarithmic presentation of the data.,width=264,height=226 ]  at present , this beautiful and unambiguous result ( presented also in ) lends credit to our proposed measure of networks complexity . to enforce it we a stability check would be in order .then , in figure [ rndeffects ] we report the outcomes of having chosen at random the values of the non - vanishing entries of the adjacency matrix , that is , where is a random variable of zero mean and variance equal to .of course negative values of the are excluded .the comparison with the results obtained with confirms the robustness of the entropy defined in eq ..   of networks as a function of the number of randomly chosen links of weight equal to ( red points ) , with a gaussian random of zero mean and variance ( black points).,width=264,height=226 ]here we go beyond the random graph model and apply the proposed measure complexity defined in eq . to complex networks with the aim of comparing our results with other already known .the general idea that a system is complex when it does not coincide with the summation " of all its parts , has been formalized in within the framework of information geometry . with this approach ,a hierarchy of exponential families is provided , which is widely studied in information geometry , modeling networks of progressively increasing order of the interactions between their parts .the model known as exponential random graphs ( erg ) is the distribution over a specified set of graphs that maximizes the gibbs entropy under some suitable constraints ; more precisely , let us suppose to be given a collection of graph observables ( such as nodes and edges ) measured in empirical observations of some real - world network , or of any other network as well .in addition , assume that we have an estimate of the expectation value of each observable .consider now a graph , and the probability of that graph within a given ensemble .then we wonder which is the best choice for so that the expectation value of each of our graph observables within that distribution is equal to its observed value .the answer is obtained by maximizing gibbs entropy   under the constraints and , where is the value of for the graph .this procedure leads to where is the graph hamiltonian and is the partition function . here , are lagrangian multipliers .this equation defines the exponential random graph model .this model has been employed to quantify the degree of interaction of all the parts of a given system . still in ref. ,simple exponential random graphs are considered in order to describe `` typical '' graphs , i.e. the graphs that are most probable in the ensemble defined by this model , and that correspond to the lowest `` energy '' characterizing the model . in particular , in ref. the authors consider the simple erg model with nodes , where only the interactions between triangles and -chains are taken into account , that is , only a subset of the family of all graphs with nodes are considered .then , the convex hull of all the possible expectation values of the probabilities of the triangles and of the -chains is derived . those graphs that correspond to the minimal `` energy ''are found to lie on the lower boundary of the mentioned convex hull .the geometric entropy proposed in the present work has been computed to provide a `` pointwise '' description of the complexity of the single members of a given family of graphs .the outcomes of these computations allow to rank the members of a given family of graphs according to their degree of complexity , of course on the basis of the proposed way of measuring it .the results are summarized into table .they suggest that going up along the lower boundary of the previously mentioned convex hull ( that is moving along a line representing a given family ) the degree of complexity increases .= [ circle , draw , fill = black!50 , inner sep=0pt , minimum width=4pt ]  [ ht ]    in 30  ( : 2 ) node ( + 60:2 ) node ( : 2 ) ( :0 ) node ( + 60:2 ) ( :0 )  ( + 17:2 ) node ( + 17:2 ) ( : 4 ) ( + 17:2 ) ( + 30:4 ) ( : 4 ) node ( + 30:4 ) node  ;  & & & & +    in 0,120  ( : 2 ) node ( + 120:2 ) ( + 240:2 ) node ( : 2 ) ( :0 ) node ( + 240:2 ) ( :0 ) ( + 90:2 ) node  ;  & & & & +    in 0 ( : 2 ) node ( + 72:2 ) node ( + 144:2 ) node ( + 216:2 ) node ( + 288:2 ) node ( + 360:2 ) node ( : 2 ) ( + 144:2 ) ( : 2 ) ( + 216:2 ) ( + 72:2 ) ( + 216:2 ) ( + 72:2 ) ( + 288:2 ) ( + 144:2 ) ( + 288:2 ) ( + 144:2 ) ( + 360:2 ) ( + 35:2 ) node ;  & & & & +  [ ht ]    in 0,60,120,180,240,300  ( : 2 ) node ( + 60:2 ) ( : 2 ) ( :0 )  ;  & & & & +  moreover , the result of table shows that our entropic measure is capable of distinguishing among different families of networks .in fact , while the graphs of table represent typical graphs on the minimal energy boundary , the graph of the table is a typical graph on the maximal energy boundary .notice that the results in table [ erg ] indicate that the network with two triangles ( 2-simplices ) is less complex than the network with one tetrahedron ( 3-simplex ) plus two points ( 0-simplices ) , which is less complex than the network with one 4-simplex plus one point ( 0-simplex ) ; in other words , network complexity is nontrivially influenced by network topology ( homology ) . a first account of this fact is given in ref. .real networks usually differ from the erds - rnyi random graphs in their degree distribution .given an undirected network with adjacency matrix , the degree of a node is just the sum of the -th row s entries , .it represents the number of connections that the node has .the degree distribution of a network is then defined to be the fraction of nodes in the network with degree .the degree distribution clearly captures information about the structure of a network .for example , in the binomial erds - rnyi random graphs , usually indicated as simple ( not - complex ) networks , one finds that most nodes in the network have similar degrees ; this model , in which each of the nodes is connected with independent probability , has a binomial distribution of degrees , namely  . however , real world networks usually have very different degree distributions .that is , most of the nodes have a relatively small degree ( low connectivity ) , while a few of them have a very large degree ( i.e. are connected to many other nodes ) .these large - degree nodes are often referred to as hubs .a first step toward testing the effectiveness of our geometric entropy in quantifying the complexity of real networks is to compare networks where each node has the same given degree to networks containing hubs .when each node of a network has the same degree , the network is called a -regular graph .one of the most widely used method to generate these special networks is the configuration model .this is specified in terms of a sequence of degrees ; for a network of nodes we have a desired degree sequence , which specifies the degree of each node , for .the average vertex degree is the ratio between the total number of links in a given network and the number of nodes .it represents a first level of characterization of the topological complexity .we consider it as benchmark to strengthen the validation of our proposal .we have numerically computed the entropy given by eq . for networks of number of nodes , constructed as _ random _ -regular graphs of two different values of , that is , and .a random -regular graph is a random graph with the uniform distribution over all -regular graphs .the computed value of the geometric entropy increases with , as is reported in table [ uniform ] .this result is very good because it is in agreement with the obvious fact that the larger the more complex the network ..the value of for random -regular graphs . [ cols="^,^,^,^,^,^",options="header " , ]the geometric framework so far put forward paves the way to interesting developments . a relevant generalization made possible by the riemannian geometric framework consists in considering the time evolution of a network . in order to do this, one should drop the simplifying assumptions of the present work by adding to the of the diagonal covariance matrix also the entries of the adjacency matrix as local coordinates of the statistical manifold of eqs . and . in this waythe dimension of increases from to .  denoting with the local coordinates of , where , there is a natural way of tackling the dynamical evolution of the network associated with , that is , through the geodesic flow given by the following set of equations where the are the standard christoffel connection coefficients   the physical time parametrization of the arc length is derived by means of the metric tensor as where the are the variation rates of the local coordinates expressed with respect to the physical time .let us remark that the dynamical evolution described by eq .encompasses also the time evolution of the weights of the links of a network , including their appearance and disappearance , thus a - priori allowing the computation of the time variation of its complexity .the fitting of empirical data concerning the true evolution of a real network by means of the model dynamics given by eq.s and could allow to get relevant information about the laws that drive the network evolution ( conservation , extremalization , optimisation of some quantities and so on ) .another prospective and remarkable application of the differential geometrical approach put forward in the present work , and notably related with the dynamical equations , concerns the study of the stability properties of a network .in fact , by setting , where are small functional perturbations , after substitution into eq . and using one can work out the _ tangent dynamics equations _ in the form of a system of first order linear differential equations  these equations , numerically integrated together eqs . and , are the natural tool to investigate the stability of either stationary or non stationary states of a network , for example - for a stationary state - to investigate a network stability / instability under addition or deletion of one or more links and so on .summarizing , the present work contributes the fascinating subject of quantifying the degree of complexity of systems that are commonly defined as complex " .there is a large number of such definitions that are already available .perhaps this history begins with kolmogorov s definition of algorithmic complexity which , in spite of its theoretical beauty , is hardly applicable in practice . sincethen the many definitions put forward were adapted to the specific systems / problems tackled .however , the number of categories in which all these definitions can be gathered is rather limited .of course borrowing the concept of physical entropy from statistical mechanics is the most inspiring and seducing way to proceed .in fact , in physics , entropy is just a measure of disorder and conversely negentropy - as defined by l.brillouin a long time ago - is a measure of the degree of order in a system and is also the physical equivalent of shannon s information entropy , again , as shown by l.brillouin . whence a vast literature addressing the quantification of complexity on the basis of shannon s information entropy which , on the other hand , has its inspiring model in boltzmann s entropy of kinetic theory .however , what was still missing was a general definition of an entropic measure of complexity accounting for both the structure of any given network and for its statistical complexity , that is , for the complexity of the probability distributions of the entities constituting the network .the new definition put forward in the present work embraces both these aspects .it is still inspired to statistical mechanics , however , instead of being modeled on boltzmann entropy is rather modeled on the microcanonical ensemble definition of entropy .the phase space volume being replaced by the volume of a state manifold " ( that is a riemannian manifold whose points correspond to all the possible states of a given network ) . the state manifold is defined through a suitable definition of a metric which is partly borrowed from the so - called information geometry and partly is an original proposal put forward in the present work .the result is a constructive way of associating a differentiable and handy mathematical object to any simple undirected and weighted graph or network .another novelty consists in having directly tested by means of numerical computations the validity and effectiveness of the proposed entropic - geometric measure of complexity . in order to do this we needed , so to speak , a paradigmatic example of a major change of complexity .a possible natural choice is suggested by the observation that phase transitions are the most impressive examples in nature of emergent phenomena - theoretically well understood - associated with a sharp qualitative and quantitative change of complexity of a physical system when a control parameter exceeds a critical value .this kind of phenomenon exists also in complex networks .in fact , random graphs undergo a well known phase transition as proved by the erds - rnyi theorem : a paradigmatic - and at present unique - example of an analytically known major variation of the degree of complexity of a network .this kind of check is unprecedented and very successful , in fact , the entropic - geometric measure of complexity proposed here displays both a pattern and its size - dependence which are typically found for the order parameter of a second - order phase transition in physics .then , since the random graphs undergoing the erds - rnyi transition are not considered genuinely complex networks , the proposed entropic - geometric measure of complexity has been applied to small exponential random graphs , to different versions of random -regular graphs with and without hubs generated according to the configuration model . moreover , the ability of detecting the transitions predicted by molloy reed criterium in power law random graphs is an important confirmation of the consistency of the geometric entropy measure of complexity proposed in the present .finally , this has been applied to some real networks and compared to three other measures of complexity , one describing the modularity structures , one homogeneity , and the heterogeneity of the real networks .the outcomes of these applications compose a consistent scenario validating the meaningfulness and effectiveness of the proposed measure of complexity .finally , the differential - geometric framework put forward opens some fascinating perspectives of application to the study of the time evolution of complex systems .we are indebited with r. quax for providing us with data for power - law rg .we also thank m. rasetti for useful discussions .this work was supported by the seventh framework programme for research of the european commission under fet - proactive grant topdrim ( fp7-ict-318121 ) .

Abstract: a central issue of the science of complex systems is the quantitative characterization of complexity . in the present work we address this issue by resorting to information geometry . actually we propose a constructive way to associate to a - in principle any - network a differentiable object ( a riemannian manifold ) whose volume is used to define an entropy . the effectiveness of the latter to measure networks complexity is successfully proved through its capability of detecting a classical phase transition occurring in both random graphs and scale free networks , as well as of characterizing small exponential random graphs , configuration models and real networks .   * keywords : * complex systems ; riemannian geometry ; entropy ; probability theory .