human activity often takes the form of exchanges .these exchanges typically consist of goods that can be quantified by value , but also opinions or other types of information may be traded . the former define a market economy .there have been several proposals to model such markets , see , for example the review by farmer .most of the proposed models aim at reproducing the fat tails and volatility clustering in a stock or currency market .earlier we have proposed a market model ( fat cat " model ) where agents trade products according to individual price estimates .these estimates were dynamically adjusted as a function of the trading encounters of each agent .the model mentioned above is one of many similar models that could be considered for such a market , each model being distinguished by a strategy that defines which product a buyer should select from a seller and how he should price it later . in the present paperwe study the interplay of a few strategies . the relative performance of these strategies is quantified by the wealth of agents employing them .we organized the paper by first reviewing the fat cat " model in section ii .the extension of this model to the case of other strategies is described in section iii .then , in section iv we discuss how the model could be extended to cases where agents are able to change their strategies according to their performance and give our concluding remarks .we picture our model as a cartoon of the trading situations found in a real market . as shown in fig .1 , this minimalistic model consists of a system where agents trade a set of different products .each agent is assigned one of three possible strategies selected to be either based on profit ( i.e.  greed " , which is the strategy adopted in the original fat cat " model ) , on the need for a particular product , or on a random selection without regard to the level of profit or need .  other strategies could be explored ; for example , agents could act as garbage collectors " , buying whatever product has the lowest possible price . in this paperwe limit ourselves to the three , probably most basic , strategies outlined in fig . 1 .each of the agents starts with units of goods .the goods are randomly selected , for each agent , among the different products .thereby we form the stock of each agent that together with some initial amount of money , define the initial state of the economy .we describe below the dynamics that arises from the interactions among these agents .  during the time evolution of the system, each agent has , at each time step , an amount of money , and a stock of the different products , where .the prices of the different items in the stock of agent are denoted which initially are taken to be integers uniformly drawn in the interval $ ] . in all caseswe have verified that the evolution of the system does not depend on this particular choice .agents then meet and exchange products and adjust prices .price adjustment is such that large differences in pricing between agents are lowered , but also such that price differences are induced by some noise when they are small , as in real markets .  as in our simulations for the original fat cat " model , we assume that at each time step the following procedure takes place :  * * 1 * buyer and seller are selected at random among the agents . if the seller has no products to offer , then another seller is chosen .* * 2 * the buyer selects a product in the seller s stock according to his strategy .+ \i ) if strategy is profit " then he selects the product which maximizes , ( i.e. his profit ) .+ \ii ) if strategy is need " then he selects the product which he has the least in his stock ( minimizes ) .+ \iii ) if strategy is random " then he selects a random product in the seller s stock for which . + the selected product called ( best buy ) . * * 3a * if the buyer does not have enough money , ( i.e. if , we return to the first step and choose a new pair of agents . * * 3b * if the buyer has enough money we proceed .+ if , the transaction is performed at the seller s price .this means that we adjust : , , , . * * 3c * if , the transaction is not performed . in this case, the seller lowers his price by one unit , , and the buyer raises his price by one unit , + .the prices are always non - negative integers . also note that since , as defined in step 3 above , the price offered by the buyer can not be higher than the amount of money it has, we are not allowing for the agents to get into debt . in casethere are several products that fulfill the selection criterion in 2 , a random one of these is chosen .one should emphasize that due to the price adjustments performed in unsuccessful encounters , the prices never reach equilibrium , and different agents typically assign different prices to the same product .  to quantify the system ,we define the total wealth of an agent as the amount of money plus the value of all goods in the agent s possession : the value of product is defined as the average of what all agents consider its value to be : and the value of all of agent s goods , is defined as the fat cat " model s profit " rule was studied extensively in our earlier paper .there it was found to lead to persistency and fat tails in the agents wealth fluctuations with time , hence its name .we now study the other strategies and their interactions .the three strategies given above lead to different wealth of the respective agents .we will now study this in some detail .our first step is to illustrate how the wealth of the agents gets distributed in a system where all agents employ the same strategy . for that purposewe consider a system composed of 50 agents , 50 different products .each agent is given 40 units of products and 20 units of money .the result , shown in fig .2 , is that there are appreciable differences in the wealth distribution according to the strategy . for a profit - minded system ( the original fat cat " model ) , there is a long tail of wealthy agents , which becomes less pronounced when the strategy is based on the stock needs . in this casethere are no rich agents , but instead a large concentration of middle - wealth agents . if the strategy adopted is to buy a random product , the long tail disappears and the middle - wealth peak is shifted to lower values than in the previous case .the situation can change when the economic conditions , as defined by the number of units of products and money in the economy , are changing .we will show that there is no best rule for all situations this is shown in fig .3 , where we have changed the number of units of products , while keeping all other model values the same , in markets composed of equal numbers of agents employing two different strategies . in this case , where , each agent has a few of many possible products , thus representing an antique dealer market where there are many special items and no one can have everything .we see , in the upper panel , that for this market the need " strategy , is better than profit " above 10 units . however , below 10 units , the result is unstable , one strategy leading to better results than the other depending on the initial conditions . in the middle panelwe see that the random " agents have a higher average wealth than the profit " motivated ones when the number of units is above 45 , the situation is reversed for a number of units below 35 , and there is unpredictability in between these two values .finally , the lower panel shows that need " works better than a random " strategy for all number of units in the range shown . for other parameter values, namely when the number of units of products and money is large , the random " strategy outperforms need " .in this last case , namely for , each agent typically has many copies of all products , which represent a mass production market with many copies of few items ; a supermarket world . in this case, a pairwise comparison of the three strategies shows an interesting situation , depicted in fig . 4 : need " outperforms profit " , which outperforms random " , which outperforms need " .so , in this case no strategy is better than both of the other strategies when studied pairwise .  in fig .5 we explore the triple market further , implementing a market where there is only one agent with need " strategy among 25 agents with profit " and 25 agents with noise " as strategies , so that there are all together 51 agents . without the single need " strategy , random" would outperform profit " with a large margin .a single need " agent in the system will slowly collect a large fraction of all products in the market .this is because it has a large competition advantage over the profit " , which overcompensates its disadvantage to the random " ones . if the number of profit " agents were reduced and that of the random " agents increased , the need " agent would do worsethis illustrates the fact that the number of agents employing the different strategies is also determinant in the relative success of the agents adopting them .as we have seen , one may have different strategies for different agents . an interesting direction to extendthis model is to let the agents to select the strategies they adopt in order to improve their performance .a further extension development would be to allow the strategies themselves to evolve . in the followingwe show a preliminary example of how the agents could choose strategies .a simple mechanism consists in updating , at fixed time intervals , the strategy of the poorest agent in the system .this agent just changes his present strategy to any of the other two , at random . the resulting time evolution for a system composed of 45 agents initially equally distributed according to their strategies is shown in fig . 6 .we see that , after an initial transient , the number of agents following the need " and random " strategies becomes approximately equal and constant for a considerable period of time , while the profit " strategy is followed by a small fraction of the agents .an increase in the profit " agents leads to a change in the conditions which lead the random " strategy to almost disappear from the system , to a dominance of need " , and the persistency of profit " at a very low level . from the fig .4 we see that were it not for the existence of profit "- thinking agents , random " would dominate over need " .the results shown in fig .6 also demonstrate that the hierarchy paper stone scissors " illustrated in fig .4 , for the same system , does not seem to lead , in the time interval considered , to alternations in the number of agents employing each strategy. there could be several reasons for this .one is that these changes take longer and longer time to alter enough the market conditions to significantly modify the performance of the different strategies .another is that while in fig .4 one measures the average wealth of the agents using each of the three strategies , the criterium for changing strategies is based on the performance of poor agents , _i.e. _ on the agents wealth distribution in the region close to the origin . as suggested by fig .2 , a strategy yielding a higher average wealth than another , may also have a larger number of poor agents than the other .these issues need to be further explored for an appropriate strategy selection procedure and will be discussed in the future .compared to earlier models of market dynamics the model presented here has some new and related key features : there is local optimization of utility ( estimated market value ) and all trades are done locally without the equilibrizing effects of a global information pool .this gives arbitrage possibility which drives a dynamic and evolving market .earlier market models , such as the minority game , have a global information pool , and lack dynamical signals that could be associated with stock market fluctuations .the evolving boolean network for minority games of paczuski , bassler and corral , on the other hand , works with local information exchange , but the reward function is still global . also the frame of minority games makes it difficult to treat a multi - product market , which we believe is important for understanding real stock markets .  the model we propose is for a market composed of agents , goods and money ( or people , prices and products ) .we have demonstrated that such a market easily shows persistent fluctuations of wealth with time , and seen that the persistency is closely related to an interplay of having many products that influence each others trade probability .a similar result was obtained in the simpler model in , where it was demonstrated that persistency could arise even without money .the setup proposed here with agents and products with individual local prices allows for individual strategies of the agents .this opens for evolution of strategy as a part of the financial market , and we have seen that evolving strategies indeed give a dynamics where wealth is often rapidly redistributed .

Abstract: we study an economic model where agents trade a variety of products by using one of three competing rules : need " , greed " and noise " . we find that the optimal strategy for any agent depends on both product composition in the overall market and composition of strategies in the market . in particular , a strategy that does best on pairwise competition may easily do much worse when all are present , leading , in some cases , to a  paper , stone , scissors " circular hierarchy .   agent - based models ; economic models ; strategic games ;


most of the modeling performed in the area of systems biology aims at achieving a quantitative description of intracellular pathways .hence , most available models describe a typical cell on the basis of experimental data .unfortunately , experimental data are in general obtained using cell population experiments , e.g. western blotting .if the considered population is highly heterogeneous , meaning that there is a large cell - cell variability , fitting a single cell model to cell population data can lead to biologically meaningless results . to understand the dynamical behavior of heterogeneous cell populations it is crucial to develop integrated cell population models .modeling on the population scale has already been addressed by and .these authors demonstrated that populations can show a bimodal response if stochasticity in biochemical reactions is considered .but besides stochasticity in biochemical reactions there are other reasons which can also lead to heterogeneity in populations .examples are unequal partitioning of cellular material at cell division , genetic and epigenetic differences .  for the purpose of this paper ,we describe heterogeneity in populations by differences in parameter values of the model describing the single cell dynamics .the network structure is assumed to be identical in all cells , as this usually represents the physical interactions among molecules , which should be independent of the cell s state .this parametric approach is well suited for genetic and epigenetic differences .the distribution of parameter values within the cell population of interest is described by a multivariate probability density function , which is part of the model specification .  in the following the problem of estimating the parameter distribution functionis studied .therefore , we consider high - throughput experimental methods such as flow cytometry , which can be used to measure concentration distributions within cell populations by suitable fluorescent labeled antibodies .classical flow cytometry devices can measure several thousand cells per second .to estimate the parameter distributions , in a first step , an appropriate population model has to be found . in the literature mathematical models of cell populationsare either described as cell ensembles , or as a non - linear partial differential equation ( pde ) for the distribution of the state variables . in case of ensemble models ,a differential equation is assigned to each cell , making an in depth theoretical analysis difficult .pde models , which describe the time evolution of the distributions of the state variables based on the single cell models , are easy to handle from a theoretical point of view but hard to simulate for a large state dimension of the single cell model .therefore , only low dimensional pde models of populations have been studied in literature so far .  in this paper a pde model for the state distribution within a heterogeneous cell populationis derived .given the solution of this pde the probability density of measuring a certain output can be determined . as for the estimationonly the measured outputs are required , a numerical method for computing the output distribution is outlined .this methods employs a particle - based approach and classical density estimation .based on these efficient computation scheme for the population response an estimation method is developed . a statistical model of the measured output distributionis derived from the single cell measurement obtained at every measurement instance .therefore , again kernel density estimators are used as they have better asymptotic properties than commonly used naive estimators . given a model and the output distribution estimated from the measurement , a -norm minimizationis performed over the set of possible parameter distributions . by employing the model properties and a parameterization of the parameter distributionthis optimization problem is convex and can be solved efficiently .the paper is structured as follows . in section 2 , the problem of estimating the parameter distribution is introduced . in section 3 ,we present the statistical model for the measured data and the simulation model for state and output distribution . section 4gives a short overview of the used identification procedure before in section 5 the proposed method is applied to a caspase activation model with artificial data ._ notation : _ consider the -dimensional hypersurface .the integral of a function , with , over is written as furthermore , the .th unit vector is denoted by .for the purpose of this work , a model of a biochemical reaction network in a population of cells is given by the collection of differential equations with state variables , measured variables , and parameters .the index specifies the individual cells within the population .the parameters can be kinetic constants , e.g. reaction rates or binding affinities .the cell - cell interaction of the considered pathway is assumed to be negligible , as it is the case in many _ in vitro _ lab experiments .  in the following heterogeneity within the cell populationis introduced , modeled by differential parameter values and initial conditions among individual cells .the distribution of parameters and initial conditions is given by a probability density function with .for ease of notation , we write .the probability density function is part of the model specification and the parameters and initial conditions of cell are subject to the probability distribution   as outlined in section [ sec : intro ] , for the study of cell populations high - throughput cell population measurements are available . using these experimental techniques protein concentrations within thousands of cells can be measured at every measurement instance , , .this yields the measurement data where is the measured output of the cell and is the index set of the cells measured at time .note that the cells can not be tracked over time , and are removed from the population in order to obtain the measurements .thus , no single - cell time series data are available . on the other hand ,the samples are independent and equally distributed and is assumed to be large , such that an approximation of the output distribution is possible .  like most measurement devices ,also high - throughput fluorescence measurements are subject to noise . for the rest of the paper , noise consisting of a relative and an absolute partis considered , in which is the measured output and is a vector of -normally distributed random variables with probability density functions yielding the joint probability density -normally distributed random variables are chosen here , since they are a good model for the commonly seen noise distributions of the considered measurement device and conserve the positivity of all variables . for notational simplicity the measurement errors of the different concentrations are assumed to be uncorrelated .this constraint can be removed easily .  given this setupthe problem we are concerned with is :  given the measurement data , , the cell population model , and the noise model , determine the parameter distribution .unfortunately , estimation of using a cell population model with a finite number of cells and discrete sampled data is fairly difficult as no single cell trajectories are available .a far more natural approach would be to use a density description , as the available measurement data can be interpreted as samples drawn from the probability density function of the output .this interpretation is also quite appealing from a point of modeling as the number of cells considered in a standard lab experiment is of the order of and hence nevertheless too large to be simulated on an individual basis . in the next chapter a pde model for the probability density of the output and a density model for the measurement datais derived .as outlined in the previous section , a continuous statistical model for the measurement data , as well as for the evolution of the state and output density would be preferable .these two aspects are addressed in the following .the data collected by the considered measurement devices are samples drawn from the distribution of the measured output , as mentioned in section [ sec : prob ] .let be the distribution of the measured outputs at time .as is considered to be a probability density , classical density estimation methods can be employed for estimating from the given samples .  in this work, the problem of determining from is approached using kernel density estimators .kernel density estimators are non - parametric approaches to estimate probability distributions from sampled data .they are widely used and can be thought of as placing probability bumps at each observation , as depicted in figure [ fig : kde ] .these bumps are the kernel function , with . note that here only the equations for the one dimensional case are given .the extension towards higher dimensions is straightforward and can be found in . in this work ,a gaussian kernel given by with standard deviation is used . in this context, is also called smoothing parameter in the literature .given the kernel an estimator of the probability density for a given set of samples is where is the cardinality of .the selection of the smoothing parameter is crucial and depends strongly on . in this work chosen according to the least - squares cross - validation method .as is considered to be of order , it can be assumed that the the estimated output distribution ins close to the actual output distribution .   for the measured outputs ( o ) and the associated gaussian kernels (  ) .]   as outlined previously , a continuous model for the output density is desirable for the purpose of parameter identification .therefore , a pde model for the cell population is derived in the next step .  at firstthe single cell model is transformed in an extended state space model in which the parameters are appended to the state vector , ^t \in \mathbb{r}^{n+q} ] . for the 2-dimensional casethis is depicted in figure [ fig : fem ] .   of the extended state space , with fluxes across the boundaries.,width=264 ]  for this infinitesimal volume the flux and storage balance is , the left hand side of the equation represents the storage term and the right hand side the fluxes across the boundaries .the fluxes and are given by the surface integral of the product of boundary distribution and entering velocity , determined by the single cell dynamics , in which .next , and are used to derive the pde for the time evolution of .therefore , at first the storage term is expanded using its taylor series , yielding here it is assumed that  . in a second stepthe flux difference is rewritten , the first line follows from the definition of and the taylor series expansion of . to obtain the second linethe integration is carried out .the final reformulation is the expansion of the time integral in , resulting in substituting and in the flux balance and dividing by then yields , given this the pde governing the evolution of is obtained by taking the limits and , leading to for sufficiently smooth .this final equation is somehow what we expected , a transport equation with position dependent transport direction and velocity , according to the single cell dynamics .the initial condition of is the initial distribution on the extended state space ,   from the state distribution , the output distribution is computed as the integral of the state distribution along , where .the resulting partial differential equation system is where and .this pde is of first order , quasilinear and known as liouville s equation .the solution always exists for sufficiently smooth  .  as the measurements are noise corrupted , the distribution of measured outputs is different from the actual output distribution .it is defined by where ^t| \mathrm{diag}(\eta^1 ) y + \eta^2 = \psi \rbrace ] , with the parameter value .a sketch of the single cell model is given in figure [ fig : caspase network ] .heterogeneity is modeled by a -normally distributed production rate of the inhibitor of apoptosis iap , , and a -normally distributed amount of tnf - receptor complexes on the cell membrane , tnfr .these two quantities were chosen as it is known from experiments that there is a high cell - cell variability .especially the concentration of iaps contained in a cells is highly variable , and a variation in iap production is known to affect cell death considerably . in the following the possibility of estimating the distributions of and ) ] for )$ ] .the obtained estimation result is depicted in figure [ fig : ex est ] .it is obvious that the estimated parameter distribution approximates the real parameter distribution very well , especially considering the finite number of degrees of freedom .hence , even though there is an average measurement error of 20 on the single cell measurement , due to good statistics at the population level , the actual parameter distributions can be estimated accurately .furthermore , this study shows that in principle , measuring one concentration can give enough information to estimate several parameter distributions , if the output distribution is sensitive with respect to these parameters .heterogeneity in cell populations is an important issue for research in systems biology .however , so far only few models describing heterogeneous populations of cells with more than one state variable have been developed . in this papera partial differential equation model describing the time evolution of the state distribution is derived .we focused hereby in particular on the distribution of the measured outputs .  in the second part of the paper , the model of the noise corrupted measured outputs and its particular properties are used to estimate the parameter distributions underlying the heterogeneity .therefore , a density - based statistical model of the sampled single cell used in combinations with -norm based convex optimization .  finally , we applied the developed estimation method to artificial data of a medium size bistable system modeling the caspase activation cascade .it could be shown that the proposed method yields good estimation results in case of a setup which is realistic in terms of noise and amount of available data .the authors acknowledge financial support from the german federal ministry of education and research ( bmbf ) within the forsys - partner program ( grant nr . 0315 - 280a ) , from the german research foundation within the cluster of excellence in simulation technology ( exc 310/1 ) at the university of stuttgart , and from center systems biology ( csb ) at the university stuttgart .t. eissing , h. conzelmann , e.d .gilles , f. allgwer , e. bullinger , and p. scheurich .bistability analyses of a caspase activation model for receptor - induced apoptosis ._ j. of biol ._ , 279 ( 35):0 3689236897 , 2004 .t. eissing , s. waldherr , e. bullinger , c. gondro , o. sawodny , f. allgwer , p. scheurich , and t. sauter .sensitivity analysis of programmed cell death and implications for crosstalk phenomena during tumor necrosis factor stimulation . in _ proc .ieee conf .( cca ) _ , pages 174652 , 2006 .t. luzyanina , d. roose , t. schenkel , m. sester , s. ehl , a. meyerhans , and g. bocharov .numerical modelling of label - structured cell population growth using cfse distribution data .biol . and med ._ , 4(26):0 114 , 2007 .t. luzyanina , d. roose , and g. bocharov .distributed parameter identification for label - structured cell population dynamics model using cfse histogram time - series data . _ j. math ._ , 59:0 581603 , 2009 .n.v . mantzaris . from single - cell genetic architecture to cell population dynamics : quantitatively decomposing the effects of different population heterogeneity sources for a genetic network with positive feedback architecture ._ biophys .j. _ , 92:0 42714288 , 2007 .

Abstract: in many biological processes heterogeneity within cell populations is an important issue . in this work we consider populations where the behavior of every single cell can be described by a system of ordinary differential equations . heterogeneity among individual cells is accounted for by differences in parameter values and initial conditions . hereby , parameter values and initial conditions are subject to a distribution function which is part of the model specification . based on the single cell model and the considered parameter distribution , a partial differential equation model describing the distribution of cells in the state and in the output space is derived .  for the estimation of the parameter distribution within the model , we consider experimental data as obtained from flow cytometric analysis . from these noise - corrupted data a density - based statistical data model is derived . using this data model the parameter distribution within the cell population is computed using convex optimization techniques .  to evaluate the proposed method , a model for the caspase activation cascade is considered . it is shown that for known noise properties the unknown parameter distributions in this model are well estimated by the proposed method . + * keywords : * parameter estimation , cell population , kernel - density estimation , flow cytometry , convex optimization


in this study , we address the issue of predicting the glaucomatous visual field loss based on patient datasets .glaucoma is an eye disease that eventually losses the visual field .the goal of the present study is to accurately predict the glaucoma progression using small visual field datasets . because measurements of visual field is expensive , early prediction of glaucoma progression based on limited measurements is important for real clinical settings .a conventional approach for predicting visual field loss is _ patient - wise linear regression _ and _ clustering - based method _ . in patient - wise linear regression , we construct a predictor for each individual patient .however , as the number of measurements for each patient is very small , we can not produce a good predictor for individuals .liang et al . proposed a clustering - based method for glaucoma progression which utilizes spatiotemporal disease patterns .each patient is categorized into a cluster based on each spatiotemporal pattern .then , for each cluster , a linear regression predictor is constructed . a predictor formed from a single clusteris called a _cluster - based predictor _ ; the clustering - based method selects an appropriate pattern from a pool of cluster - based predictors .the prediction accuracy of this method is highly dependent on the quality of the clustering method .furthermore , a clustering - based method will not work well if the target patient is not a typical member of clusters .the aim of the present study is to overcome this weakness in clustering - based methods .we propose a novel framework for aggregating a number of cluster - based predictors .this significantly improves the prediction accuracy over that of a single cluster - based predictor in the case of real glaucoma patient datasets .we present a schematic illustration of our approach in fig .[ fig : predictingglaucoma ] .in addition to conventional patient - wise linear regression and clustering - based methods , various techniques have been developed to predict the visual field loss in glaucoma patients .bengtsson et al . introduced the _ visual field index _ ( vfi ) , and showed that only five measurements were needed to produce a correlation coefficient of 0.84 with the vfi obtained using all measurements .russell et al . used bayesian linear regression to derive the _ mean sensitivity _ index .they showed that their approach outperformed the ordinary linear regression method when there were fewer than nine measurements .murata et al . proposed a variational bayesian learning method .maya et al .proposed a multi - task learning method to predict the visual field loss , using matrix decomposition .maya et al . also proposed a hierarchical mdl - based clustering method to detect progressive patterns of glaucoma .recently , tomoda et al . proposed a prediction method utilizing information of eye pressure .other methods were reviewed by liang et al . , including those reported by noureddin et al . , fitzke et al . , chan et al . , chan et al . , and mayama et al . .however , the best predictor for each patient will differ according to his / her disease features . in this context, it is crucial to investigate a strategy to aggregate these prediction methods and automatically to generate the best prediction .thus , the present study addresses the issue of combining several prediction methods to obtain a better prediction than that produced using a single method .such aggregating strategies have worked well in other applications , for example , the issue of predicting prostate specific antigen from short time series .we propose a new aggregating strategy based on a previously proposed _ aggregating algorithm _( , ) . in this method ,each predictor is treated as an _ expert _ , and the prediction is obtained by taking a weighted average of all experts predictions . in theory , this method performs almost as well as the best expert in terms of the worst - case regret .however , in practice , the aggregating algorithm outperforms all of the experts in terms of the instantaneous prediction loss .  the novelty of the present study is a novel aggregating algorithm which is adapted to glaucoma visual loss prediction: we modify the original aggregating algorithm as follows to enable its practical application to the specialized setting of glaucoma visual loss prediction :  \1 ) _ learning rate adaptation _ :the learning rate is estimated using training examples to maximize the improvement rate .this indicates the improvement in prediction loss relative to the patient - wise linear prediction .this learning rate achieves better performance than the theoretically designed learning rate .\2 ) _ batch learning adaptation _ : the expert weights are determined in a batch process , rather than the online process employed in the original aggregating algorithm .the modified algorithm optimizes the use of all patient data , thereby producing better predictions than the original method .\3 ) _ hierarchical aggregation _ : we propose an aggregation process , the _ hierarchical aggregating strategy_. this method first aggregates all cluster - based predictors from a single clustering method to obtain intermediate predictors , then aggregates these intermediate predictors ( see fig . [fig : schematic_ill](c ) ) .when we have several clustering - based methods , we can aggregate several types of experts .a facile idea is aggregating all cluster - based predictors over all the clustering methods at once ( see fig . [fig : schematic_ill](b ) ) .we call this simple method the _ flat aggregating strategy_. although the performance of the flat aggregation is better than clustering - based methods , our analysis indicates that this hierarchical aggregating strategy exhibits much better performance than the flat aggregating strategy .the significance of this study can be summarized as follows :  \a ) _ better use of predictors in the area of glaucoma visual field loss _ :we demonstrate that our proposed method delivers better prediction accuracy than the flat aggregating strategy , the conventional clustering - based methods , and patient - wise linear regression .\b ) _ a new framework for aggregating visual field loss predictors _ : there are a number of promising methods for predicting glaucoma visual field loss , and many more are expected to be proposed in the future .thus , our method provides a general strategy whereby new methods do not have to compete with existing methods .instead , our approach allows numerous methods to collaborate in the hierarchical aggregation framework .therefore , by adding new methods to the pool of experts , our hierarchical aggregation approach obtains better predictions than those given by previous techniques .the remainder of this paper is organized as follows . in section [ sec :overview ] , we describe the prediction of visual fields .section [ sec : reviewaa ] reviews the aggregating algorithm , and section [ sec : generatingexperts ] discusses the construction of experts from clustering - based methods . in section [ sec : algorithm_acm ] , we present the details of our modification of the existing aggregating algorithm , and present our experimental results in section [ sec : experiment ] .section [ sec : conclusion ] contains our concluding remarks and discussion .in the present study , the patient visual field comprises 74 scalar real values .each value is the _ total deviation _ ( td ) , which corresponds to the visual loss at each local point of the visual field .we denote these values in vector form as , where ( ) is the td of the point in the visual field , and is the number of points . in our algorithm ,observations of the target patient s visual field are given in this vector form .the prediction problem is formalized as follows : at time , given past data , we predict with using . through a clustering method , we generate a _ cluster - based predictor _ from the cluster containing .these predictors enables us to produce a prediction using the above vector formalism .we introduce the basic framework of an existing aggregating algorithm .this method predicts at time by combining various _ expert _ predictions during each time step , , where is the expert s prediction at time and is the number of experts .the weights are updated automatically using the loss function of expert prediction for outcome as follows : where is a constant called the _ learning rate_. the weight assigned to an expert decreases when the difference between its prediction and the observation is larger .this difference is evaluated using loss functions , e.g. , .the performance of this aggregating algorithm is evaluated based on the _ regret _ , which is defined as .the upper bound of the regret is known to be  .the optimal learning rate is derived by minimizing the regret to .we call this theoretically optimal learning rate the _ regret _ ( rg)-_optimal _ . for more details , see .in this section , we explain how to construct experts from existing clustering methods . in the process of prediction with a single clustering method ,a cluster - based predictor is generated for each cluster .the most suitable predictor then gives the prediction .such clustering - based methods often produce good predictions , but it is not clear whether the selected predictor is optimal . if the target patient is not adequately represented by a single cluster , none of the cluster - based predictors will produce sufficiently accurate predictions . in this case , combining the contributions from several clusters may improve the prediction accuracy . therefore , we can think of cluster - based predictors as experts , and aggregate them to produce better predictions .the process of generating experts from clustering - based methods is illustrated in fig .[ fig : schematic_ill](a ) .liang et al . proposed a spatiotemporal clustering algorithm that predicts the progress of glaucoma using past patient information .the algorithm comprises a spatial feature clustering method and a prediction module that uses the temporal characteristics of each spatial cluster .this scenario is illustrated in fig .[ fig : spatiotemporal ] .we employ the spatial feature clustering method ( called _ uv - em _ ) proposed in .this is an _ expectation - maximization _ ( em )algorithm that learns the spatial cluster centers and temporal feature vectors of glaucoma patients visual fields .  from the clinical knowledge that td decreases linearly over time , we assume that the clustering - based predictors can be written as follows : where is the predicted tds at time , and and denote the progression rate and initial td , respectively .these predictors are constructed using the following two methods .the first method is _ temporal - shift linear regression_(_tslr _ ) , where the disease progression rate of all patients belonging to the same spatial cluster is assumed to be the same .however , the initial state of the disease can be different in each patient .thus , we obtain a cluster - based predictor through linear regression by applying optimal time - shifts to the patients within a cluster . the second method is _slope clustering_(_sc _ ) , where there are at most progression rates for local visual points in each spatial cluster.  these predictions are more accurate than the traditional patient - wise lr predictors , which implies that these clustering - based methods represent the characteristic of glaucoma patient datasets .hereafter , we denote uv - em + sc as sc and uv - em + tslr as tslr for simplicity ( for more details , see ) .   next , we explain how to construct experts using tslr , sc , and patient - wise lr .each expert is a linear function of time , as in ( [ eq : linearpredictioncluster ] ) . under tslr, is as the cluster - based predictor s gradients . for sc, is the same as that of patients within the training datasets , and are selected from .patient - wise lr gives by linearly regressing each patient .the initial disease state is determined as , where is the date , is the observation , and is the number of observations .this is obtained by minimizing the errors .the pair , defines an expert in the proposed framework .to utilize the experts introduced above , we modify the original aggregating algorithm as follows : 1 ) the learning rate is selected to ensure that it is practical ; 2 ) the weight is determined in a batch process , rather than an online process ; 3 ) a hierarchical aggregation algorithm is applied to the clustering - based methods .the pseudo - code is shown in alg .[ alg : clusters ] .  [ !t ]  * input * : observables for the target patient .determine based on past patient datasets using tslr , sc , and lr .determine from the observations .determine rg - optimal and ir - optimal values of using the experts .determine the weights of the experts based on these values of .* output * : make prediction using the aggregating algorithm .   as described in section [ sec : reviewaa ] , the existing algorithm derives a theoretical rg - optimal .our goal is to minimize the instantaneous prediction loss at a desired time point ; thus , the regret is not appropriate .the _ root mean squared error _ ( _ rmse _ ) was used to measure the prediction accuracy as where are the predictions and are the observations .the performance of the prediction method is measured in terms of the _ improvement rate_(_ir _ ) , which is defined as where is the number of observation points ( $ ] ) and is the number of patients in the test dataset .the value of is if , and otherwise . here , is the rmse of the patient using patient - wise lr , and is the rmse using prediction method . the larger the value of ir , the greater the improvement of relative to the patient - wise lr . when has the same accuracy as the patient - wise lr , ir .we empirically determine the _ improvement - rate _( ir)_-optimal _ by maximizing the ir .   in the usual online algorithm ,the expert weights are updated after each new observation . for the glaucoma datasets ,samples are usually obtained at intervals of a few months .thus , it is natural to use all available data points to determine the weights of the experts .our method employs the following batch update rule : , which replaces the traditional rule ( [ eq : update_online ] ) .we propose a hierarchical aggregating strategy , where all the predictors given by each single clustering - based method are first aggregated to construct intermediate predictors .these intermediate predictors are then aggregated to construct the final predictor ( see fig .[ fig : schematic_ill](c ) ) .the purpose of the proposed method is to adequately deal with experts generated from a small expert source .the number of experts generated from each clustering - based method is naturally different . in the flat aggregating strategy ,a good expert belonging to a small community may not greatly affect the prediction because a large community gathers considerable expert weights .on the contrary , in the hierarchical aggregating strategy , experts within each community are aggregated to generate intermediate predictors . then, the final prediction is given by aggregating intermediate predictors that only one intermediate predictor belongs to each community .therefore , such effects on the biased community size may be canceled at the second aggregation . in the real glaucoma datasets ,the number of experts generated from clustering - based methods is widely distributed , therefore , our proposed method showed the best performance among all the methods as shown in the next section [ sec : experiment ] .we compare the effectiveness of our proposed hierarchical aggregating strategy against a single cluster - based predictor for glaucoma datasets .the dataset used in this study is the same as that used by liang et al .we divided the dataset into two parts , a learning set ( , 977 patients ) and a test set ( , 109 patients ) . in the learning period, we constructed experts using clustering methods , and used them to determine rg - optimal and ir - optimal values . in the test period, we predicted the final observation point for each patient .the prediction errors were evaluated using ten - fold cross - validation .we set the number of spatial clusters to and the number of slope clusters to , as reported in . in our dataset , .clusters were omitted if they contains fewer than three patients .lr , sc , and tslr gave 977 , 977 , and 38 experts , respectively .for all calculations , we used the loss function .the ir - optimal is obtained through ten - fold cross - validation using portions of the learning dataset . as shown in fig .[ fig : obt_eta ] , ir attained peaks at different values of .the ir - optimal generally agrees with the optimal obtained using the test dataset .however , the rg - optimal are clearly smaller than the optima .[ tbl : tslrandsc ]  [ cols="^,>,>,>,>,>,>,>,>,>",options="header " , ]   hierarchical aggregation outperformed the flat aggregating strategy for the following reasons . if there are a few experts , those with larger weights ( i.e. , better experts ) in a clustering method will lose their influence in the flat aggregating strategy , because the initial weight becomes small during the flat aggregation process over a large number of experts .such experts will survive in the hierarchical aggregating strategy , because this approach gives equal initial weights to all clustering methods , regardless of how many experts they include .in other words , the hierarchical aggregating strategy has a mechanism whereby good predictors in a small community ( i.e. , a clustering - based method with a small number of experts ) can thrive .this works particularly well when the number of experts varies widely across all of the cluster - based methods .this was the case in our experiments where the three clustering - based methods had 977 , 977 , and 38 experts for lr , sc , and tslr , respectively , and tslr produced a very good predictor ( see table [ tbl : all ] ) .when the number of experts does not deviate widely , the advantages of the hierarchical strategy are not so pronounced .such a situation might cause the data to be overfitted through the repeated aggregation . hence , whether flat or hierarchical aggregation is preferable depends on the distribution of the experts in the clustering - based methods , as well as the nature of the data itself .in the present study , we have developed prediction algorithms that aggregate the experts generated by clustering - based methods .our aggregation framework has been then applied to the prediction of glaucoma progression .there are three main differences between our proposed method and the existing approach .first , we have used an empirically optimized learning rate .second , the expert weights are determined by using a batch process .third , we have examined the hierarchical aggregating strategies for multiple clustering - based methods .we found that the hierarchical aggregating strategy gives consistently better predictions than the traditional patient - wise linear regression , single clustering - based methods , the best expert , and the flat aggregation .the findings reported here may contribute to knowledge discovery and data mining by connecting clustering methods and predictions through aggregation algorithms .our method for aggregating experts generated several clustering methods that obtained better prediction results than the original clustering - based method .this suggests that the prediction accuracy could be further improved by embedding additional clustering methods .  from a clinical significance viewpoint ,our method may help to establish good predictions of glaucoma progression .in addition to its good prediction performance , our framework of aggregating cluster - based predictors may be sufficiently flexible to include other clustering methods and predictors , because the algorithm does not assume any specific constraints. this flexibility will allow clinicians to add novel prediction methods .therefore , our method will contribute to improving the future performance of glaucoma predictions .moreover , our proposed method is not limited to glaucoma progress prediction .therefore , our proposed aggregating strategies for clustering - based methods can be applied in a wide range of areas .              c. mayama , m. araie , y. suzuki , k. ishida , t. yamamoto , y. kitazawa , m. shirakashi , h. abe , h. tsukamoto , h. k. mishima , k. yoshimura , and y. ohishi .statistical evaluation of the diagnostic accuracy of methods used to determine the progression of visual field defects in glaucoma ., 111(11):21172125 , 2004 .k. morino , y. hirata , r. tomioka , h. kashima , k. yamanishi , n. hayashi , s. egawa , and a. kazuyuki . predicting disease progression from short biomarker series using expert advice algorithm ., 5:8953 , 2015 .r. a. russell , r. malik , b. c. chauman , d. p. crabb , and d. f. garway - heath . improved estimates of visual field progression using bayesian linear regression to integrate structural information in patients with ocular hypertension ., 53(6):27602769 , 2012 .

Abstract: this study addresses the issue of predicting the glaucomatous visual field loss from patient disease datasets . our goal is to accurately predict the progress of the disease in individual patients . as very few measurements are available for each patient , it is difficult to produce good predictors for individuals . a recently proposed clustering - based method enhances the power of prediction using patient data with similar spatiotemporal patterns . each patient is categorized into a cluster of patients , and a predictive model is constructed using all of the data in the class . predictions are highly dependent on the quality of clustering , but it is difficult to identify the best clustering method . thus , we propose a method for aggregating cluster - based predictors to obtain better prediction accuracy than from a single cluster - based prediction . further , the method shows very high performances by hierarchically aggregating experts generated from several cluster - based methods . we use real datasets to demonstrate that our method performs significantly better than conventional clustering - based and patient - wise regression methods , because the hierarchical aggregating strategy has a mechanism whereby good predictors in a small community can thrive . + * keywords : glaucoma , hierarchical aggregating strategy *


the aim of the present paper is to present a range of cluster validation indexes that provide a multivariate assessment covering different complementary aspects of cluster validity . herei focus on `` internal '' validation criteria that measure the quality of a clustering without reference to external information such as a known `` true '' clustering .furthermore i am mostly interested in comparing different clusterings on the same data , which is often referred to as `` relative '' cluster validation .this can be used to select one of a set of clusterings from different methods , or from the same method ran with different parameters such as different numbers of clusters .  in the literature ( for an overviewsee halkidi _ _ et al.__ ) many cluster validation indexes are proposed .usually these are advertised as measures of global cluster validation in a univariate way , often under the implicit or explicit assumption that for any given dataset there is only a single best clustering .mostly these indexes are based on contrasting a measure of within - cluster homogeneity and a measure of between - clusters heterogeneity such as the famous index proposed by calinski and harabasz , which is a standardised ratio of the traces of the pooled within - cluster covariance matrix and the covariance matrix of between - cluster means .  in hennig ( see also hennig )i have argued that depending on the subject - matter background and the clustering aim different clusterings can be optimal on the same dataset .for example , clustering can be used for data compression and information reduction , in which case it is important that all data are optimally represented by the cluster centroids ; or clustering can be used for recognition of meaningful patterns , which are often characterised by clear separating gaps between them . in the former situation , large within - cluster distances are not desirable , whereas in the latter situation large within - cluster distances may not be problematic as long as data objects occur with high density and without gap between the objects between which the distance is large .see figure [ fxyunif ] for two different clusterings on an artificial dataset with 3 clusters that may be preferable for these two different clustering aims .  given a multivariate characterisation of the validity of a clustering , for a given application a user can select weights for the different characteristics depending on the clustering aim and the relevance of the different criteria . a weighted average can then be used to choose a clustering that is suitable for the specific application .this requires that the criteria measuring different aspects of cluster validity and normalised in such a way that their values are comparable when doing the aggregation . although it is easy in most cases to define criteria in such a way that their value range is  ] is a normalised version .when different criteria are aggregated ( see section [ sagg ] ) , it is useful to define them in such a way that they point in the same direction ; i will define all normalised indexes so that larger values are better . for this reason subtracted from 1 .there are alternative ways of measuring whether within - cluster dissimilarities are overall small .all of these operationalise cluster homogeneity in slightly different ways .the objective function of k - means clustering can be written down as a constant times the average of all squared within - cluster euclidean distances ( or more general dissimilarities ) , which is an alternative measure , giving more emphasis to the biggest within - cluster dissimilarities .most radically , one could use the maximum within - cluster dissimilarity . on the other hand one could use quantiles or trimmed means in order to make the index less sensitive to large within - cluster dissimilarities , although i believe that in most applications in which within - cluster similarity is important , these should be avoided and the index should therefore be sensitive against them .apart from within - cluster homogeneity , the separation between clusters is most often taken into account in the literature on cluster validation ( most univariate indexes balance separation against homogeneity in various ways ) .separation as it is usually understood can not be measured by averaging all between - cluster dissimilarities , because it refers to what goes on `` between '' the clusters , i.e. , the smallest between - cluster dissimilarities , whereas the dissimilarities between pairs of farthest objects from different clusters should not contribute to this .the most naive way to measure separation is to use the minimum between - cluster dissimilarity .this has the disadvantage that with more than two clusters it only looks at the two closest clusters , and also in many applications there may be an inclination to tolerate the odd very small distance between clusters if by and large the closest points of the clusters are well separated .i propose here an index that takes into account a portion , say , of objects in each cluster that are closest to another cluster .  for every object let .let be the values of for ordered from the smallest to the largest , and let be the largest integer .then the -separation index is defined as obviously ,  ] is a suitable normalisation .   in some applications clustersare used for information reduction , and one way of doing this is to use the cluster centroids for further analysis rather than the full dataset .it is then relevant to measure how well the observations in a cluster are represented by the cluster centroid .the most straightforward method to measure this is to average the dissimilarities of all objects to the centroid of the cluster they re assigned to .let be the centroids of clusters .then , some clustering methods such as k - means and partitioning around medoids ( pam , kaufman and rousseeuw ) are centroid - based , i.e. , they compute the cluster centroids along with the clusters .centroids can also be defined for the output of non - centroid - based methods , most easily as which corresponds to the definition of pam .again , there are possible variations .k - means uses squared euclidean distances , and in case of euclidean data the cluster centroids do not necessarily have to be members of , they could also be mean vectors of the observations in the clusters .  again , by definition ,  ] .another way in which the clustering can be used for information reduction is that the clustering can be seen as a more simple summary or representation of the dissimilarity structure .this can be measured by correlating the vector of pairwise dissimilarities{i < j}\right) ] , where , and denotes the indicator function . with denoting the sample pearson correlation , this index goes back to hubert and schultz , see also halkidi _ _ et al.__ for alternative versions . ] .the idea that a cluster should be homogeneous can mean that there are no `` gaps '' within a cluster , and that the cluster is well connected .a gap can be characterised as a split of a cluster into two subclusters so that the minimum dissimilarity between the two subclusters is large .the corresponding index measures the `` length '' ( dissimilarity ) of the widest within - cluster gap ( an alternative would be to average widest gaps over clusters ) :  ] .  a version of this taking into account density values is defined in section [ sdens ] .widest gaps can be found computationally by constructing the within - cluster minimum spanning trees ; the widest distance occurring there is the widest gap .a very popular idea of a cluster is that it corresponds to a density mode , and that the density within a cluster goes down from the cluster mode to the outer regions of the cluster .correspondingly , there should be density valleys between different clusters .the definition of indexes that measure such a behaviour is based on a density function that assigns a density value to every observation . for euclidean data ,standard density estimators such as kernel density estimators can be used . for general dissimilarities ,i here propose a simple kernel density estimator .let be the -quantile of the vector of dissimilarities , e.g. , for , the 10% smallest dissimilarities are .define the kernel and density as these can be normalised to take a maximum of 1 : alternatively , with being the dissimilarity to the nearest neighbour would be another simple dissimilarity - based density estimator , although this has no trivial upper bound ( , even before normalising by its within - cluster maximum , is bounded by ) .one could also standardise by the within - cluster maxima if clusters with generally lower densities should have the same weight as high density clusters , but lower density values rely on fewer observations and are therefore less reliable .three different aspects of density - based clustering are measured by three different indexes :  1 .the density should decrease within a cluster from the density mode to the `` outskirts '' of the cluster ( ) .2 .  cluster boundaries should run through density `` valleys '' , i.e. , high density points should not be close to many points from other clusters ( ) .3 .  there should not be a big gap between high density regions within a cluster ( ; gaps as measured by may be fine in the low density outskirts of a cluster ) .  the idea for as follows . for every cluster ,starting from the cluster mode , i.e. , the observation with the highest density , construct a growing sequence of observations that eventually covers the whole cluster by always adding the closest observation that is not yet included .optimally , in this process , the within - cluster density of newly included points should always decrease .whenever actually the density goes up , a penalty of the squared difference of the densities is incurred . the index aggregates these penalties .the following algorithm computes this , and it also constructs a set that collects information about high dissimilarities between high density observations and is used for the definition of below :  initialisation : :   , . for   :step 1 : :   , where   .step 2 : :  let .if   : , if   go to step 1 , if then go to  step 5 .otherwise : step 3 : :  find   . ,   .step 4 : :  if , back  to step 2 .step 5 : :     collects the penalties from increases of the within - cluster densities during this process .the definition of does not take into account whether the neighbouring observations that produce high density values for are in the same cluster as .but this is important , because it would otherwise be easy to achieve a good value of by cutting through high density areas and distributing a single high density area to several clusters .a second index can be defined that penalises a high contribution of points from different clusters to the density values in a cluster ( measured by below ) , because this means that the cluster border cuts through a high density region . normalising : a penalty is incurred if for observations with a large density there is a large contribution to that density from other clusters : both and are by definition . also , the maximum contribution of any observation to any of and is , because the normalised -values are .these are penalties , so low values are good , and normalised versions are defined as an issue with is that it is possible that there is a large gap between two observations with high density , which does not incur penalties if there are no low - density observations in between .this can be picked up by a version of based on the density - weighted gap information collected in above .this is suggested instead of if a density - based cluster concept is of interest :  ].   sometimes different clusters should not ( only ) be characterised by gaps between them ; overlapping regions in data space may be seen as different clusters if they have different within - cluster density levels , which in some applications could point to different data generating mechanisms behind the different clusters , which the researcher would like to discover .such a cluster concept would require that densities within clusters are more or less uniform .this can be characterised by the coefficient of variation cv of either the within - cluster density values or the dissimilarities to the nearest within - cluster neighbour ( say ) .the latter is preferred here because as opposed to the density values , is clean from the influence of observations from the other clusters .define for , assuming : using this , low values are good .the maximum value of the coefficient of variation based on observations is ( katsnelson and kotz ) , so a normalised version is .   in some clustering applications , particularly where clustering is done for `` organisational '' reasons such as information compression , it is useful to have clusters that are roughly of the same size .this can be measured by the entropy : large values are good .the entropy is maximised for fixed by , so it can be normalised by .   in casethat there is a preference for a lower number of clusters , one could simply define ( already normalised ) with the maximum number of clusters of interest .if in a given application there is a known nonlinear loss connected to the number of clusters , this can obviously be used instead , and the principle can be applied also to other free parameters of a clustering method , if desired .sometimes the meaning of `` homogeneity '' for a cluster is defined by a homogeneous probability model , e.g. , gaussian mixture model - based clustering models all clusters by gaussian distributions with different parameters , requiring euclidean data .historically , due to the central limit theorem and quetelet s `` elementary error hypothesis '' , measurement errors were widely believed to be normally / gaussian distributed ( see stigler ) .under such a hypothesis it makes sense in some situations to regard gaussian distributed observations as homogeneous , and as pointing to the same underlying mechanism ; this could also motivate to cluster observations together that look like being generated from the same ( approximate ) gaussian distribution .indexes that measure cluster - wise gaussianity can be defined , see , e.g. , lago - fernandez and corbacho .one possible principle is to compare a one - dimensional function of the observations within a cluster to its theoretical distribution under the data distribution of interest ; e.g. , coretto and hennig compare the mahalanobis distances of observations to their cluster centre with their theoretical -distribution using the kolmogorow - distance .this is also possible for other distributions of interest .clusterings are often interpreted as meaningful in the sense that they can be generalised as substantive patterns .this at least implicitly requires that they are stable .stability in cluster analysis can be explored using resampling techniques such as bootstrap and splitting the dataset , and clustering from different resampled datasets can be compared .this requires to run the clustering method again on the resampled datasets and i will not treat this here in detail , but useful indexes have been defined using this principle , see , e.g. , tibshirani and walther and fang and wang .hennig lists further potentially desirable characteristics of a clustering , for which further indexes could be defined :  * areas in data space corresponding to clusters should have certain characteristics such as being linear or convex .* it should be possible to characterise clusters using a small number of variables .* clusters should correspond well to an externally given partition or values of an external variable ( this could for example imply that clusters of regions should be spatially connected ) . *variables should be approximately independent within clusters .the required cluster concept and therefore the way the validation indexes can be used depends on the specific clustering application .the users need to specify what characteristics of the clustering are desired in the application .the corresponding indexes can then be aggregated to form a single criterion that can be used to compare different clustering methods , different numbers of clusters and other possible parameter choices of the clustering .the most straightforward aggregation is to compute a weighted mean of selected indexes with weights expressing the relative importance of the different methods : assuming that large values are desirable for all of , the best clustering for the application in question can be found by maximising .this can be done by comparing different clusterings from conventional clustering methods , but in principle it would also be an option to try to optimise directly .the weights can only be chosen to directly reflect the relative importance of the various aspects of a clustering if the values ( or , more precisely , their variations ) of the indexes are comparable , and give the indexes equal influence on if all weights are equal . in section [ saspects ]i proposed tentative normalisations of all indexes , which give all indexes the same value range $ ] .unfortunately this is not good enough to ensure comparability ; on many datasets some of these indexes will cover almost the whole value range whereas other indexes may be larger than 0.9 for all clusterings that any clustering method would come up with .therefore , section [ sstupid ] will introduce a new computational method to standardise the variation of the different criteria .another issue is that some indexes by their very nature favour large numbers of clusters ( obviously large within - cluster dissimilarities can be more easily avoided for large ) , whereas others favour small values of ( separation is more difficult to achieve with many small clusters ) .the method introduced in section [ sstupid ] will allow to assess the extent to which the indexes deliver systematically larger or smaller values for larger .note that this can also be an issue for univariate `` global '' validation indexes from the literature , see hennig and lin .  if the indexes should be used to find an optimal value of , the indexes in should be chosen in such a way that indexes that systematically favour larger and indexes that systematically favour smaller are balanced .the user needs to take into account that the proposed indexes are not independent .for example , good representation of objects by centroids will normally be correlated with having generally small within - cluster dissimilarities . including both indexes will assign extra weight to the information that the two indexes have in common ( which may sometimes but not always be desired ) .there are alternative ways to aggregate the information from the different indexes .for example , one could use some indexes as side conditions rather than involving them in the definition of .for example , rather than giving entropy a weight for aggregation as part of , one may specify a certain minimum entropy value below which clusterings are not accepted , but not use the entropy value to distinguish between clusterings that fulfil the minimum entropy requirement .multiplicative aggregation is another option .as explained above , the normalisation in section [ saspects ] does not provide a proper calibration of the validation indexes .here is an idea for doing this in a more appropriate way .the idea is that random clusterings are generated on and index values are computed , in order to explore what range of index values can be expected on , so that the clusterings of interest can be compared to these .so in this section , as opposed to conventional probability modelling , the dataset is considered as fixed but a distribution of index values is generated from various random partitions .completely random clusterings ( i.e. , assigning every observation independently to a cluster ) are not suitable for this , because it can be expected that indexes formalising desirable characteristics of a clustering will normally give much worse values for them than for clusters that were generated by a clustering method .therefore i propose two methods for random clusterings that are meant to generate clusterings that make some sense , at least by being connected in data space .the methods are called `` stupid k - centroids '' and `` stupid nearest neighbours '' ; `` stupid '' because they are versions of popular clustering methods ( centroid - based clustering like k - means or pam , and single linkage / nearest neighbour ) that replace optimisation by random decisions and are meant to be computable very quickly .centroid - based clustering normally produces somewhat compact clusters , whereas single linkage is notorious for prioritising cluster separation totally over within - cluster homogeneity , and therefore one should expect these two approaches to explore in a certain sense opposite ways of clustering the data .stupid k - centroids works as follows .for fixed number of cluster draw a set of cluster centroids from so that every subset of size has the same probability of being drawn . is defined by assigning every observation to the closest centroid :    again , for fixed number of cluster draw a set of cluster initialisation points from so that every subset of size has the same probability of being drawn . is defined by successively adding the not yet assigned observation closest to any cluster to that cluster until all observations are clustered :  initialisation : :  let .let   step 1 : :  let .if   , find   ,  otherwise stop .step 2 : :  let . forthe   with , let   , updating   accordingly .go back to step 1 .  at the end , .the random clusterings can be used in various ways to calibrate the indexes . for any value of interest , clusterings  on generated , say .as mentioned before , indexes may systematically change over and therefore may show a preference for either large or small . in order to account for this , it is possible to calibrate the indexes using stupid clusterings for the same , i.e. , for a clustering with .consider an index of interest ( the normalised version is used here because this means that large values are good for all indexes ) .then , where .a desired set of calibrated indexes can then be used for aggregation in ( [ eagg ] ) .an important alternative to ( [ ecali ] ) is calibration by using random clusterings for all values of together .let be the numbers of clusters of interest ( most indexes will not work for ) , , . with this,  does not correct for potential systematic tendencies of the indexes over , but this is not a problem if the user is happy to use the uncalibrated indexes directly for comparing different values of ; a potential bias toward large or small values of in this case needs to be addressed by choosing the indexes to be aggregated in ( [ eagg ] ) in a balanced way .this can be checked by computing the aggregated index also for the random clusterings and check how these change over the different values of .another alternative is to calibrate indexes by using their rank value in the set of clusterings ( random clusterings and clusterings to compare ) rather than a mean / standard deviation - based standardisation .this is probably more robust but comes with some loss of information .the first example is the artificial dataset shown in figure [ fxyunif ] .four clusterings are compared ( actually many more clusterings with between 2 and 5 were compared on these data , but the selected clusterings illustrate the most interesting issues ) .the clusterings were computed by k - means with and , single linkage cut at and pam with .the k - means clustering with and the single linkage clustering are shown in figure [ fxyunif ] .the k - means clustering with puts the uniformly distributed widespread point cloud on top together in a single cluster , and the two smaller populations are the second cluster .this is the most intuitive clustering for these data for and also delivered by most other clustering methods .pam does not separate the two smaller ( actually gaussian ) populations for , but it does so for , along with splitting the uniform point cloud into three parts ..normalised index values for four clusterings on artificial data .[ cols="<,>,>,>,>",options="header " , ]   table [ ttetrac ] shows the corresponding results with calibration using all random clusterings .this does not result in a different ranking of the clusterings , so this dataset does not give a clear hint which of the two calibration methods is more suitable , or , in other words , the results do not depend on which one is chosen .the multivariate array of cluster validation indexes presented here provides the user with a detailed characterisation of various relevant aspects of a clustering .the user can aggregate the indexes in a suitable way to find a useful clustering for the clustering aim at hand .the indexes can also be used to provide a more detailed comparison of different clustering methods in benchmark studies , and a better understanding of their characteristics .the methodology is currently partly implemented in the `` fpc''-package of the statistical software system r and will soon be fully implemented there .most indexes require and the approach can therefore not directly be used for deciding whether the dataset is homogeneous as a whole ( ) .the individual indexes as well as the aggregated index could be used in a parametric bootstrap scheme as proposed by hennig and lin to test the homogeneity null hypothesis against a clustering alternative .research is still required in order to compare the different calibration methods and some alternative versions of indexes .a theoretical characterisation of the indexes is of interest as well as a study exploring the strength of the information overlap between some of the indexes , looking at , e.g. , correlations over various clusterings and datasets .random clustering calibration may also be used together with traditional univariate validation indexes .further methods for random clustering could be developed and it could be explored what collection of random clusterings is most suitable for calibration ( some work in this direction is currently done by my phd student serhat akhanli ) .this work was supported by epsrc grant ep / k033972/1 .a. m. bowcock , a. ruiz - linares , j. tomfohrde , e. minch , j. r. kidd , and l. l. cavalli - sforza .high resolution of human evolutionary trees with polymorphic microsatellites ._ nature _ , 368 , 455457 , 1994 . t. calinski and j. harabasz . a dendrite method for cluster analysis ._ communications in statistics _ 3 , 1-27 , 1974 .p. coretto and c. hennig .robust improper maximum likelihood : tuning , computation , and a comparison with other methods for robust gaussian clustering . _ journal of the american statistical association _ 111 , 16481659 , 2016 . y. fang and j. wang .selection of the number of clusters via the bootstrap method ._ computational statistics and data analysis _, 56 , 468 - 477 , 2012 .p. franck , e. cameron , g. good , j .- y .rasplus and b. p. oldroyd . nest architecture and genetic differentiation in a species complex of australian stingless bees ._ molecular ecology _ , 13 , 23172331 , 2004 . m. halkidi , m. vazirgiannis and c. hennig .method - independent indices for cluster validation and estimating the number of clusters . in _ handbook of cluster analysis _ , c. hennig , m. meila , f. murtagh , r. rocci ( eds . ) , crc / chapman & hall , boca raton , 703730 , 2016 . b. hausdorf and c. hennig . species delimitation using dominant and codominant multilocus markers . _ systematic biology _ , 59 , 491503 , 2010 . c. hennig .how many bee species ?a case study in determining the number of clusters . in _ data analysis , machine learning and knowledge discovery _ , m. spiliopoulou , l. schmidt - thieme , r. janning ( eds . ) , springer , berlin , 4149 , 2013 .c. hennig .what are the true clusters ?_ pattern recognition letters _ 64 , 5362 , 2015 . c. hennig .clustering strategy and method selection . in _ handbook of cluster analysis _ , c. hennig , m. meila , f. murtagh , r. rocci ( eds . ) , crc / chapman & hall , boca raton , 703730 , 2016 . c. hennig and c .- j .flexible parametric bootstrap for testing homogeneity against clustering and assessing the number of clusters . _ statistics and computing _ 25 , 821833 , 2015 . l. j. hubert and p. arabie .comparing partitions . _ journal of classification _ , 2 , 193218 , 1985 . l. j. hubert and j. schultz .quadratic assignment as a general data analysis strategy ._ british journal of mathematical and statistical psychology _ 29 , 190-241 , 1976 . l. kaufman and p. j. rousseeuw ._ finding groups in data _ , wiley , new york , 1990 .j. katsnelson and s. kotz . on the upper limits of some measures of variability ._ archiv fr meteorologie , geophysik und bioklimatologie , series b _ 8 , 103-107 , 1957 . l. f. lago - fernandez and f. corbacho .normality - based validation for crisp clustering ._ pattern recognition _ 43 , 782-795 , 2010 . s. stigler ._ the history of statistics : the measurement of uncertainty before 1900_. harvard university press , cambridge , 1986 .r. tibshirani and g. walther .cluster validation by prediction strength ._ journal of computational and graphical statistics _ , 14 , 511528 , 2005 .

Abstract: there are many cluster analysis methods that can produce quite different clusterings on the same dataset . cluster validation is about the evaluation of the quality of a clustering ; `` relative cluster validation '' is about using such criteria to compare clusterings . this can be used to select one of a set of clusterings from different methods , or from the same method ran with different parameters such as different numbers of clusters .   there are many cluster validation indexes in the literature . most of them attempt to measure the overall quality of a clustering by a single number , but this can be inappropriate . there are various different characteristics of a clustering that can be relevant in practice , depending on the aim of clustering , such as low within - cluster distances and high between - cluster separation .  in this paper , a number of validation criteria will be introduced that refer to different desirable characteristics of a clustering , and that characterise a clustering in a multidimensional way . in specific applications the user may be interested in some of these criteria rather than others . a focus of the paper is on methodology to standardise the different characteristics so that users can aggregate them in a suitable way specifying weights for the various criteria that are relevant in the clustering application at hand .  +  + * keywords : * number of clusters , separation , homogeneity , density mode , random clustering