numerical sampling of rugged energy landscapes is notoriously difficult .transition rates between two states are exponential functions of the energy barrier between them divided by the temperature .the sampling of complex systems is a daunting task because there are many states of comparable energies separated by large barriers .one of the most widespread sampling methods at finite temperatures is the monte carlo method , where one follows one or more trajectories of a virtual brownian particle as it moves through the state space . to sample a system at a given temperature , there exists a plethora of approaches , and amongst many others particularly notable are the standard ( metropolis ) and the kinetic monte carlo methods . at low temperaturesthese stochastic schemes tend to take a long time before a satisfactory result is reached . if one is interested in the behavior of a specific problem in the low - temperature limit , the common method is simulated annealing . if , however , one samples systems where below a certain temperature the state space splits into regions which are separated by huge barriers , like for example in spin glass systems , simulated annealing does not always lead to satisfactory results .examples of accepted solutions to this problem include the exchange monte carlo method and the population annealing .although good results are obtainable for most systems using these techniques , relaxation time still stays a crucial factor . for the protein folding problem for example ,the longest simulations available on present day computers are still far from the equilibrium distributions .a possible step in resolving the problem of over - long relaxation times was proposed in ref . where a method to accelerate the sampling of continuous systems was introduced .the basic idea is to rewrite the fokker - planck equation , which describes the time evolution of the probability distribution for continuous systems , into an imaginary - time schrdinger equation , for which one artificially introduces an energy gap between the ground state and the first excited state .then the relaxation time , which is proportional to the inverse of the energy gap , is reduced .  in the present paper, we extend the idea of ref . in sec .[ sec : method ] to discrete systems and implement it into a stochastic sampling scheme .next , in sec .[ sec : results ] , we analyze how our method can be used to improve the performance of simulated annealing of the traveling salesman problem and the three - dimensional ising spin glass . in the case of the traveling salesman problem we find that the simulated annealing is significantly accelerated and the modified sampling finds the approximately optimal solution much faster than unmodified simulations .our method also leads to improvements for the three - dimensional gaussian ising spin glass .[ sec : conclusio ] concludes this paper .in this section we extend idea of ref . to discrete systems and discuss its implementation into the kinetic monte carlo algorithm .we furthermore introduce a simplified version of the kinetic monte carlo scheme to speed up the calculations and save computational resources .   the master equation whose transition rates fulfill the detailed balance condition reads , if we set , with , where is the partition function of the system and is the energy of state , we get since , as defined in eq .( [ eqn : hamab ] ) , is a real , symmetric matrix , we call it a hamiltonian and thus , eq . ( [ eqn : hamab ] ) may be regarded as an imaginary - time schrdinger equation .it has a zero eigenvalue with eigenvector , .this follows directly from the definitions of and and the detailed balance condition in eq .( [ eqn : masta ] ) .the lowest eigenvalue of is therefore zero as guaranteed by the perron - frobenius theorem .following the idea in ref . , we make the transformation , where is the projector to the state of zero eigenvalue , which is expected to shorten the relaxation time toward equilibrium .the spectrum of is then shifted by for all states except the ground state .it is easy to see that the matrix elements of the projector are , since and the eigenvectors of corresponding to different eigenvalues are orthogonal .therefore , the imaginary - time schrdinger equation with the modified hamiltonian is the solutions of this equation are , where the and are the eigenvectors and corresponding eigenvalues of , respectively .  making use of the relation , we can translate eq .( [ eqn : qlambda ] ) into the modified master equation and deduce its solution from eq .( [ eqn : qsol ] ) as for large times this decays to i.e. the boltzmann weight , as expected .however , as can be seen from eq .( [ eqn : pt ] ) , the relaxation is faster than the case where is absent .   in ref . , a similar idea was tested for continuous systems using a diffusion monte carlo calculation . in the present discrete case, a straightforward implementation of our method is through the kinetic monte carlo algorithm .let us first describe this monte carlo method on the original master equation ( [ eqn : masta ] ) in order to make is clear what parts need modifications to accommodate the -term in eq .( [ eqn : qlambda ] ) .the idea is to try to generate time ` trajectories ' of the system among its various available states in such a way as to satisfy the master equation .assume the system is in a given state .the rate ( or probability per unit time ) at which the system will escape from to any available state is given by the second term on the right - hand side of eq .( [ eqn : masta ] ) .in other words , it is equal to . therefore , the probability distribution of the escape - time from is given by the poisson distribution .the time at which the system will leave state can thus be drawn from this distribution .the probability for the system to go from to a state is obviously given by the ratio .the practical implementation of the algorithm goes as follows .assume that the system is in state at time :  * the system will make a random transition out of state at a time where is drawn from the distribution . in practice ,one draws a number uniformly distributed between 0 and 1 and takes . * the state to which the system will make the transitionis chosen with probability . a simple way to dothis is to draw all individual probabilities consecutively until eventually they fill up the interval $ ] .then one draws a uniform random number between 0 and 1 , and the state to which the system jumps is the one indexed by the transition probability which corresponds to on the interval [ 0,1 ] .  by generating many such trajectories ,one generates probability distributions which stochastically satisfy the master equation .let us note that all trajectories generated this way are _ statistically independent _ , and can thus be used to compute averages .we now turn to an implementation of the kinetic monte carlo when one introduces the parameter .let us first expand unity as where the sum is over the states accessible from a given state .if all states are accessible in principle , which is the case if the system has no intrinsic dynamics , like the ising model , then we restrict to some subset depending on , for example the nearest neighbors .next , insert eq .( [ eqn : unity ] ) into the outgoing part of the master equation , to obtain note that , even if the physical considerations do not allow non - local moves , we can define transition probabilities for such moves and thus incorporate non - local states into the list in a very straightforward manner .  the introduction of the term in eq .( [ eqn : unity ] ) and eq .( [ eqn : out ] ) makes the present implementation unsuitable to reproduce finite - temperature properties . to show this fact ,let us first recall that a frequently - used transition probability is the heat bath ( or glauber ) method which trivially fulfills the detailed balance condition .however , the modified master equation is not symmetric in its outgoing and incoming parts .the kinetic monte carlo algorithm uses only the outgoing part for sampling , and therefore , adding the terms tilts the detailed balance in favor of the energetically lower lying states . to see this, we write the detailed balance condition as inserting the transition rate ( [ glauber_trprob ] ) into the above equation , we get / z_a ) } { e^{-\beta e_a}(1+\lambda [ e^{-\beta e_a}+e^{-\beta e_c } ] /z_c)}=e^{-\beta(e_c - e_a)}\frac{z_c}{z_a}\frac{z_a+\lambda [ e^{-\beta e_a}+e^{-\beta e_c } ] } { z_c+\lambda [ e^{-\beta e_a}+e^{-\beta e_c } ] } .\ ] ] we see clearly that the detailed balance in its conventional form is not satisfied .if we recall that , which represents the sum of boltzmann factors of the subset of states accessible from state , then we can conclude that if , , there are more energetically lower lying states available from than from .this would suggest that in this implementation ( eq .( [ eqn : out ] ) ) the addition of indeed tilts the detailed balance in favor of states from where more lower lying states are accessible .this changes the finite temperature values of physical observables when compared to calculations made with at the same temperature .nevertheless , when we are interested in the ground state solution , we do not have to worry about such finite temperature differences .there may be other implementations that do not use state- or temperature - dependent renormalizations of and thus allow us to keep detailed balance .however , we reserve the finite temperature case for future studies .  using the language and example of the traveling salesman problem( see sec .[ sec : models ] ) we now introduce a simplified version of the kinetic monte carlo scheme .sampling all nearest neighbors of a given tour , as needed for the kinetic monte carlo , is neither efficient nor feasible . at every monte carlo stepwe would have to calculate transition rates , where is the number of cities in the map .for example , if we would have to make about calculations .therefore , we choose just a certain number of nearest neighbors and generate another number of non - local states to build our list .this is possible since the traveling salesman problem has no intrinsic dynamics , and thus , all other states are accessible from any given state . by taking a set of nearest neighbor states and a set of non - local states as possible jumps available, we sample in effect the local structure as well as ` far ' away states and can in this fashion overcome large barriers .we can either choose to sample less local or non - local states than in the case of the kinetic monte carlo .in such a way the computational cost is reduced .non - local and no local trials .see the text for details . ]  before performing full - scale computations , we run preliminary simulations to check if the simplified method actually works and investigate what values of various parameters are to be used in practical calculations . in fig .[ kmchy ] a ) we compare the full kinetic monte carlo ( labeled k ) and the simplified method ( labeled h ) .we perform simulated annealing calculations of a small traveling salesman map , ` gr229 ' , with 229 cities . the annealing schedule is chosen to be step - wise growing ( see fig .[ fig : schedule ] for comparison ) , starting from an already low , the inverse temperature is increased by every monte carlo steps . for the simplified method , we chose 230 local states and 230 non - local states per monte carlo step at random to build the list , and sample all nearest neighbors plus 230 non - local states for the full kinetic monte carlo .the simplified method outperforms the full kinetic monte carlo , as the simplified method finds a shorter tour within the investigated time window .  in fig .[ kmchy ] b ) we investigate the parameter dependence of the simplified method .we look at the case where we allow no nearest neighbor hops and sample non - local states per monte carlo step to see the effect of adding to the transition rates while taking too few states to build the list .we see that for and very small ( red line ) there is a slight visible deviation to shorter tours from the simulation with ( black line ) , while a larger value of ( green squares ) affects the sampling in a negative way . as an extreme casewe take only non - local states and ( blue circles ) , and see that the system does not relax .the reason for this failure at , is that , with adding to the transition rates , jumps are facilitated .since we are looking at only two possible jumps , at some point both transitions will become approximately equally likely and the system fails to relax .we see that , if too few states to which jumps are possible are chosen , the sampling is influenced in an undesirable way .we use this simplified version of the kinetic monte carlo algorithm also for the ising model , since for larger system sizes calculating the hamiltonian and exponentiating it times ( there are nearest neighbors to a given state ) becomes quickly very time - consuming . as a rule of thumb , we sample , instead of , local states per monte carlo step .in this section we begin with a short review of the models we used for our calculations to establish the terminology .then we show the results of our simulations for different scenarios .we applied our method to two models .first we treat instances of the symmetric traveling salesman problem .given a set of coordinates on a two - dimensional plane , the task is to find the shortest closed path , called a tour , connecting all points while traversing each point only once .therefore , tours are ordered lists of the coordinates , giving the rule in which order to visit the coordinates .we take the distance between two points and on a tour to be euclidean so that the total length of the tour is calculated as the sum of all segments . to formulate this as a pseudo - physical problem, we identify the tours as the states , the tour length as the hamiltonian and we choose nearest neighbor hoppings as the local dynamics .nearest neighbors of a tour are defined as tours differing by the exchange of two points , i.e. having hamming distance of two to , , see fig . [fig : tsp ] .    with only nearest neighbor hoppingsit is hardly possible to solve the traveling salesman problem to optimality .we can , however , efficiently choose non - local hoppings , reversals or transport of tour segments .the second model we treat is the three - dimensional ising model on the cubic lattice of linear size , with random bonds .the energy of a given spin configuration is calculated by , where the spins take values .the bonds are quenched random numbers drawn from the distribution , which we will call the gaussian ising spin glass , and is an external field .we take only nearest neighbor jumps for the dynamics and use no non - local cluster flips , because we want to compare our method to the exchange monte carlo which gives satisfactory results with local sampling only .the nearest neighbors of a spin configuration are the configurations which differ from by a single spin flip , .simulated annealing relies on one hand on a stochastic sampling scheme and on the other on an annealing schedule , the rule of increase of the inverse temperature with iterations .we first show on the wiggly harmonic potential discussed in ref . what the effect of introducing is and then discuss the application of our method to simulated annealing . in this spiritwe then apply simulated annealing to the traveling salesman problem and the three - dimensional gaussian ising spin glass .consider a one - dimensional system where a number of local minima is evenly distributed over the large basin , as in fig . [ fig : wiggly ] .the distance between two neighboring minima is kept constant .the random walk of a point in this scenario is governed by the master equation describing the hopping between neighboring sites , and : where is the height of the barrier separating two minima and .when we take the continuum limit , the coarse - grained master equation becomes the fokker - planck equation }{\partial x}\label{fokkerplanck}\end{aligned}\ ] ] with diffusion constant .the fastest annealing schedule that minimizes the average energy is given by  , which coincides with the generic bound for convergence to reach the global minimum . with this schedulethe average energy decays as . `faster ' schedules than this do not further minimize the average energy .we identify the cause for this inverse - log law for the wiggly harmonic potential as an instability in the associated imaginary - time schrdinger equation. then we will propose a way how a step - wise growing schedule , see fig .[ fig : schedule ] , together with the considerations of sec .[ sec : method ] can improve the performance .the average energy in this improved case scales as , where is the time step index and is some constant , which implies .    to rewrite eq .( [ fokkerplanck ] ) as an imaginary - time schrdinger equation , let us set to get \psi(x , t)\nonumber\\ & = & -d\frac{\partial^2 \psi(x , t)}{\partial x^2}+\left[d\left(\frac{\beta}{2}x\right)^2-d\frac{\beta}{2}-\frac{\dot\beta x^2}{4}\right]\psi(x , t)\label{psi},\end{aligned}\ ] ] where is the time - derivative of the schedule .we can draw on an analogy with the quantum harmonic oscillator ( ) to have , . for a meaningful analysiswe require that the separation of energy levels , which is proportional to the frequency , be greater than or at least equal to zero , or this relation is asymptotically only fulfilled for functions which grow like or slower than that . to see thiswe first integrate eq .( [ stability ] ) and have we take the large limit and ignore compared to to get the usual schedule restriction . for energy levels coalesce and eq .( [ psi ] ) becomes a free diffusion equation with time dependent , monotonically decreasing diffusion coefficient .schedules which do not fulfill the relation ( [ stability ] ) have an imaginary and do not lead to a decaying solution .to circumvent this problem we employ the method proposed in ( see also sec .[ sec : method ] ) for the accelerated sampling of boltzmann distributions .if we choose the schedule like in fig .[ fig : schedule ] , , where is the heaviside step function , then for intermediate times , the schrdinger equation ( [ psi ] ) reads \psi(x , t)= h_i\psi\label{psiho},\end{aligned}\ ] ] where and .again , in correspondence with a harmonic oscillator , we obtain and denote the eigenfunction of in eq .( [ psiho ] ) to the eigenvalue as .we now employ the sudden approximation , which uses the fact that , if the system changes too quickly , the wave function can not follow and we can use the new hamiltonian in the time evolution operator with the previous wave function as the initial condition . let us assume that a particle is in the ground state of the hamiltonian at time step , . then just after the jump to the next time step , the wave function of the particle can be described by we insert unity as into the above equation to find note that for odd , the overlap vanishes since is an even function . to keep the probabilistic interpretation of , the normalization is chosen as therefore, the normalized ground state solution of the hamiltonian reads and the new wave function up to the slowest decaying term is for our approach to be sensible we have to wait again long enough for the decay of the excited state . however , the decay constant , as we learn from eq .( [ eqn : e_i ] ) , is exponentially increasing with , and therefore , we will have to wait longer and longer as time proceeds for the system to decay .now , we set , with , the projector to the ground state. then eq .( [ psievol ] ) becomes for any monotonically growing , vanishes exponentially , so that the slowest decaying terms of eq .( [ psievol.lambda ] ) decay approximately as .therefore , the decay constant is bounded from below by and is thus independent of the index .let us investigate what the above considerations mean for the average energy at a time , shortly before the next jump of at , where we have used eq .( [ eqn : phi_0 ] ) . for the schedule the prefactor becomes thus the average energy scales as , in contrast to , as is the case for the logarithmic schedule .we now turn our attention to the traveling salesman problem .we use the ` gr666 ' and ` u1060 ' data sets from the tsp - database with 666 and 1060 cities , respectively .the inverse temperature is chosen to be stepwise growing with monte carlo steps .starting from a base value , the inverse temperature is increased by after 3000 monte carlo steps , similarly to the plot in fig .[ fig : schedule ] . for the simulation we used the simplified method described in sec .[ sec : implementation ] . at each monte carlo step , we chose nearest neighbors and non - local states randomly , where is the number of cities .the results of the simulations are shown in fig .[ satsp ] a ) for ` gr666 ' and b ) ` u1060 ' , where the tour length is plotted versus the logarithm of the monte carlo steps. the effect and advantage of using our method are clearly visible .sampling for the same amount of iterations , we find much better solutions by using the transition rates modified by , as defined in eq .( [ eqn : out ] ) , than when using . stated otherwise , we can achieve an acceleration by an order of magnitude to reach a given tour length .     )do not correspond to equilibrium properties . ]next we treat the three - dimensional gaussian ising spin glass ( see sec . [sec : models ] ) .it is clear that a nave simulated annealing , with local jumps only , fails to find a good low temperature solution of this model in a reasonable time .there are many energetically close states which are separated by huge barriers , and thus the time needed to escape from a local minimum increases very quickly as the temperature decreases .nevertheless , we would like to see how our method affects the annealing procedure .the temperature is lowered in 480 steps from to .the coupling constants of the hamiltonian are drawn from a gaussian distribution , with zero mean and variance .the external field strength is chosen as . in fig .[ xmc ] we compare the a ) magnetization and b ) edwards - anderson order parameter , , of simulated annealing ( a ) with results from exchange monte carlo calculations ( x ) .notice that the finite - temperature values of the present method , marked ( a ) , do not represent the equilibrium properties for the reason discussed in sec . [sec : implementation ] .the exchange monte carlo is usually expected to give good results although we have not checked equilibration conditions since our goal is to compare the performance of the methods under the same conditions on computational cost .the first observation from the data is that the introduction of the term significantly improves the performance of simulated annealing at the lowest temperature .the values of physical quantities , and , have come close to those of the exchange monte carlo , the latter being a benchmark .another notable fact is that the term , at least for a small value , induces no perceptible change in the exchange monte carlo .lastly , the very large value of yields close results to those of the exchange monte carlo at the lowest temperature .the results for are , nevertheless , still slightly away from the exchange monte carlo values .these facts clarify the usefulness as well as limits of the present method for this problem of ising spin glass .based on the idea of ref . , we introduced a method to accelerate stochastic sampling of discrete , classical problems .our method suggests a way to overcome the limits of standard simulated annealing .we tested our algorithm on the traveling salesman problem , where , in the framework used in this paper , we find the shortest tour an order of magnitude faster by using our method than in the conventional case .simulated annealing of the three - dimensional gaussian ising spin glass is also accelerated . in this latter case ,the performance of our method is relatively close to that of the exchange monte carlo .  throughout our investigationwe used a simplified version of the full kinetic monte carlo algorithm to reduce the computational cost of the sampling at each monte carlo step .this simplified method outperforms the full kinetic monte carlo when we are faced with a plethora of accessible states , but when the choices are limited , adding tends to have undesirable effects if it is not chosen accordingly .however , in its present form the algorithm is useful only in the search for very low temperature solutions .  in conclusion , the present method would be a useful alternative of simple simulated annealing for optimization problems .the relatively straightforward implementation using kinetic monte carlo and non - local moves would make it a method of choice for some purposes , especially where computational cost is a factor .40 a. k. hartmann and h. rieger , _optimization algorithms in physics _ ,( wiley , berlin , 2002 ) . n. metropolis , a. w. rosenbluth , m. n. rosenbluth , a. h. teller and e. teller , _ j. chem . phys . _ * 21 * , 1087 ( 1953 ) .a. f. voter , in _ radiation effects in solids , k.e .sickafus and e.a .kotomin ( eds . ) _ , ( springer , dordrecht , 2005 ) . s. kirkpatrick , c. d. gelatt and m. p. vecchi , _ science _ * 222 * , 220 ( 1983 ) . v. cerny , _ j. opt .app . _ * 45 * , 41 ( 1985 ) .k. hukushima and k. nemoto , _ j. phys .jpn . _ * 65 * , ( 1996 ) .k. hukushima and y. iba , in _ the monte carlo method in the physical sciences , j.e .gubernatis ( ed . ) _ , ( the american institute of physics , 2003 ) .a. r. fersht and v. daggett , _ cell _ * 108 * , 573 ( 2002 ) .h. orland , _jpn . _ * 78 * , 103002 ( 2009 ) .g. reinelt , _ orsa j. comp . _* 3 * , 376 ( 1991 ) .[ http://comopt.ifi.uni-heidelberg.de/software/tsplib95 ] s. shinomoto and y. kabashima , _ j. phys . a : math .* 24 * , l141 ( 1991 ) .s. geman and d. geman , _ ieee trans .* 6 * , 723 ( 1984 ) .d. s. johnson and l. a. mcgeoch , in _ local search in combinatorical optimization , e.h.l .aarts and j.k .lenstra ( eds . ) _ , ( john wiley and sons inc . ,london , 1997 ) .s. lin and b. w. kernighan , _ oper . res . _ * 21 * , 498 ( 1973 ) .r. h. swendsen and j. s. wang , _ phys .lett . _ * 58 * , 86 ( 1987 ) .u. wolff , _ phys .lett . _ * 62 * , 361 ( 1989 ) .a. l. talapov and h. w. j. blte , _j. phys a : math .* 29 * , 5727 ( 1996 ) .g. schwabl , _ quantum mechanics _ , ( springer , heidelberg , 1995 ) .

Abstract: we propose a method to reduce the relaxation time towards equilibrium in stochastic sampling of complex energy landscapes in statistical systems with discrete degrees of freedom by generalizing the platform previously developed for continuous systems . the method starts from a master equation , in contrast to the fokker - planck equation for the continuous case . the master equation is transformed into an imaginary - time schrdinger equation . the hamiltonian of the schrdinger equation is modified by adding a projector to its known ground state . we show how this transformation decreases the relaxation time and propose a way to use it to accelerate simulated annealing for optimization problems . we implement our method in a simplified kinetic monte carlo scheme and show an acceleration by an order of magnitude in simulated annealing of the symmetric traveling salesman problem . comparisons of simulated annealing are made with the exchange monte carlo algorithm for the three - dimensional ising spin glass . our implementation can be seen as a step toward accelerating the stochastic sampling of generic systems with complex landscapes and long equilibration times .


since the early work of sobol , global sensitivity analysis ( gsa ) has received a lot of attention in the computer code experiments community .these days variance - based sensitivity indices are common tools in the analysis of complex physical phenomena .several statistical estimators have been proposed and their asymptotic properties are now well understood .in addition , the case of computationally expensive codes has been investigated thoroughly with the introduction of several dedicated surrogate models .however , even if they are extremely popular and informative importance measures , variance - based indices suffer from theoretical and practical limitations .first , by definition they only study the impact of the input parameters on the variance of the output .since this is a restricted summary of the output distribution , this measure happens to be inadequate for many case studies .alternative approaches include for example density - based indices , derivative - based measures , or goal - oriented dedicated indices .second , variance - based indices do not generalize easily to the case of a multivariate output .unfortunately , computer code outputs often consist of several scalars or even time - dependent curves , which limits severely the practical use of standard indices . finally for high - dimensional problems ,a preliminary screening procedure is usually mandatory before the analysis of the computer code or the modeling with a surrogate .the computational cost of gsa is in general too high to envision its use for screening purposes and more qualitative approaches are thus needed , e.g. the morris method or group screening .  in this paper, we propose a completely original point of view in order to overcome these limitations .starting from the general framework of gsa and the concept of dissimilarity measure , we introduce a new class of sensitivity indices which comprises as a special case the density - based index of .we propose an estimation procedure relying on density ratio estimation and show that it gives access to several different indices for the same computational cost .more importantly , we highlight that other special cases lead to well - known dependence measures , including the mutual information .this link motivates us to investigate the potential of recent state - of - the - art dependence measures as new sensitivity indices , such as the distance correlation or the hilbert - schmidt independence criterion .an appealing property of such measures is that they can handle multivariate random variables very easily .we also discuss how feature selection methods based on these measures can effectively replace standard screening procedures .the structure of the paper is as follows . in section [ dissim ], we introduce the general gsa framework based on dissimilarity measures and discuss the use of csiszr f - divergences .we also emphasize the link with mutual information and propose an estimation procedure . in section [ indep ] ,we give a review of some dependence measures and show how they can be used as new sensitivity indices .we also provide examples of feature selection techniques in which they are involved .screening will then be seen as an equivalent to feature selection in machine learning .finally , several numerical experiments are conducted in section [ secexp ] on both analytical and industrial applications . in particular , we illustrate the potential of dependence measures for gsa .denote the computer code output which is a function of the input random variables , where is assumed to be continuous . in standard global sensitivity analysis, it is further assumed that the have a known distribution and are independent . as pointed out by , a natural way of defining the impact of a given input on to consider a function which measures the similarity between the distribution of and that of .more precisely , the impact of on is given by where denotes a dissimilarity measure between two random variables .the advantage of such a formulation is that many choices for are available , and we will see in what follows that some natural dissimilarity measures yield sensitivity indices related to well known quantities . however before going further , let us note that the naive dissimilarity measure where random variables are compared only through their mean values produces the unnormalized sobol first - order sensivity index .   assuming all input random variables have an absolutely continuous distribution with respect to the lebesgue measure on , the f - divergence between and is given by where is a convex function such that and and are the probability distribution functions of and , respectively .standard choices for function include for example  * kullback - leibler divergence : or ; * hellinger distance : ; * total variation distance : ; * pearson divergence : or ; * neyman divergence : or . +  plugging this dissimilarity measure in ( [ si ] ) yields the following sensitivity index : where and are the probability distribution functions of and , respectively .first of all , note that inequalities on csiszr f - divergences imply that such sensitivity indices are positive and equal zero when and are independent .also , it is important to note that given the form of , it is invariant under any smooth and uniquely invertible transformation of the variables and , see the proof for mutual information in .this is a major advantage over variance - based sobol sensitivity indices , which are only invariant under linear transformations .+ it is easy to see that the total variation distance with gives a sensivity index equal to the one proposed by : in addition , the kullback - leibler divergence with yields that is the mutual information between and .a normalized version of this sensitivity index was studied by .similarly , the neyman divergence with leads to which is the so - called squared - loss mutual information between and ( or mean square contingency ) .these results show that some previously proposed sensitivity indices are actually special cases of more general indices defined through csiszr f - divergences . to the best of our knowledge ,this is the first work in which this link is highlighted .moreover , the specific structure of equation ( [ sif ] ) makes it possible to envision more efficient tools for the estimation of these sensitivity indices .indeed , it only involves approximating a density ratio rather than full densities .this point is investigated in the next subsection . but more importantly, we see that special choices for define sensivity indices that are actually well - known dependence measures such as the mutual information .this paves the way for completely new sensitivity indices based on recent state - of - the - art dependence measures , see section [ indep ] .   coming back to equation ( [ sif ] ), the goal is to estimate where is the ratio between the joint density of and the marginals .of course , straightforward estimation is possible if one estimates the densities , and with e.g. kernel density estimators .however , it is well known that density estimation suffers from the curse of dimensionality .this limits the possible multivariate extensions we discuss in the next subsection .besides , since only the ratio function is needed , we expect more robust estimates by focusing only on it .let us assume now that we have a sample of , the idea is to build first an estimate of the ratio .the final estimator of will then be given by powerful estimating methods for ratios include e.g. maximum - likelihood estimation , unconstrained least - squares importance fitting , among others ( see * ? ? ?a k - nearest neighbors strategy dedicated to mutual information is also discussed in .given our approach focusing only on densities , it is straightforward to extend the definition of the sensitivity index in equation ( [ sif ] ) to any number of input and output variables .we can then study the impact of a given group of input variables where is a subset of on a multivariate output with the sensitivity index given by this favorable generalization was already mentioned for the special cases of the total - variation distance and mutual information by and , respectively .however , in the high - dimensional setting , estimation of such sensitivity indices is infeasible since even the ratio trick detailed above fails .this is thus a severe limitation for screening purposes .we examine efficient alternatives in section [ indep ] .  moreover , note that extending the naive dissimilarity measure ( [ naive ] ) to the multivariate output case naturally leads to consider .straightforward calculations reveal that the corresponding sensitivity index is then the sum of sobol first - order sensitivity indices on each output . showed that this multivariate index is the only one possessing desired invariance properties in the variance - based index family .we focused above on csiszr f - divergences but other dissimilarity measures exist to compare probability distributions .in particular , integral probability metrics ( ipm , * ? ? ?* ) are a popular family of distance measures on probabilities given by for two probability measures and and where is a class of real - valued bounded measurable functions on .just as the choice of function in csiszr f - divergences gives rise to different measures , the choice of generates different ipms , e.g. the wasserstein distance , the dudley metric or the total variation distance .it is interesting to note that csiszr f - divergences and ipms are very distinct classes of measures , since they only intersect at the total variation distance .unfortunately , plugging the general expression ( [ ipm ] ) of an ipm in equation ( [ si ] ) no longer yields a closed - form expression for a sensitivity index .however , we plan to study such indices in a future work since estimation of ipms appears to be easier than for csiszr f - divergences and is independent of the dimensionality of the random variables .finally , let us mention the recent work of on goal - oriented measures , where they introduce a new class of sensitivity indices where is the contrast function associated to the features of interest and of and conditionally to , respectively ( note that we only give here the unnormalized version of the index ) .it is easy to check that ( [ contrast ] ) is a special case of ( [ si ] ) .given two random vectors in and in , dependence measures aim at quantifying the dependence between and in arbitrary dimension , with the property that the measure equals zero if and only if and are independent .in particular , they are useful when one wants to design a statistical test for independence . here , we focus on the long - known mutual information criterion , as well as on the novel distance correlation measure .recently , showed that it shares deep links with distances between embeddings of distributions to reproducing kernel hilbert spaces ( rkhs ) and especially the hilbert - schmidt independence criterion ( hsic , * ? ? ? * ) which will also be discussed .finally , we will review feature selection techniques introduced in machine learning which make use of these dependence measures .   mutual information ( mi , * ? ? ?* ) is a symmetric measure of dependence which is related to the entropy . assuming and are two random vectors which are absolutely continuous with respect to the lebesgue measure on and with density functions and , respectively , one can define their marginal entropy : and similarly . denoting their joint density function, the joint entropy between and writes mi is then formally defined as interestingly , mi equals zero if and only if and are independent .this implies that mi is able to detect nonlinear dependencies between random variables , unlike the correlation coefficient .it is also easy to check that with jensen s inequality .further note that it is not a distance since it does not obey the triangle inequality .a simple modified version yielding a distance , the variation of information ( vi ) , is given by another variant is the squared - loss mutual information ( smi , * ? ? ?* ) : which is again a dependence measure verifying with equality if and only if and are independent .applications of mi , vi and smi include independent component analysis , image registration and hierarchical clustering , among many others .+ in the context of global sensitivity analysis , we have seen in section [ fdiv ] that mi and smi arise as sensitivity indices when specific csiszr f - divergences are chosen to evaluate the dissimilarity between the output and the conditional output where is one of the input variables .we will then study the two following sensitivity indices : and a normalized version of given by has already been proposed by and compared to sobol sensitivity indices by .   the distance correlation was introduced by to address the problem of testing dependence between two random vectors in and in .it is based on the concept of distance covariance which measures the distance between the joint characteristic function of and the product of the marginal characteristic functions .more precisely , denote and the characteristic function of and , respectively , and their joint characteristic function . for a complex - valued function , we also denote the complex conjugate of and .the distance covariance ( dcov ) between and with finite first moment is then defined as a weighted -distance between and given by where the weight function with constants for is chosen to ensure invariance properties , see .the distance correlation ( dcor ) between and is then naturally defined as if and equals otherwise .important properties of the distance correlation introduced in ( [ dcor ] ) include that and if and only if and are independent .interestingly , the distance covariance in ( [ dcov ] ) can be computed in terms of expectations of pairwise euclidean distances , namely \label{dcove}\end{aligned}\ ] ] where is an i.i.d .copy of .concerning estimation , let be a sample of the random vector .following equation ( [ dcove ] ) , an estimator of is then given by .\label{dcovemp}\end{aligned}\ ] ] denoting , , , , and similarly for , show that equation ( [ dcovemp ] ) can be written as and is also equal to equation ( [ dcov ] ) if one uses the empirical characteristic functions computed on the sample .the empirical distance correlation is then and satisfies .although is a consistent estimator of , it is easy to see that it is biased . an unbiased version of and a specific correction for the high - dimensional case is investigated in .further note that also study defined as \label{dcova}\end{aligned}\ ] ] with the new weight function and constants as soon as and .distance covariance is retrieved for .the very general case of and living in metric spaces has been examined by .more precisely , let and be metric spaces of negative type ( see * ? ? ?* ) , the generalized distance covariance \label{dcovmetric}\end{aligned}\ ] ] characterizes independence between and .+ coming back to sensitivity analysis , just like we defined a new index based on mutual information , we can finally introduce an index based on distance correlation , i.e. which will measure the dependence between an input variable and the output . since distance correlation is designed to detect nonlinear relationships , we except this index to quantify effectively the impact of on .besides , considering that distance covariance is defined in arbitrary dimension , this index generalizes easily to the multivariate case : for evaluating the impact of a group of inputs on a multivariate output .[ rempf ] the limiting case in ( [ dcova ] ) interestingly leads to , see .this turns out to be another original way for defining a new sensitivity index .indeed , recall that sobol first - oder sensitivity index actually equals where is an independent copy of obtained by fixing , see .the idea is then to replace the covariance ( obtained with ) by dcov ( with ) : where pf stands for pick - and - freeze , since this index generalizes the pick - and - freeze estimator proposed by and is able to detect nonlinear dependencies , unlike the correlation coefficient .the hilbert - schmidt independence criterion proposed by builds upon kernel - based approaches for detecting dependence , and more particularly on cross - covariance operators in rkhss . here , we only give a brief summary and introduction on this topic and refer the reader to for details .let the random vector have distribution and consider a rkhs of functions with kernel and dot product .similarly , we can also define a second rkhs of functions with kernel and dot product associated to the random vector with distribution . by definition ,the cross - covariance operator associated to the joint distribution of is the linear operator defined for every and as -\mathbb{e}_xf(x)\mathbb{e}_yg(y).\ ] ] in a nutshell , the cross - covariance operator generalizes the covariance matrix by representing higher order correlations between and through nonlinear kernels . for every linear operator and provided the sum converges , the hilbert - schmidt norm of is given by where and are orthonormal bases of and , respectively .this is simply the generalization of the frobenius norm on matrices . the hsic criterion is then defined as the hilbert - schmidt norm of the cross - covariance operator : \label{hsic}\end{aligned}\ ] ] where the last equality in terms of kernels is proven in .an important property of is that it equals if and only if and are independent , as long as the associated rkhss and are universal , i.e. they are dense in the space of continuous functions with respect to the infinity norm . examples of kernels generating universal rkhss are e.g. the gaussian and the laplace kernels .it is interesting to note the similarity between the generalized distance covariance of equation ( [ dcovmetric ] ) and the hsic criterion ( [ hsic ] ) .actually , recently studied the deep connection between these approaches and show that if the kernels and generate the metrics and , respectively ( see * ? ? ?in particular , the standard distance covariance ( [ dcove ] ) is retrieved with the ( universal ) kernel which generates the metric .  assume now that is a sample of the random vector and denote and the gram matrices with entries and . propose the following consistent estimator for : where is the centering matrix such that .besides , it is easy to check that can be expressed just like the empirical distance covariance ( [ dcovemp ] ) : .\end{aligned}\ ] ] an unbiased estimator is also introduced by .+ we can finally propose a sensitivity index generalizing ( [ sdcor ] ) : where the kernel - based distance correlation is given by and the kernels inducing and have to be chosen within the class of universal kernels .the multivariate extension of is straightforward .the impact of the choice of kernels has previously been studied by in the context of independence hypothesis tests .  instead of working with the cross - covariance operator , work with the normalized cross - covariance operator ( nocco ) defined as ,see for the existence of this representation .just as the hsic criterion , the associated measure of dependence is given by .interestingly , is independent of the choice of kernels and is actually equal to the squared - loss mutual information ( [ smi ] ) under some assumptions , see . despite the advantage of being kernel - free , using in practice unfortunately requires to work with an estimator with a regularization parameter , which has to be selected . nevertheless , it is still interesting to use this approach for approximating smi efficiently , since dimensionality limitations related to density function estimation no longer apply .the pick - and - freeze estimator defined in remark [ rempf ] can be readily generalized with kernels : where this time only the kernel acting on needs to be specified .the kernel point of view in hsic also provides an elegant and powerful framework for dealing with categorical inputs and outputs , as well as functional ones .the categorical case is common practice in feature selection , since the target output is often represented as labels .appropriate kernels include for example where is the number of samples with label , see e.g. . from a gsa perspective, this implies that we can evaluate the impact of the inputs on level sets of the output by a simple change of variable for a given threshold .we can note the resemblance with the approach of if one uses a contrast function adapted to exceedance probabilities .  as a matter of fact , it is also possible to design dedicated semi - metrics for functional data which can be incorporated in the definition of the kernels , see e.g. .for example , let be such a semi - metric defined on when the output variable is of functional type .the kernel associated to is then given by where is a kernel acting on .the same scheme applies to functional inputs as well , see for an illustration in the context of surrogate modeling where the semi - metric is a cheap and simplified computer code .however , a theoretical shortcoming lies in our current inability to check if such semi - metric kernels are universal , which implies that we can not claim that independence can be detected . despite this deficiency ,we show in section [ secexp ] that from a practical perspective , the use of a semi - metric based on principal components can efficiently deal with a functional output given as a 2d map .   in machine learning ,feature selection aims at identifying relevant features ( among a large set ) with respect to a prediction task .the goal is to detect irrelevant or redundant features which may increase the prediction variance without reducing its bias . as a matter of fact, this closely resembles the objective of factor screening in gsa .the main difference is that in gsa , input variables are usually assumed to be independent , whereas in feature selection redundant features , i.e. highly dependent factors , precisely have to be filtered out .this apparently naive distinction actually makes feature selection an interesting alternative to screening when some input variables are correlated .but it is important to note that it is also a powerful option even in the independent case .we do not intend here to give an exhaustive review of feature selection techniques , but rather detail some approaches which make use of the dependence measures we recapped above .we hope that it will illustrate how they can be used as new screening procedures in high dimensional problems .+ literature on feature selection is abundant and entails many approaches . in the high dimensionalsetting , model - based techniques include for example the lasso or sparse additive models , see for a selective overview .generalizations for the ultra - high dimensional case usually replace penalty - based techniques to focus on marginal regression , where an underlying model is still assumed ( e.g. linear or non - parametric additive ) .another line of work for the ultra - high dimensional setting are model - free methods , where only dependence measures are used to identify relevant features . except for the very specific hsic lasso technique ,here we only focus on pure dependence - based approaches .let us first introduce the concept of max - dependency .denote the set of available features , the target output to predict and any measure quantifying the dependence between two random vectors .the max - dependency scheme for feature selection involves finding features which jointly have the largest dependency with , i.e. one has to solve the following optimization problem solving ( [ maxd ] ) is however computationally infeasible when and are large for cardinality reasons .near - optimal solutions are then usually found by iterative procedures , where features are added one at a time in the subset ( forward selection ) . on the other hand , the dependence measure must also be robust to dimensionality , which is hard to achieve in practice when the number of samples is less than .consequently , marginal computations which only involve terms are usually preferred .the max - relevance criterion serves in this context as a proxy to max - dependency , where the optimization problem writes but when the features are dependent , it is likely that this criterion will select redundant features . to limit this effect, one can add a condition of min - redudancy expressed as the final scheme combining ( [ maxr ] ) and ( [ minr ] ) , called minimal - redundancy - maximal - relevance ( mrmr ) , is given by forward and backward procedures for mrmr are investigated by where is chosen as the mutual information .similarly , forward and backward approaches where mi is replaced with hsic is introduced by .a purely marginal point of view is studied by where the authors propose the dcor criterion ( [ dcor ] ) . in a nutshell, the dcor measure is computed between and each factor , and only the features with dcor above a certain threshold are retained .a sure screening property of this approach is also proven . extend this work by considering a modified version of the hsic dependence measure ( supremum of hsic over a family of universal kernels , denoted sup - hsic ) .even if the sure screening procedure of this generalized method is proven , the authors mention that every feature selection technique based on marginal computations fails at detecting features that may be marginally uncorrelated with the output but are in fact jointly correlated with it . as a result, they propose the following iterative approach :  1 .compute the marginal sup - hsic measures between and each feature , and select the inputs with a measure above a given threshold .let be the subset of selected features .2 .  compute sup - hsic between and for each .augment with features having a measure greater than the sup - hsic criterion between and .repeat until the subset of selected features stabilizes or when its cardinality reaches a given maximum value .  as pointed out previously, another drawback of marginal computations which is not taken care of by the above scheme is that redundant variables are not eliminated . but design another iterative procedure to deal with this case . finally , let us note that in the examples of section [ secexp ] , we will only study the above iterative technique since we focus on independent input factors .we plan to investigate in particular the full mrmr approach for problems with correlated inputs in a future work .instead of working with forward and backward approaches , propose a combination of the lasso and the hsic dependence measure .denote for and the centered gram matrices computed from a sample of following the notations of section [ sechsic ] .the hsic lasso solves the following optimization problem with constraints and where stands for the frobenius norm and is a regularization parameter .interestingly , the first term of equation ( [ hsiclasso ] ) expands as using that , are symmetric and is idempotent , which highlights the strong correspondence with the mrmr criterion ( [ mrmr ] ) . the authors show that ( [ hsiclasso ] ) can be recast as a standard lasso program and propose a dual augmented lagrangian algorithm to solve the optimization problem .they also discuss a variant based on the dependence measure .we mentioned before that feature selection techniques based on dependence measures have been particularly designed for the ultra - high dimensional case , which is not the common setting of screening problems in gsa .nevertheless , we illustrate in section [ secexp ] that they perform remarkably well on complex benchmark functions , while requiring very few samples of the output variable .this reveals their high potential for preliminary screening of expensive computer codes .in this section , we finally assess the performance of all the new sensitivity indices introduced before on a series of benchmark analytical functions and two industrial applications .all benchmark functions can be found in the virtual library of simulation experiments available at http://www.sfu.ca/~ssurjano/index.html . for easier comparison, we first summarize the proposed indices in table [ tabsi ] ( si stands for sensitivity index and see for rbd - fast ) .[ [ standard - gsa . ] ] standard gsa . + + + + + + + + + + + + +  for the first experiments , we focus on gsa problems where the dimensionality is not too large ( less than input variables ) .the objective is to compare the information given by the new indices with sobol first - order and total indices .* decreasing function with , .we compute the sensitivity indices based on csiszr f - divergences , dcor , pick - and - freeze dcor , hsic and pick - and - freeze hsic ( gaussian kernels ) with a sample of size and we repeat this calculation 100 times . herewe use a simple kernel density estimator since we only study first - order indices .results are given in figure [ figlink ] ., , , and for function , , 100 replicates.,scaledwidth=90.0% ]  analytical first - order sis are for , which is coherent with the estimates at the top left .as expected , indices given by csiszr f - divergences , , , and provide the same information as variance - based ones in this simple case of a linear model . however , let us note that dcor and hsic detect non - influential factors very easily and robustly .* function with , ( the original function has constraint but we do not consider it here ) .conclusions are similar for the function , where only the first three inputs have a large impact on the output with very small interactions ( total sis almost equal first - order sis ) , see figure [ figloep ] .note that and recover since interactions are small .again , dcor and hsic clearly identify inputs which are independent of the output ., , , and for function , , 100 replicates.,scaledwidth=90.0% ]  * ishigami function  with , and constants taken from .this time we also compute since encompasses a strong iteration term .we use a sample of size for computing , , and , but now we also need a sample of size for with rbd - fast .estimates obtained with 100 replications are reported in figure [ figishi1 ] .while first - order sis indicate that has a negligible impact , it actually influences the output through an interaction term which is naturally accounted for by the total index .is is interesting to note that all other indices detect the impact of , as was pointed out by for the total - variation index .however , one can observe the striking adequacy between and ( unlike ) .this clearly shows that distance correlation has the potential to detect any interaction effect since it is specifically designed for nonlinear dependence .an additional appealing property is that its estimation does not depend on the number of inputs , unlike .this is a major advantage for expensive computer codes .finally , tends to bring the same information as . but recall that it has the same limitation as concerning computational cost due to the pick - and - freeze technique .the same comments apply to and .  , , , and for function , , 100 replicates.,scaledwidth=90.0% ]  we also investigate hsic on level sets of the ishigami function to compare our results with .more precisely , we use a categorical kernel and use the change of variable .figure [ figishi2 ] shows that we can recover the fact that input factor is more important than and for this level set function , as was observed by .   and for function , , 100 replicates.,scaledwidth=90.0% ]  [ [ screening . ] ] screening .+ + + + + + + + + +  we now propose to study the performance of feature selection as an alternative to screening for problems where the number of input variables is large ( more than ) .we will deliberately limit the number of samples in order to be as close as possible to a real test case on an expensive code .* function where , , , and is an integer controlling the number of influential inputs .we select , and compute , , and since all other indices are too expensive to compute in this setting ( recall that ) .first remark that first - order sis identify the influential inputs in mean , but there are many replicates for which they are confounded with non - influential ones . on the contrary, completely fails ate detecting them : it will then be excluded from the other tests on screening .notably , dcor and hsic perfectly discriminate the first five factors and identify the remaining ones as independent from the output .we also use the hsic lasso ( [ hsiclasso ] ) , where for each replicate we use a bootstrap procedure to evaluate the probability of selection of each input factor . here , hsic lasso performs very well since it selects the first 5 inputs factors almost every time .  , , and hsic lasso for function , , 100 replicates.,scaledwidth=90.0% ]  * function with , and are taken from .only the first eight factors are influential .  herewe also illustrate the feature selection method based on the iterative hsic scheme detailed in section [ secfs ] .results for are given in figure [ figsoblev1 ] .as expected , is unable to detect correctly the impact of the inputs . on the other hand , dcor and hsicaccurately estimate higher dependence for the first input factors than for the remaining ones .similarly , hsic lasso never selects the last inputs as influential ones .the iterative feature selection based on hsic performs well but tends to select more inputs than necessary .  , , and hsic lasso for function , , 100 replicates.,scaledwidth=90.0% ]  to go further and to compare our results with the ones obtained by , we repeat this experiment with ( approximately the sample size used by ) .this time first - order sis slightly detect the influential inputs , but the more interesting fact is that dcor , hsic and hsic lasso give even better results and almost perfectly identifies them .finally , the iterative hsic scheme now almost always discards the non - influential inputs .  , , and hsic lasso for function , , 100 replicates.,scaledwidth=90.0% ]   [ [ acquisition - strategy - for - reservoir - characterization .] ] acquisition strategy for reservoir characterization .+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +  in the petroleum industry , reservoir characterization aims at reducing the uncertainty on some unknown physical parameters of an oil reservoir by using all the data collected on the field , e.g. well logs , seismic images or dynamic data at the wells ( pressures , ... ) .basically , engineers solve a bayesian inverse problem where an initial prior distribution assumed on the parameters is updated by incorporating all field observations to produce a posterior distribution . in the end , this posterior distribution is used to predict the expected oil recovery of the reservoir in the future .however , it may be expensive to collect data and usually one wants to gather relevant observations only .this principle is at the core of so - called data acquisition strategies .for example , given the prior distribution , a natural idea is to get data which , when incorporated in the bayesian procedure , will reduce the most the uncertainty of the obtained posterior distribution .it is easy to see that this idea actually corresponds to performing a sensitivity analysis of the parameters when data varies , where the difference between the prior and the posterior distribution is given by the measure of uncertainty reduction one chooses , i.e. the dissimilarity function in equation ( [ si ] ) .since the number of both uncertain parameters and observations can be large , we can greatly capitalize on the advantages of dcor and hsic measures in arbitrary dimension to perform this task .our example here makes use of the punq reservoir test case , which is an oil reservoir model derived from real field data . in this simplified model ,seven variables which are characteristic of media , rocks , fluids or aquifer activity , are considered as uncertain ( permeability multipliers , residual oil saturations , ... ) and are assigned a uniform prior distribution . for illustration purposes, we assume that collectable data only consist of gas - oil ratios measurements at a given well .we generate a sample of size of the prior distribution , and propagate them through a fluid - flow simulator to get a sample of the simulated gas - oil ratios at the well over 10 years .they are given in figure [ figpunq ] , top .for each day in these 10 years , we compute the dcor measure between the parameters and the simulated ratio , see figure [ figpunq ] , bottom .this information makes it possible to pick up the days where measurements should be collected in order to reduce as much as possible the uncertainty on the parameters : at the beginning of the reservoir production ( before 500 days ) or around 3 years after .obviously , this procedure generalizes to any number of measurements and can be performed sequentially on many observations thanks to the properties of dcor .   of the collectable data ( top ) and dcor measure between the parameters and the data at each time step ( bottom ) for the punq test case.,scaledwidth=90.0% ]  [ [ screening - for - contamination - migration - in - waste - storage - site . ] ] screening for contamination migration in waste storage site .+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +  the marthe test case investigated here concerns prediction of the transport of strontium 90 in a porous water - saturated medium for evaluating the contamination of an aquifer in a temporary storage of radioactive waste .twenty input parameters mainly representative of the geological uncertainty are considered as random , and a set of 300 simulations is available at http://www.gdr-mascotnum.fr/benchmarks.html .accessible outputs are strontium 90 concentrations simulated at ten different wells , as well as the concentrations on a complete 2d map of the area ( discretized on pixels ) .we place ourselves in a screening setting where we use only simulations to identify the influential inputs . to estimate the variability of our results , we pick at random these 50 samples among the 300 available and repeat the procedure 100 times .we use the hsic dependence measure with a gaussian kernel first in its standard form by considering the vector of concentrations at the 10 observation wells .but we also take advantage of a kernel designed for the 2d maps as was mentioned in section [ secfunc ] .namely we use the pca semi - metric and vary the number of principal components ( 1 , 5 and 20 explaining , and of the total variance , respectively ) .results are given in figure [ figmarthe ] .first note that they are coherent with the ones obtained by where the authors used the 300 simulations to build a surrogate model . here, we then get the same detection of influential inputs but with only 50 simulations ( parameters i3 , kd1 , kd2 ) .in addition , the pca kernel leads to more discriminating indices as soon as the explained variance is sufficient ( 5 pcs ) .this clearly illustrates the potential of hsic for functional data .in this paper , we introduced a new class of sensitivity indices based on dependence measures which overcomes the insufficiencies of variance - based methods in gsa .we demonstrated that when the output distribution is compared with its conditional counterpart through csiszr f - divergences , sensitivity indices arise as well - known dependence measures between random variables .we then extended these indices by using recent state - of - the - art dependence measures , such as distance correlation and the hilbert - schmidt independence criterion .we also emphasized the potential of feature selection techniques relying on such dependence measures as alternatives to screening in high dimension .interestingly , these new sensitivity indices are very robust to dimensionality , have low computational cost and can be elegantly extended to functional and categorical output or input variables .this opens the door to new and powerful tools for gsa and factors screening for high dimensional and expensive computer codes .balasubramanian , k. , sriperumbudur , b. lebanon , g. 2013 , ultrahigh dimensional feature screening via rkhs embeddings , _ in _ ` proceedings of the sixteenth international conference on artificial intelligence and statistics ' , pp . 126134 .cukier , r. i. , fortuin , c. m. , shuler , k. e. , petschek , a. g. schaibly , j. h. 1973 , ` study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients .i theory ' , _ the journal of chemical physics _ * 59 * , 38733878 .gretton , a. , bousquet , o. , smola , a. schlkopf , b. 2005a , measuring statistical dependence with hilbert - schmidt norms , _ in _s. jain , h. simon e. tomita , eds , ` algorithmic learning theory ' , vol .3734 of _ lecture notes in computer science _ , springer berlin heidelberg , pp .6377 .ishigami , t. homma , t. 1990 , an importance quantification technique in uncertainty analysis for computer models , _ in _ ` first international symposium on uncertainty modeling and analysis ' , ieee , pp .398403 .manceau , e. , mezghani , m. , zabalza - mezghani , i. roggero , f. 2001 , combination of experimental design and joint modeling methods for quantifying the risk associated with deterministic and stochastic uncertainties - an integrated test study , _ in _ ` 2001 spe annual technical conference and exhibition , new orleans , 30 september-3 october ' .paper spe 71620 .morris , m. d. , moore , l. m. mckay , m. d. 2006 , ` sampling plans based on balanced incomplete block designs for evaluating the importance of computer model inputs ' , _ journal of statistical planning and inference _ * 136*(9 ) , 32033220 .peng , h. , long , f. ding , c. 2005 , ` feature selection based on mutual information criteria of max - dependency , max - relevance , and min - redundancy ' , _ pattern analysis and machine intelligence , ieee transactions on _ * 27*(8 ) , 12261238 .saltelli , a. , annoni , p. , azzini , i. , campolongo , f. , ratto , m. tarantola , s. 2010 , ` variance based sensitivity analysis of model output .design and estimator for the total sensitivity index ' , _ computer physics communications _ * 181*(2 ) , 259270 .sriperumbudur , b. k. , fukumizu , k. , gretton , a. , lanckriet , g. r. schlkopf , b. 2009 , kernel choice and classifiability for rkhs embeddings of probability distributions , _ in _ ` advances in neural information processing systems ' , pp . 17501758 .suzuki , t. , sugiyama , m. , sese , j. kanamori , t. 2008 , ` approximating mutual information by maximum likelihood density ratio estimation ' , _ journal of machine learning research - proceedings track _ * 4 * , 520 .volkova , e. , iooss , b. van dorpe , f. 2008 , ` global sensitivity analysis for a numerical model of radionuclide migration from the rrc `` kurchatov institute '' radwaste disposal site ' , _ stochastic environmental research and risk assessment _ * 22*(1 ) , 1731 .

Abstract: global sensitivity analysis with variance - based measures suffers from several theoretical and practical limitations , since they focus only on the variance of the output and handle multivariate variables in a limited way . in this paper , we introduce a new class of sensitivity indices based on dependence measures which overcomes these insufficiencies . our approach originates from the idea to compare the output distribution with its conditional counterpart when one of the input variables is fixed . we establish that this comparison yields previously proposed indices when it is performed with csiszr f - divergences , as well as sensitivity indices which are well - known dependence measures between random variables . this leads us to investigate completely new sensitivity indices based on recent state - of - the - art dependence measures , such as distance correlation and the hilbert - schmidt independence criterion . we also emphasize the potential of feature selection techniques relying on such dependence measures as alternatives to screening in high dimension .


atomic laser cooling and trapping experiments typically require several independently tuned laser beams distributed in frequency across atomic ground and excited state manifolds .for example , in the 5s ground state and 5p excited state hyperfine manifolds ( connected by the 780 nm d2 line ) have widths of 6.8 ghz and 500 mhz , respectively . in the course of a typical experiment it is often necessary to rapidly and accurately scan or jump the frequency of one or more lasers over these ranges , while maintaining absolute frequency reference to a particular transition .to meet these goals , we have developed a scheme to electronically lock a rapidly , widely and accurately tunable frequency difference between two lasers based on digitally counting their optical beat frequency .here we report on two particular implementations of the scheme : a `` small- '' lock , in which a slave laser is locked 120 mhz above a stable master laser ; and a `` large- '' lock , in which the slave is locked 6.7 ghz above the master , using nearly the same electronics .the remainder of the paper is structured as follows . in section [ sec : compare ] we review a number of published alternative schemes for locking the frequency difference between two lasers . in section [ sec : description ] we present a general description of our method , specializing to the two configurations realized in our laboratory ( viz . , the small- and large- locks described above ) . in section[ sec : signaltonoise ] we analyze theoretically the performance limits set by quantization error for a frequency servo loop based on sampled digital counting . in section [ sec : experiment ] we present the results of experiments assessing the stability , accuracy and agility of the locks .we conclude in section [ sec : conclusion ] with a brief discussion of potential improvements .a large number of techniques to stabilize the frequency difference between two lasers have been developed over the long history of laser spectroscopy . herewe review two broad categories that are most closely related to our method , and ( not coincidentally ) are often employed in laser cooling and trapping laboratories .  in _ optical injection locks_ a weak `` seed '' beam derived from a master laser is launched into the cavity of a slave laser , forcing the slave to lase on a longitudinal mode having frequency equal to that of the seed .frequency offsets between master and slave of 100 s of mhz can be achieved by frequency shifting the seed beam with an acousto - optic modulator ( aom ) . in this case , not only is the frequency difference between the lasers locked , but their optical fields are in fact phase coherent .disadvantages are that careful optical mode matching of the seed beam to the slave laser is required ; that the dynamical tuning range is limited by the modulation bandwidth of the aom ( typically less than 50 mhz ) ; and that the range of possible offset frequencies is limited by the small selection of commercially available aoms .  replacing the aom with an electro - optic modulator ( eom ) can extend the dynamical tuning range beyond 100 mhz .eoms are available over a wide range of operating frequencies up to 10 s of ghz . however , these benefits come at considerably higher cost and additional complexity , to the extent that optical injection locking with eom modulation is typically used only when phase - coherence is essential.  it is also possible to use fm sidebands imposed on the master laser as seed beams. while a large range of offsets can be achieved with this technique , it is only suitable for diode master lasers , and when only one or possibly two slaves are to be locked to one master , as it is challenging to isolate the effects of the spatially overlapping and co - propagating sidebands .  _ electronic offset locks _ , the family to which ours belongs , work by electronically detecting the optical beat note between beams from the master and slave lasers superimposed on a fast photodiode , then processing this signal to steer the frequency of the slave laser to maintain the desired offset .an electronic offset lock designed around a commercial frequency - to - voltage ( f - to - v ) ic is described in ref . [ 4 ] .unfortunately , commercial f - to - v chips are limited in operating frequency to at most 1 mhz , necessitating a deleteriously large prescaling factor to realize our small- lock , and rendering unfeasible our large- lock .furthermore , the charge pump output stage of commercial f - to - v chips typically exhibits a large ( 20% ) ripple that leads to undesirable dithering of the slave laser frequency even with aggressive low - pass post - filtering .the hybrid analog+digital locking scheme we demonstrate here realizes high performance f - to - v conversion by using fast ac cmos electronics to digitally sample and count the beat note frequency and a 10-bit dac to generate an analog error signal suitable for processing by analog servo electronics .  ref .[ 5 ] describes an electronic offset lock based on a rf mach - zehnder interferometer . the rf beat note is split , transmitted through two unequal lengths of coaxial cables and recombined in an rf power combiner .the length of one cable is manually adjusted to produce , through total destructive interference , a node in the total transmitted power when the difference frequency between the two lasers is the desired value .this scheme has the advantages of large capture range and a continuous selection of possible offset frequencies .unfortunately , since the offset frequency is tuned by a manual adjustment of the cable length , real - time , rapid tuning of the offset is impossible without additional frequency shifting components ( e.g. , aoms ) .another serious drawback is the lack of absolute frequency reference for the offset , which is determined by the physical dimensions of cables .finally , electronic optical phase - locked - loops ( opll ) have been used to lock two lasers with a tunable offset .our lock is well suited to experiments that do not warrant the complexity of an opll , or that involve broad - line lasers for which the wideband control demands of on opll would be difficult to meet .( our control loop has a bandwidth of  khz , but there is no critical level that must be met for stable operation , unlike for an opll . ) also , our scheme allows for multi - ghz tunable offsets without the need for a tunable local oscillator in the microwave range , as is typically employed in oplls .our hybrid analog+digital electronic frequency offset lock is appropriate when a single , low - cost , flexible solution is required for multiple locks spanning a wide range of offsets from one stable master laser .[ fig : blockdiagram ] shows the essential optical and electronic components of our scheme .a few milliwatts of light from the master laser ( optical frequency ) and slave laser ( optical frequency ) are combined on a fast photodetector .the induced photocurrent comprises a dc component proportional to the total incident optical power and an ac component oscillating at the optical beat note frequency .the ac signal is amplified , prescaled in the case of the large- lock , shifted via heterodyne mixing , and ultimately counted with digital electronics .the digital frequency counting starts with a fast zero - crossing detector generating a train of logic pulses synchronous with its analog input .the number of logic pulses ( typically 200 ) in each sample window of duration is counted by a gated 12-bit binary counter . at the conclusion of each sample windowthe counter s data is latched into a digital - to - analog converter ( dac ) that generates an output voltage proportional to the average frequency difference between the two lasers during that window .subtracting from a computer - controlled set - point voltage in a differencing amplifier produces an error signal suitable for subsequent processing by a pid ( proportional - integral - differential ) servo controller .the servo controller generates an electronic control signal that steers the slave laser frequency to maintain the desired offset with respect to the master .we implement the digital electronics using ac cmos logic , which has a theoretical maximum operating frequency of  mhz .since both our small- and large- locks are intended to maintain values of at or exceeding this limit , we must take steps to translate the frequency of the optical beat note down into the working range of our counting electronics .for example , to count the  mhz beat note of our small- lock , we heterodyne the optical beat note with a local oscillator of frequency  mhz in an rf mixer , then digitally count the _ down_-shifted copy of the beat note at  mhz .besides shifting the beat note into the working range of the counting electronics , the heterodyning step has other significant benefits .first , it provides an attractive method to tune the master - slave frequency difference by adjusting while leaving the reference voltage centered in the circuit s capture range. second , by counting the _ up_-shifted copy of the beat note from the rf mixer , we can lock a master - slave offset frequency for which the optical beat note would otherwise be too low in frequency to digitally count with the fidelity required for a high - bandwidth servo loop .to count the  ghz beat note of our large- lock we use a high bandwidth photoreceiver ( discovery semiconductor dsc - r402 - 89 ) and divide the frequency of the optical beat note by 32 using a high frequency divider ic , producing a signal at the prescaled frequency  mhz .the prescaled signal is then rf - heterodyned as in the small- lock , though with  mhz , to produce a ( down - shifted ) signal at  mhz , suitable for digital counting .[ subsec : digital ] the post - heterodyne signal at is input to a high speed zero - crossing detector that generates a synchronous ac cmos digital signal. this signal is binned and counted in sample windows of duration at a rate of 195.3 ksamples / s .the sample clock is derived from a 25.000 mhz crystal oscillator divided by 256 with an 8-bit binary counter . as shown in figs .[ fig : blockdiagram ] and [ fig : timingdiagram ] , a multiple - output shift register generates three auxiliary logic signals from the 195.3 khz sample clock , shifted by + 2 , + 3 , and + 4 cycles of the 25 mhz crystal oscillator ( i.e. , by 80 , 120 and 160 ns , respectively ) .the unshifted and `` + 3 '' signals are xord to generate the `` hold '' signal ; `` + 2 '' and `` + 3 '' are xord to generate the `` latch '' signal ; and `` + 3 '' and `` + 4 '' are xord to generate the `` reset '' signal .the rising edge of reset asynchronously clears the 12-bit counter ; counting commences 40 ns later , after the fall of reset , and continues for ( the `` sample window '' ) , until hold toggles high .( a logical - high hold suspends counting by preventing falling edges from the zero - crossing detector from arriving at the 12-bit counter . )as long as reset and hold are both low , the 12-bit counter will count logic pulses from the zero - crossing detector .the 80 ns delay between the close of the counting window and the start of data conversion by the dac ( on the rising edge of latch ) is included to satisfy a minimum data setup time specification of our dac .the result of this gating logic is a livetime fraction of approximately 97% .the lowest 10 bits of the 12-bit counter serve as the inputs to a 10-bit dac .the dac is operated in buffered , single - ended output configuration with an output voltage range of  v. to generate the `` error '' signal for the analog pid servo loop , is subtracted from the computer generated set - point voltage in a differencing amplifier .we choose separately for the small- and large- configuration so that is approximately centered in the capture range when the servo loop is locked .( color online . ) detecting frequency fluctuations by discretely sampled counting . ( a ) pictorial representation of the optical beat note frequency modulated by the noise waveform of ( b ) .( b ) representative fm noise waveform , identifying various symbols used in the text . ]there is a fundamental limit to the frequency stability of a lock such as ours that employs sampled digital frequency counting .suppose that counts ( i.e. , zero crossings ) of a noiseless carrier at frequency are detected in sample windows of duration ( neglecting deadtime ) .introduce monochromatic fm noise on the carrier . for simplicitywe restrict our calculation to noise frequencies with period an integer number of sample windows : .this fm noise on the carrier causes _ additional _counts during the noise waveform s positive half - cycle ( `` nwphc , '' from to ( see fig . [fig : signaltonoise ] ) , where .we define the _ minimum detectable noise amplitude _ as that fluctuation of the carrier frequency that causes exactly to accumulate over the nwphc .with the perspective that the `` signal '' we wish our circuit to detect ( and subsequently to correct via feedback ) is the instantaneous frequency fluctuation , and that the `` noise '' that disturbs this effort is the unavoidable count uncertainty in the least significant bit of the binary counter , we say that the fluctuation will be sensed by the circuit with a quantization error limited ( `` qel '' ) signal - to - noise ratio neglecting all other sources of noise .a fluctuation smaller than would go unnoticed ( and therefore uncorrected ) by the circuit , while larger fluctuations would be sensed ( and corrected ) with .  using relations developed in the preceding discussion, we find in a sampled digital frequency counter such as ours , the minimum detectable frequency fluctuation of the carrier decreases with increasing fm noise period ( decreasing fm noise frequency ) .this imposes a fundamental stability limit to a carrier whose frequency is controlled by a feedback loop that employs such a counter . in our lock ,the `` carrier '' is the optical beat note at .the qel implies that fluctuations of the average optical beat note frequency between successive time - averages of duration can not be smaller than and thus will exhibit a ( qel ) relative rms deviation of no less than   the division of the optical beat note by 32 in the large- lock increases and therefore the _ absolute _ frequency instability , by the same factor .however , since the large- lock prescaled frequency  mhz is larger than the unscaled optical beat note  mhz of the small- lock , the qel _ relative _ instability limit is only a factor of higher ( i.e. , worse ) for the large- lock .[ fig : allan ] shows lines demarcating the qel instability limits for both locks , in both relative and absolute units .here we report the results of several experiments to characterize the performance of both small- and large- locks .to facilitate comparison of the two configurations , the same master laser , slave laser and lock electronics were used in all experiments , save for the explicit differences indicated in fig .[ fig : blockdiagram ] .( color online . )long - term locking .time series of the optical beat note frequency , acquired over a 5 hour period , for the small- ( a ) and large- locks ( b ) .for comparison , time series are also shown for the slave laser unlocked . ]( color online . )optical beat note linewidth .rf spectrum analyzer traces of the optical beat note for small- ( a ) and large- ( b ) locks for the servo loops both locked and unlocked .fwhm linewidths are given for the locked data .for these measurements , the resolution bandwidth of the spectrum analyzer was 10 khz ( 100 khz ) for the small- ( large-) locks , respectively .the full - screen sweep time was approximately 20 ms for all traces . ]the master and slave lasers are both grating stabilized , external - cavity diode lasers with  mhz intrinsic `` fast '' linewidths . fig .[ fig : longterm ] shows a 5 hr time series of the optical beat note frequency for both locks under continuous operation , measured at 10 s intervals with an rf spectrum analyzer .we evaluated the short time stability of the optical beat note by directly measuring its linewidth over a 10 ms integration time using an rf spectrum analyzer ( see fig .[ fig : noise ] ) . the beat note linewidth was measured to have fwhm of 1.5 mhz ( 4.0 mhz ) for the small- ( large-) locks , respectively .the broadening of the large- linewidth is qualitatively consistent with quantization error limited performance ( see sec . [sec : signaltonoise ] ) .  to characterize frequency stability over longer times , a second, independent copy of the lock electronics was used to make an `` out of loop '' measurement of the frequency allan deviation of the beat note for integration times ; these data are presented in fig .[ fig : allan ] . to make these measurements , fluctuations of the error signal voltage of the out - of - loop counting boxwere recorded using a digital multimeter , and the measured voltage noise subsequently converted to frequency fluctuations .the deadtime fraction for the measurements is and is ignored in the analysis , except for the points at  s ( see fig .[ fig : allan ] caption ) . for the large- lock ,the data is fit to a white - noise - limited model for with a result .a 1/f noise dominated `` flicker floor '' model is fit to the data for with the result .similar fits are made to the small- lock data , yielding for and for .these fits are shown in fig .[ fig : allan ] .we could not measure allan deviations for integration times sufficiently short to probe the onset of qel performance for either lock . by extrapolating the data we estimate that the small- ( large-) locks would reach qel at  s (  s ) , respectively .  fig . [fig : allan ] also shows measurements of the allan deviation of the reference voltage , which in our system is supplied by a pc - based multifunction card , which are consistent with the observed level of the `` flicker floor '' for both locks .inspection of the raw time - series data show that the increase in instability of at  s is the result of rare , discrete jumps in .we do not have an explanation for the jumps , which are much smaller than can be attributed to lsb noise in the pc card s output dac . in the future we plan to replace the pc card with a higher - performance model , upgrade electrical connections and install opto - isolators to break ground loops , in an attempt to suppress these jumps .( color online . )frequency stability .absolute and relative frequency allan deviation data are shown as red solid squares for the small- ( a ) and large- ( b ) locks , for integrations times .. points at 900 s are not true allan deviations but rather the average deviation of 13 discrete measurements of the beat note taken at 15 minute intervals over 3 hours .the dashed and dashed - dot lines are fits to the measured allan deviation points , as described in the text .heavy black lines in both figures show the quantization error limited ( qel ) relative instability discussed in sec .[ sec : signaltonoise ] .solid black circles are the measured relative allan deviation of the computer - controlled reference voltage .open black diamonds show the counting noise floor of the counting electronics as measured in a separate experiment in which the optical beat note was replaced by the output of a high - quality electronic synthesizer set to the same frequency , with the high speed divider either removed ( a ) , or set to divide by 16 ( b ) .there is no contradiction in these measurements lying below the qel line , as they were made in open loop , whereas the qel applies only to the closed loop stability of the _ locked _ laser beat note .error bars are 1-standard deviation statistical uncertainties . ]we quantify the tuning agility of the lock as the average frequency slewing rate over the time it takes the beat note to track from 10% to 90% of its asymptotic value ( `` 10 - 90 '' time ) , following a large step in .[ fig : jump ] shows traces of , proportional to , from which we extract a 10 - 90 time of 71  ( 690  ) after a 70 mhz ( 416 mhz ) step , for an average slew rate of 1.0 mhz/ ( 1.7 mhz/ ) for the small- ( large-) locks , respectively .these results are consistent with the designed 100 khz bandwidth of the servo loop .( color online . )tuning agility .oscilloscope traces of , proportional to , show the response of the slave laser to an abrupt step of , corresponding to jumps of 70 mhz in the small- lock ( a ) and 416 mhz in the large- lock ( b ) .the stair - case pattern visible in in ( a ) directly reflects the discrete sampling and counting of the beat note .the steps are not visible in ( b ) due to its compressed time axis . ]the optical beat note can be accurately tuned over a wide range by adjusting the heterodyne local oscillator frequency .[ fig : accuracy ] shows data demonstrating the range and accuracy of closed - loop tuning of the slave laser in which is changed in 20 mhz steps for both the small- and large- locks .( for the large- lock , because of the division - by-32 prescaling of the optical beat note , the tuning steps are  mhz . ) while the 80 mhz capture range of the servo loop limits the size of an individual abrupt step , smooth adjustments to permit tuning limited on the high - end only by the bandwidth of the rf photodetector and on the low - end by the steepness of the post - heterodyne low - pass filter. as discussed in section [ sec : description ] , we reach the lowest lockable beat notes by selecting the up - shifted heterodyne component , while choosing the polarity of the servo loop appropriate to the sign of so as to yield overall negative feedback .( color online . )tuning range .the frequency of the locked optical beat note , for the small- lock ( a ) and large- lock ( b ) versus heterodyne local oscillator frequency , counted using an rf spectrum analyzer .a positive ( negative ) beat note corresponds to the slave laser frequency higher ( lower ) than the master laser .solid red circles ( solid blue squares ) correspond to locking to the down- ( up-)shifted copy of the beat note from the heterodyne stage .open symbols represent measurements made with a different value of the post - heterodyne rf low - pass filter so as to lock the lowest optical beat notes for a given configuration .curves through the data are lines of slope=1 ( = 32 ) through the lowest measured beat note of each series , for the small- ( large-) locks , respectively . ]we have designed and implemented a hybrid analog+digital scheme to lock the difference frequency between two lasers .the long term drift , short term stability , tuning range , tuning accuracy and overall flexibility of the scheme make it well suited to applications where the frequencies of several independent lasers need to be stabilized to an absolute reference _ and _ rapidly and accurately tuned over several ghz .improvements to the digital electronics , including optimized component layout , better ground plane practices , and the use of stripline leads for signal transport should allow the present design to reach the maximum clock rate for ac cmos logic of 120 mhz , for a modest increase in performance . a large jump in performancecould be realized by switching to a higher speed logic family , such as ecl , which would enable count rates of 500 mhz or more , improving the signal - to - noise , capture range , and closed - loop bandwidth of the lock .it would then be feasible to implement fully digital - domain processing of the error signal , which could greatly improve both the long - term stability and absolute accuracy of the lock .  in conclusion ,we have demonstrated a method to lock the offset frequency between two lasers based on the direct digital counting of the optical beat note , which is well suited to atomic laser cooling and trapping experiments that call for many widely tunable laser beams , each with absolute frequency reference .

Abstract: we demonstrate a hybrid analog+digital electronic lock to stabilize a dynamically tunable rf frequency offset between two lasers . our method features an 80 mhz capture range ,  ghz tuning range , frequency agility of 1 mhz/ , and low (  ppm ) drift in the absolute optical frequency difference after  s. with this scheme , multiple slave lasers can easily be referenced to one stable master laser , while each remains rapidly and accurately tunable over the wide frequency ranges encountered in typical laser cooling and trapping experiments .