relays are widely used to extend the coverage of wireless networks without expensive payload backhaul .however , it was shown that relays can not provide degrees of freedom ( dof ) gain in conventional wireless networks .recently , it has been discovered that if relays are allowed to deliver cached information beside instantaneous information , the dof of the network increases .the benefits of caching stem from the fact that a currently - delivered data file may be requested again in the future .caching schemes were first proposed for improving the performance of operating systems .after that , various caching applications have been proposed .many web caching schemes have been presented such as with the aim of sustaining the data flow of the internet . in ,caching was used to reduce the number of hops from the source to the destination , and hence , the delay was reduced . in ,the aim of caching was to reduce the peak traffic by duplicating some contents in distributed memories across the network .this duplication occurs over the off - peak hours .the duplicated contents can be used during the peak hours to reduce network congestion .small cell caching was used to improve the area spectral efficiency of video transmission in cellular systems .an information theoretic formulation of the caching problem was presented in . however , all these works separated caching from the physical layer ( phy ) .  in , it was shown that the one - shot sum - dof grows linearly with the aggregate cache size in the network .also , it was recently shown in that phy caching can provide dof gain in a single cell with a caching relay , as the topology of the relay channel is transformed into a broadcast channel when the requested content exists in the base station ( bs ) and the relay station ( rs ) simultaneously .however , the results in can not be directly extended to the case of multiple cells .for example , consider a simple network with single antenna users , two non - cooperative -antenna bss and two non - cooperative -antenna rss each with cache of size ( one for each bs ) .assuming that the bss do not have any contents in common , the two bss can not use the channel simultaneously , and hence , we have to assign the channel for only one of them at each time instant .thus , we have no dof gain over in this case .however , in this paper we show that using a single shared rs with antennas and cache of size instead of two separate rss can greatly improve the dof beyond those achieved by direct extension of the scheme in .   with the help of the shared rs ,while the dashed lines in ( b ) represents the transmission of parity bits not in the rs cache where dof.,scaledwidth=45.0% ]  in this paper , we propose a novel scheme to exploit _ cache - induced opportunistic mimo cooperation _ in relay networks with multiple non - cooperative bss .in particular , we exploit the benefit of having a shared relay station ( rs ) which significantly increases the dof gain over the results of .this dof improvement can be attributed to the ability of a bs to use the shared rs when the other bs can not benefit from the existence of the rs as illustrated in fig .[ fig:1 ] .we propose a cache - assisted relaying protocol using maximum distance separable ( mds ) coding which improves the opportunity of mimo cooperation between the bss and the rs .next , we propose a global - optimal cache content placement solution and a low - complexity self - learning algorithm to find a near - optimal cache content placement solution that does not require prior knowledge of the popularity of the content files .the mathematical notation used in this paper is as follows .the superscripts and denote the transpose and the hermitian transpose respectively .the indicator function returns when the event is true , otherwise it returns .consider a cached relay network composed of two non - cooperative -antenna bss ( designated as bs1 and bs2 ) and a -antenna shared relay with cache of size .each bs is connected to a gateway .there are single antenna users requesting files from any of the content gateways via the relay network where .we denote the total number of available content files by .we assume that the rs stores a fraction of each content file .each user has the freedom to request any of the available files and the bs which connected to the gateway containing the requested file serves the user subsequently possibly with the assistance of the relay .we assume that each user has a long sequence of requested files .time is slotted with time slots indexed by where the duration of each time slot is .the index of the file requested by user is denoted by whose size is given by .we define the user request profile ( urp ) as the vector whose entry contains the index of the file requested by user , i.e. , .we assume that has a general distribution which changes over a slow timescale .we also assume that the packets of the content files are delivered to each bs via a high speed backhaul connection and that the rs has no fixed - line backhaul connection to any of the bss .let denote the set containing the network nodes where , and denote respectively the bs , rs , and the user .we use to denote the matrix containing the coefficients of the channel from transmitter node to receiver node where .the channel matrices have i.i.d .complex gaussian entries with zero - mean and unit - variance and are independent of each other .we consider a block fading channel where all the channel matrices remain constant within a time slot but are i.i.d . across the time slots .the transmission power of each bs is bounded by while the transmission power of the rs is bounded by where .   in this subsection, we present the proposed cache - assisted relaying protocol .each bs divides each of its available content files into segments of size bits each .these segments of the file are encoded into parity bits using an mds rateless code , where is the _ cache content placement variable_. the parity bits will be cached at the rs .an mds rateless code generates a codeword of arbitrary length from information bits , such that if the decoder receives any parity bits , it can recover the original information bits .the mds rateless code can be implemented using raptor codes with a small redundancy overhead .as discussed in , mds encoding highly improves the probability of mimo cooperative transmission .after that , the cache - assisted relaying protocol follows the following phases   the parity bits for every segment of the th content file for are transmitted offline to the rs and stored in the rs cache .we define the _ cache content placement vector _ as the vector whose component is the cache content placement variable of the file , i.e. , ^t ] , i.e. , the first user requests the second file while the twelfth user requests file number 6 .hence , , , ^t ] .we also define as the _ parity usage state _ of user , where means that user has not received all the cached parity bits for the currently requested segment and means that user has received all the cached parity bits for the currently requested segment .we denote the set of users that can be cooperatively served by the bs and the rs by .let us define the _ cache state _ of the bs by where .we also define the _ total cache state s _ as when , the bs whose can cooperate with the rs to transmit data streams to different users that belong to the set achieving a large dof gain as shown in fig .[ fig:1](a ) . hence , there are two transmission modes based on the total cache state + * 1)bs - only transmission mode ( ): * since the two bss can not use the channel simultaneously , the channel is assigned for one bs at each time slot . since we assume that each user has a long sequence of requested files , i.e. , , we have two cases : + 1 ) and , : in this case we assign the channel to the bs which has , while the other bs will transmit nothing , where denotes the index of the chosen bs to use the channel .+ 2 ) both : in this case we choose a bs at random with probability to use the channel .+ the chosen bs randomly selects a subset of users ( denoted by ) from the set and transmits some parity bits that are not in the rs cache to the users in using zero - forcing ( zf ) beamforming as illustrated in fig .[ fig:1](b ) . in this case, the received signal at user is where is the normalized additive white gaussian noise ( awgn ) , is the data symbol intended for user and is the zf beamforming vector satisfies the data rate of user in this case ( ) is given by where is the number of non - cached bits that are not delivered to user for the currently requested segment .* 2)bs - rs mimo cooperative transmission mode ( ): * we also have two cases : + 1 ) and , : in this case we choose the bs which has to use the channel .+ 2 ) both : in this case we choose a bs at random with probability to use the channel .the chosen bs randomly selects a subset of users ( denoted by ) from the set .the bs and the rs cooperatively transmit some cached parity bits to the users in using zf beamforming as illustrated in fig .[ fig:1](a ) .let denote the augmented channel vector containing the coefficeints of the channels from the bs and rs to the user , i.e. , that . in this case, the received signal at user can be written as where is the cached data symbol intended for user and is the zf beamforming vector satisfies the data rate of user in this case ( ) is given by where is the number of cached bits that are not delivered to user for the currently requested segment ._ example 2 _ : assume the same setup of example 1 and suppose that and . then , , , because , , and hence , .the transmission will occur according to case 1 in the bs - rs mimo cooperative transmission mode .  at the beginning of each time slot, each bs determines and , based on the cache content placement vector and the user requests .after that , each bs determines based on ( [ eqs1 ] ) then sends it to the rs .the rs determines the total cache state based on ( [ eqs3 ] ) and selects a bs to use the channel based on the discussion we made earliers .then , the rs sends the local csi to the selected bs which in turn forms the global csi and .the selected bs computes all the control variables ( ) and sends them to the rs .then , the transmission begins based on the total cache state .+   each user stores the received parity bits in the reassembling buffer until its size reaches .the received segment is then mds decoded and the buffer is cleared to store the next segment .in traditional relay networks , the dof is defined as the ratio of the sum rate and snr as snr tends to . in ,the dof definition of a cached relay network was introduced as where the expectation is taken over the urp , cache state and the channel matrices .is the sum rate received by the users . to have a non - zero dof, we have to scale with the snr .thus , we assume  we also assume that .the following theorem characterizes the dof of our cached relay network .  for given , and as , the dof of the cached relay network with two bss is given by where is the fraction of time at which bs - rs mimo cooperative transmission is used for a given urp .refer to appendix [ proof1 ] .   from theorem 1, we see that the overall performance of the cached relay network depends strongly on the probability of bs - rs mimo cooperative transmission mode .it was previously shown in that mds - coded caching significantly improves the probability of bs - rs mimo cooperative transmission mode over naive random caching .the following theorem gives an expression for the bs - rs mimo cooperative transmission probability .  for given , urp and as , the fraction of time at which bs - rs mimo cooperative transmission is used can be given by : where , , , and can be obtained from by algorithm [ alg1 ] .refer to appendix [ proof2 ] .we define as the _ modified cache content placement vector _ for the bs .we can get from by the following algorithm  * step 0 : if * , put all the entries of in , and extend the size of to be by adding zeros at the end of it , then terminate the algorithm . +* else * , put the largest entries of in with any order .then , replace the largest entries of by zeros .+ * step 1 : * get the largest entry of , and add it to the smallest entry in , i.e. , , where is the index of the smallest entry in . then , replace the largest entry in by zero and repeat * step 1 * until all the entries of become zeros ._ example 3 _ : let us consider the same setup of example 2 .let us assume that the vectors and are given by ^t ] .then , ^t,\tilde{\mathbf{q}}_2=[0.1,0.2,0.4,0,0,0]^t$ ] .it is observed that if and , then , and hence , ( [ coop_prob ] ) can be reduced to the cooperative transmission probability in the case of single bs which was given in .  for clarity , we have considered a case with two bss and a simple antenna configurations to highlight the main ideas .however , the results and algorithms can be easily extended for arbitrary number of bss with more general antenna settings .   in this subsectionwe formulate the dof optimization problem .we aim to find the cache content placement vector based on the content popularity statistics that maximizes the dof . from ( [ dof ] ) , to maximize the dof , we need to maximize .we denote as the feasible set of cache content placement vectors .the dof optimization problem is formulated as is a stochastic optimization problem .moreover , the objective function is neither concave nor convex for unknown urp distribution .we propose a global - optimal solution for which requires the knowledge of the distribution of and a low - complexity algorithm to find a near - optimal solution which does not require explicit knowledge of the urp distribution .   for a given distribution of , the objective function of problem is a convex function .problem involves the maximization of a convex function over a feasible set given by a polyhedron . as a result ,the optimal solution is one of the vertices of the polyhedron .let us assume that can take the value with probability , .we denote as the modified cache content placement vectors for the realization . to solve our optimization problem , we transform into an equivalent deterministic optimization problem by adding auxiliary variables and non - negative slack variables .we assume that .thus we have where is the feasible set of .is a polyhedron and is a convex maximization problem with linear constrains .the global - optimal solution of has the following property  given and the distribution of , the global - optimal solution of is a vertex of the polyhedron .refer to appendix [ proof3 ] .  from theorem 3, we can solve by enumerating the vertices of .majthay and whinston suggested a finite algorithm for solving this problem .unfortunately , the complexity of the majthay - whinston algorithm grows exponentially with the number of files .moreover , since the size of the support of grows exponentially with the number of users , the complexity of the majthay - whinston algorithm also grows exponentially with .also , it is not easy in practice to obtain the distribution of . to overcome these challenges , we propose a low - complexity algorithm that does not require explicit knowledge of the distribution of .sampling - based algorithms have been previously used to solve stochastic optimization problems .we follow on this direction to solve the problem using independent samples of which denoted by . then , the sample average approximation problem of is is a polyhedron and ( [ optpro ] ) is a convex maximization problem with linear constrains .we propose a low - complexity algorithm ( algorithm [ alg2 ] ) to solve this problem .the idea of algorithm [ alg2 ] is inspired from the simplex method which can be founded in .* step 0 : * obtain realizations of urp .let be the initial vertex of the polyhedron .set the step index . +* step 1 : * starting at , if there is no adjacent vertex which has a greater value for , terminate the algorithm . otherwise , assign the adjacent vertex where is maximum to .put and repeat * step 1*.  in step 1 of algorithm [ alg2 ] , we utilize the subroutine in to find the adjacent vertices of a vertex .the simplex method is efficient , as well as algorithm [ alg2 ] .in this section , we numerically evaluate the performance of our proposed scheme and compare it with various baselines .we begin with a simple simulation to compare the results of algorithm [ alg2 ] with the global - optimal solution results .a cached relay network with two bss is considered with content files ( files for each bs ) and users .we assume that the sizes of all the content files are equal .each bs has antennas and the rs has antennas .we assume that the power constrains of the bss and the rs are equal , i.e. , .each user requests a content file independent on the other users .the content popularity of each content gateway is modeled by the zipf distribution with popularity skewness parameter .we assume that a user requests a file from the content gateway of bs1 or bs2 with probability .  , and ,scaledwidth=48.0% ]  , and ,scaledwidth=48.0% ]  in fig .[ fig:3 ] , we plot the dof gain versus the cache capacity .the results of our proposed near - optimal solution using urp samples are compared with the proposed global - optimal solution results and with other various schemes .the first scheme is the _infinite caching _ scheme where the rs can store all the content files .the _ uniform caching _ scheme where is the second one .the third is the _ mds - coded caching with separate rss _ scheme where each bs has a separate rs as in . from fig .[ fig:3 ] , we can see that the results of the low - complexity algorithm are very close to the global solution results for a moderate value of .now , we change our setup to be , and . as and increase, the complexity of the majthay - whinston algorithm increases rapidly and the global - optimal solution takes a very long time .thus , we use algorithm [ alg2 ] to calculate the optimizing vector using samples .  from fig .[ fig:2 ] , we can observe the significant improvement in the dof gain due to having a shared rs instead of two separate rss .we can also observe the improvement in the dof gain due to using mds - coded caching .also , we can see that the gap between the results of and the proposed scheme increases as the cache capacity increases .these results demonstrate that our proposed solution has significant performance gain over the case where there are two separate rss and also over the other caching schemes .this improvement occurs even in the case where the cache capacity is small compared to the total content size .in this paper , we have proposed a cache - assisted relaying protocol for cached relay networks with multiple bss .no cooperation between the bss is needed .we have exploited the benefit of having a shared rs to provide large dof gain .an optimal and a low - complexity near - optimal algorithms for the cache content placement problem have been proposed with the objective of maximizing the dof gain .numerical results have been provided to show the performance improvement of the proposed scheme over various baseline schemes .the average instantaneous sum rate received by the users can be given by where is the index of the chosen bs to use the channel .the beamforming vectors and satisfy where is the identity matrix . from the definition of the dof of cached relay networks in ( [ dofdef ] ) , we have      from the power constraint , it can be easily shown that on the other hand , we have where are finite valued factors which depend on the channel coefficients .assume that and that the ratio is finite .hence , and thus we have from and , we have   we first assume that bs1 has a priority in the cooperative mode , i.e. , bs1 chooses the cooperative mode whenever it is possible regardless of the cache state of bs2 .let denote the number of time slots required to transmit the current segment of user which served by bs1 . as , we have and the average rate of user can be given by   let , . in this casecan be expressed as the first term in ( [ eq3 ] ) is the rate of user in the case of , while the second term is the rate in the case of non - cooperative transmission where both and . in the non - cooperative mode , when only bs1 can use the channel , while when bs1 uses the channel with probability , and hence , the factor appears .using mds - coded caching aligns the transmission of the cached parity bits as much as possible and the total number of transmitted cached parity bits in time slots is equal to . dividing this number of bits by the average cooperative time yields from , and , we have we can take the limit as and use lemma 1 to get in a similar way if we assume that user k is served by bs2 , we will get from and , we have  hence , the probability of the bs - rs mimo cooperative transmission given that bs1 has a priority in the cooperative mode is given by if we assume that bs2 has a priority in the cooperative mode , we will get in a similar way  and hence , the probability of the bs - rs mimo cooperative transmission given that bs2 has a priority in the cooperative mode is given by where , given that bs2 has a priority in the cooperative mode . to preserve fairness between the two bss , we take the average performance of the two assumptions .thus we have    from ( [ p_e ] ) , it can be shown that .thus , is finite over . in the case of or ,the expression of in ( [ p_e ] ) can be reduced to which was shown to be convex over . in the case of and ,the hessian of is a block - diagonal matrix whose diagonal block  is given by ( [ hess ] ) . for simplicity , we use instead of . also , the determinant of is given as from ( [ d_e ] ) , we have , and hence , thus , is positive - semidefinite .hence , the hessian of is positive - semidefinite , so is a convex function over .using and property , has at least one global - optimal solution which must be a vertex of .v. r. cadambe and s. a. jafar , `` degrees of freedom of wireless networks with relays , feedback , cooperation , and full duplex operation , '' _ ieee transactions on information theory _ , vol . 55 , no . 5 , pp . 23342344 , 2009 .s. borst , v. gupta , and a. walid , `` distributed caching algorithms for content distribution networks , '' in _ international conference on computer communications , infocom_.1em plus 0.5em minus 0.4emieee , 2010 , pp . 19 .n. naderializadeh , m. a. maddah - ali , and a. s. avestimehr , `` fundamental limits of cache - aided interference management , '' in _ information theory ( isit ) , 2016 ieee international symposium on_.1em plus 0.5em minus 0.4em ieee , 2016 , pp .20442048 .w. han , a. liu , and v. k. lau , `` improving the degrees of freedom in mimo relay networks via phy caching , '' in _ international workshop on signal processing advances in wireless communications , spawc_.1em plus 0.5em minus 0.4emieee , 2015 , pp .550554 .l. breslau , p. cao , l. fan , g. phillips , and s. shenker , `` web caching and zipf - like distributions : evidence and implications , '' in _ international conference on computer communications , infocom _ ,1.1em plus 0.5em minus 0.4emproc .ieee , 1999 , pp .

Abstract: the ability of physical layer relay caching to increase the degrees of freedom ( dof ) of a single cell was recently illustrated . in this paper , we extend this result to the case of multiple cells in which a caching relay is shared among multiple non - cooperative base stations ( bss ) . in particular , we show that a large dof gain can be achieved by exploiting the benefits of having a shared relay that cooperates with the bss . we first propose a cache - assisted relaying protocol that improves the cooperation opportunity between the bss and the relay . next , we consider the cache content placement problem that aims to design the cache content at the relay such that the dof gain is maximized . we propose an optimal algorithm and a near - optimal low - complexity algorithm for the cache content placement problem . simulation results show significant improvement in the dof gain using the proposed relay - caching protocol .   physical layer caching , relay networks , cooperative mimo , dof .


we present a sequential monte carlo algorithm for adaptive sampling from a binary distribution .a monte carlo algorithm is said to be adaptive if it adjusts , sequentially and automatically , its sampling distribution to the problem at hand .besides sequential monte carlo , important classes of adaptive monte carlo are adaptive importance sampling ( e.g. * ? ? ?* ) and adaptive markov chain monte carlo ( e.g. * ? ? ?a central aspect of adaptive algorithms is their need for a parametric family of auxiliary distributions which should have the following three properties : ( a ) the family is sufficiently flexible to guarantee a reasonable performance in the context of the specific algorithm ; ( b ) it allows to quickly draw independent samples ; ( c ) it can , with reasonable effort , be calibrated using past simulations .  for problems in continuous sampling spaces ,the typical example is the multivariate normal distribution , which clearly fulfils ( b ) and ( c ) , and complies with ( a ) in many practical problems . in this paper, we propose an analogue for high - dimensional binary sampling spaces .our objective is to construct a parametric family for sequential monte carlo on the binary sampling space , where is too large to allow for exhaustive enumeration of the whole space .since there is no multivariate binary family which we can easily parametrise by its first and second order moments like the multivariate normal , the construction of suitable proposal distributions seems more difficult for the discrete adaptive sampling problem than for its continuous counterpart .the major application for our algorithm is variable selection in linear regression models . in this context, a binary vector encodes whether each of possible covariates are included in the linear regression model or not . in a bayesian framework , and for a judicious choice of prior distributions, we can explicitly calculate the posterior distribution up to a constant .we want to sample from this distribution in order to approximate quantities like the expected value , that is the marginal probability of inclusion of each variable .often , the marginal probabilities provide a richer picture of the posterior distribution than a collection of modes found using stochastic optimisation techniques .our sequential monte carlo approach to variable selection views a well studied problem from a different angle and provides new perspectives .the reason is two - fold .firstly , there is growing evidence that global methods , which track a population of particles , initially well spread over the sampling space , are often more robust than local methods based on markov chain monte carlo .the latter are more prone to get trapped in the neighbourhood of local modes .we largely illustrate this effect in our simulations in section [ sec : simu ] .  secondly , global methods have the property to be easily parallelisable .parallel implementations of monte carlo algorithms have gained a tremendous interest in the very recent years , due to the increasing availability of multi - core ( central or graphical ) processing units in standard computers .the paper is organised as follows .  in section [ sec: var sel ] , we recapitulate the basics of bayesian variable selection in linear regression models as the motivating application .  in section [ sec : mcmc ] , we briefly review the principal markov chain monte carlo methods which are commonly used to integrate with respect to a binary distributions .  in section [ sec :smc ] , we describe an alternative approach to the same problem using sequential monte carlo methods .the key ingredient of this algorithm is a parametric family which is flexible enough to come close to the target distribution .  in section [ sec : bv ] , we extensively discuss approaches for constructing rich parametric families on binary spaces .this is the core of our work .some of the binary models discussed are not suitable in the framework of our sequential monte carlo algorithm but mentioned for completeness of the survey .  in section [ sec : simu ] , we construct two examples of variable selection problems which yield challenging posterior distributions . we show that standard markov chain techniques fail to produce reliable estimates of the marginal probabilities while our sequential monte carlo approach successfully copes with the integration problem .[ [ notation ] ] notation + + + + + + + +  for a vector , we write for the sub - vector indexed by . we write if the indices are a complete sequence .we denote by the sub - vector .we write for .  for a matrix , we denote its components by , its determinant by .the operator transforms the vector into a diagonal matrix . for a finite set ,we denote by the number of elements in .the standard linear normal model postulates that the relationship between the observed explained variable and the observations \in\r^{m , d} ]  * for * * do * + * if * * then * +   [ [ performance ] ] performance + + + + + + + + + + +  we refer to the ratio as the acceptance probability of the metropolis - hastings step . in binary spaces , however , accepting a proposal does not imply we are changing the state of the chain , since we are likely to re - propose the current state .we are actually interested in how fast the chain explores the state spaces , precisely its mutation probability .   for this section ,let be constant .algorithm [ algo : generic mg ] collapses to changing a single component . instead of independently drawing the index , we could also iterate through a uniformly drawn permutations of the index set .kernels of this kind are often referred to as metropolised gibbs samplers , since they proceed component - wise as does the classical gibbs sampler , but also involve a metropolishastings step . in the sequel, we discuss some special cases .[ [ classical - gibbs ] ] classical gibbs + + + + + + + + + + + + + + +  the gibbs sampler sequentially draws each component from the full marginal distribution , which corresponds to by construction , the acceptance probability is while the mutation probability is only , where is a copy of the current state with component altered .  [ [ adaptive - metropolised - gibbs ] ] adaptive metropolised gibbs + + + + + + + + + + + + + + + + + + + + + + + + + + +  an adaptive extension of the metropolised gibbs has been proposed by .the full marginal distribution is approximated by a linear predictor . in their notation , where is the estimated mean , the estimated covariance matrix and a design parameter which ensures that is a probability .analogously to our vector notation , denotes the matrix without the row and column .we obtain the estimates from the past trajectory of the chain and update them periodically .the mutation probability is of the same order as that of the gibbs kernel , but adaption largely avoids the computationally expensive evaluations of .the non - adaptive gibbs sampler already requires evaluation of just to produce the proposal .in contrast , the adaptive metropolised gibbs samples from a proxy and only evaluates if .[ [ modified - metropolised - gibbs ] ] modified metropolised gibbs + + + + + + + + + + + + + + + + + + + + + + + + + + +  in comparison to the classical gibbs kernel , we obtain a more efficient chain using the simple form since we always propose to change the current state , the acceptance and mutation probabilities are the same. they amount to , where is a copy of the current state with component altered . comparing the mutation probabilities of the two kernels , we see that the modified metropolised gibbs chain moves , on average , faster than the classical gibbs chain .the modified metropolised gibbs easily generalises to the case where may take values larger than one .suppose , for example , we propose to change components simultaneously , where is a truncated geometric distribution .note that we suggest , on average , to change approximately components .in other words , for larger values of , we are more likely to propose further steps in the sampling space .large step proposals improve the mixing properties of the chain and help to escape from the attraction of local modes .they are , however , less likely to be accepted than single component steps which leads to a problem - dependent trade - off . in our numerical examples , we could not observe any benefit from block updating , and we do not further consider it to keep the comparison with our sequential monte carlo method more concise .   we can construct a fast mixing markov chain based on independent proposals . let be some distribution with , that is for all .we propose a new state and accept it with probability the associated markov chain has the unique invariant measure .this kernel is referred to as the independent metropolis - hastings kernel , since the proposal distribution is not a function of the current state .the mutation rate is the acceptance rate minus , so the two notions practically coincide in large spaces .obviously , in order to make this approach work , we need to choose sufficiently close to , which implies high acceptance rates on average . in absence of reliable prior information , however , we are not able to produce such a distribution .we shall , however , use precisely this markov kernel as part of our sequential monte carlo algorithm . in this context, we can calibrate sequences of proposal distributions to be close to our current particle approximation .in this section , we show how to estimate the expected value with respect to a probability mass function defined on using sequential monte carlo . this general class of algorithms alternates importance sampling steps , resampling steps and markov chain transitions , to recursively approximate a sequence of distributions , using a set of weighted ` particles ' which represent the current distribution . in the following , we present a version which is tailored to work on binary spaces .  for readers not familiar with sequential monte carlo , the following algorithm described might seem rather complex at first glance .we introduce the steps separately before we look at the complete algorithm .we give comprehensive instructions which correspond exactly to our implementation in order to make our results plausible and easily reproducible for the reader .the first ingredient of sequential monte carlo is a smooth sequence of distributions , which ends up at the distribution of interest .the intermediary distributions are purely instrumental : the idea is to depart from a distribution with broad support and to progress smoothly towards the distribution of interest .  [[ initial - distribution ] ] initial distribution + + + + + + + + + + + + + + + + + + + +  theoretically , we can use any with that can sample from as initial distribution .numerical experiments taught us , however , that premature adjustment of , for example using markov chain pilot runs , leads to faster but less robust algorithms .thus , in practice , we recommend the uniform distribution for its simplicity and reliability . therefore ,in the sequel , we let .  [[ intermediate - distributions ] ] intermediate distributions + + + + + + + + + + + + + + + + + + + + + + + + + +  we construct a smooth sequence of distributions by judicious choice of an associated real sequence increasing from zero to one .the most convenient and somewhat natural strategy is the geometric bridge  alternatively , one could use a sequences of mixtures or , in a bayesian context , a sequences of posterior distributions where data is added as increases , that is see . in the following ,we use the geometric bridge for its computational simplicity and present a procedure to determine a suitable sequence .suppose we have already produced a sample of size from .we can roughly approximate by the empirical distribution })\,\delta_{\v x_k^{[\,t-1]}}(\v\gamma),\ ] ] where the corresponding importance function is   note that is the step length at time . as we choose larger , that is further from , the weights become more uneven and the accuracy of the importance approximation deteriorates . * for all *  * for all *   if we repeat the weighting steps until we reach , we obtain a classical importance sampling estimate with instrumental distribution .the idea of the sequential monte carlo algorithm , however , is to control the weight degeneracy such that we can intersperse resample and move steps before loosing track of our particle approximation .[ [ effective - sample - size ] ] effective sample size + + + + + + + + + + + + + + + + + + + + +  we measure the weight degeneracy through the effective sample size criterion , see .in our case , we have .\ ] ] the effective sample size is if all weights are equal and if all mass is concentrated in a single particle .  for a geometric bridge ,the effective sample size is merely a function of .we can thus control the weight degeneracy by judicious choice of the step lengths .we pick a step length such that the effective sample size equals a fixed value , see . since is continuous and monotonously increasing in ,we solve using bi - sectional search , see procedure [ algo : step ] .this approach is numerically more stable than a newton - raphson iteration , for the derivative involves fractions of sums of exponentials which are difficult to handle .let be the unique solution to .we can construct an associated sequence setting .thus , the number of steps depends on the complexity of the integration problem at hand and is not known in advance .  in other words , for fixed ,the associated sequence is a self - tuning parameter . in our simulations ,we always choose , which yields convincing results on both example problems in section [ sec : simu ] .smaller values significantly speed up the sequential monte carlo algorithm but lead to a higher variation in the results .  * if * * then * * else *    suppose we have a sample of size from with importance weights as defined in .we can obtain a sample which is approximately distributed according to by drawing from the empirical approximation defined in . * sample * )$ ]   for the implementation of the resampling step , there exist several recipes .we could apply a multinomial resampling which is straightforward .there are , however , more efficient ways like residual , stratified and systematic resampling .we use the latest in our simulations , see procedure [ algo : resample ] .  in the resulting unweighted particle system of size , the particles with small weights have vanished while the particles with large weights have bee multiplied .there are approaches that resample a weighted particle system of size from an augmented system of size , see , but these techniques are computationally more demanding without visibly improving our numerical results .theoretically , one would expect a rao - blackwellisation effect but its analysis is beyond the scope of this paper .  in any case, if we repeat the weighting and resampling steps several times , we rapidly deplete our particle reservoir reducing the number of different particles to a very few .thus , the particle approximation will be totally inaccurate .the key to fighting the decay of our approximation is the following move step .the resampling step provides an unweighted particle system of containing multiple copies of many particles .the central idea of the sequential monte carlo algorithm is to diversify the resampled system , replacing the particles by draws from a markov kernel with invariant measure  .since the particle is , approximately , distributed according to , a draw is again , approximately , distributed according to .we can repeat this procedure over and over without changing the target of the particle approximation .note that , even if the particles are equal after resampling , the particles are almost independent after sufficiently many move steps . in order to make the algorithm practical , however , we need a transition kernel which is rapidly mixing and therefore diversifies the particle system within a few steps . therefore , the locally operating markov kernels reviewed in section [ sec : mcmc ] are not suitable .in fact , our numerical experiments suggest that making a sequential monte carlo algorithm work with local kernels is practically impossible.  therefore , we use a metropolis - hastings kernel with independent proposals as described in section [ sec : ind mh ] .precisely , we construct a kernel employing a parametric family on which , for some , is sufficiently close to to allow for high acceptance probabilities .  for this purpose, we fit a parameter to the particle approximation of according to some convenient criterion .the choice of the parametric family is crucial to a successful implementation of the sequential monte carlo algorithm .we come back to this issue in section [ sec : bv ] . * sample * * for all * +  +   [ [ particle - diversity ] ] particle diversity + + + + + + + + + + + + + + + + + +  we need to determine how often we move the particle system before we return to the weight - resample step .an easy criterion for the health of the particle approximation is its particle diversity ,\ ] ] that is the proportion of distinct particles .note that the particle diversity is a quality criterion which has no simple analogue in continuous sampling spaces .  for optimal results, we recommend to keep on moving the particle system until the particle diversity can not be augmented any longer . in the first steps of the algorithm, is still close to the uniform distribution , and we manage to raise the particle diversity up to one . as is approaching a strongly multi - modal target distribution , however , the particle diversity reaches a steady - state we can not push it beyond .  clearly , even if we could draw a particle system independently from , the particle diversity would be a lot smaller than one , since we would draw the modes of several times .[ [ aggregated - weights ] ] aggregated weights + + + + + + + + + + + + + + + + + +  shifting weights between identical particles does not affect the nature of the approximation but it changes the effective sample size which seems paradoxical at first sight . for reasons of parsimoniousness , we could just keep a single representative for identical particles and aggregate the associated weights to the sum without changing the quality of the particle approximation .there are , however , three reasons why we refrain from doing so .firstly , it is vital not to confuse the weight disparity induced by reweighting according to the progression of and the weight disparity due to aggregation of the weights of multiply sampled states . from the aggregated system, we can not tell whether the effective sample size is determined by the gap between and , that is the step length , or by the presence of particle copies due to the mass of being very concentrated .therefore , it seems more difficult to control the smoothness of the sequence of distributions and find a suitable sequence .secondly , aggregation is an additional computational effort equivalent to keeping the particle system sorted . here, we trade in computing time for memory while the required memory is proportional to the number of particles and not critical in the context of our algorithm .thirdly , the straightforward way to implement repeated move steps is breaking up the particles into multiple copies corresponding to their weights and moving them separately .consequently , instead of permanently splitting and aggregating the weights we might just allow for multiple copies of the particles .   finally , we summarize the complete sequential monte carlo method in algorithm [ algo : smc ] .note that , in practice , the sequence is not indexed by but rather by , that is the counter is only given implicitly .* sample * * for all * .  [ cols= "< , < , < " , ]n. chopin is supported by the anr grant anr-008-blan-0218 `` bigmc '' of the french ministry of research .we would like to thank pierre jacob and two anonymous referees for their valuable comments on this paper .we acknowledge the statlib data archive and the uci machine learning repository for providing the data sets used in this work .

Abstract: a monte carlo algorithm is said to be adaptive if it automatically calibrates its current proposal distribution using past simulations . the choice of the parametric family that defines the set of proposal distributions is critical for good performance . in this paper , we present such a parametric family for adaptive sampling on high - dimensional binary spaces .  a practical motivation for this problem is variable selection in a linear regression context . we want to sample from a bayesian posterior distribution on the model space using an appropriate version of sequential monte carlo .   raw versions of sequential monte carlo are easily implemented using binary vectors with independent components . for high - dimensional problems , however , these simple proposals do not yield satisfactory results . the key to an efficient adaptive algorithm are binary parametric families which take correlations into account , analogously to the multivariate normal distribution on continuous spaces .   we provide a review of models for binary data and make one of them work in the context of sequential monte carlo sampling . computational studies on real life data with about a hundred covariates suggest that , on difficult instances , our sequential monte carlo approach clearly outperforms standard techniques based on markov chain exploration .


high - dimensional time series include video streams , electroencephalography ( eeg ) and sensor network data .dynamical models describing such data are desired for forecasting ( prediction ) and controller design , both of which play an important role , e.g. , in autonomous systems , machine translation , robotics and surveillance applications . a key challenge is system identification , i.e. , finding a mathematical model of the dynamical system based on the information provided by measurements from the underlying system . in the context of state - space modelsthis includes finding two functional relationships between ( a ) the states at different time steps ( prediction / transition model ) and ( b ) states and corresponding measurements ( observation / measurement model ) . in the linear case ,this problem is very well studied , and many standard techniques exist , e.g. , subspace methods  , expectation maximization  and prediction - error methods  . however , in realistic and practical scenarios we require non - linear system identification techniques .  learning non - linear dynamical models is an inherently difficult problem , and it has been one of the most active areas in system identification for the last decades  . in recent years , sequential monte carlo ( smc ) methods have received attention for identifying non - linear state - space models  , see also the recent surveys  .while methods based on smc are powerful , they are also computationally expensive .learning non - linear dynamical models from very high - dimensional sensor data is even more challenging .first , finding ( non - linear ) functional relationships in very high dimensions is hard ( un - identifiability , local optima , overfitting , etc . ) ; second , the amount of data required to find a good function approximator is enormous .fortunately , high - dimensional data often possesses an intrinsic lower dimensionality .we will exploit this property for system identification by finding a low - dimensional representation of high - dimensional data and learning predictive models in this low - dimensional space . for this purpose ,we need an automated procedure for finding compact low - dimensional representations / features .the state of the art in learning parsimonious representations of high - dimensional data is currently defined by deep learning architectures , such as deep neural networks  , stacked / deep auto - encoders  and convolutional neural networks  , all of which have been successfully applied to image , text , speech and audio data in commercial products , e.g. , by google , amazon and facebook .typically , these feature learning methods are applied to static data sets , e.g. , for image classification .the auto - encoder gives explicit expressions of two generative mappings : 1 ) an encoder mapping the high - dimensional data to the features , and 2 ) a decoder mapping the features to high - dimensional reconstructions . in the machine learning literature, there exists a vast number of other well studied nonlinear dimensionality reduction methods such as the gaussian process latent variable model ( gp - lvm ) , kernel pca  , laplacian eigenmaps  and locally linear embedding .however , all of them provide at most one of the two mappings and .  in this paper , we combine feature learning and dynamical systems modeling to obtain good predictive models for high - dimensional time series .in particular , we use deep auto - encoder neural networks for automatically finding a compact low - dimensional representation of an image . in this low - dimensional feature space, we use a neural network for modeling the nonlinear system dynamics .the embedding and the predictive model in feature space are learned _jointly_. an simplified illustration of our approach is shown in [ fig : model illustration ] .an encoder maps an image at time step to a low - dimensional feature . in this feature space ,a prediction model maps the feature forward in time to .subsequently , the decoder can be used to generate a predicted image at the next time step .this framework needs access to both the encoder and the decoder , which motivates our use of the auto - encoder as dimensionality reduction technique .consequently , the contributions of this paper are ( a ) a model for learning a low - dimensional dynamical representation of high - dimensional data , which can be used for long - term predictions ; ( b ) experimental evidence demonstrating that jointly learning the parameters of the latent embedding and the predictive model in latent space can increase the performance compared to a separate training .we consider a dynamical system where control inputs are denoted by and observations are denoted by . in the context of this paper , the observations are pixel information from images .we assume that a low - dimensional latent variable exists that compactly represents the relevant properties of . since we consider dynamical systems , a low - dimensional representation of a ( static ) image is insufficient to capture important dynamic information , such as velocities .therefore , we introduce an additional latent variable , the _ state_. in our case , the state contains features from multiple time steps ( e.g. , and ) to capture velocity ( or higher - order ) information .therefore , our transition model does not map features at time to time ( as illustrated in [ fig : model illustration ] ) , but the transition function maps states ( and controls ) to states at time .the full dynamical system is given as the state - space model  [ eq : model ]   where each measurement can be described by a low - dimensional feature representation .these features are in turn modeled with a low - dimensional state - space model in and , where the state contains the full information about the state of the system at time instant  , see also [ fig : probabilistic_model ] . here , and are sequences of independent random variables and are the model parameters .  0.46  ( x1 ) at ( -1,0 ) ; ( x2 ) at ( 1,0 ) ; ( x0 ) at ( -2.5,0 ) ; ( d ) at ( -2.5,-1.5 ) ; ( x3 ) at ( 2.5,0 ) ;( y1 ) at ( -1,1.5 ) ; ( y2 ) at ( 1,1.5 ) ; ( z1 ) at ( -1,3.2 ) ; ( z2 ) at ( 1,3.2 ) ; ( u0 ) at ( -2.5,-1 ) ; ( u1 ) at ( -1,-1.5 ) ; at ( -2.5,3.2 ) high - dim . + data ; at ( -2.5,1.5 ) features ; at ( -2.5,0 ) hidden + state ; at ( -2.5,-1.5 ) control + inputs ; ( x1 ) ( x2 ) node[midway , below ] ; ( x0 ) ( x1 ) ; ( x2 ) ( x3 ) ; ( x1 ) ( y1 )node[midway , left ] ; ( y1 ) ( z1 ) node[midway , left ] ; ( x2 ) ( y2 )node[midway , left ] ; ( y2 ) ( z2 ) node[midway , left ] ; ( u0 ) ( x1 ) ; ( u1 ) ( x2 ) ;  0.46  ( u0 ) at ( -4,-1.5 ) ; ( u1 ) at ( -1,-1.5 ) ; ( y0 ) at ( -4,1.5 ) ; ( y1 ) at ( -1,1.5 ) ; ( y2 ) at ( 1,1.5 ) ; ( d1 ) at ( -2.5,-1.5 ) ; ( d2 ) at ( -2.5,1.5 ) ; ( d3 ) at ( -2.5,3.2 ) ; ( z0 ) at ( -4,3.2 ) ; ( z1 ) at ( -1,3.2 ) ; ( z2 ) at ( 1,3.2 ) ; at ( -4.5,3.2 ) high - dim .+ data ; at ( -4.5,1.5 ) features ; at ( -4.5,-1.5 ) control + inputs ; ( y1 ) ( y2 ) ; ( u1 ) ( y2 ) ; ( u0 ) ( y2 ) node[midway , below ] ; ( z1 ) ( y1 )node[midway , left ] ; ( z0 ) ( y0 )node[midway , left ] ; ( y2 ) ( z2 ) node[midway , left ] ; ( y0 ) .. controls ( -1,0.5 ) .. ( y2 ) ;   to identify parameters in dynamical systems , the prediction - error method has been applied extensively within the system identification community during the last five decades .it is based on minimizing the error between the sequence of measurements and the predictions , usually the one - step ahead prediction . to achieve this, we need a predictor model that relates the prediction to all previous measurements , control inputs and the system parameters .  in general , it is difficult to derive a predictor model based on the nonlinear state - space model , and a closed form expression for the prediction is only available in a few special cases  .however , by approximating the optimal solution , a predictor model for the features can be stated in the form where includes all past features and control inputs , is a nonlinear function and is the corresponding model parameters . note that the predictor model no longer has an explicit notion of the state .the model introduced in  is indeed very flexible , and in this work we have restricted this flexibility somewhat by working with a nonlinear autoregressive exogenous model ( narx ) , which relates the predicted current value of the time series to the past values of the time series , as well as the past values of the control inputs .the resulting narx predictor model is given by where is a nonlinear function , in our case a neural network .the model parameters in the nonlinear function are normally estimated by minimizing the sum of the prediction errors .however , since we are interested in a good predictive performance for the high - dimensional data rather than for the features , we transform the predictions back to the high - dimensional space and obtain a prediction , which we use in our error measure .an additional complication is that we do not have access to the features .therefore , before training , the past values of the time series have to be replaced with their feature representation , which we compute from the pixel information . here , is an approximate inverse of , which will be described in more detail the next section .this gives the final predictor model    which is also illustrated in [ fig : predictor_model ] .the corresponding prediction error will be    /[count= ] in 1,2,3,missing,4 ( input- ) at ( 0,2.5- ) ;  in 1,missing,2 ( hiddena- ) at ( 2,2 - 1.25 ) ;  in 1,missing,2 ( hiddenb- ) at ( 4,1.5- ) ;  in 1,missing,2 ( hiddenc- ) at ( 6,2 - 1.25 ) ;  /[count= ] in 1,2,3,missing,4 ( output- ) at ( 8,2.5- ) ;  [count = i ] in 1,2,3,m ( input - i ) + + ( -1.2,0 ) node [ above , midway ] ;  [count = i ] in 1,m at ( hiddenb-i.north ) ;  [count = i ] in 1,2,3,m ( output - i ) + + ( 1.9,0 ) node [ above , midway ] ;  iin 1, ... ,4 in 1, ... ,2 ( input - i ) ( hiddena- ) ;  iin 1, ... ,2 in 1, ... ,2 ( hiddena - i ) ( hiddenb- ) ; iin 1, ... ,2 in 1, ... ,2 ( hiddenb - i ) ( hiddenc- ) ;  iin 1, ... ,2 in 1, ... ,4 ( hiddenc - i ) ( output- ) ;  [count = from 0 ] in input layer + ( high - dim .data ) , , hidden layer + ( feature ) , , output layer + ( reconstructed ) at ( 2,3 ) ; [count = from 0 ] in , , , , at ( 2,-2.8 ) ;  we use a deep auto - encoder neural network to parameterize the feature mapping and its inverse .it consists of a deep encoder network and a deep decoder network .each layer of the encoder neural network computes , where is a squashing function and and are free parameters .the control input to the first layer is the image , i.e. , .the last layer is the low - dimensional feature representation of the image , where  ] and assumes a continuous gray - value in $ ] .0.46  0.46   we simulated 400 frames of a pendulum moving in a plane with pixels in each frame and the torque of the pendulum as control input . to speed up training ,the image input has been reduced to prior to model learning ( system identification ) using pca . with these 50 dimensional inputs ,four layers have been used for the encoder as well as the decoder with dimension 50 - 25 - 12 - 6 - 2 .hence , the features have dimension .the order of the dynamics was chosen as to capture velocity information .for the predictor model we used a two - layer neural network with a 6 - 4 - 2 architecture .we evaluate the performance in terms of long term predictions .these predictions are constructed by concatenating multiple 1-step ahead predictions .more precisely , the -step ahead prediction is computed iteratively as  [ eq : multi_step_prediction ]   where is the features of the data point at time instance .we assumed that the applied future torques were known .the predictive performance on two exemplary image sequences of the validation data of our system identification models is illustrated in [ fig : results ] , where control inputs ( torques ) are assumed known .the top rows show the ground truth images , the center rows show the predictions based on a model using joint training , the bottom rows show the corresponding predictions of a model where the auto - encoder and the predictive model were trained sequentially according to . for the model based on jointly training all parameters , we obtain good predictive performance for both one - step ahead prediction and multiple - step ahead prediction .in contrast , the predictive performance of learning the features and the dynamics separately is worse than the predictive performance of the model trained by jointly optimizing all parameters .although the auto - encoder does a perfect job ( left - most frame , 0-step ahead prediction ) , already the ( reconstructed ) one - step ahead prediction is not similar to the ground - truth image .this can also be seen in table [ tab : result ] where the reconstruction error is equally good for both models , but for the prediction error we manage to get a better value using joint training than using separate training .let us have a closer look at the model based on separate training : since the auto - encoder performs well , the learned transition model is the reason for bad predictive performance .we believe that the auto - encoder found a good feature representation for reconstruction , but this representation was not ideal for learning a transition model .1 : prediction error and reconstruction error for separate and joint training . [ cols="<,<,<",options="header " , ]   0.46  0.46  [ fig : latentboth ] displays the `` decoded '' images corresponding to the latent representations using joint and separate training , respectively . in the joint training the feature values line up in a circular shape enabling a low - dimensional dynamical description , whereas separate training finds feature values , which are not even placed sequentially in the low - dimensional representation .separate training extracts the low - dimensional representations without context , i.e. , the knowledge that these features constitute a time - series . on the other hand ,joint training enables the extraction of features that can also model the dynamical behavior in a compact manner .  in this particular data set, the data points clearly reside on one - dimensional manifold , encoded by the pendulum angle. however , a one - dimensional feature space would be insufficient since this one - dimensional manifold is cyclic , see [ fig : latentboth ] , compare also with the period of an angle .therefore , we have used a two - dimensional latent space .further , only along the manifold in the latent space where the training data reside the decoder produces reasonable outputs .this can be further inspected by zooming in on a smaller region as displayed in [ fig : latent_example ] .plot coordinates ( 0 , 96.1978 ) ( 1 , 96.1541 ) ( 2 , 95.9451 ) ( 3 , 95.5584 ) ( 4 , 95.1162 ) ( 5 , 94.6709 ) ( 6 , 94.2409 ) ( 7 , 93.8712 ) ( 8 , 93.5397 ) ;  plot coordinates ( 0 , 96.0393 ) ( 1 , 88.8115 ) ( 2 , 88.9518 ) ( 3 , 89.823 ) ( 4 , 89.5416 ) ( 5 , 88.8983 ) ( 6 , 89.0535 ) ( 7 , 89.572 ) ( 8 , 89.5154 ) ;  plot coordinates ( 0 , 100 ) ( 1 , 92.9682 ) ( 2 , 90.9634 ) ( 3 , 90.0684 ) ( 4 , 89.5501 ) ( 5 , 89.2161 ) ( 6 , 88.9749 ) ( 7 , 88.8094 ) ( 8 , 88.6715 ) ; plot coordinates ( 0 , 92.301 ) ( 1 , 92.123 ) ( 2 , 91.9723 ) ( 3 , 91.8444 ) ( 4 , 91.7392 ) ( 5 , 91.654 ) ( 6 , 91.5849 ) ( 7 , 91.5264 ) ( 8 , 91.4775 ) ;  to analyze the predictive performance of the two training methods , we define the fitting quality as as a reference , the predictive performance is compared with a naive prediction using the previous frame at time step as the prediction at as .the result for a prediction horizon ranging from to is displayed in [ fig : prediction ] .clearly , joint learning ( blue ) outperforms separate learning in terms of predictive performance for prediction horizons greater than 0 . even by using the last available image frame for prediction ( const ., brown ) , we obtain a better fit than the model that learns its parameter sequentially ( red ) .this is due to the fact that the dynamical model often predicts frames , which do not correspond to any real pendulum , see [ fig : results ] , leading to a poor fit .furthermore , joint training gives better predictions than the naive prediction .the predictive performance slightly degrades when the prediction horizon increases , which is to be expected .finally we also compare with the subspace identification method  ( black , starred ) , which is restricted to linear models .such a restriction does not capture the non - linear , embedded features and , hence , the predictive performance is sub - optimal .   in this experimentwe simulated 601 frames of a moving tile in a pixels image .the control inputs are the increments in position in both of the two cartesian directions . as in the previous experiment ,the image sequence was reduced to prior to the parameter learning using pca . a four layer 50 - 25 - 12 - 8 - 2 autoencoderwas used for feature learning , and an 8 - 5 - 2 neural network for the dynamics .  as in the previous example, we evaluate the performance in terms of long - term predictions .the performance of joint training is illustrated in [ fig : results_2dtile ] on a validation data set .the model predicts future frames of the tile with high accuracy . in [ fig : latent_2dtile ] , the feature representation of the data is displayed .the features reside on a two - dimensional manifold encoding the two - dimensional position of the moving tile .the four corners in this manifold represent the four corners of the tile position within the image frame .this structure is induced by the dynamical description .the corresponding feature representation for the case of separate learning does not exhibit such a structure , see the supplementary material .  in [ fig: prediction_2dtile ] , the prediction performance is displayed , where our model achieves a substantially better fit than naively using the previous frame as prediction .from a system identification point of view , the prediction error method , where we minimize the one - step ahead prediction error , is fairly standard .however , in a future control or reinforcement learning setting , we are primarily interested in good prediction performance on a longer horizon in order to do planning .thus , we have also investigated whether to additionally include a multi - step ahead prediction error in the cost .these models achieved similar performance , but no significantly better prediction error could be observed either for one - step ahead predictions or for longer prediction horizons .+  plot coordinates ( 0 , 94.1555 ) ( 1 , 94.0422 ) ( 2 , 93.868 ) ( 3 , 93.6945 ) ( 4 , 93.5584 ) ( 5 , 93.4499 ) ( 6 , 93.3689 ) ( 7 , 93.298 ) ( 8 , 93.2361 ) ;  plot coordinates ( 0 , 100 ) ( 1 , 87.1909 ) ( 2 , 86.3216 ) ( 3 , 86.0445 ) ( 4 , 85.8246 ) ( 5 , 85.6132 ) ( 6 , 85.5812 ) ( 7 , 85.4631 ) ( 8 , 85.3723 ) ;  instead of computing the prediction errors in image space , see , we can compute errors in feature space to avoid a decoding step back to the high - dimensional space according to .however , this will require an extra penalty term in order to avoid trivial solutions that map everything to zero , eventually resulting in a more complicated and less intuitive cost function .although joint learning aims at finding a feature representation that is suitable for modeling the low - dimensional dynamical behavior , the pre - training initialization as described in section [ sec : pretraining ] does not . if this pre - training yields feature values far from `` useful '' ones for modeling the dynamics , joint training might not find a good model .the network structure has to be chosen before the actual training starts .especially the dimension of the latent state and the order of the dynamics have to be chosen by the user , which requires a some prior knowledge about the system to be identified . in our examples , we chose the latent dimensionality based on insights about the true dynamics of the problem .in general , a model selection procedure will be preferable to find both a good network structure and a good latent dimensionality .we have presented an approach to non - linear system identification from high - dimensional time series data .our model combines techniques from both the system identification and the machine learning community .in particular , we used a deep auto - encoder for finding low - dimensional features from high - dimensional data , and a nonlinear autoregressive exogenous model was used to describe the low - dimensional dynamics .the framework has been applied to a pendulum moving in the horizontal plane .the proposed model exhibits good predictive performance and a major improvement has been identified by training the auto - encoder and the dynamical model jointly instead of training them separately / sequentially .possible directions for future work include ( a ) robustify learning by using denoising autoencoders to deal with noisy real - world data ( b ) apply convolutional neural networks , which are often more suitable for images ; ( c ) exploiting the learned model for learning controller purely based on pixel information ; ( c ) sequential monte carlo methods will be investigated for systematic treatments of such nonlinear probabilistic models , which are required in a reinforcement learning setting .  in a setting where we make decisions based on ( one - step or multiple - step ahead ) predictions , such as optimal control or model - based reinforcement learning ,a probabilistic model is often needed for robust decision making as we need to account for models errors  .an extension of our present model to a probabilistic setting is non - trivial since random variables have to be transformed through the neural networks , and their exact probability density functions will be intractable to compute .sampling - based approaches or deterministic approximate inference are two options that we will investigate in future .this work was supported by the swedish foundation for strategic research under the project _ cooperative localization _ and by the swedish research council under the project _probabilistic modeling of dynamical systems _( contract number : 621 - 2013 - 5524 ) .mpd was supported by an imperial college junior research fellowship .in the second experiment in the paper , results the joint learning of prediction and reconstruction error have been reported .the joint learning brings structure to feature values , which is not present if the autoencoder is learn separately , see fig . [fig : latent_2dtile_sep ] .

Abstract: modeling dynamical systems is important in many disciplines , e.g. , control , robotics , or neurotechnology . commonly the state of these systems is not directly observed , but only available through noisy and potentially high - dimensional observations . in these cases , system identification , i.e. , finding the measurement mapping and the transition mapping ( system dynamics ) in latent space can be challenging . for linear system dynamics and measurement mappings efficient solutions for system identification are available . however , in practical applications , the linearity assumptions does not hold , requiring non - linear system identification techniques . if additionally the observations are high - dimensional ( e.g. , images ) , non - linear system identification is inherently hard . to address the problem of non - linear system identification from high - dimensional observations , we combine recent advances in deep learning and system identification . in particular , we jointly learn a low - dimensional embedding of the observation by means of deep auto - encoders and a predictive transition model in this low - dimensional space . we demonstrate that our model enables learning good predictive models of dynamical systems from pixel information only .


the simplest parabolic equation is the canonical diffusion equation , with some initial condition .the conventional , forward , weighted monte - carlo solution to eq .( [ eq : canonical_diffusion ] ) is given by where the stochastic variables are found by following the stochastic trajectories given by the forward monte - carlo difference equation of motion : where , , and is a zero - mean , unit - variance gaussian random number , .the points where the particles are launched , , must be chosen so that , as given by eq .( [ eq : forward_solution ] ) , approximates .the algorithm is illustrated in fig .[ fig : forward ] below .  ) .,width=453 ]  the recently introduced _backward _ monte - carlo method  is both strikingly similar and fundamentally different from its older , forward sibling .the backward solution to eq . ( [ eq : canonical_diffusion ] ) is where the stochastic variables are found by following the stochastic trajectories given by the _backward _ monte - carlo difference equation of motion :   ) .,width=453 ]  comparing the forward and backward solutions , eqs .( [ eq : forward_solution ] ) and ( [ eq : backward_solution ] ) , respectively , the appearance of -functions in the forward solution is the most striking difference . to understand why the backward method produces a smooth solution , i.e. without -functions , we have to go back to its derivation  , which relies heavily on the feynman - kac formula  .in fact , eq . ( [ eq : backward_solution ] ) , supplemented by eq .( [ eq : backward_motion ] ) , is just the numerical approximation of the stochastic feynman - kac representation of eq .( [ eq : canonical_diffusion ] ) .the feynman - kac formula relates a parabolic equation to its _ naturally associated stochastic differential equation _this sde governs microscopic motion going _ backward _ in time , and eq .( [ eq : backward_motion ] ) is its difference approximation . as a consequence of time running in different directions in the macroscopic and microscopic world , we get the natural initial condition , in eq .( [ eq : backward_motion ] ) ; i.e. we launch the particles from the point in extended phase space where we want to know the solution to the parabolic equation and let them work their way backward in time to sample from the initial condition [ see fig . [fig : backward ] above ] .as is shown in appendix [ app : fmc ] , the forward monte - carlo method can be derived by forcing the naturally associated sde to govern microscopic motion going _ forward _ in time .this coercion is responsible for both the -functions in the forward monte - carlo solution , eq . ( [ eq : forward_solution ] ) , and the `` fuzzy initial condition '' , , imposed on the forward monte - carlo equation of motion , eq . ( [ eq : forward_motion ] ) [ see appendix [ app : fmc ] for details ] .+ however , the backward algorithm was not developed for esthetic reasons .the motivation originally came from the frustration over the extremely inefficient use of test particles of the forward method in cases where only the solution in a small part of phase space contributes significantly to the physics of interest .an example of this type of problem is the heating of fusion plasmas by cyclotron - resonance absorption on ions , where only the high - energy tail of the solution is of relevance ; but most test particles are wasted on the almost maxwellian bulk , which has been simulated with e.g. the fido forward monte - carlo code  .earlier claims  that the backward method is vastly superior in these cases will be substantiated in section [ sec : model ] .+ before that , the sources of numerical error , for both the forward and the backward method , will be identified and briefly discussed in the next section below . in section [ sec : model ] we will compare the forward and backward monte - carlo solutions to a simple model equation . in section [ sec : higher ], it will be shown how a minor modification of the backward monte - carlo difference equation of motion , eq .( [ eq : backward_motion ] ) , leads to improved time convergence of the backward solution , eq .( [ eq : backward_solution ] ) .finally , a summary of the main findings of this article follows in section [ sec : summary ] .the greatest disadvantage of the monte - carlo method is the unavoidable statistical error caused by the use of random numbers . with the conventional , forward monte - carlo method , the -functions in the solution [ see eq .( [ eq : forward_solution ] ) ] make binning of the particles necessary . with the number of particles in each bin proportional to , the statistical error becomes with obvious notations .backing out the solution by linear interpolation between the bins introduces an error .+ as was mentioned previously , the backward method was developed for cases where the solution is of interest only in a point or small part of phase space . in such situationsit reduces the statistical error to . even in cases where the backward method is used to calculate global solutions , andthe statistical error becomes , it does not have a finite - bin - size error because the absence of -functions in the solution [ see eq .( [ eq : backward_solution ] ) ] makes binning superfluous .+ the finite time step , , also introduces an error term .an analysis of this time - step error requires some care ; some mathematical subtleties are involved .there are two ways through which the time - step error can enter the numerical solution . + first , directly through the numerical approximation of the feynman - kac formula itself .i.e , even if we knew the _ exact _ stochastic variables , the backward solution , eq .( [ eq : backward_solution ] ) , would have an error due to the finite time step . to estimate the magnitude of this error, we need to go back to the derivation of it s formula in appendix a of ref . where terms of order and higher were neglected .however , in the same way that the -term , which was kept , becomes zero after averaging , so does the -term .so , even if the were known to all orders , , the backward solution eq .( [ eq : backward_solution ] ) , would have an error .the error of the forward solution , eq .( [ eq : backward_solution ] ) would of course be of the same order .+ the other way in which the finite time step introduces an error into the solution is through an error in . again , going back to ref . , we see that the drift term , , on the right - hand side of the backward monte - carlo equation of motion , eq .( [ eq : backward_motion ] ) , has an error , and the diffusive term , , has an error .the different character of these two error terms should be noted ; the -term is deterministic whereas the -term is stochastic . it can be shown ( see e.g. ref . ) that after averaging over all the , the approximation eq . ( [ eq : backward_motion ] ) introduces an error into the backward solution .again , the situation is equivalent for the forward method .+ in section [ sec : higher ] we will discuss how the time - step error of the backward solution can be reduced .but before that , we will study the backward and forward solutions to a simple model equation in the next section .to validate and benchmark the backward method and to compare it to the conventional forward method , we will solve a simple model equation that has an analytic solution .we have chosen the lorentz equation that has been used to model pitch - angle scattering in plasmas  : where the diffusion coefficient and the initial condition is , where is the legendre polynomial .the analytic solution is .+ in the following we will use the parameters : , , and ; while , , and will be varied so that their impact on the error can be studied .before we get into a detailed analysis of accuracy and convergence , we show the solutions ( with , , and ) to the lorentz equation ( [ eq : lorentz ] ) in fig .[ fig : bulk+tail ] below .  solving for over the whole interval , both the backward , , and the forward , monte - carlo solutions are indistinguishable from the exact solution .however , solving only in the subinterval , the backward solution is a much better approximation of the exact solution due to its much smaller statistical error . note especially that the forward solution drops to zero for not a single particle finds its way into the last few bins [ compare also with the sketch in fig .[ fig : forward ] ] . the ability to focus in on a small subinterval and calculate efficiently ( i.e. without wasting the vast majority of the particles ) the local solution is one of the main strengths of the backward monte - carlo method .+ for a detailed error analysis we need a well - defined measure of the error .we will use the local relative error . in the following we will plot the logarithm of the error against the logarithm of one of the parameters , , and , while keeping the other two parameters fixed .first we compare the statistical error of the forward and the backward solutions .to single out the statistical error from the finite bin - size and time - step errors we make the latter two small by choosing , and . with spanning six decades ( ) , we plot the error in the left frame of fig .[ fig : n - scaling ] below .the dotted lines are least - square fits whose slopes approximate the exponents that determine the scaling of the error . with the forward method ,the bins adjacent to are empty for small ( ) . as a result , for these values of , and the slope becomes flat .the low- data points have thus been excluded from the least - square fit to . as can be seen , the statistical error [ with fixed ] then scales as predicted .the exponents -0.43 and -0.41 are as close to the theoretical value of -1/2 as could be expected , given the fact that the other small but non - zero error terms always tend to flatten the slope .the forward and backward methods thus converge at the same rate as is increased , but the forward method needs times more particles to achieve the same accuracy as the backward method .this can easily translate into orders of magnitude in terms of execution time . + next , we study the bin - size error ( and the statistical error ) by varying over four decades ( ) while keeping fixed and .the result is shown in the right frame of fig .[ fig : n - scaling ] above .the first observation is that the backward solution is completely unaffected by the bin size ; is constant and the slope is zero .the error in the forward solution , , exhibits a more interesting behavior with different scalings for small and large . with few binsthere are many particles in each bin , so the statistical error is small .the bin - size error , , caused by backing out the solution by linear interpolation between these huge bins , however , becomes large . for small ,the slope of is -1.90 , in excellent agreement with the theoretical value of -2 . with many bins ,the bin - size error becomes small , but with few particles in each bin , the statistical error [ with fixed ] becomes dominant as is evidenced by the slope ( ) for large .the shape of the graph illustrates the conflict between resolution and statistics that is inherent for the forward monte - carlo method .the backward solution , however , can be calculated in two points arbitrarily close without affecting the statistical error .this is yet another of the main strengths of the backward method. + turning now to the time - step error , we fix and and vary . as can be seen in fig . [ fig : t - low ] below ,  ) with the forward ( blue ) and backward ( red ) methods . ]both the forward and the backward methods converge as predicted ( the respective slopes are ) down to a time step . at this pointthe statistical error becomes dominant .as was discussed in section [ sec : errors ] , the dominant time - step error in the monte - carlo equations of motion , eqs .( [ eq : forward_motion ] ) and ( [ eq : backward_motion ] ) , comes from the diffusive term .there is thus reason to suspect that a higher - order diffusive term might result in better overall time convergence , by reducing the time - step error to . in appendix [ app : trapezoidal ]it is shown that the next - higher - order monte - carlo equation of motion is in the left frame of fig .[ fig : t - high ] below the time - convergence study presented in fig .[ fig : t - low ] above is repeated using the higher - order approximation of eq .( [ eq : backward_motion_higher ] ) . in the right frameexactly the same experiment is repeated with ten times more particles to further reduce the statistical error .somewhat surprisingly , the forward solution does not converge at all .the backward solution , however , exhibits the faster time convergence we had hoped for ( the slope is ) .+ milshtein , who was not explicitly concerned with solving parabolic equations , did investigate the scaling of the error of expectation values similar to the ones of interest here  .his results are consistent with the error of the forward and backward lower - order solutions . however , for the higher - order methods introduced in this section , milshtein s results would still indicate a error for both the forward and backward solutions , in clear disagreement with the scalings of fig .[ fig : t - high ] .we hope to be able to resolve the discrepancies in future work .we have shown that the backward monte - carlo method works as expected , i.e. it dramatically reduces the statistical error in situations where the solution is sought only in a small part of phase space . furthermore , even in cases where we solve for the global solution , the backward method removes the conflict between resolution and statistics .this has great practical significance e.g. when the gradient of the solution is of interest .+ we have also shown how a remarkably simple modification of the backward monte - carlo equation of motion leads to improved time convergence .[ sec : summary ]  this research was sponsored by the oak ridge national laboratory , managed by ut battelle , llc , for the u.s .department of energy undercontract de ac0500or22725 .research was supported in part by anappointment to the ornl postdoctoral research associates program , administered jointly by oak ridge national laboratory and the oak ridge institute for science and education .+ the author wishes to thank his colleagues in the fusion energy division radiofrequency theory group ( don batchelor , lee berry , mark carter , and fred jaeger ) for helpful comments during the work on this article .the aim of this appendix is to derive the conventional , forward , weighted monte - carlo method in the same manner as the backward method was derived  .the starting point is the _ forward _ sde : solving this sde is trivial ; following exactly the same procedure as for the backward sde , we get where is a zero - mean , unit - variance gaussian random number , . when the sde is backward , the obvious initial condition is . for the forward sde ( [ eq : forward_sde ] ) ,we simply do not know which initial condition to impose ; we will have to leave undefined for now .+ as before  , we use the it formula to differentiate and obtain where again we have identified , , and used [ eq . ( [ eq : canonical_diffusion ] ) ] . using the macroscopic initial condition , , we get at this stage of the derivation of the _ backward _ method , we used the microscopic initial condition and found that the equivalent of the lhs of eq .( [ eq : proto2 ] ) was in fact a solution to eq .( [ eq : canonical_diffusion ] ) : [ ] . but here , in the forward derivation , we can not do that because ! going forward in time , is an unknown ; we could use the monte - carlo equation of motion , eq .( [ eq : proto0 ] ) , to find , but we do not have an initial condition !so , is there any way to get from eq.([eq : forward_sde ] ) ?the answer is yes ; there is a ( rather contrived ) way .+ if we let be a distribution , we can write it as \ . \label{eq : proto3}\ ] ] substituting eq .( [ eq : proto2 ] ) into eq .( [ eq : proto3 ] ) , we get : \ .\label{eq : proto4}\ ] ] the numerical approximation of this expectation value is the forward weighted monte - carlo solution : now , we still need to know what the microscopic initial condition on should be .the best we can do is to make sure that .note that `` approximately equal '' must be given a very liberal definition because is a jagged sum of -functions , whereas is in general a smooth function .the stochastic variables can now be found by following the stochastic trajectories given by the forward monte - carlo equation of motion , eq .( [ eq : proto0 ] ) , with the `` fuzzy initial condition '' : diffusive term of the monte - carlo equations of motion , eqs .( [ eq : forward_motion ] ) and ( [ eq : backward_motion ] ) , is an approximation of the it integral where is the stochastic process that solves the sde where is a wiener process , and we impose the initial condition . the approximation eq .( [ eq : sigma_integral_low ] ) replaces with its zero - order taylor expansion , . to reduce the error in eq .( [ eq : backward_motion ] ) to we taylor expand to first order , and approximate the solution to eq .( [ eq : sde ] ) with \ .\label{eq : x_approx}\ ] ] the it integral over becomes : \right \ }dw(s ) + \mathcal{o}(\delta{}t^{3/2 } ) = \nonumber \\ & \sigma&(x_{t } ) \int_{t}^{t + \delta{}t}\!dw(s ) + \sigma{}(x_{t } ) \ , \sigma{}'(x_{t } ) \int_{t}^{t + \delta{}t } [ w(s ) - w(t ) ] \ ,dw(s ) + \mathcal{o}(\delta{}t^{3/2 } ) \ , \nonumber \\ { }\label{eq : sigma_integral_high}\end{aligned}\ ] ] where is and hence was neglected . the first term on the rhs of eq .( [ eq : sigma_integral_high ] ) is just the low - order approximation eq .( [ eq : sigma_integral_low ] ) .the second term is the next - order correction . to calculate this correction term ,we first need some intermediate results . + following bjrk , we introduce the stochastic variables  \ , \label{eq : i_n}\ ] ] and  \ , \label{eq : b_n}\ ] ] where .it trivially follows that [ w(t_{k+1 } ) - w(t_{k } ) ] = w^{2}(t ) \, \label{eq : b+i}\ ] ] and ^{2 } = s_{n}(t ) \ .\label{eq : b - i}\ ] ] in appendix a of ref . we showed that subtracting eq .( [ eq : b - i ] ) from eq .( [ eq : b+i ] ) and taking the limit , we obtain \ .\label{eq : w_integral}\ ] ]  applying eq .( [ eq : w_integral ] ) to eq .( [ eq : sigma_integral_high ] ) we get + \nonumber \\ & \half & \sigma{}(x(t ) ) \ , \sigma{}'(x(t ) ) \ ,\{[w(t + \delta{}t ) - w(t)]^{2 } - \delta{}t \ } + \mathcal{o}(\delta{}t^{3/2 } ) \ . \nonumber \\ { } \label{eq : sigma_integral_high_eval}\end{aligned}\ ] ] in the absence of advection , as is the case for eq .( [ eq : canonical_diffusion ] ) , , and the higher - order monte - carlo equations of motion become where we have used the fact that ; is a zero - mean , unit - variance gaussian random number ; ; and and . + in ref . milshtein derived eq .( [ eq : sigma_integral_high_eval ] ) by introducing + c_{3 } [ w(t + \delta{}t ) - w(t)]^{2 }  \ , \nonumber\ ] ] and finding the , and that minimize $ ] .j. carlsson , t. hellsten , and l .- g .eriksson , in _ theory of fusion plasmas _ , e. sindoni , f. troyon , and j. vaclavik , eds ., editrice compositori , bologna , 1994 , pp .351356 ; t. hellsten , j. carlsson , and l .- g .eriksson , phys .* 74 * ( 1995 ) 36123615 ; j. carlsson , l .- g .eriksson , and t. hellsten , nucl .fusion * 37 * ( 1997 ) 719723 ; j. hedin , j. carlsson , t. hellsten , and a. jaun , plasma phys .fusion * 40 * ( 1998 ) 10851095 ; j. carlsson , t. hellsten , and j. hedin , phys .plasmas * 5 * ( 1998 ) 28852892 ; l .- g .eriksson , m. mantsinen , d. borba , a. fasoli , r. heeter , s. sharapov , d. f. h. start , j. carlsson , a. gondhalekar , t. hellsten , and a. korotkov , phys .* 81 * ( 1998 ) 12311234 .

Abstract: the recently introduced backward monte - carlo method [ johan carlsson,`arxiv : math.na/0010118 ` ] is validated , benchmarked , and compared to the conventional , forward monte - carlo method by analyzing the error in the monte - carlo solutions to a simple model equation . in particular , it is shown how the backward method reduces the statistical error in the common case where the solution is of interest in only a small part of phase space . the forward method requires binning of particles , and linear interpolation between the bins introduces an additional error . reducing this error by decreasing the bin size increases the statistical error . the backward method is not afflicted by this conflict . finally , it is shown how the poor time convergence can be improved for the backward method by a minor modification of the monte - carlo equation of motion that governs the stochastic particle trajectories . this scheme does not work for the conventional , forward method . + _ pacs : _ 02.70.lq ; 02.60.lj +  linear parabolic partial differential equation ; monte - carlo method ; time convergence