structural credit risk models such as the first - passage - time models proposed by or are widely used in the analysis of defaultable corporate securities . in these models , a firm defaults if a random process representing the firm s asset value hits some threshold that is typically linked to value of the firm s liabilities .first - passage - time models offer an intuitive economic interpretation of the default event .however , in the practical application of these models a number of difficulties arise : to begin with , it might be difficult for investors in secondary markets to assess precisely the value of the firm s assets .moreover , for tractability reasons is frequently modelled as a diffusion process . in that casethe default time is a predictable stopping time , which leads to unrealistically low values for short - term credit spreads . for these reasonsproposed a model where secondary markets have only incomplete information on the asset value .more precisely they consider the situation where the market obtains at discrete time points a noisy accounting report of the form ; moreover , the market knows the default history of the firm .duffie and lando show that in this setting the default time admits an intensity that is proportional to the derivative of the conditional density of the asset value at the default threshold .this well - known result provides an interesting link between structural and reduced - form models .moreover , the result shows that by introducing incomplete information it is possible to construct structural models where short - term credit spreads take reasonable values .the subsequent work of discusses the pricing of the firm s equity in structural models with unobservable asset value .moreover , it is shown that the valuation of the firm s equity and debt leads to a stochastic filtering problem : one needs to determine the conditional distribution of the current asset value given the -field representing the information available to the market at time .consider this problem in the setup of duffie and lando where new information on the asset value arrives only at discrete points in time .working with a markov - chain approximation approximation for they derive a recursive updating rule for the conditional distribution of the approximating markov chain via elementary bayesian updating ; the discrete nature of the information - arrival is crucial for their arguments .  neither nor study the price dynamics of traded securities under incomplete information .hence in these papers it is not possible to analyze the pricing and the hedging of derivative securities such as options on corporate bonds or on the stock .the main goal of the present paper is therefore to develop a proper theory of derivative asset analysis for structural credit risk models under incomplete information .more precisely , we make the following contributions .first , in order to obtain realistic price dynamics for the traded securities , we model the noisy observations of the asset value by a continuous time process of the form for some brownian motion independent of . we show that this leads to price processes with non - zero instantaneous volatility , whereas the discrete information arrival considered by duffie and lando or frey and schmidt generates asset prices that evolve deterministically between the news - arrival dates .moreover , modeling as a continuous time processes is in line with the standard literature on stochastic filtering such as .second , we show that in order to derive the price dynamics of traded securities we need to determine the dynamics of the conditional distribution of given . this is a challenging stochastic filtering problem , since under full observation , the default time is predictable .hence standard filtering techniques for point process observations such as the results from do not apply .we therefore transform the original problem to a new filtering problem where the observations consist only of the noisy asset information ; the signal process in this new problem is on the other hand given by the asset value process stopped at the first exit time of the solvency region . using results of on the filtering of stopped diffusion processes we derive a stochastic partial differential equation ( spde ) for the conditional density of given , denoted , and we discuss the numerical solution of this spde via a galerkin approximation . extending to our more general information structure ,we show that admits an intensity process such that the intensity at time is proportional to the spatial derivative of at .armed with these results we finally study derivative asset analysis in our setup : we identify the price dynamics of the traded securities ; we consider the pricing of options on traded assets ; we derive risk - minimizing dynamic hedging strategies for these claims , and we discuss model calibration .the paper closes with a small simulation study illustrating the theoretical results .incomplete information and filtering methods have been used before in the analysis of credit risk .structural models with incomplete information were considered among others by , , , , and .the last contribution is related to the present paper . working in a similar setup as ours, cetin uses probabilistic arguments to establish the existence of a default intensity in the market filtration , and he derives the corresponding filter equations . he does not discuss the existence of the conditional density and financial applications are discussed only in a peripheral manner . from a purely mathematical point of view our paper is also closely related to who deal with the filtering of partially observed diffusions up to the first exit time of a domain .the relation between our results and those of krylov and wang are best explained once the mathematical details of our setup have been introduced , and we refer to section [ sec : filtering ] , remark [ rem : krylov ] for a deeper discussion of similarities and differences between the two papers .reduced - form credit risk models with incomplete information have been considered previously by , and , among others .the modelling philosophy of the present paper is inspired by , but the mathematical analysis differs substantially .in particular , in the default times of the firms under consideration do admit an intensity under full information . hence the filtering problem that arises in the pricing of credit derivatives can be addressed via a straightforward application of the innovationsapproach to nonlinear filtering .the remainder of the paper is organized as follows . in section [ sec : model ]we introduce the model ; the relation between traded securities and stochastic filtering is discussed in section [ sec : pricing - basic - securities ] ; section [ sec : filtering ] is concerned with the stochastic filtering of the asset value ; in section [ sec : dynamics ] we derive the dynamics of corporate securities ; section [ sec : applications ] is concerned with derivative asset analysis ; the results of numerical experiments are given in section [ sec : simulations ] .[ [ acknowledgements . ] ] acknowledgements .+ + + + + + + + + + + + + + + + +  financial support from the german science foundation ( dfg ) and from the vienna science and technology fund ( wwtf ) , project ma14 - 031 , is gratefully acknowledged .moreover , we thank andreas peterseil for useful research assistance .we begin by introducing the mathematical structure of our model .we work on a filtered probability space and we assume that all processes introduced below are -adapted . since we are mainly interested in the pricing of derivative securities we assume that is the risk - neutral pricing measure .we consider a company with nonnegative asset value process .the company is subject to default risk and the default time is modelled as a first passage time , that is in practice might represent solvency capital requirements imposed by regulators ( see example [ ex : fin - institution ] ) or it might correspond to an endogenous default threshold as in ( see example [ ex : duffie - lando ] ) . by denote the _ default state _ of the firm at time , that is if and only if the firm has defaulted by time ; the associated _ default indicator process _is denoted by .[ ass : assets - and - dividends ]  \1 ) the risk free rate of interest is constant and equal to .\2 ) the firm pays _ dividends _ at equidistant deterministic time points , ( for instance semi - annual dividend payments ) .the set of dividend dates is denoted by .the dividend payment at is a random percentage of the _ surplus _ ( the part of the asset value that can be distributed to shareholders without sending the company into immediate default ) . denoting by the dividend payment at , it holds that here s is an iid sequence of noise variable that are independent of , take values in and that have density function .we assume that is bounded and twice continuously differentiable on  ] is given by provided of course that the discounted cash flow stream is integrable .note that is defined as conditional expectation with respect to the -field .the use of the risk - neutral pricing formula  ensures that the discounted gains from trade of every traded security are martingales , which is sufficient to exclude arbitrage opportunities .hedging arguments based on martingale modelling are presented in section [ subsec : hedging ] .finally we describe two economic settings that can be embedded in our framework .[ ex : fin - institution ] our first example is that of a financial institution that is subject to financial regulation .we assume that the institution has issued shares to outside shareholders .it is run by a management team that knows the asset value .management is prevented from actively trading the shares of the institution , for instance because of insider trading regulation .outside stock and bond investors on the other hand are unable to discern the exact asset value from public information .the dividend policy of the firm is of the form . in this examplewe let so that dividend payments do reduce the asset value of the firm .we assume that the institution is subject to capital adequacy rules such as the basel iii or the solvency ii rules . loosely speaking these rulesrequire that the ratio of the equity capital of the firm over its total asset value must be larger than a given threshold .if we denote by the value of the firms liabilities , this translates into the condition that and hence that we assume that regulators actively monitor that the state of the firm is in accordance with the capital adequacy rule  and that management provides them with correct information about the asset value .if falls below the threshold , regulators shut down the financial institution and there will be a default .hence the default time is a first passage time with default threshold given in .note that in this setup default is enforced by regulators with privileged access to information .[ ex : duffie - lando ] the well - known model of can be embedded in our setup as well .duffie and lando consider a firm that is operated by risk - neutral equity owners who have complete information about .the firm issues some debt in the form of a consol bond in order to profit from the tax shield of debt , but there are no traded shares and no dividend payments to outside investors .equity owners are prohibited from trading in bond markets by insider trading regulation . in this setupthe owners of the firm have the option to stop servicing the firm s debt , in which case the firm defaults . following ,duffie and lando show that the optimal default time ( for the equity owners ) is a first passage time , but now with endogenously determined default threshold .note that in this example the random variables can be viewed as additional information on that arrives at discrete time points , such as earnings announcements .this interpretation corresponds to a value of for the parameter in .moreover , it is not necessary to assume that the rvs are of the special form ; we only require that for fixed the mapping is smooth and bounded .in this section we explain the relation between the prices of traded securities and stochastic filtering and we discuss several examples .[ [ traded - securities . ] ] traded securities .+ + + + + + + + + + + + + + + + + +  we assume that a couple of securities related to the firm are traded .more precisely , the set of traded securities consists so - called _ basic debt securities _ and of the stock of the firm .we now describe the payoff stream of these securities in more detail .first we refer to an asset as a _ basic debt security _ if its cash - flow stream can be expressed as a linear combination of the following two building blocks  * a _ survival claim _ with generic maturity date .this claim pays one unit of account at , provided that . * a _ payment - at - default claim _ with generic maturity date .this claim pays one unit directly at , provided that .it is well known that bonds issued by the firm and credit default swaps on the firm can be expressed as linear combination of these building blocks , see for instance .next we discuss the modelling of the firm s stock .the shareholders of the firm receive the dividend payments made by the firm at dividend dates . hence the cumulative cashflow stream received by the shareholders up to time equals .the risk neutral pricing formula  thus implies that the value of the firm s stock gives the market capitalization of the firm at time , that is value of the entire outstanding stock . since we assume that the number of outstanding shares is constant we use the symbol also for the price process of a single share . ]is given by   [ [ relation - to - stochastic - filtering . ] ] relation to stochastic filtering .+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +  consider now a traded security with cash - flow stream and ex - dividend price . in the sequel we mostly consider the pre - default value of the security given by ( pricing for is largely related to the modelling of recovery rates which is of no concern to us here ) .using iterated conditional expectations we get that by the markov property of , for basic debt securities and for the stock the inner conditional expectation can be expressed as a function of time and of the current asset value , that is see below for details . the function is called the _ full - information value _ of the security . in section [ sec : filtering ]we show that on the -measurable set the conditional distribution of given admits a density and we derive an spde for this density .substituting into gives that this relation provides an important relationship between prices of traded securities and stochastic filtering . in this paper this relationship will be used in two ways .first , at any given time point an estimate of is backed out from the price of traded securities observed at time ( the necessary calibration methodology is described in section [ subsec : calibration ] ) .second , in order to derive the price dynamics of traded securities under the risk - neutral measure we determine the dynamics of using filtering methods ; this gives the dynamics of the pre - default value of the traded securities .this approach is akin to the use of factor models in term structure modelling where prices of traded securities are used to estimate the current value of the factor process and where bond price dynamics are derived from the dynamics of the factor process . in fact , in section [ subsec : derivatives ] it will be shown that in our setup the price at time of a non - path - dependent option on traded assets is a function of so that our model can be viewed as factor model with infinite - dimensional factor process .note that the modelling approach outlined above leads to filtering problems under the risk - neutral measure and that it is not necessary to associate with publicly observable economic data ; is simply an abstract representation of the continuously released economic information on the asset value .this differs from the ` classical ' application of stochastic filtering in statistical inference .a typical problem in the latter context would be as follows : the process is identified with a specific set of economic data that contains noisy information of and filtering techniques are employed to estimate the conditional distribution of under the historical measure given the observed trajectories of , and up to time .such an approach could be used to estimate the firm s real - world default probability , similar in spirit to the well - known public firm edf model  .it is worth mentioning that the mathematical results developed in sections [ sec : filtering ] and [ sec : dynamics ] cover also applications of this type .[ [ full - information - value - of - traded - securities . ] ] full information value of traded securities .+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +  next we discuss the computation of the full information value for basic debt securities and for the stock .we concentrate on the case , so that there is a downward jump in at the dividend dates ; for the asset value is a geometric brownian motion and the ensuing computations are fairly standard .we begin with a survival claim with payoff and associated full - information value . since is a -martingale we get the following pde characterization of : first , between dividend dates solves the boundary value problem with boundary condition for , where we let for  second , at a dividend date it holds that finally , one has the terminal condition for .these conditions can be used to compute numerically by a backward induction over the dividend dates ; see for instance for details . moreover, we will need the pde characterization of to derive the price dynamics of a survival claim under incomplete information in section [ subsec : filtering - dynamics ] . recall that a payment - at - default claim with maturity pays one unit directly at , provided that .the pde characterization is similar to the case of a survival claim ; however , now the boundary condition is , and the terminal value is , . by definitionthe full information value of all basic debt securities can be computed from and .next we consider the stock of the firm .it follows from that the full information value of the firm s stock is given by the next lemma whose proof is given in appendix [ app : proofs ] shows that can be interpreted as value of all future dividend payments ( up to ) .[ lemma : finite - stock - price ] under assumption [ ass : assets - and - dividends ] it holds that .note that lemma [ lemma : finite - stock - price ] implies that .it follows that the stock price is finite as well ( this is not a priori clear since is the expected value of an infinite payment stream ) .using the fact that is a martingale we obtain the following pde characterization for : between dividend dates solves the pde with boundary condition ; at the dividend date  satisfies the relation since we assumed equidistant dividend dates it holds that for so that it is enough to compute for .an explicit formula for is not available ; the main problem is the fact that the downward jump in at a dividend date combines arithmetic and geometric expressions .there are essentially two options for computing numerically . on the one hand onecan rely on monte carlo methods . in order to speed up the simulation explicit pricing formulas for in a black scholes model with continuous dividend stream can be used as control variate .alternatively , it is possible to use pde methods in order to compute .we omit the details since the numerical computation of option prices is not central to our analysis .fix some horizon date , for instance the largest maturity date of all outstanding derivative securities related to the firm .recall from the previous section that in order to derive the price dynamics of traded securities we need to determine the dynamics of the conditional density , .this problem is studied in the present section .following the usual approach in stochastic filtering we start with a characterization of the conditional distribution of given ( the filter distribution ) in weak form .more precisely , given a function on such that for all we want to derive the dynamics of the conditional expectation this is sufficient for our purposes since we are only interested in the dynamics of the filter distribution prior to default .the problem is a challenging filtering problem since the default time does not admit an intensity under full information .hence standard filtering techniques for point process observations as in do not apply .this issue is addressed in the following proposition where , loosely speaking , is transformed to a filtering problem with respect to the background filtration .[ prop : reduction - to - background ] denote by the asset value process stopped at the default boundary , by the noisy asset information corresponding to the signal process and by the cumulative dividend process corresponding to .then we have for such that for all   for notational simplicity we ignore the dividend observation in the proof so that .the first step is to show that where the filtration is generated by the noisy observations of the _ stopped _asset value process ; the proof of this identity is given in appendix [ app : proofs ] .second , using the dellacherie formula ( see for instance lemma 3.1 in ) and the relation , we get as claimed .  with the notation , proposition [ prop : reduction - to - background ] shows that in order to evaluate the right side of one needs to compute for generic such that for all conditional expectations of the form this is a stochastic filtering problem with signal process given by ( the asset value process stopped at the first exit time of the halfspace ) .in the sequel we study this problem using results of on the filtering of diffusions stopped at the first exit time of some _ bounded _ domain , first for the case without dividends and in section [ subsec : filtering - wrt - fi ] for the general case .  in order to be in a situation where the results of are applicable we fix some large number and replace the unbounded halfspace with the bounded domain . for thiswe define the stopping time and we replace the original asset value process with the stopped process .applying proposition [ prop : reduction - to - background ] to the process leads to a filtering problem with signal process .more precisely , one has to compute conditional expectations of the form where , with a slight abuse of notation , and note that is the first exit time of from the domain .moreover , it holds by definition that , i.e.  is equal to the asset value process stopped at the boundary of the bounded domain .hence the state space of is given by  ] such that .then it holds that for ,   the proof of the proposition is given in appendix [ app : proofs ] .we continue with a few comments .denote by the first exit time of the cum - dividend asset value process from .clearly , as and thus .hence the conditional probability that reaches the upper boundary is controlled uniformly for all subfiltrations of by the first exit time of a geometric brownian motion from ; this can be used to choose when implementing of the model .the ucp convergence in the second statement ensures that the difference between the prices of traded securities in the model based on and in the original model can be controlled uniformly in  ] of .  2 .the solution has additional regularity : it holds that a.s . and that the trajectories of belong to ,h^1_0(s^x)\big) ] , . using that by assumption [ ass : information ] , we obtain that the stochastic integral with respect to can be written as hence it holds that moreover , .an approximation argument shows that these properties hold also for , see for instance , so that by proposition [ prop : sigma - t ] .[ rem : krylov ] it is interesting to compare our results to those of krylov and wang. translated to our setting they consider the following problem : their signal process is a non - degenerate diffusion on ; denoting by the first exit time of from ( in our notation ) , the observation filtration is given by and by the filtration generated by the indicator .krylov and wang then derive an spde for the conditional density of given and the information and they show that where and are given by similar expressions as in theorem [ thm : spde - for - u ] .however , krylov and wang do not compute the dynamics of the conditional probability , an expression that is crucial for the computation of default intensities ( see theorem [ thm : default - intensity ] ) .   in this subsectionwe compute the conditional distribution of with respect to the market filtration .the key part is to include the dividend information and the jumps of the asset value process at the dividend dates in the analysis .we recall some notation : the dividend dates are denoted by , ; denotes the dividend paid at and the conditional density of given is . in the sequel we let for notational convenience .moreover , we let for some smooth and strictly positive reference density on that we use in the construction of the model via a change of measure .note that the choice of has no economic implications , as we are only interested in the distribution of the asset value prior to default .we use an extension of the reference probability argument from section [ subsec : preliminaries ] . consider a product space , , and so that supports a -brownian motion .suppose that supports a a brownian motion and an independent random measure with compensating measure equal to let , .denote by the solution of the sde and define the state process by .the indicator function in the dynamics of is included as under the asset value may become negative due to a downward jump at a dividend date . note that under the measure that we construct next such jumps have probability zero .  in order to revert to the original model dynamics we introduce the density martingale where is as in and where satisfies since and are probability densities we get   hence and we obtain that since and are orthogonal we get that the next lemma , whose proof is given in appendix [ app : proofs ] , shows that is in fact the appropriate density martingale to consider ( is the horizon date fixed at the beginning of section [ sec : filtering ] ) .[ lemma : density - for - dividends ] it holds that define the measure by . then under the random measure has -compensator , the triple has the joint law postulated in assumption [ ass : assets - and - dividends ] .similarly as in we get from the generalized bayes rule ( , proposition iii.3.8 ) that where .[ [ dynamics - of - the - unnormalized - density . ] ] dynamics of the unnormalized density .+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +  the form of in suggests the following dynamics of the unnormalized density : between dividend dates , that is on , , solves the spde with initial value ; at the density is first updated to second , for there is a shift to account for the downward jump in the asset value , that is where we let for . in theorem [ prop : dividend - information ] belowwe show that this is in fact correct . as a first stepwe describe the dynamics of by means of an spde .denote for and by the function , where we let for .consider the spde with initial condition .the interpretation of is analogous to the previous section : for it holds that   the next result extends theorem [ thm : spde - for - u ] to the case with dividends .  [ prop : dividend - information ] 1 . there is a unique positive solution of the spde .define and then it holds that .[ [ filtering - with - respect - to - mathbbfmathbbm . ] ] filtering with respect to .+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +  finally we return to the filtering problem with respect to the filtration .  [corr : cond - density ] define the norming constant by and let and .then it holds for that   combining proposition [ prop : reduction - to - background ] and theorem [ prop : dividend - information ] we get    the spde is a stochastic partial differential equation and thus an infinite - dimensional object . in order to solve the filtering problem numerically and to generate price trajectories of basic corporate securities one needs to approximate by a finite - dimensional equation . a natural way to achievethis is the galerkin approximation method .we first explain the method for the case without dividend payments .consider linearly independent basis functions generating the subspace , and denote by the projection on this subspace with respect to . in the galerkin methodthe solution of the equation with initial condition is used as an approximation to the solution of .since projections are self - adjoint , we get that for  hence if belongs to ( the orthogonal complement of ) . since moreover conclude that for all .hence is of the form , and we now determine an sde system for the dimensional process . using we get for  define now the matrices , and with , and . equating and ,we get the following system of sdes for  with initial condition .equation  can be solved with numerical methods for sdes such as a simple euler scheme or the more advanced splitting up method proposed by .further details regarding the numerical implementation of the galerkin method are given among others in or in chapter 4 of .conditions for the convergence are well - understood , see for instance : the galerkin approximation for the filter density converges for if and only if the galerkin approximation for the deterministic forward pde converges .  in the case with dividend informationthe galerkin method is applied successively on each interval , .denote by the approximating density over the interval .following the initial condition for the interval is then given by that is by projecting the updated and shifted density onto .in this section we identify the price process of traded corporate securities .it turns out that these price processes are of jump - diffusion type , driven by a brownian motion ( the martingale part in the semimartingale decomposition of ) , by the compensated random measure corresponding to the dividend payments and by the compensated default indicator process .   as a first step we derive the -semimartingale decomposition of the default indicator process andwe show that admits an -intensity .[ thm : default - intensity ] the -compensator of is given by the process where and where the default intensity is given by here is conditional density of given introduced in corollary [ corr : cond - density ] .we mention that a similar result was obtained in for the case where the noisy observation of the asset value process arrives only at deterministic time points .we use the following well - known result to determine the compensator of ( see for instance section 2.3 of ) .  [ prop : compensator - via - hazard - approach ] let and suppose that for all . denote the doob - meyer decomposition of the bounded -submartingale by . define the process via then is an -martingale . in particular ,if is absolutely continuous , that is if , has the default intensity .  in order to apply the proposition we need to compute the doob - meyer decomposition of the submartingale .here we get theorem [ prop : dividend - information ] gives and .next we consider the term . by definitionit holds that hence we get that is a -local martingale ; see for instance , corollary iii.3.10 .it s product rule therefore gives that furthermore we have the claim thus follows from proposition [ prop : compensator - via - hazard - approach ] and from the definition of in corollary [ corr : cond - density ] .   in this sectionwe derive the dynamics of the traded security prices . in line with standard notationwe denote for \times s^x ) \to { \mathbb{r}} ] it holds that on we finally obtain the result for .mutatis mutandis these arguments also apply to the default claim and to the stock price .the additional term in the drift of stems from the fact that ; the additional integral with respect to in the dynamics of the stock price is due to the different behaviour of at a dividend date , see .of course this term is quite intuitive : the expected downward jump in the stock price at a dividend date is just equal to the expected dividend payment .[ [ comments - and - extensions . ] ] comments and extensions .+ + + + + + + + + + + + + + + + + + + + + + + +  \1 . the processes that drive the price dynamics of the traded assets ( the brownian motion , the compensated random measure and the compensated default indicator ) are directly related to the arrival of information on the market .hence theorem [ thm : asset - price - dynamics ] formalizes the idea that the prices of traded corporate securities are driven by the arrival of new information on the value of the underlying firm .\2 . in order to study dynamic hedging strategies we need the dynamics and the predictable quadratic variation of the cum dividend price or _gains process _ of the traded assets .the survival claim has no intermediate cash flows and we have ; for the default claim it holds that ; for the stock we have .note that theorem [ thm : asset - price - dynamics ] implies that the discounted gains processes of all three assets are martingales as they have to be given that we work directly under a martingale measure .next we turn to the quadratic variations . according to theorem [ thm : asset - price - dynamics ] , the discounted gains process of the traded asset has a martingale representation of the form and the integrands are explicitly given in the theorem .define a measure on by letting ) = b(t):= t + \sum_{n = 1}^\infty \delta_{\{t_n\ } } ( [ 0,t ] ) ] one has , the abstract bayes formula gives moreover , using we have for  ] with initial condition ( a given constant ) is given by ,  ] .in particular , for a selffinancing strategy it holds that for all .finally we define the _ remaining risk process _ of the strategy by consider now a claim with square integrable -measurable payoff and an admissible strategy with ( note that this condition can always be achieved by a proper choice of the cash position ) .then is a measure for the precision of the hedge , in particular , if is a selffinancing hedging strategy for .an admissible strategy is called _ risk - minimizing _ if and if moreover for any  ] and such that and are .moreover , for all the random variable is square integrable .note that is left continuous and that is right continuous .show that the strategy that minimizes the remaining risk over all admissible discrete trading strategies with terminal value can be described as follows : for the random vector , is determined from the regression equation where and where .the cash position is given by so that for all . in order to compute onemay therefore generate realisations of and of via monte carlo ; can then be computed from these simulated data via standard regression methods .the fact that pricing formulas and hedging strategies depend on the current filter density raises the issue of model calibration . as explained in section [ sec : pricing - basic - securities ] , we view the process generating as abstract source of information so that the density process is not directly observable for investors . on the other hand ,pricing formulas need to be evaluated using only publicly available information .hence we have to back out an estimate of from prices of traded securities at time .we now explain how this can be achieved by means of a quadratic optimization problem with linear constraints in case that a galerkin approximation of the form with smooth basis functions is used to approximate the filter density .assume that we observe prices of basic corporate securities with full information value , . in order to match the observed pricesperfectly , the vector of fourier coefficients need to satisfy the following linear constraints moreover , it should hold that in order to prevent from becoming negative .typically , so that the constraints do not determine the fourier coefficients uniquely . in that case one needs to apply a regularisation procedure . following who face a similar issue in the calibration of the implied copula model to cdo tranche spreads, we propose to minimize the -norm of the second derivative of over all nonnegative that satisfy the constraints ; this produces a maximally smooth initial density .denote by the second derivative of and define the symmetric and positive definite matrix by . since minimization of the -norm of thus leads to the quadratic optimization problem this problem can be solved with standard optimization software ; a numerical example is discussed in section [ sec : simulations ] .in this section we illustrate the model with a number of numerical experiments .we are particularly interested in the impact of the noisy asset information on the asset price dynamics under incomplete information .we use the following setup for our analysis : dividends are paid annually ; the dividend size is modelled as where is beta - distributed with mean equal to and standard deviation equal to .the process is two - dimensional with and ; around the kink at ; details do not matter . ] for this choice of models the idea that the market obtains additional information on the asset value of the firm as soon as the asset value is less than one standard deviation away from the default threshold , perhaps because the firm is monitored particularly closely in that case .the remaining parameters are given in in table [ table : parameters ] ..[table : parameters ] parameters used in simulation study . [ cols="<,^,^,^,^",options="header " , ]   in order to generate a trajectory of the filter density with initial value and related quantities such as the stock price we proceed according to the following steps .  1 .generate a random variable , a trajectory of the asset value process with and the associated trajectory of the default indicator process .2 .  generate realizations and of the cumulative dividend process and of the noisy asset information , using the trajectory generated in step 1 as input .3 .  compute for the observation generated in step 2 a trajectory of the unnormalized filter density with initial value , using the galerkin approximation described in section [ subsec : finite - dim - approx ] .return and , .  for details on the numerical methodology including the choice of the basis functions , numerical methods to solve the sde system  arising from the galerkin method and tests for the accuracy of the numerical implementation we refer to chapter 4 of .next we describe the results of our numerical experiments . in figure[ fig : stock - price - only - dividends ] we plot a trajectory of the stock price and of the corresponding full information value for the case where the market has only dividend information ( ) .this can be viewed as an example of the discrete noisy accounting information considered in .we see that has very unusual dynamics ; in particular it evolves deterministically between dividend dates .   of the stock ( dashed line ) and of the stock price ( normal line , label shat ) for ( only dividend information ) .[ fig : stock - price - only - dividends],width=510,height=264 ]  the next graphs use the parameter values and .they show in particular that more realistic asset price dynamics can be obtained by introducing the continuous noisy asset information .we begin by plotting the evolution of the filter density in figure [ fig : evol - of - pi ] .it can be seen that evolves continuously between dividend dates , and that the density jumps at a dividend date .moreover , the mode of is close to the default boundary immediately prior to default .figure [ fig : stock - price - c1 - 4-c2 - 0 ] shows a typical stock price trajectory together with the full information value .clearly , for  has nonzero volatility between dividend dates .a comparison of the two trajectories moreover shows that the stock price jumps to zero at the default time ; this reflects the fact that the default time has an intensity under incomplete information so that default comes as a surprise to the market .  for comparison purposes we finally consider the parameter set ; this corresponds to the situation where the market receives a lot of information on the asset value whenever is close to the default boundary . in this scenario defaultis almost predictable " and the model behaves similar to a structural model .this can be seen from figure [ fig : defint - comparison ] where we plot the default intensity for both parameter sets .note that for , the default intensity is close to zero most of the time and very large immediately prior to default ( in fact almost twice as large as in the case where . ) .finally we present the results of a small calibration exercise where was calibrated to five - year cds spreads of lehman brothers using the methodology described in the previous section .the data range over the period september 2006 to september 2008 ( lehman filed for bankruptcy protection on september 15 , 2008 ) .since under full information cds spreads are homogeneous of degree zero in and , we took the default threshold equal to so that the numbers on the -axis can be viewed as ratio of asset over liabilities .it can be seen clearly that prior to default the mass of is concentrated close to the default threshold .   a simulated realisation of the conditional density .note how the mass of concentrates at the default threshold shortly before ., width=510,height=283 ]   of the stock ( dashed line ) and of the stock price ( normal line , label shat ) for ( noisy asset information ) .[ fig : stock - price - c1 - 4-c2 - 0],width=510,height=264 ]   ( dashed ) and for ( straight line).,width=510,height=264 ]  [ fig : defint - comparison ]  .[ fig : calibration - lehman],width=491,height=302 ]this paper has developed a theory of derivative asset analysis for structural credit risk models under incomplete information using stochastic filtering techniques .in particular we managed to derive the dynamics of traded securities which enabled us to study the pricing and the hedging of derivatives .to conclude we briefly mention a couple of financial problems where this theory could prove useful .  to begin with itmight be interesting to study contingent convertible bonds , also known as cocos , in our setup .a coco is a convertible bond that is automatically triggered once the issuing company ( typically a financial institution ) enters into financial distress . at the trigger eventthe bond is either converted into equity or into an immediate cash - payment that is substantially lower than the nominal value of the bond .modelling the trigger mechanism adequately is a crucial part in the analysis of cocos .the cocos that have been issued so far have a so - called accounting trigger based on capital adequacy ratios .it is difficult to include this directly into a formal pricing model ; many pricing approaches therefore model the conversion time as a first passage time of the form for a conversion threshold and a set of monitoring dates .this valuation approach is however difficult to apply in practice , since investors are not able to track the asset value continuously in time , see for instance .our setup is where is not fully observable is well - suited for dealing with this issue in a consistent manner .first results in this direction can be found in chapter 5 of .our framework could also be used to study derivative asset analysis for sovereign bonds .several fairly recent papers have proposed structural models with endogenous default for sovereign credit risk , see for instance  or .roughly speaking , in these models default is given by a first passage time , where the process is a measure of the expected future economic performance of the sovereign and where the threshold process is chosen by the sovereign in an attempt to balance the benefits accruing from lower debt services against the adverse economic implications of a default such as reduced access to capital markets .it is reasonable to assume that and are not fully observable for outside investors , for instance because it is hard to predict the outcome of the sovereign s decision process in detail .hence one is led to a model of the form with asset value and default threshold .the results of the present paper can be used to derive the dynamics of sovereign credit spreads in this setup ; this is important for the pricing of options on sovereign bonds and for risk management purposes .in the next corollary we state the filtering equations for .[ thm : filter - equations ] for \times s^x) ] the directional derivative of in direction , that is } ( t , \pi ) = \frac{d}{ds } { \big \vert}_{s=0 } \widetilde{c}^h ( t , \pi + sv )  \,.\ ] ] suppose that satisfies the regularity conditions of  ( essentially this means that the first and second directional derivative of exists in every point ). then theorem 3.1 of ) gives the following martingale representation for the discounted option price : } ( s , \pi(s))\ , dm_{s , i}^{z } \label{eq : martingale - representation-2 }  - \int_0^{t \wedge \tau } \!\ !\widetilde{c}^h(s , \pi(s))\ , d m^y_s \\ + & \int_0^{t \wedge \tau } \!\ ! \int_{{\mathbb{r}}^+ } \widetilde{c}^h \big ( s , \frac { s_{\kappa y } \big ( \pi(s- ) \varphi(y,\cdot ) \big ) }{ ( \widehat{\varphi ( y)})_{s- } } \big ) - \widetilde{c}^h\big ( s , \pi(s- ) \big ) \ , ( \mu^d - \gamma^{d , { \mathbb{f}}^{\mathbb{m}}})(dy , ds)\,.\end{aligned}\ ] ] in the it formula for spdes is used to determine the integrands with respect to ; the integrands with respect to and with respect to can be determined by elementary arguments . all integrands ( and hence the risk - minimizing hedging strategy ) are functions of the current filter density ; the practical computation of the directional derivative of in the integral with respect to ( and of the other integrands ) can be done with monte carlo .note that we have not established the regularity of required in ; this very technical issue is left for future research .  as a toy example we consider the problem of hedging an option with payoff using the stock as hedging instrument . to simplify the notation we consider the case where is a one - dimensional process .first , we get from proposition [ prop : hedging ] that between dividend dates , that is for \setminus \mathcal{t}^d ] that due to the presence of the indicator in we may replace with in that equation , so that we obtain by the definition of conditional expectations .a similar argument shows that , which gives the equality as claimed .now we turn to statement 2 . for the purposes of the proof we make the dependence of the stopped asset value process on explicit and we write .clearly , on it holds that , . using proposition [ prop : reduction - to - background ] we thus get  converges to zero for so that we concentrate on .the difficulty in estimating this probability is the fact that we have to compare conditional expectations with respect to different filtrations .similarly as in robust filtering , we address this problem using the reference probability approach . to ease the notation we introduce the abbreviations and moreover, we set and we let . the density martingale is defined in analogously , but with instead of . in view of proposition [ prop :reduction - to - background ] and we need to show that for , the key tool for this is the following lemma .fix constants .we have to show that for sufficiently large , we first show that we may assume without loss of generality that and are bounded .to this we choose some constant such that where this is possible since is bounded away from zero on every compact subset of and since for fixed the mapping is bounded by . by definition it holds on that and .moreover , is no larger than hence we assume from now on that and are bounded by .we continue with some useful estimates on .doob s maximal inequality gives similarly we get for ( the cum - dividend asset value ) that of course , similar estimates hold for and for . since on the set , and , it holds that hence we get now note that by assumption .hence can be estimated by in order to complete the proof of the lemma we finally show that the expression converges to zero for , that is converges to zero in and hence also in probability .now note that our previous estimates and imply that the random variables are uniformly bounded in and hence uniformly integrable , so that the claim follows from the lebesgue theorem .the same argument obviously applies to the second term in which proves the lemma .finally we return to the proof of and hence of proposition [ prop : bounded - domain ] .fix constants and choose in such a way that for with it holds that and .choose finally large enough so that for it holds and ; here this is possible by lemma [ lemma : bounded - domain ] .let and note that .by definition of we have for and the estimates and . nowthe mean value theorem from standard calculus gives for and with and the estimate applying this estimate we get for that   in order to show that we use induction over the dividend dates . for , and by the novikov criterion ( recall that is bounded by assumption ) .suppose now that the claim holds for .we get that moreover , so that as well .  in order to show that has -compensator we use the general girsanov theorem ( see for instance , theorem 3.40 ) : a process such that exists for is a -local martingale if and only if is a -local martingale .consider now some bounded predictable function \times{\mathbb{r}}^+ \to { \mathbb{r}} ] .hence we get that and thus the relation as claimed .we start with the case without dividends and we assume that is time - independent .recall from corollary [ corr : cond - density ] that with . using andwe get partial integration shows that the drift term in equals .hence , similarly , we obtain that hence we get , using the it product formula and the fact that  \\ = & \big ( \big ( \pi(t ) , \mathcal{l}_x f\big ) _ { s^x } - \frac{1}{2 } \sigma^2 k^2  \frac{d\pi}{dx}(t , k ) f(k )  \big ) \ , dt + \pi_t(a^\top f ) \ ,+ & \big ( \frac{1}{2 } \sigma^2 k^2 \frac{d\pi}{dx}(t , k ) \,\pi_t f + \sum_{j=1}^{l } ( \pi_t a_j)^2 \pi_t f \big ) \ , dt\ - ( \pi_t f ) ( \pi_t a^\top ) \ , dz_t - \big ( \sum_{j=1}^{l } ( \pi_t a_j ) ( \pi_t a_j f)\big ) \ , dt\,.\end{aligned}\ ] ] rearranging terms and using that gives the claim of the lemma for the case of time - independent and no dividends . for time - dependent have so that we obtain the additional term in the drift of the dynamics of .finally , we consider the case with dividend payments . between dividenddates the dynamics of can be derived by similar arguments as before . to derive the jump of at recall that we have , using the form of ,   and similarly for .now , by assumption [ ass : assets - and - dividends]2 we know that  a.s , so that .hence we get which gives the form of the integral with respect to in .schweizer , m. : 2001 , a guided tour through quadratic hedging approaches , _ in _ e. jouini , j. cvitanic and m. musiela ( eds ) , _ option pricing , interest rates and risk management _ , cambridge university press , cambridge , pp .

Abstract: the paper studies derivative asset analysis in structural credit risk models where the asset value of the firm is not fully observable . it is shown that in order to compute the price dynamics of traded securities one needs to solve a stochastic filtering problem for the asset value . we transform this problem to a filtering problem for a stopped diffusion process and we apply results from the filtering literature to this problem . in this way we obtain an spde - characterization for the filter density . moreover , we characterize the default intensity under incomplete information and we determine the price dynamics of traded securities . armed with these results we study derivative asset analysis in our setup : we explain how the model can be applied to the pricing of options on traded assets and we discuss dynamic hedging and model calibration . the paper closes with a small simulation study .   + rdiger frey lars rsler , dan lu + _ institute of statistics and mathematics , vienna university of economics and business ( wu ) + _  [ [ keywords . ] ] keywords . + + + + + + + + +  structural credit risk models , incomplete information , stochastic filtering , derivative asset analysis for corporate securities


modern methods in data analysis for high energy physics ( hep ) usually rely on learning algorithms such as neural networks .the popularity of the latter has led to many out of the box solutions that are easy to use and widely employed in the physics community ( e.g. the root based package tmva ) .however , it is often ignored that even the most advanced algorithms strongly depend on the quality of the input .if the input is poorly prepared , the performance will suffer as well .this lecture will present some basic means to prepare the input and evaluate the value of a feature .when the input is well prepared good classification results with simple classification schemes can often be achieved , some of which will be introduced here such as bayesian schemes and rule based classification .the advantages of using a simple algorithm is , that the resulting functions are easy to interpret , compared with complex schemes like neural networks or support vector machines .for example results of a rule based classification can be used and interpreted as a normal cut based analysis .this lecture will also present overtraining and performance evaluation as points that are common to all classification tasks .all of these are done with a special regard to the specifics of problems that are encountered in particle physics : outliers in the data caused by noise , training data not readily available and produced by simulations which are not always accurate . for further reading and are recommended .a typical problem at the track level in particle physics where statistical learning schemes are nowadays routinely employed consists of the classification of particles .the input to a classification algorithm is made up of an ordered set of attributes which characterize the event that should be classified .the terms feature and attribute may be used interchangeably .the features in particle classification problems usually consist of measurable quantities like energy loss and particle momentum .in addition features that incorporate prior knowledge about the process are important .this information usually follows from constraints given by the laws of physics .since the processes under consideration are governed by the laws of physics there is usually a lot of background knowledge , i.e. that the features can not have unphysical values .incorporating this kind of knowledge into a classification scheme is very difficult , but often improves a classifier considerably .it can be used to exclude noisy events . an easy way to incorporate prior knowledgeis often to include the output of existing algorithms as a feature so that only corrections have to be learned . in this way experiencealready acquired can be used in a new approach , and combined with state of the art learning algorithms .   after the identification of the set of features that contains information about the class membership , the so called feature vector ( fv ) can be formed by ordering the features .the fvs of the instances of the classification problem exist in the feature space , a vector space of dimension , being the number of features .the output of a classification algorithm is 1 or -1 if the task is a two class problem . in that wayit is a function , that maps the feature space onto . . this function defines a border in feature space .depending on the side of the border the instances are , the classifier assigns a class .the easiest kind of classification is of course linear classification where the borders are straight lines .this is the case of a cut based analysis .the advantage here is that the cuts are easy to interpret .every algorithm that can separate two classes can readily be used for multiple classes by building one classifier for each class . that is done by training the algorithm on the training set . if the algorithm has several parameters which have to be tuned , additionally the so called validation set is used , and the performance is tested on the test set ( see sect [ overtrainingsect ] ) .fig . [ fvlayout ] illustrates , that it is desirable to process the data such that the layout in feature space ( fs ) is reasonable .i.e. the topology in fs is simple .usually it is easy to improve the separability of the data by preprocessing , which often renders the use of a complex algorithm for classification superfluous .the following sections describe several simple actions which should be taken as a first step .an instance can only be classified or used for training of a classification algorithm , if it is an element of the feature space . therefore missing values , that would reduce the dimensionality of the feature vector have to be filled .an intuitive choice is to use values at the borders of the range of values .maintaining the view of a border in feature space that is defined by the algorithm , it is obvious that this is not necessarily appropriate . by choosing extreme values during trainingthe problem occurs , that an algorithm tries to place the border such that this instance is correctly classified .especially if the border is a smooth function this can have many undesirable effects , in particular , since the knowledge of the unphysical nature of the value is available beforehand . during classificationan extreme value can place the feature vector in a part of the feature space with few training examples which makes a correct classification harder .thus it is desirable to place fvs in densely populated part of the fs , such that the other features can determine the class , for example by using the mean value for the missing value .this is obviously the simplest choice .there exist more elaborate ways , e.g. estimating the value from the distribution of the rest of the fv by the expectation - maximization algorithm or taking the mean of the nearest neighbors in fs where the distance is measured by the other attributes .an introduction can be found in .many well - known data classification algorithms require discrete data .this includes trees but also rule finders and ultimately cut based analysis .given data that take on values from a continuous range , the question imposes itself , what the optimal way of discretization is , e.g. by trying different cuts . the straightforward way is to choose equal steps . if the data is not distributed equally in the given range , it is better to use a spacing such that the number of data points in each bin is similar .but since these cuts should be used in the end to distinguish one class of data from another it is advisable to choose the cuts in such a way that it is still possible to decide on the class membership given the discretized attribute .ideally that would mean , that in each bin there are instances of only one class .this might , however , generate too many possible divisions .common modern algorithms thus employ a strategy where the the data is sorted , the possible cut - points are determined , the data is divided at the optimal cut - point given a specific criterion and the discretization is halted if a stopping criterion is fulfilled .possible cut - points are the points between label changes , the optimality criteria can be similar to the feature selection criteria entropy and information gain that will be talked about later in this lecture , because the choice of discretization steps implies the choice of new features .the stopping criterion is often mdl ( minimal description length ) . for details on popular algorithmssee : , .data normalization can be achieved simply by scaling the data to a specific range , e.g. [ -1,1 ] , or fixing variance to one and the mean to zero for each feature .another way is to use the softmax algorithms which scales data close to its mean value almost linearly , but includes a smooth nonlinearity further away .data normalization simplifies the search for optimal parameters during classification .widely different ranges for different features can lead to numerical instabilities during computation and different weights for different features .outliers are a problem often encountered in physics measurements .they can be caused , for example , by electronics noise or mislabeling .many outliers can significantly degrade the performance of classifiers , so treatment of those should be taken seriously . if an algorithm is used which scales the data automatically , the presence of outliers can concentrate the meaningful training examples in a small range of the allowed values .some strategies can be found in , however , it should be kept in mind that outlier treatment also depends on the problem at hand .this section will deal with some basic aspects in feature selection .the first topic that will be introduced is decorrelation for which the motivation is twofold .firstly , the assumption of statistical independence facilitates the classification task at hand .after decorrelation it is possible to consider each feature independently . simple bayesian schemes in principle even require independent features .secondly , strongly correlated features can be combined , which leads to a reduction of the dimensionality of the feature space . having this dimensionality as low as possible is very important , since the volume of the feature space grows exponentially with each dimension , which in turn means that the density of training examples reduces exponentially .of course this makes discrimination harder .in addition the computing time of algorithms grows strongly with the number of dimensions .this is called the curse of dimensionality . in this part of the lectureit will be shown how after decorrelation the variance of a feature can be used as a measure for the importance .other measures are the information gain and significance of a feature by which will also be introduced .the problem of reducing the dimensionality of the feature vector by making a minimal error is of interest for many applications .consequently , solutions for this problem have been developed .the most widely known is the principal component analysis or karhunen - loeve - transformation ( klt ) . the idea ofthe klt is that for the removal of a dimension with the minimal error , it is necessary to remove the directions in which the data has the minimal variance .if only linear transformations of the fs are allowed , the solution is unique .an example is shown in the data distribution in fig .[ kltpic ] . therethe information is mainly contained in one direction . describing a two dimensional vector only with the component in the direction of the maximum variance leads to the smallest expected error which can be achieved with a linear transformation :  a feature vector is denoted by $ ] , the correlation matrix of the feature vectors by .then the unitary transformation is constructed such that the feature vector after application of the transformation becomes and the correlation matrix is given by .thus the feature vectors are now uncorrelated .  assuming that the eigenvalues are ordered according to their size, the klt has some desirable features ( proof given in ) :  * consists of eigenvectors of , * it can be shown that the expected error is , where the first dimensions , corresponding to the variances and eigenvalues of  are omitted , * the projection onto a subspace spanned by the first eigenvectors is optimal , * the sum of the eigenvalues , which are the variances in the direction of the corresponding eigenvector , can be used as a measure of the information contained in this feature ,  and some undesirable :  * it is data dependent , since the covariance matrix has to be estimated from data . also , there has to be enough data to be able to invert the matrix , * there exists no fast algorithm , * it is not optimized for class separability , this is incorporated into fishers linear discriminant , which will not be discussed here .since the klt is not optimized for class separability it is ad hoc problematic to identify the direction of maximal variance with the direction of maximal importance .possibly the direction for optimal separability is one of small variance . however , even if the full dimensionality is kept , the decorrelation is already of importance for many tasks .for example maximum likelihood methods assume that the data is not correlated .but the klt is only one transformation that can make the feature space more amiable for classification .depending on the application , other representations may be useful .popular examples are the fourier transform or the haar transform which leads to robustness against transformations of the data by integration over these .the best feature for classification is obviously one that already determines the class membership of the example .then the information gained by knowing the correct classification equals zero , since the outcome is already known , whereas without knowing the value of the feature the information gain was big , since the outcome was completely uncertain .this is an extreme example , and in most of the cases finding a feature like this is impossible but the generalization demonstrates that the better the feature is , the more the outcome of the classification is fixed .the actual classification will not be surprising , if the probability of a specific outcome is already 99% after the knowledge of a certain attribute .this means that the quality of the attribute can be measured in a way by the uncertainty of the outcome after knowing the attribute compared to the one before knowing it . to put this in a less colloquial language , the significance of an attribute for classification can be measured by computing the information gained from the knowledge of the value of the attribute , the so called information gain .in information theory the information contained in the outcome of a random draw is defined as if there are possible outcomes , each with a probability . the unit of information is a bit . during classification a feature vectoris randomly drawn and the class membership is to be determined .assuming there are only two different classes with equal probability ( ) the information contained in the outcome is thus this could be for example the toss of a fair coin .the information contained in the outcome is 1 bit , as expected .if the coin is not fair , and it is known for example that the outcome will be in 99% of the cases heads , the information contained in the outcome will be less and the information gain in bit of this knowledge will be .[ difffsdivides ] illustrates how the knowledge of one discrete attribute devides the feature space in different partitions in a classification problem .each of the partitions has a different probability for the two classes . before , without knowing anything , the information contained in the outcome of the classification of one of the examples was 1 bit , assuming that there are the same number of examples for both classes . with the prior knowledge of a feature , the partition of the feature space in which the example that is to be classified lies is known .depending on the distribution of classes in this partition the classification provides significantly less information . if the partition contains only one class , the information gained from knowing the class label in addition to the partition is even zero .  to compute the information left after knowing in which partition the example lies ( by knowing the attribute a ), the information still needed to determine the classification in each partition weighted by relative number of examples in this partition can be computed : for a set containing positive and negative examples the computation of the information is straightforward : thus the information gain of is .information gain is a popular measure for the significance of an attribute for classification , the higher the value the better the attribute .given a set of possible attributes , meaningless attributes can be excluded by computing the information gain for each .this value can also be used to order the attributes in the nodes of decision trees .another ansatz for the determination of the significance of an attribute is the question if the probability distribution function ( pdf ) changes significantly in one of the partitions of fig .[ difffsdivides ] . for the difference between the expected number of positive and negative examplesgiven the pdf can be computed with the actual number of positive and negative examples .this is done by computing the between the null hypothesis , the feature is irrelevant and the pdf will be unchanged , and the hypothesis that the feature is relevant and the pdf is changed . if the impact of the attribute on the pdf is big , the will also be big .thus the significance of the attribute can be deduced from the that can be computed in a straightforward way if the attribute splits the set of examples l - fold .first the expectation values for the number of positive and negative examples in set ( ) is computed : if the attribute is irrelevant the deviation should follow a distribution : the significance of the attribute then follows from the actual .this section introduces some basic classification schemes which follow from basic statistics .the problem to be solved is illustrated in fig .[ diffclassifications ] . given the feature space with the training examples the task is to find the partition of the space that minimizes the classification error .usually the more general concept of classification cost is used .the definition of an appropriate cost function then allows to weight different errors by their importance . in basic classificationusually the cost is defined as the fraction of misclassified examples : where is the classification function giving the assigned classlabel or for the test examples .this is not always appropriate for physics problems , since here the goal is often to maximize a figure of merit . if is taken as the signal and the background this has for example the form .more problems with the use of basic cost function occur if the number of training examples in each class is not weighted correctly . therefore statistical learning toolboxes usually give the user the choice of the cost function for example in the form of a matrix assigning a different cost for each possible confusion .   often , when classifying physics data , it is a problem to find appropriate training examples .either the examples are from simulated ( mc ) data , or from real data , but then the labeling is most of the time not 100% perfect . in both cases , it is advisable to study the effects of this on the performance of the classifier . if it performs good on the training data but gets worse if there are some training examples with the wrong label or are a little bit shifted in feature space , one can not expect a good performance on real data. most classifiers also have parameters that control the generalizing power .this is done to avoid overtraining ( see section [ overtrainingsect ] ) .for example rule based classifiers or decision trees have a pruning stage .the algorithm should be used in a way that maximizes its ability for generalization .the most basic , and optimal , classifier tries to maximize the probability for correct classification .it thus chooses the class according to this is the so called maximum a posteriori ( map ) classifier .given evidence , it gives the probability of each possible classification .however , to evaluate this expression the conditional probability distribution has to be known for each possible feature vector which is usually impossible .therefore the formulation is recast with the help of bayes theorem .this says it is possible to express the a posteriori probability in terms of the probability of the values for the feature vectors given the class and the a priori probabilities for each class and the probability for the specific feature vector as : the standard example for this theorem , given in the following , comes from medicine .given certain symptoms , the map classifier gives the probability for an illness .after the reformulation with the help of bayes theorem it can be seen that this depends heavily on the a priori probability for having the illness .given a rare illness that only occurs once in ten thousand cases , its probability can be computed after a positive test .given that the test is correct 98 % of the time , so for a rare illness approximately for 2 % percent of all cases it gives a positive result , the probability for the illness even after positive test is , .  by using the bayes theorem surprisingly good resultscan be achieved on many datasets .the classification question is reduced to the question     for actual classification it is necessary to make certain assumptions regarding the occurring probability distributions .the natural choice for the conditional pdfs is that the features given a class are normally distributed around the mean value for that feature given the class .[ twogaussians ] shows the one - dimensional case . herethe classification is done by simply evaluating each pdf at the value of the feature and choosing the class that has the higher probability there .the a priori probability of the classes can in principle be computed by sampling .but it is also possible to use the assumption of equal probabilities for all classes and uncorrelated features , which can be achieved by the application of the klt , each distributed with the same variance , if this is reasonable for a given problem .the resulting classifier is called maximum likelihood estimator ( mle ) . under the assumption of uncorrelated entries in the feature vector ,the probability is just given by the product of the probabilities of the entries , which have a gaussian distribution around the mean value of the given class , thus : so the classification is reduced to the computation of the euclidean distance of the feature vector to the mean of the class . in a two dimensional feature space the lines of equal probability are circles . if the covariance matrix is known and if it is not equal unity it is possible to use a different metric , the mahalanobis distance .this incorporates the covariance matrix by using where is the expectation value and is the mean of .the distance induced by this is the so called mahalanobis distance , the distance between two points , in a vector space can thus be computed by and with this notation as illustrated in figure [ figwitheuklidmahaldist ] , the lines of equal probability in a two dimensional space are now elliptic .     in this sectionautomated cut finding with the ripper algorithm is introduced , which was not included in the original lecture .many statistical learning tools also incorporate rule based classification , which can easily translated into a cut based approach .the advantages are of course that simple rules can be interpreted more easily , interpretation of the data is facilitated and most people have more confidence in cuts than in black box algorithms like neural network .the ripper algorithm was introduced by william w. cohen . and has been known for a long time in the machine learning community . a similar approach is realized by decision trees which are not discussed here . in general ,rule based algorithm need discretized attributes .thus they have to be combined with one of the strategies for discretization .examples were covered in section [ sectionwithdiscretization ] .the ripper algorithm builds a ruleset for classification .the ruleset is initialized with the empty set in the beginning , .the goal of the algorithm is to grow rules by adding conditions ( connected by the logical and ) so each rule covers a convex space in feature space .an example is positively classified if any rule matches . thereafter comes the building stage .this consists of repeatedly executing the following steps until a stopping criterion is reached :  * grow phase - grow one rule by greedily adding new conditions according to the highest information gain of the attribute until the rule is perfect .i.e. it covers all positive examples and no negative . *prune phase - tune the rules to the training set , since they are 100% perfect .this is the problem of overtraining ( see section [ overtrainingsect ] ) .it is thus necessary to prune the set .all finite sequences of conditions are considered , if a sequence covers positive and negative examples , and the value is below a certain threshold , it is pruned .this makes sure that the rules are not customized to single examples .the threshold can be chosen according to variance in the data .the algorithm stops if the rule lengths are too different or all positive examples are covered or the error rate is too high . in the end ,the initial ruleset is optimized .the set of rules obtained in this way can then be easily translated in cuts .experience shows that taking care during the discretization step is very important , because measurements are not spread evenly over the allowed range of values and if the discretization is too coarse in regions where they are dense and small differences matter , the best algorithm can not achieve a good performance .the performance of an algorithm on a training set is not necessarily a good indicator of its performance on real data .a perfect performance often comes from a specialization on the given training data .this leads to a loss of generality and thus most of the time to a poor performance on real data .this problem is known as  overtraining  and is always an issue in machine learning and consequently all algorithms have measures to control the generalization capability .support vector machines ( svms ) have a solid theoretic background by which , in principle , the generalization ability can be computed .other algorithms have more ad - hoc parameters , trees and rule finders use pruning , neural networks often use weight decay .the principle is that of occam s razor .this states that it is always advisable to use the simplest theory that describes the data . especially with little training data or with data that is uncertain , as it is often the case with physics data ,a simple model should be used .this follows also from a kind of bayesian reasoning : a complex theory , a theory that takes a lot of bits to describe , is more unlikely , so more evidence is needed to support it .see for example fig .[ figwithoverfit ] . herethe training performance of a linear function is worse , but looking at the data it seems to be the correct description , especially if expected scattering of the data is considered .there exists an elaborate theory that connects the predictive power of a function and the training error to the expected error , the vapnik - chervonenkis theory , and svms are build according to this principle , but to explain this would be too much detail .it is enough to keep this problem in mind , especially in physics , and evaluate the performance in a way that includes the generalization ability .for this an independent test set is needed , that had nothing to do with the training of the algorithm . if the algorithm has also meta parameters , which have to be determined in an optimal way , in principle three sets are necessary .one for training the algorithm with a specific set of parameters , one to determine the optimal parameters , the so called validation set , and one to evaluate the whole algorithm with the chosen parameters .otherwise there is also the risk that parameters are tuned to the training set .it is also advisable to incorporate a bit of scattering in the training examples to see how the algorithm performs . to compute the expected error , it is common practice to either just split the data 3 ways or do the so called  n - fold cross - validation  . herethe data is split in folds , then the training is performed on sets and the remaining set is used for testing .this is done * n * times and the expected error is computed as the mean error .it can be shown that the higher , the better the estimation is .the most extreme case being the leave - one - out ( loo ) error , where the sets only have one element .getting to know existing tools is very important for two reasons .firstly , of course , it is not necessary to invent anything that has already been implemented and there are a lot of existing toolboxes which can be used .the second reason is , that toolboxes provide the opportunity to try many different strategies which enables the user to get a feeling what might work for a given problem .this is one of the most important things that one has to do while implementing a successful strategy for solving a classification task .therefore two toolboxes are presented in this section that make it possible , even for a rather unexperienced user , to try out different algorithms for data normalization , preparation and classification , the weka toolbox and the toolkit for multivariate analysis ( tmva : ) .the weka tool has the advantage of implementing almost every popular algorithm and combining this in a unified graphical user interface , providing a workplace where different algorithms can be combined and tested .in particular , all algorithms presented so far are implemented , so that it is possible to try them right away . on the other hand tmva developed using the root software ,is particularly aimed at analyzing particle physics data .for example it allows the definition of the signal significance as the figure of merit to be optimized and is fully integrated in the root environment . as of this writting ,tmva is still under heavy development but stable and usable .it is part of root , but since its development cycles are currently faster , it should be downloaded separately .easy to use graphical user interfaces allow to choose optimal working points , inspect the data and compare the performance of different classifiers .it contains a scripting as well as a c++ interface .there is also a complete users guide available .10 http://tmva.sourceforge.net r. duda , p. hart , d. stork , _ pattern classification _( wiley , new york , ny , 2000 ) s. theodoridis and k. koutroumbas , _ pattern recognition _( academic press , 1999 ) z. ghahramani and m. i. jordan , supervised learning from incomplete data via an em approach , _ advances in neural information processing systems _ , * 6 * , ( 1994 ) 120 .fayyad and k.b .irani , multi interval discretization of continuous - valued attributes for classification learning , proc . of the 13th int .joint conf . on artificial intelligence , ( morgan kaufmann 1993 ) , p. 1022 .igor kononenko , the minimum description length based decision tree pruning , proc . of the 5th pacific rim international conf . on artificial intelligence , topics in artificial intelligence , ( springer - verlag , london 1998 ) lecture notes in computer science , vol .1531 , p. 228 . p.j. huber , _ robust statistics _ ( wiley , new york , ny , 1981 ) . r.a .fisher , the use of multiple measurements in taxonomic problems , _ annals of eugenics _ ( 1936 ) , vol . 7 , part ii , p. 179 .william w. cohen , fast effective rule induction , proc .twelfth int .conf . on machine learning , 1995 ( morgan kaufman 1995 ) , p. 115 . b. schlkopf and a.j .smola , _ learning with kernels _ , ( mit press , cambridge , ma , 2002 ) .this report , chapter : support vector machines o. chapelle , v. vapnik , o. bousquet and s. mukherjee . choosing multiple parameters for support vector machines , _machine learning _ ,* 46 * ( 2002 ) , 1 - 3 , 131 .ian h. witten and eibe frank ( 2005 ) _ data mining : practical machine learning tools and techniques , 2nd edition _ ( morgan kaufmann , san francisco , 2005 ) .

Abstract: this document introduces basics in data preparation , feature selection and learning basics for high energy physics tasks . the emphasis is on feature selection by principal component analysis , information gain and significance measures for features . as examples for basic statistical learning algorithms , the maximum a posteriori and maximum likelihood classifiers are shown . furthermore , a simple rule based classification as a means for automated cut finding is introduced . finally two toolboxes for the application of statistical learning techniques are introduced .


one of the most fundamental tasks in quantum mechanics is that of quantum state tomography , i.e. , reliably reconstructing an unknown quantum state from measurements . specifically in the context of quantum information processing in most experiments one has to eventually show what state had actually been prepared . yet, surprisingly little attention has so far been devoted to the observation that standard methods of quantum state tomography scale very badly with the system size .only quite recently , novel more efficient methods have been introduced which solve this problem in a more favorable way in the number of measurement settings that need to be performed .this development is more timely than ever , given that the experimental progress with controlled quantum systems such as trapped ions is so rapid that traditional methods of state reconstruction will fail : e.g. , ions can already be controlled in their quantum state .hence , further experimental progress appears severely challenged as long as ideas of reconstruction can not keep up .such new methods are based on ideas of quantum compressed sensing  inspired by recent work on low - rank matrix completion  or on ideas of approximating unknown quantum states with matrix - product states . indeed ,using methods of quantum compressed sensing , one can reduce the number of measurement settings from in standard methods to for a quantum system with hilbert space dimension , if the state is of rank .this is efficient in the sense that the number of measurements required is only slightly greater ( by an factor ) than the number of degrees of freedom in the unknown state .these ideas have so far been tailored to the situation where observables are taken from an orthonormal operator basis , which is not always the natural situation at hand . in this paper, we introduce a theory of state reconstruction based on quantum compressed sensing that allows for continuous families of measurements , referred to as _ tight frames _ , which can be thought of over - complete , non - orthogonal generalization of operator bases .these settings are particularly important in the context of continuous - variables , which are notably used to describe quantum optical systems beyond the single - photon regime .these have drawn a considerable amount of research , both experimentally and theoretically , due to very desired features such as easy preparation and highly efficient detection .note that when talking about a measurement , we always mean the estimation of an expectation value of an observable for which , of course , several repetition of some experimental procedure are necessary . in this paperwe are mainly concerned with the number of distinct observables or _measurement settings _ that are needed for tomography .  in this work ,we make significant progress towards a full theory of efficient state reconstruction via compressed sensing :  1 .we introduce _ new incoherence properties _ for tight frames , that are sufficient to ensure efficient compressed sensing for low - rank states .this uses an extension of the `` golfing '' proof technique of .we give examples of tight frames that satisfy these properties .in addition , we show that , if one only wishes to reconstruct `` typical '' or `` generic '' low - rank states , there is a much larger class of tight frames that also lead to efficient compressed sensing .we also describe a way to _ certify _ a successful reconstruction of the state , making our protocol unconditional and heralded . in this way, one does not need to make any a priori assumptions on the unknown state .our method uses convex duality , and is different from other approaches to certification that focus mainly on pure states .also , we discuss the _ robustness _ of the procedure under decoherence , imperfect measurements , and statistical noise . we show that , as long as all those effects are small , it is possible to certify that the reconstructed state is close to the true state .we show that , using an incoherent tight frame , and a slightly larger number of measurements , one can achieve _universal _ state reconstruction : a single fixed set of measurements can simultaneously distinguish among all possible low - rank states .this is a qualitatively stronger claim than those shown above , and it is obtained using a different technique , based on the `` restricted isometry property '' ( rip ) .this implies strong error bounds , showing that our procedure for state reconstruction is robust to statistical noise , and that it works even when the true state is full - rank with rapidly decaying eigenvalues ( in which case our procedure returns a low - rank approximation to the true state ) .we show how our theory can be applied in realistic experimental scenarios , involving pointwise measurements of the wigner function , and homodyne detection .we demonstrate _ numerically _ that compressed sensing outperforms the naive approach to tomography not only in the asymptotic limit of large systems but also for system sizes commonly accessible in present day experiments .this article is organized as follows : we start by introducing quantum compressed sensing in the general setting described by tight frames in section 2 . after discussing a suitable notion of efficiency ,we show in section 3 that efficient compressed sensing is possible if the tight frame fulfills certain incoherence properties .section 4 is devoted to certified compressed sensing .we discuss how to certify the success of the reconstruction without prior assumptions on the tight frame , both in the ideal case and under the effects of errors . in section 5we show universal state reconstruction and error bounds . in section 6 , we investigate applications of the formalism to two common classes of quantum optical experiments ; and in section 7 , numerical data , showing the efficiency for small systems , is presented .consider a quantum system with hilbert space dimension . in most cases of interest, is very large , but the states one wants to reconstruct are approximately low - rank , that is , they are well - approximated by density matrices having rank .( pure states correspond to the case where . )when dealing with continuous - variable systems , we will truncate the infinite - dimensional hilbert space and choose to be some large but finite cutoff .this is unavoidable , if one wants to do tomography as one can not reconstruct a state that contains an infinite number of completely independent parameters .however , in most experimentally relevant situations , e.g. , continuous - variable light modes with finite mean energy , all states can be arbitrarily well approximated by finite - dimensional ones .we will elaborate on this claim when discussing other sources of errors such as decoherence or imperfect measurements .compressed sensing contains two key ideas .first , rather than measuring all degrees of freedom , it is sufficient to measure a randomly chosen subset of about degrees of freedom , provided these degrees of freedom satisfy certain _incoherence properties_. secondly , one can reconstruct the state using an efficient algorithm .the obvious approach of searching for the lowest - rank state compatible with the measurement results leads to a computationally intractable problem ( generally np - hard ) .instead , one can perform a _ convex relaxation _ , and minimize instead of the rank . here stands for the schatten -norm : , , and are respectively the trace norm , frobenius norm , and spectral ( or operator ) norm .let us denote the measured observables , i.e. hermitian matrices , by , and suppose that we estimate their expectation values ( by measuring many copies of the unknown state ) .knowing these expectation values ( for an unknown state ) is equivalent to knowing the value of the _ sampling operator _ , where we define where is the hilbert - schmidt scalar product . in all of our compressed sensing schemes , will be chosen independently at random from some distribution .the sampling operator is a linear super - operator on the -dimensional real vector space of hermitian matrices , or operators , .such super - operators will always be denoted by capital script letters .sometimes we will use the notation , multiplying the `` matrix '' by the `` vector '' ; this means the same thing as .let be the unknown state .in the ideal case , with perfect measurements and no statistical noise , we measure exactly .then the procedure to reconstruct can be written as note that a quantum state is a hermitian matrix with the additional properties and .however , the reconstruction procedure ( [ eq : reconstruction ] ) does not make use of this property and is , therefore , also applicable in more general settings , e.g. matrix completion .this problem can be stated as a semi - definite program ( sdp ) and , therefore , solved efficiently with many well - developed tools .  in the case of noisy data , we measure approximately , that is , we measure some such that , for some norm and tolerance that are chosen depending on the kind of noise that is expected . the constraint in ( [ eq : reconstruction ] ) can then be replaced by , which implies .we remark that equation ( [ eq : reconstruction ] ) is the key to certifying our estimate for .notice that if the solution of ( [ eq : reconstruction ] ) is unique and fulfills , then it must be the case that .we will show later on how one can test the uniqueness of the solution , without assuming anything about .( this can be adapted to work with noisy data , without assuming anything about the noise . )   when we talk about a compressed sensing scheme , we mean any protocol based on the reconstruction procedure ( [ eq : reconstruction ] ) , with _ some _ choice of measurements described by the sampling operator ( [ eq : rdef ] ) . in refs . , the observables were required to be chosen uniformly at random from an operator basis .we substantially generalize these techniques , using the notion of a tight frame , which naturally captures many useful quantum measurements :  let be a probability measure on some set , and for every [ def : tfcondition ] , let be an observable , i.e. , a hermitian operator , and let be the ( unnormalized ) orthogonal projector which acts as .we say that is a tight frame if   this can also be written as where is drawn according to . because we deal with randomly drawn operators very often, will usually denote a random element of that has distribution .note that we do not require that for all as it will be convenient in many applications .however , we do require a weaker normalization condition : = 1 |\alpha|>0 $ ) } \\  & \leq e^{-|\alpha|^2/2 } ( 1+|\alpha|^2)^{n+1 } |\alpha|^{-2 }  = e^{-|\alpha|^2/2 } ( 1+|\alpha|^2)^n ( 1+|\alpha|^{-2 } ) .\end{split}\ ] ] then we can bound the scaled truncated operator as follows : ( 1+|\alpha|^{-2 } ) .\end{split}\ ] ] let be the quantity inside the exponential ; we will upper - bound it . note the following identity , for any : ( by approximating to first order at the point ) set and , then we have then using the fact that , we get plugging this into ( [ eqn - onb - step ] ) , we get ( 1+|\alpha|^{-2}).\ ] ] using the fact that and , we have that ( 1+|\alpha|^{-2 } )  \leq \sqrt{2}\sigma \exp(-1 ) \tfrac{17}{16 }  < \sqrt{2}\sigma.\ ] ] since the operator norm is upper - bounded by the -norm , this implies ( [ eqn - onb ] ) .

Abstract: we significantly extend recently developed methods to faithfully reconstruct unknown quantum states that are approximately low - rank , using only a few measurement settings . our new method is general enough to allow for measurements from a continuous family , and is also applicable to continuous - variable states . as a technical result , this work generalizes quantum compressed sensing to the situation where the measured observables are taken from a so - called tight frame ( rather than an orthonormal basis ) hence covering most realistic measurement scenarios . as an application , we discuss the reconstruction of quantum states of light from homodyne detection and other types of measurements , and we present simulations that show the advantage of the proposed compressed sensing technique over present methods . finally , we introduce a method to construct a certificate which guarantees the success of the reconstruction with no assumption on the state , and we show how slightly more measurements give rise to `` universal '' state reconstruction that is highly robust to noise .