the success of prominent contemporary cryptosystems is attributed to the degree of difficulty in computing integer factorizations [ 1 ] and discrete logarithms [ 2 , 3 ] , respectively .this paper describes a novel strategy to encode covert information via unitary projections into the _ null spaces _ of the eigenstructures of a hierarchy of host statistical distributions ( multiple _ pdf _ s ) . the multiple _ pdf _ s are inferred from an incomplete set of constraints ( physical observable s ) using the maximum entropy ( maxent ) principle [ 4 ] . here, a multiple _ pdf _ model is defined as a hierarchial ensemble of _ pdf _s . the index is defined as the _ hierarchy index_. this paper defines the host _ pdf _s to deviate from the equilibrium state , with increasing values of .the case of incomplete constraints corresponds to scenarios where the number of constraints ( physical observable s ) is less than the dimension of the distribution . in a discretesetting , the maxent lagrangian for a single host _ is [ 4 ] :  , where the lagrange multiplier ( lm ) corresponds to the _ pdf _ normalization condition .the lm s correspond to physical constraints of the form . here, is an operator , and , are the incomplete constraints .this paper employs geometric moment constraints , i.e. .solution of ( 1 ) yields : ;n = 1, ... ,n , \\  and , \\  e^{\lambda _o^\nu  }  = \sum\limits_{n = 1}^n { e^ { - \lambda _ i^\nu  x_n^i } } ; i = 1, ... ,m . \\\end{array}\ ] ] solution of ( 2 ) for a given input set of lm s , is referred to as the _ forward maxent problem_. inference of _ pdf _ s and the concomitant lm s , given an input set of , is referred to as the _ inverse maxent problem_.  statistical distributions inferred from incomplete constraints possess eigenstructures that are ill - conditioned .these demonstrate extreme sensitivity to perturbations .this sensitivity is exploited to formulate a principled framework to encrypt / decrypt code .the _ null spaces _ of these eigenstructures constitute an `` invisible reservoir '' into which covert information may be projected .the projection is achieved without altering the host statistical distributions .the advantage of statistical encryption is that the dimension of the code that can be encrypted increases with the size of the statistical distribution .the projection of code into a host statistical distribution is also a promising candidate to implement steganography [ 5 ] , and , related disciplines in information hiding [ 6 ] .a recent study has treated the statistical encryption / decryption of covert information , using the fisher information as the measure of uncertainty [ 7 ] .qualitative distinctions vis - - vis an equivalent maxent formulation [ 8 ] have been established .the encoding strategies in [ 7 ] and [ 8 ] are _ independent _ of the _ pdf _ of the host statistical distribution .encoding covert information into multiple _ pdf _ s allows code of dimension greater than that of any single host _pdf _ to be selectively encrypted into the multiple _ pdf _ s .an example of this is the encryption of multi - dimensional code defined in matrix form , such as image data . in this case , each dimension of the code ( column / row vector of the matrix ) may be selectively encoded into individual components ( single host _ pdf _s ) of a multiple host _ pdf _ model .each single host _pdf _ is taken to be one dimensional in the _continuum_.  conversely , the dimension of each host in a multiple _ pdf _ model may be chosen by the designer to be less than the dimension of the code .this is tantamount to effecting a tradeoff between the dimension of a single host _ pdf _ , and , the number of hosts comprising a multiple _ pdf _ model . _the success of employing multiple pdf s to enhance the security of the covert information , is critically contingent upon the encryption / decryption strategy being dependent upon the pdf s of the statistical hosts_. this feature permits the _ pdf _ dependent statistical encryption / decryption strategy to possess immense qualitative flexibility , as compared with _pdf _ independent models [ 7 , 8 ] .numerical simulations demonstrate impressive quantitative performance in securing covert information .consider constraints . in a discretesetting , these are expectation values of a random variable : evoking the dirac _ bra - ket _ notation [ 9 ] , the _ pdf _ which is a column vector ( _ ket _ ) , where is the standard basis in , is expressed as . defining the column vector of observable s as with components , and , an operator given by . defining vectors as the expansion , where is a basis vector in , ( 3 ) may be expressed simply as :   the physical significance of the constraint operator in ( 4 ) is as follows .inference of the _ pdf _ from physical observable s employing ( 2 ) is achieved by specifying .setting , , the constitute the elements of the rows and columns of the operator .the unity element in enforces the normalization of .consequently , .  _the operator is independent of the host pdf_. this qualitative deficiency may be rectified by defining : here , is a row vector ( _ bra _ ) , and , is a constant parameter introduced to adjust the condition number of , and hence its sensitivity to perturbations . in ( 5 ) , dependence upon the host _ pdf _ is `` injected '' into the operator by the incorporation of .specifically , each element of the _ ket _ is defined by .re - defining ( 4 ) in terms of ( 5 ) yields : , where signifies the expectation evaluated at the _ hierarchy index _ .expanding , and evoking the _ pdf _ normalization : , yields :   the operator ill - conditioned and rectangular .thus , ( 6 ) becomes : where , is the pseudo - inverse [ 10 ] of , and lies in .all necessary data dependent information resides in .  the _ null space _term in ( 7 ) is of particular importance since the code is encoded into it via unitary projections . here, is explicitly data independent signifies the _ null space _ of an ill - conditioned operator , unless explicitly specified as being the routine to calculate the normalized basis of an ill - conditioned operator ( eg . section 3.1 ) . ] .however , it is critically dependent on the solution methodology employed to solve ( 7 ) . to define the unitary projections of the embedded code ,the operator is introduced . here, is the conjugate transpose of . for real matrices , , where is the transpose of .it is legitimate to project the covert information into instead of [ 7 ] .the loss of information " caused by floating point errors could make the evaluation of prohibitively unstable for many applications .specifically , this operation squares the condition number , resulting in the large singular values being increased and the small singular values decreased .this instability is used to the advantage of the designer to increase the security of the covert information .  assuming the availability of and , the normalized eigenvectors corresponding to the eigenvalues in the _ null space _ of having value zero ( _ zero eigenvalues _ ) is : .the are hereafter referred to as the basis of .the unitary decryption and encryption operators operators for each are : respectively .note that , where is the identity operator .  given a code to be encrypted in a host _ pdf _ having _ hierarchy index _ , the components are given by .the _ pdf _ of the embedded code is :   the total _pdf _ comprising the host _ pdf _ and the _ pdf _ of the code is : since , . in the decryption stage , the reconstructed host _pdf _ s : , are first obtained .the _ pdf _ s of the embedded code is recovered from :   the encrypted code is recovered by the operation :   the above theory does not , by itself , constitute the strategy to encrypt / decrypt code .this is achieved in two manners , i.e , _ symmetric _ and _ asymmetric _ cryptographic strategies [ 11,12 ] . before proceeding further , the concept of a key in the encryption of covert information is briefly explained .the distribution of keys is an issue of primary concern in cryptography and allied disciplines .a key may be a program , a number , or a string of numbers that enables the legitimate recipient of the message ( decrypter ) to access the covert information . in cryptography ,a secret , shared , or private key is an encryption / decryption key known only to the entities that exchange secret messages .  in traditional secret key cryptography , a key would be shared by the communicators so that each could encrypt and decrypt messages .the risk in this system is that if either party loses the key or it is stolen , the system is broken .secret key cryptography is also susceptible to a number of malicious attacks , the most common being the _ plaintext attack _ [ 11,12 ] . by definition ,plaintext attack _ is one where the prior messages have intercepted and decrypted in order to decrypt other messages .a more recent alternative is to use a combination of public and private keys . in this system, a public key is used together with a secret key .the rsa protocol [ 1 ] is a prominent example of a public key infrastructure ( pki ) .a pki often employs a _key ring strategy_. specifically , one key is kept secret while the others are made public .pki is the preferred approach on the internet .the secret key system is sometimes known as _ symmetric _cryptography and the public key system as _ asymmetric _ cryptography .  in this model , an operator is formed by perturbing select elements of by . in _cryptography , only a single element of is perturbed .the security of the code may be ensured by adopting an _asymmetric _ cryptographic strategy . here, more than one element of is perturbed .each , a threshold .the extreme sensitivity to perturbations of causes the eigenstructure of to substantially differ from that of , this assertion is valid even for infinitesimal perturbations . to distinguish operations involving the ill - conditioned operators , the following change of notation is effected in ( 9)-(12 ) : , , , , , and , .the are hereafter referred to as the basis of .determination of the threshold is a vital task in defining the cryptographic keys in this model .this is accomplished by the designer ( encrypter ) who performs a simultaneous encryption / decryption without effecting perturbations to the operator . specifically , using ( 2 ), the host _ pdf _ s are inferred by solving an _ inverse maxent problem_. the code having dimension is formed .the designer implements ( 9)-(12 ) for each _ hierarchy index _ . the threshold for the cryptographic key / keys is : .the process of encryption occurs after the host _ pdf _ s have been inferred from incomplete constraints for _ multiple pdf _ s .this procedure is detailed in section 4 .the terminology in cryptography and allied disciplines refers to two communicating parties as alice and bob , and , an eavesdropper as eve . in this study, the author performs the role of both alice and bob by implementing the encryption on an ibm rs-6000 workstation cluster and , the decryption on an ibm thinkpad running v 7.01 .  herein , the implementation of the encryption / decryption strategy by effecting perturbations to the operator is described .the procedures are to be implemented for each _ hierarchy index _ .this section presents the implementation of the encryption / decryption strategies in point form , for the sake of clarity and brevity . the host _ pdf _ is inferred from incomplete constraints by solving ( 2 ) as an _ inverse maxentproblem_. the constraint operators and are evaluated formed , for an _ a - priori _ specified value of the parameter , from ( 5 ) .the operator is formed by perturbing one or more elements of by , the cryptographic key / keys . the basis , are evaluated by operating on with the routine that employs svd , or with an equivalent routine . the code is generated , and , is encoded into the _ null space _ of using ( 9 ) . the total _ pdf _ is obtained using ( 10 ) .the statistical encryption model provides two separate manners in which information may be transferred from the encrypter to the decrypter , via a _public channel_. the first mode is to transmit the constraint operators and the total _ pdf _s .an alternate mode is to transmit the lm s obtained on solving the _ inverse maxent problem_(section 3.1 ) , and , the total _ . owing to the large dimensions of the constraint operators , the latter transmission strategy is more attractive .the values of parameters for each _ hierarchy index _ , and , the cryptography key / keys are transmitted through a _secure / covert channel_. the key / keys are labeled in order to identify the elements of the operator that are perturbed . in the case of _ asymmetric _ cryptography , some of the keys may be declared public , while keeping the remainder private . the legitimate receiver recovers the key / keys and the parameter from the _ covert channel_. the operators , , and , are constructed . the host _ pdf _ may be recovered in two distinct manners , depending upon the transmission strategy employed .note that both methods to reconstruct the host _ pdf _ require the total _ pdf _ to be provided by the encrypter .first , the scaled incomplete constraints , defined in ( 6 ) , are obtained by solving . here , is a basis vector in .this procedure is possible because .thus , .the host _ pdf _ are then computed for each _ hierarchy index _ by solving the _ inverse maxent problem _ , using the re - scaled set of incomplete constraints .alternatively , the host _ pdf _ may be obtained by solving ( 2 ) as a _forward maxent problem _ , given the values of the lm s obtained from the _ inverse maxent problem _ ( section 3.1 ) . both methods allow the reconstructed host _ pdf _s to be obtained with a high degree of precision . the reconstructed code _ pdf _ is recovered using ( 11 ) . the reconstructed code is obtained using ( 12 ) .it is important to note that the success of the encryption / decryption strategy is critically dependent upon the exact compatibility of software available to the encrypter and decrypter .of special importance is the compatibility of the routines to calculate the basis .this section provides numerical simulations to analyze the theory and implementation of the statistical encryption / decryption strategy , presented in section s 2 . and 3 ., respectively . to demonstrate the efficacy of the theory presented in this paper , it is judicious to compare the _ pdf _ dependent model with a _ pdf _ independent model [ 8 ] .these are characterized by the constraint operators ( described in ( 5 ) and ( 6 ) ) , and , ( described in ( 4 ) ) , respectively .the following numerical studies perform the encryption / decryption strategy for the case of _ asymmetric _ cryptography .section 4.1 demonstrates the inference of the host _ pdf _ s from incomplete constraints , using an _ inverse maxent _ procedure .  the comparative analysis between the _ pdf _ dependent model and the _ pdf _ independent model is accomplished using two separate metrics , that define the security of the covert information .pdf _ s that deviate further from the equilibrium state often possess a constraint operator having a higher condition number , as compared with _ pdf _ s that are closer to the equilibrium state . increasingthe condition number of the constraint operator ( or , ) represents one way of securing the integrity of the covert information .the condition numbers of the constraint operators are obtained using the routine .this provides a measure of the sensitivity of and to perturbations .  in the _ pdf _ dependent model ,the security of the code is enhanced by the increased sensitivity of to perturbations of select elements of the operator , resulting in the operator with _ null space _ denoted by . within the framework of this model ,a more relevant metric of the extreme sensitivity of to perturbations , brought about by the introduction of the cryptographic keys , is the distortion of the code _ pdf _ . here , is evaluated from ( 9 ) , using ( the basis of ) .the distorted code _ pdf _ is , which is calculated from ( 9 ) using ( the basis of ) .note that the codes and are identical _kets_.  for the _ pdf _ dependent model ,the _ rms error of encryption _ between and is defined as : . here , , and, is the dimension of .section 4.3 will exemplify the utility of in resolving a dichotomy .specifically , it will be demonstrated that a high value of is the reason for host _ pdf _ s possessing constraint operators with lower condition numbers , sometimes providing a greater degree of security to the encoded covert information , than host _ pdf _ s possessing constraint operators with higher condition numbers .another measure of the security of the covert information , that is operationally advantageous , is the rms error of the difference between the encrypted code , and , the code reconstructed without the cryptographic keys .for the _ pdf _ dependent model , the _ rms error of reconstruction _ is : . here , , and , is the dimension of .  the rationale for evaluating the is to obtain a measure of the error of reconstruction by an _ unauthorized eavesdropper ( eve ) _ , who does not possess the correct cryptographic keys .eve is , however , assumed to be in possession of the reconstructed code _such an attack may be simulated by choosing the wrong set of keys , and , incorrectly constructing the perturbed operators and .alternatively , an analogous scenario may be simulated by assuming that the reconstruction is performed without keys ( _ sans _ keys ) . in this case, the reconstructed code without keys is : . here, is the basis of .section 4.2 presents the comparative analysis using the performance metrics described above .analysis of the results is provided in section 4.3 .host _ pdf _ column vectors ( _ kets _ ) corresponding to _ hierarchy indices _ and , each of dimension 401 , are independently inferred in the event space \ ] . in order to establish a degree of uniformity in the comparisons , two identical _ kets _ of the code ,each having dimension i.e. , are created for projection into the _ null spaces _ of the perturbed operators and , respectively .this `` emulates '' the _ selective _ projection of a code comprising of a single _ ket _ of dimension , into and , for each _ hierarchy index_.  a further measure of uniformity in the comparative analysisis sought by specifying the perturbations to both and as ( first row , third column ) and ( second row , seventh column ) , respectively .all numerical examples in this paper have a threshold for perturbations .simulations for the case of the _ pdf _ independent model are described in section 4.2.1 .two distinct case studies for the _ pdf _ dependent model are described in sections 4.2.2 and 4.2.3 , respectively .the results are presented in table 1 through table 3 .therein , sample values of the original code , the code reconstructed _ with _ the keys , and , the code reconstructed _ without _ the keys are presented .these comprise the elements of the respective arrays ( _ kets _ ) .as is evident from the numerical simulations , the reconstructed code _ with _ the keys is exactly similar to the original code .the corresponding is zero .on the other hand , the code reconstructed without the keys bears no resemblance to the original code .   here , , , and , .the values of the encrypted and reconstructed codes are independent of the _ hierarchy index_.these results are consistent with expectations since the nature of the host _ pdf _ is not reflected in the constraint operator .select code values are presented in table 1 ..select code values for the _ pdf _ independent model . - the original code, - the code reconstructed _ with _ the keys , and , - the code reconstructed _ without _ the keys . [ cols="^,^,^",options="header " , ]   the case presented in section 4.2.2 follows expectations .specifically , the constraint operators possess larger condition numbers than the constraint operator of the _ pdf _ independent model , presented in section 4.2.1 .further , the and in the _ pdf _ dependent model are greater those for the _ pdf _ independent model .this implies that the _ pdf _ dependent model provides greater security to the covert information , than the _ pdf _ independent model .the case presented in section 4.2.3 represents an interesting scenario , which poses a dichotomy of sorts .conventional logic would expect an increase in the condition numbers of to coincide with an increase in the sensitivity of , and thus , an increase in the values of .section 4.2.3 conclusively demonstrates the fact that the condition number of and the do not necessarily increase simultaneously .specifically , the case presented in section 4.2.3 demonstrates a significant _ decrease _ in the condition numbers of , that is accompanied by a significant _ increase _ in .this dichotomy may be explained by the fact that an _ increase _ in the value of is always accompanied by a corresponding _ increase _ in the rms error of encryption : .this trend is evident in each of the case studies presented in this paper , and , has been consistently observed in numerous simulation exercises . as indicated in the introduction of section 4 , the rms error of encryption ( ) represents a more relevant metric to assess the sensitivity of to perturbations , as compared to the condition number of , within the framework of this statistical encryption / decryption model .this section is concluded by providing a glimpse of the effects of the instability of the eigenstructures of the ill - conditioned operators .figure 2 depicts the error of encryption : - for the case study presented in section 4.2.3 , corresponding to .the highly oscillatory ( `` chaotic '' ) profile in figure 2 is indicative of the extreme sensitivity of to perturbations applied to .it may be noted that the code , projected into each host _ pdf _ , comprises of a _ ket _ of dimension 396 created by a random number generator . while one might expect the randomness inherent in to be factored into the error of encryption , further investigation concerning the oscillatory behavioris required .the _ ket _ is subjected to a sorting operation , by employing the routine .this results in a _ of dimension 396 , containing progressively increasing values defined in $ ] .the resulting error of encryption using the sorted code is depicted in figure 3 .  as is observed, the error of encryption still displays a highly pronounced oscillatory behavior , despite the randomness in the code being mitigated .akin to the case depicted in figure 2 , this oscillatory behavior is reflective of the instability induced in by perturbing 2 elements of ( having dimension ) by and , respectively .figure 4 depicts the total _pdf _ defined by ( 10 ) , corresponding to the case study presented in section 4.2.3 for . as is expected , the total _pdf _ exhibits a highly oscillatory ( `` chaotic '' ) behavior .this is in stark contrast to the smooth curves of the host _ pdf _ s depicted in figure 1.  - . ]  - with sorted code . ]a novel strategy to project covert information into a hierarchy of statistical hosts has been presented .this has been accomplished within the ambit of the maxent principle .the encryption / decryption strategy relates the projection of the covert information to the host _ pdf _ s .this feature permits the statistical encryption / decryption strategy to possess immense qualitative flexibility , and , provide enhanced security to the covert information , as compared to a _ pdf _ independent model [ 8 ] .the _ pdf _ dependent model , presented herein , sometimes demonstrates an increased rms error of reconstruction for decreased values of the condition number of the constraint operator .this seemingly counter - intuitive result is adequately explained with the aid of the rms error of encryption .the rms error of encryption is demonstrated to be a viable and relevant metric to assess the sensitivity of to perturbations .the statistical encryption / decryption strategy is platform independent , and the process of recovery of the covert information is accomplished with a very high degree of precision .the small amounts of data being transmitted through the covert channel as a consequence of the transmission of the cryptographic keys and the _ hierarchy indices _ s ( see section 3.2 ) , augers well for a coupling between the statistical encryption / decryption model , and a quantum key distribution protocol [ 13 , 14 ] .a study extending the present work , by describing the statistical encryption / decryption strategy within the framework of a fisher - schrdinger model , has been recently completed .herein , the fisher information has been employed as the measure of uncertainty .the host _ pdf _ s satisfy a time independent schrdinger - like equation ( tisle ) with an empirical _ pseudo - potential _ , that approximates a time independent schrdinger equation ( tise ) physical potential [ 7 , 15 ] .the tisle inherits the _ energy states _ of the tise , within an information theoretic context .the encryption of covert information is tantamount to projection of the code into different _ energy states _ of the tisle .the _ hierarchy indices _ in the present paper are replaced by the tisle _ energy states_.the fisher - schrdinger model provides a quantum mechanical connotation to the statistical encryption / decryption strategy .coupling the fisher - schrdinger model with a quantum key distribution protocol holds forth the prospect of achieving a self - consistent _ hybrid _ statistical / quantum mechanical cryptosystem .this work will be shortly presented for publication .ongoing work is directed towards a two - pronged objective .first , the projection of the covert information into has been provided with an information theoretic basis .next , models to extend the work presented in this paper to the case of steganography and information hiding have been developed .  by definition , steganography involves hiding data inside other ( host ) data . in steganography ,the covert information is concealed in a manner such that no one apart from the intended recipient knows of the existence of the message .this is in contrast to cryptography , where the existence of the message is obvious , but the meaning is obscured .the extension of the statistical encryption / decryption model presented in this paper to steganography involves encrypting image data into a _cover image_. in this case , the average pixel intensities of the _ cover image _ constitute the incomplete constraints .these results will be presented in future publications .r. l. rivest , a. shamir , l. adelman , `` on digital signatures and public key cryptosystems , '' _ mit laboratory for computer science technical memorandum _ , * 82 * , massachusetts institute of technology ( 1977 ) .r. c. venkatesan , `` encryption of covert information through a fisher game '' , in _ exploratory data analysis using fisher information _ , b. r. frieden and r. a. gatenby ( eds . ) , springer - verlag , london , 2006 ( in press ) .        c. h. bennett and g. brassard , `` quantum cryptography : public key distribution and coin tossing '' , in _ proc .conference on computers , systems and signal processing _ , bangalore , india , ieee press , new york , 175 - 179 , 1984 .

Abstract: a novel strategy to encrypt covert information ( code ) via unitary projections into the _ null spaces _ of ill - conditioned eigenstructures of multiple host statistical distributions , inferred from incomplete constraints , is presented . the host _ pdf _ s are inferred using the maximum entropy principle . the projection of the covert information is dependent upon the _ pdf _ s of the host statistical distributions . the security of the encryption / decryption strategy is based on the extreme instability of the encoding process . a self - consistent procedure to derive keys for both _ symmetric _ and _ asymmetric _ cryptography is presented . the advantages of using a multiple _ pdf _ model to achieve encryption of covert information are briefly highlighted . numerical simulations exemplify the efficacy of the model .   statistical encryption / decryption , projections , ill - conditioned eigenstructures , inference , maximum entropy .   pacs : 05.20.-y , 02.50.tt , 02.30.zz , 07.05.kf , 89.70.+c


in this paper , we propose an iterative algorithm to solve the following general optimization problem : where is a closed and convex set , and is a proper and differentiable function with a continuous gradient .we assume that problem ( [ eq : original_function ] ) has a solution .problem ( [ eq : original_function ] ) also includes some class of nondifferentiable optimization problems , if the nondifferentiable function is convex : because problem ( [ eq : original_function_nonsmooth ] ) can be rewritten into a problem with the form of ( [ eq : original_function ] ) by the help of auxiliary variables :   we do not assume that is convex , so ( [ eq : original_function ] ) is in general a nonconvex optimization problem .the focus of this paper is on the development of efficient iterative algorithms for computing the stationary points of problem ( [ eq : original_function ] ) .the optimization problem ( [ eq : original_function ] ) represents general class of optimization problems with a vast number of diverse applications .consider for example the sum - rate maximization in the mimo multiple access channel ( mac ) , the broadcast channel ( bc ) and the interference channel ( ic ) , where is the sum - rate function of multiple users ( to be maximized ) while the set characterizes the users power constraints . in the context of the mimo ic , ( [ eq : original_function ] ) is a nonconvex problem and np - hard . as another example , consider portfolio optimization in which represents the expected return of the portfolio ( to be maximized ) and the set characterizes the trading constraints .furthermore , in sparse ( -regularized ) linear regression , denotes the least square function and is the sparsity regularization function .commonly used iterative algorithms belong to the class of descent direction methods such as the conditional gradient method and the gradient projection method for the differentiable problem ( [ eq : original_function ] ) and the proximal gradient method for the nondifferentiable problem ( [ eq : original_function_nonsmooth ] ) , which often suffer from slow convergence . to speed up the convergence , the block coordinate descent ( bcd ) method that uses the notion of the nonlinear best - responsehas been widely studied ( * ? ? ?in particular , this method is applicable if the constraint set of ( [ eq : original_function ] ) has a cartesian product structure such that the bcd method is an iterative algorithm : in each iteration , only one variable is updated by its best - response ( i.e. , the point that minimizes with respect to ( w.r.t . )the variable only while the remaining variables are fixed to their values of the preceding iteration ) and the variables are updated sequentially .this method and its variants have been successfully adopted to many practical problems .when the number of variables is large , the convergence speed of the bcd method may be slow due to the sequential nature of the update .a parallel variable update based on the best - response seems attractive as a mean to speed up the updating procedure , however , the convergence of a parallel best - response algorithm is only guaranteed under rather restrictive conditions , c.f . the diagonal dominance condition on the objective function  , which is not only difficult to satisfy but also hard to verify . if is convex , the parallel algorithms converge if the stepsize is inversely proportional to the number of block variables .this choice of stepsize , however , tends to be overly conservative in systems with a large number of block variables and inevitably slows down the convergence .a recent progress in parallel algorithms has been made in , in which it was shown that the stationary point of ( [ eq : original_function ] ) can be found by solving a sequence of successively refined _ approximate problems _ of the original problem ( [ eq : original_function ] ) , and convergence to a stationary point is established if , among other conditions , the approximate function ( the objective function of the approximate problem ) and stepsizes are properly selected .the parallel algorithms proposed in are essentially descent direction methods . a description on how to construct the approximate problem such that the convexity of the original problem is preserved as much as possible is also contained in to achieve faster convergence than standard descent directions methods such as classical conditional gradient method and gradient projection method .despite its novelty , the parallel algorithms proposed in suffer from two limitations .firstly , the approximate function must be strongly convex , and this is usually guaranteed by artificially adding a quadratic regularization term to the original objective function , which however may destroy the desirable characteristic structure of the original problem that could otherwise be exploited , e.g. , to obtain computationally efficient closed - form solutions of the approximate problems .secondly , the algorithms require the use of a decreasing stepsize . on the one hand ,a slow decay of the stepsize is preferable to make notable progress and to achieve satisfactory convergence speed ; on the other hand , theoretical convergence is guaranteed only when the stepsize decays fast enough . in practice, it is a difficult task on its own to find a decay rate for the stepsize that provides a good trade - off between convergence speed and convergence guarantee , and current practices mainly rely on heuristics .the contribution of this paper consists in the development of a novel iterative convex approximation method to solve problem ( [ eq : original_function ] ) .in particular , the advantages of the proposed iterative algorithm are the following :  \1 ) the approximate function of the original problem ( [ eq : original_function ] ) in each iteration only needs to exhibit a weak form of convexity , namely , pseudo - convexity .the proposed iterative method not only includes as special cases many existing methods , for example , , but also opens new possibilities for constructing approximate problems that are easier to solve .for example , in the mimo bc sum - rate maximization problems ( sec . [ sub : mimo - broadcast - channel ] ) , the new approximate problems can be solved in closed - form .we also show by a counterexample that the assumption on pseudo - convexity is tight in the sense that if it is not satisfied , the algorithm may not converge .\2 ) the stepsizes can be determined based on the problem structure , typically resulting in faster convergence than in cases where constant stepsizes and decreasing stepsizes are used .for example , a constant stepsize can be used when is given as the difference of two convex functions as in dc programming .when the objective function is nondifferentiable , we propose a new exact / successive line search method that is carried out over a properly constructed differentiable function .thus it is much easier to implement than state - of - the - art techniques that operate on the original nondifferentiable objective function directly .  in the proposed algorithm ,the exact / successive line search is used to determine the stepsize and it can be implemented in a centralized controller , whose existence presence is justified for particular applications , e.g. , the base station in the mimo bc , and the portfolio manager in multi - portfolio optimization .we remark that also in applications in which centralized controller are not admitted , however , the line search procedure does not necessarily imply an increased signaling burden when it is implemented in a distributed manner among different distributed processors . for example , in the lasso problem studied in sec .[ sub : lasso ] , the stepsize based on the exact line search can be computed in closed - form and it does not incur any additional signaling as in predetermined stepsizes , e.g. , decreasing stepsizes and constant stepsizes .besides , even in cases where the line search procedure induces additional signaling , the burden is often fully amortized by the significant increase in the convergence rate .the rest of the paper is organized as follows . in sec .[ sec : preliminaries ] we introduce the mathematical background .the novel iterative method is proposed and its convergence is analyzed in sec .[ sec : proposed - method ] ; its connection to several existing descent direction algorithms is presented there . in sec .[ sec : applications ] , several applications are considered : the sum rate maximization problem of mimo bc , the energy efficiency maximization of a massive mimo system to illustrate the advantage of the proposed approximate function , and the lasso problem to illustrate the advantage of the proposed stepsize .the paper is finally concluded in sec .[ sec : concluding - remarks ] .  _notation : _ we use , and to denote a scalar , vector and matrix , respectively .we use to denote the -th element of ; is the -th element of where , and denotes all elements of except : .we denote as the element - wise inverse of , i.e. , .notation and denotes the hadamard product between and , and the kronecker product between and , respectively .the operator {\mathbf{a}}^{\mathbf{b}} ] : {\mathbf{a}}^{\mathbf{b}}\triangleq\max(\min(\mathbf{x},\mathbf{b}),\mathbf{a}) ] .we denote as the smallest integer that is larger than or equal to .we denote as the vector that consists of the diagonal elements of and is a diagonal matrix whose diagonal elements are as same as .we use to denote the vector whose elements are equal to 1 .in this section , we introduce the basic definitions and concepts that are fundamental in the development of the mathematical formalism used in the rest of the paper .  * stationary point*. __ a point is a stationary point of ( [ eq : original_function ] ) if condition ( [ eq : stationary - point ] ) is the necessary condition for local optimality of the variable . for nonconvex problems , where global optimality conditions are difficult to establish , the computation of stationary points of the optimization problem ( [ eq : original_function ] )is generally desired .if ( [ eq : original_function ] ) is convex , stationary points coincide with ( globally ) optimal points and condition ( [ eq : stationary - point ] ) is also sufficient for to be ( globally ) optimal .* descent direction*. _ _ the vector is a descent direction of the function at if ( [ eq : def - descent - direction ] ) is satisfied , the function can be decreased when is updated from along direction .this is because in the taylor expansion of around is given by : where the first order term is negative in view of ( [ eq : def - descent - direction ] ) . for sufficiently small , the first order term dominates all higher order terms .more rigorously , if is a descent direction , there exists a such that note that the converse is not necessarily true , i.e. , for arbitrary functions does not necessarily imply that is a descent direction of at .* quasi - convex function*. _ _ a function is quasi - convex if for any  ] is an appropriate stepsize that can be determined by either the exact line search ( also known as the minimization rule ) or the successive line search ( also known as the armijo rule ) .since , and  ] : {0}^{1}=\begin{cases } 1 , & \textrm { if } \left.\nabla_{\gamma}f(\mathbf{x}^{t}+\gamma(\mathbb{b}\mathbf{x}^{t}-\mathbf{x}^{t}))\right|_{\gamma=1}\geq0,\\ 0 , & \textrm { if } \left.\nabla_{\gamma}f(\mathbf{x}^{t}+\gamma(\mathbb{b}\mathbf{x}^{t}-\mathbf{x}^{t}))\right|_{\gamma=0}\leq0,\\ \gamma^{\star } , & \textrm { otherwise}. \end{cases}\ ] ] if , the constrained optimization problem in ( [ eq : minimization - rule ] ) is essentially unconstrained . in some applications it is possible to compute analytically , e.g. , if is quadratic as in the lasso problem ( sec .[ sub : lasso ] ) . otherwise , for general convex functions , can be found efficiently by the bisection method as follows .restricting the function to a line , the new function is convex in  .it thus follows that if and if .given an interval  ] denotes the projection of onto .this is equivalent to defining in ( [ eq : mapping_definition ] ) as follows : which is the first - order approximation of augmented by a quadratic regularization term that is introduced to improve the numerical stability .a generalization of ( [ eq : gradient - projection - approximate - function ] ) is to replace the quadratic term by where  .  * proximal gradient method * : if is convex and has a lipschitz continuous gradient with a constant , the proximal gradient method for problem ( [ eq : original_function_nonsmooth ] ) has the following form ( * ? ? ?4.2 ) : where . in the context of the proposed framework ( [ eq : nonsmooth - approximate - problem-1 ] ), the update ( [ eq : proximal - gradient - update ] ) is equivalent to defining as follows : and setting the stepsize for all . according to theorem [ thm : convergence ] and the discussion following assumption ( a6 ) , the proposed algorithm converges under a constant unit stepsize if is a global upper bound of , which is indeed the case when in view of the descent lemma ( * ? ? ?a.24 ) .* jacobi algorithm * : in problem ( [ eq : original_function ] ) , if is convex in each where ( but not necessarily jointly convex in ) , the approximate function is defined as  where for . the -th component function in ( [ eq : jacobi - approximate - function ] ) is obtained from the original function by fixing all variables except , i.e. , , and further adding a quadratic regularization term . since in ( [ eq : jacobi - approximate - function ] ) is convex , assumption ( a1 ) is satisfied . based on the observations that we conclude that assumption ( a3 ) is satisfied by the choice of the approximate function in ( [ eq : jacobi - approximate - function ] ) .the resulting approximate problem is given by this is commonly known as the jacobi algorithm .the structure inside the constraint set , if any , may be exploited to solve ( [ eq : jacobi - approximate - problem ] ) even more efficiently .for example , the constraint set consists of separable constraints in the form of for some convex functions . since subproblem ( [ eq : jacobi - approximate - problem ] ) is convex , primal and dual decomposition techniques can readily be used to solve ( [ eq : jacobi - approximate - problem ] ) efficiently ( such an example is studied in sec . [ sub : mimo - broadcast - channel ] ) .  to guarantee the convergence ,the condition proposed in is that for all in ( [ eq : jacobi - approximate - function ] ) unless is strongly convex in each .however , the strong convexity of in each is a strong assumption that can not always be satisfied and the additional quadratic regularization term that is otherwise required may destroy the convenient structure that could otherwise be exploited , as we will show through an example application in the mimo bc in sec .[ sub : mimo - broadcast - channel ] . in the case , convergence of the jacobi algorithm ( [ eq : jacobi - approximate - problem ] ) is only proved when is jointly convex in and the stepsize is inversely proportional to the number of variables  , namely , . however , the resulting convergence speed is usually slow when is large , as we will later demonstrate numerically in sec .[ sub : mimo - broadcast - channel ] .  with the technical assumptions specified in theorem [ thm : convergence ] ,the convergence of the jacobi algorithm with the approximate problem ( [ eq : jacobi - approximate - problem ] ) and successive line search is guaranteed even when .this is because in ( [ eq : jacobi - approximate - function ] ) is already convex when for all and it naturally satisfies the pseudo - convexity assumption specified by assumption ( a1 ) .  in the casethat the constraint set has a cartesian product structure ( [ eq : original - function - cartesian ] ) , the subproblem ( [ eq : jacobi - approximate - problem ] ) is naturally decomposed into sub - problems , one for each variable , which are then solved _ in parallel_. in this case , the requirement in the convexity of in each can even be relaxed to pseudo - convexity only ( although the sum function is not necessarily pseudo - convex in as pseudo - convexity is not preserved under nonnegative weighted sum operator ) , and this leads to the following update : and    where can be interpreted as variable s best - response to other variables when .the proposed jacobi algorithm is formally summarized in algorithm [ alg : jacobi ] and its convergence is proved in theorem [ thm : cartesian ] .  *data : * and for all .  repeat the following steps until convergence :  1 .  for ,compute using ( [ eq : jacobi - approximate - problem - cartesian ] ) .2 .  compute by the exact line search ( [ eq : minimization - rule ] ) or the successive line search ( [ eq : armijo - rule ] ) .update according to set .[ thm : cartesian]consider the sequence generated by algorithm [ alg : jacobi ] . provided that is pseudo - convex in for all and assumptions ( a4)-(a5 ) are satisfied .then any limit point of the sequence generated by algorithm [ alg : jacobi ] is a stationary point of ( [ eq : original - function - cartesian ] ) .see appendix [ sec : appendix - of - theorem - cartesian ] .the convergence condition specified in theorem [ thm : cartesian ] relaxes those in : only needs to be pseudo - convex in each variable and no regularization term is needed ( i.e. , ) . to the best of our knowledge ,this is the weakest convergence condition on jacobi algorithms available in the literature .we will show in sec .[ sub : energy - efficiency - maximization ] by an example application of the energy efficiency maximization problem in massive mimo systems how the weak assumption on the approximate function s convexity proposed in theorem [ thm : convergence ] can be exploited to the largest extent . besides this, the line search usually yields much faster convergence than the fixed stepsize adopted in even under the same approximate problem , cf .[ sub : mimo - broadcast - channel ] .  *dc algorithm * : _ _ if the objective function in ( [ eq : original_function ] ) is the difference of two convex functions and : the following approximate function can be used : since is convex and , assumption ( a6 ) is satisfied and the constant unit stepsize can be chosen .in this subsection , we study the mimo bc capacity computation problem to illustrate the advantage of the proposed approximate function .  consider a mimo bc where the channel matrix characterizing the transmission from the base station to user is denoted by , the transmit covariance matrix of the signal from the base station to user is denoted as , and the noise at each user is an additive independent and identically distributed gaussian vector with unit variance on each of its elements .then the sum capacity of the mimo bc is  where is the power budget at the base station .problem ( [ eq : mimo - bc ] ) is a convex problem whose solution can not be expressed in closed - form and can only be found iteratively . to apply algorithm [ alg : successive - approximation - method ] , we invoke ( [ eq : jacobi - approximate - function])-([eq : jacobi - approximate - problem ] ) and the approximate problem at the -th iteration is where . the approximate function is concave in and differentiable in both and , and thus assumptions ( a1)-(a3 ) are satisfied .since the constraint set in ( [ eq : mimo - bc - approx ] ) is compact , the approximate problem ( [ eq : mimo - bc - approx ] ) has a solution and assumptions ( a4)-(a5 ) are satisfied .problem ( [ eq : mimo - bc - approx ] ) is convex and the sum - power constraint coupling is separable , so dual decomposition techniques can be used . in particular, the constraint set has a nonempty interior , so strong duality holds and ( [ eq : mimo - bc - approx ] ) can be solved from the dual domain by relaxing the sum - power constraint into the lagrangian : where and is the optimal lagrange multiplier that satisfies the following conditions : , , , and can be found efficiently using the bisection method .  the problem in ( [ eq : mimo - bc - dual ] ) is uncoupled among different variables in both the objective function and the constraint set , so it can be decomposed into a set of smaller subproblems which are solved in parallel : and and exhibits a closed - form expression based on the waterfilling solution . thus problem ( [ eq : mimo - bc - approx ] ) also has a closed - form solution up to a lagrange multiplier that can be found efficiently using the bisection method . with the update direction , the base station can implement the exact line search to determine the stepsize using the bisection method described after ( [ eq : minimization - rule ] ) in sec .[ sec : proposed - method ] .we remark that when the channel matrices are rank deficient , problem ( [ eq : mimo - bc - approx ] ) is convex but not strongly convex , but the proposed algorithm with the approximate problem ( [ eq : mimo - bc - approx ] ) still converges .however , if the approximate function in is used [ cf .( [ eq : jacobi - approximate - function ] ) ] , an additional quadratic regularization term must be included into ( [ eq : mimo - bc - approx ] ) ( and thus ( [ eq : mimo - bc - dual - decomp ] ) ) to make the approximate problem strongly convex , but the resulting approximate problem no longer exhibits a closed - form solution and thus are much more difficult to solve .  * simulations . *the parameters are set as follows .the number of users is and , the number of transmit and receive antenna is ( 5,4 ) , and db . the simulation results are averaged over 20 instances .we apply algorithm [ alg : successive - approximation - method ] with approximate problem ( [ eq : mimo - bc - approx ] ) and stepsize based on the exact line search , and compare it with the iterative algorithm proposed in , which uses the same approximate problem ( [ eq : mimo - bc - approx ] ) but with a fixed stepsize ( is the number of users ) .it is easy to see from figure [ fig : mimo - bc - sum - capacity ] that the proposed method converges very fast ( in less than 10 iterations ) to the sum capacity , while the method of requires many more iterations .this is due to the benefit of the exact line search applied in our algorithm over the fixed stepsize which tends to be overly conservative . employing the exact line searchadds complexity as compared to the simple choice of a fixed stepsize , however , since the objective function of ( [ eq : mimo - bc ] ) is concave , the exact line search consists in maximizing a differentiable concave function with a scalar variable , and it can be solved efficiently by the bisection method with affordable cost . more specifically , it takes 0.0023 seconds to solve problem ( [ eq : mimo - bc - approx ] ) and 0.0018 seconds to perform the exact line search ( the software / hardware environment is further specified in sec . [ sub : lasso ] ) .therefore , the overall cpu time ( time per iteration of iterations ) is still dramatically decreased due to the notable reduction in the number of iterations . besides, in contrast to the method of , increasing the number of users does not slow down the convergence , so the proposed algorithm is scalable in large networks .mimo bc : sum - rate versus the number of iterations . ]we also compare the proposed algorithm with the iterative algorithm of , which uses the approximate problem ( [ eq : mimo - bc - approx ] ) but with an additional quadratic regularization term , cf .( [ eq : jacobi - approximate - function ] ) , where for all , and decreasing stepsizes where is the so - called decreasing rate that controls the rate of decrease in the stepsize .we can see from figure [ fig : mimo - bc - variable - error ] that the convergence behavior of is rather sensitive to the decreasing rate .the choice performs well when the number of transmit and receive antennas is 5 and 4 , respectively , but it is no longer a good choice when the number of transmit and receive antenna increases to 10 and 8 , respectively .a good decreasing rate is usually dependent on the problem parameters and no general rule performs equally well for all choices of parameters .we remark once again that the complexity of each iteration of the proposed algorithm is very low because of the existence of a closed - form solution to the approximate problem ( [ eq : mimo - bc - approx ] ) , while the approximate problem proposed in does not exhibit a closed - form solution and can only be solved iteratively .specifically , it takes ( version 2.0 ) 21.1785 seconds ( based on the dual approach ( [ eq : mimo - bc - dual - decomp ] ) where is found by bisection ) .therefore , the overall complexity per iteration of the proposed algorithm is much lower than that of .mimo bc : error versus the number of iterations . ]   in this subsection , we study the energy efficiency maximization problem in massive mimo systems to illustrate the advantage of the relaxed convexity requirement of the approximate function in the proposed iterative optimization approach : according to assumption ( a1 ) , only needs to exhibit the pseudo - convexity property rather than the convexity or strong convexity property that is conventionally required .consider the massive mimo network with cells and each cell serves one user .the achievable transmission rate for user in the uplink can be formulated into the following general form : where is the transmission power for user , is the covariance of the additive noise at the receiver of user , while and are positive constants that depend on the channel conditions only .in particular , accounts for the hardware impairments , and accounts for the interference from other users .  in 5 g wireless communication networks ,the energy efficiency is a key performance indicator . to address this issue, we look for the optimal power allocation that maximizes the energy efficiency : where is a positive constant representing the total circuit power dissipated in the network , and specifies the lower and upper bound constraint , respectively .problem ( [ eq : ee - problem - formulation ] ) is nonconvex and it is a np - hard problem to find a globally optimal point . therefore we aim at finding a stationary point of ( [ eq : ee - problem - formulation ] ) using the proposed algorithm .to begin with , we propose the following approximate function at : where and note that the approximate function consists of component functions , one for each variable , and the -th component function is constructed as follows : since is concave in ( shown in the right column of this page ) but is not concave in ( as a matter of fact , it is convex in ) , the concave function is preserved in in ( [ eq : ee - approximate - function - individual ] ) with fixed to be while the nonconcave functions are linearized w.r.t . at . in this way, the partial concavity in the nonconcave function is preserved in .similarly , since in the denominator is linear in , we only set .note that the division operator in the original problem ( [ eq : ee - problem - formulation ] ) is kept in the approximate function ( [ eq : ee - approximate - function ] ) . although it will destroy the concavity ( recall that a concave function divided by a linear function is no longer a concave function ) , the pseudo - concavity of is still preserved , as we show in two steps .step 1 : the function is concave in . for the simplicity of notation , we define two constants and .the first - order derivative and second - order derivative of w.r.t . are since when , is a concave function of in the nonnegative orthant  .  step 2 : given the concavity of , the function is concave in .since the denominator function of is a linear ( and thus convex ) function of , it follows from ( * ? ? ?* lemma 3.8 ) that is pseudo - concave .{\underline{p}_{k}}^{\overline{p}_{k}},\label{eq : ee - closed - form-3}\ ] ]  then we verify that the gradient of the approximate function and that of the original objective function are identical at .it follows that therefore assumption ( a3 ) in theorem [ thm : cartesian ] is satisfied .assumption ( a2 ) is also satisfied because both and are continuously differentiable for any .  given the approximate function ( [ eq : ee - approximate - function ] ) , the approximate problem in iteration is thus assumptions ( a4 ) and ( a5 ) can be proved to hold in a similar procedure shown in the previous example application . since the objective function in ( [ eq : ee - problem - formulation ] ) is nonconcave , it may not be computationally affordable to perform the exact line search . instead , the successive line search can be applied to calculate the stepsize . as a result, the convergence of the proposed algorithm with approximate problem ( [ eq : ee - approximate - problem - overall ] ) and successive line search follows from the same line of analysis used in the proof of theorem [ thm : cartesian ] .  the optimization problem in ( [ eq : ee - approximate - problem - overall ] )can be decomposed into independent subproblems ( [ eq : ee - approximate - problem - decomposition ] ) that can be solved in parallel : where . as we have just shown ,the numerator function and the denominator function in ( [ eq : ee - approximate - problem - decomposition ] ) is concave and linear , respectively , so the optimization problem in ( [ eq : ee - approximate - problem - decomposition ] ) is a fractional programming problem and can be solved by the dinkelbach s algorithm ( * ? ? ?* algorithm 5 ) : given ( can be set to 0 ) , the following optimization problem in iteration is solved :  [ eq : ee - closed - form ] where is the numerator function in ( [ eq : ee - approximate - problem - decomposition ] ) .the variable is then updated in iteration as   it follows from the convergence properties of the dinkelbach s algorithm that at a superlinear convergence rate .note that problem ( [ eq : ee - closed - form-1 ] ) can be solved in closed - form , as is simply the projection of the point that sets the gradient of the objective function in ( [ eq : ee - closed - form-1 ] ) to zero onto the interval  ] is the so - called soft - thresholding operator and or more compactly : thus the update direction exhibits a closed - form expression .the stepsize based on the proposed exact line search ( [ eq : nonsmooth - stepsize-1 ] ) is {0}^{1}.\label{eq : lasso - approximate - stepsize}\end{aligned}\ ] ] the exact line search consists in solving a convex quadratic optimization problem with a scalar variable and a bound constraint , so the problem exhibits a closed - form solution ( [ eq : lasso - approximate - stepsize ] ) .therefore , both the update direction and stepsize can be calculated in closed - form .we name the proposed update ( [ eq : lasso - approximate - problem])-([eq : lasso - approximate - stepsize ] ) as soft - thresholding with exact line search algorithm ( ) .  the proposed update ( [ eq : lasso - approximate - problem])-([eq : lasso - approximate - stepsize ] ) has several desirable features that make it appealing in practice .firstly , in each iteration , all elements are updated in parallel based on the nonlinear best - response ( [ eq : lasso - approximate - problem ] ) .this is in the same spirit as and the convergence speed is generally faster than bcd or the gradient - based update .secondly , the proposed exact line search ( [ eq : lasso - approximate - stepsize ] ) not only yields notable progress in each iteration but also enjoys an easy implementation given the closed - form expression .the convergence speed is thus further enhanced as compared to the procedures proposed in where either decreasing stepsizes are used or the line search is over the original nondifferentiable objective function in ( [ eq : lasso ] ) :   * computational complexity*. the computational overhead associated with the proposed exact line search ( [ eq : lasso - approximate - stepsize ] ) can significantly be reduced if ( [ eq : lasso - approximate - stepsize ] ) is carefully implemented as outlined in the following .the most complex operation in ( [ eq : lasso - approximate - stepsize ] ) is the matrix - vector multiplication , namely , in the numerator and in the denominator . on the one hand ,the term is already available from , which is computed in order to determine the best - response in ( [ eq : lasso - approximate - problem ] ) . on the other hand ,the matrix - vector multiplication is also required for the computation of as it can alternatively be computed as : where then only an additional vector addition is involved . as a result, the stepsize ( [ eq : lasso - approximate - stepsize ] ) does not incur any additional matrix - vector multiplications , but only affordable vector - vector multiplications .operation flow and signaling exchange between local processor and the central processor .a solid line indicates the computation that is locally performed by the central / local processor , and a solid line with an arrow indicates signaling exchange between the central and local processor and the direction of the signaling exchange . ]  * signaling exchange*. when can not be stored and processed by a centralized processing unit , a parallel architecture can be employed .assume there are ( ) processors .we label the first processors as local processors and the last one as the central processor , and partition as ,\ ] ] where and .matrix is stored and processed in the local processor , and the following computations are decomposed among the local processors :  [ eq : decompose ]   where .the central processor computes the best - response in ( [ eq : lasso - approximate - problem ] ) and the stepsize in ( [ eq : lasso - approximate - stepsize ] ) .the decomposition in ( [ eq : decompose ] ) enables us to analyze the signaling exchange between local processor and the central processor involved in ( [ eq : lasso - approximate - problem ] ) and ( [ eq : lasso - approximate - stepsize ] ) ) and ( [ eq : lasso - approximate - stepsize ] ) can also be implemented by a parallel architecture without a central processor . in this case , the signaling is exchanged mutually between every two of the local processors , but the analysis is similar and the conclusion to be drawn remains same : the proposed exact line search ( [ eq : lasso - approximate - stepsize ] ) does not incur additional signaling compared with predetermined stepsizes . ] .  the signaling exchange is summarized in figure [ fig : lasso - signaling - exchange ] .firstly , the central processor sends to the local processors ( * s1.1 * ) is set to , so . ] , and each local processor for first computes and then sends it back to the central processor ( * s1.2 * ) , which forms ( * s1.3 * ) as in ( [ eq : decompose-2 ] ) and calculates as in ( [ eq : r(x ) ] ) ( * s1.4 * ) and then as in ( [ eq : lasso - approximate - problem ] ) ( * s1.5 * ) .then the central processor sends to the local processor for ( * s2.1 * ) , and each local processor first computes and then sends it back to the central processor ( * s2.2 * ) , which forms ( * s2.3 * ) as in ( [ eq : decompose-1 ] ) , calculates as in ( [ eq : lasso - approximate - stepsize ] ) ( * s2.4 * ) , and updates ( * s3.1 * ) and ( * s3.2 * ) according to ( [ eq : s_(t+1 ) ] ) . from figure[ fig : lasso - signaling - exchange ] we observe that the exact line search does _ not _ incur any additional signaling compared with that of predetermined stepsizes ( e.g. , constant and decreasing stepsize ) , because the signaling exchange in * s2.1-s2.2 * has also to be carried out in the computation of in * s3.2 * , cf .( [ eq : s_(t+1 ) ] ) .we finally remark that the proposed successive line search can also be applied and it exhibits a closed - form expression as well .however , since the exact line search yields faster convergence , we omit the details at this point .  convergence of ( proposed ) and ( state - of - the - art ) for lasso : error versus the number of iterations . ]  * simulations*. * * we first compare in figure [ fig : lasso_decreasing_fails ] the proposed algorithm with  in terms of the error criterion defined as : {-\mu\mathbf{1}}^{\mu\mathbf{1}}\bigr\vert_{2}.\label{eq : lasso - error}\ ] ] note that is a solution of ( [ eq : lasso ] ) if and only if  . is implemented as outlined in ; however , the selective update scheme is not implemented in because it is also applicable for and it can not eliminate the slow convergence and sensitivity of the decreasing stepsize .we also remark that the stepsize rule for is  , where is the decreasing rate and .the code and the data generating the figure can be downloaded online .  note that the error plotted in figure [ fig : lasso_decreasing_fails ] does not necessarily decrease monotonically while the objective function always does .this is because and are descent direction methods . for ,when the decreasing rate is low ( ) , no improvement is observed after 100 iterations . as a matter of fact, the stepsize in those iterations is so large that the function value is actually dramatically increased , and thus the associated iterations are discarded in figure [ fig : lasso_decreasing_fails ] .a similar behavior is also observed for , until the stepsize becomes sufficiently small . when the stepsize is quickly decreasing ( ) ,although improvement is made in all iterations , the asymptotic convergence speed is slow because the stepsize is too small to make notable improvement . for this example, the choice performs well , but the value of a good decreasing rate depends on the parameter setup ( e.g. , , and ) and no general rule performs equally well for all choices of parameters . by comparison , the proposed algorithm is fast to converge and exhibits stable performance without requiring any parameter tuning .we also compare in figure [ fig : lasso_other_algorithms ] the proposed algorithm with other competitive algorithms in literature :  ,  ,  and  .we simulated of because it exhibits guaranteed convergence .the dimension of is ( the left column of figure [ fig : lasso_other_algorithms ] ) and ( the right column ) .it is generated by the matlab command with each row being normalized to unity .the density ( the proportion of nonzero elements ) of the sparse vector is 0.1 ( the upper row of figure [ fig : lasso_other_algorithms ] ) , 0.2 ( the middle row ) and 0.4 ( the lower row ) . the vector is generated as where is drawn from an i.i.d . gaussian distribution with variance .the regularization gain is set to , which allows to be recovered to a high accuracy .the simulations are carried out under matlab r2012a on a pc equipped with an operating system of windows 7 64-bit home premium edition , an intel i5 - 3210 2.50ghz cpu , and a 8 gb ram .all of the matlab codes are available online .the comparison is made in terms of cpu time that is required until either a given error bound is reached or the maximum number of iterations , namely , 2000 , is reached .the running time consists of both the initialization stage required for preprocessing ( represented by a flat curve ) and the formal stage in which the iterations are carried out . for example , in the proposed algorithm , is computed , so matrix - matrix multiplication between and is not required .] in the initialization stage since it is required in the iterative variable update in the formal stage , cf .( [ eq : lasso - approximate - problem ] ) .the simulation results are averaged over 20 instances .we observe from figure [ fig : lasso_other_algorithms ] that the proposed algorithm converges faster than all competing algorithms .some further observations are in order .time versus error of different algorithms for lasso . in the left andright column , the dimension of is and , respectively . in the higher , middle and lower column ,the density of is 0.1 , 0.2 and 0.4 . ] the proposed algorithm is not sensitive to the density of the true signal . when the density is increased from 0.1 ( left column ) to 0.2 ( middle column ) and then to 0.4 ( right column ) , the cpu time increases negligibly . the proposed algorithm scales relatively well with the problem dimension .when the dimension of is increased from ( the left column ) to ( the right column ) , the cpu time is only marginally increased . the initialization stage of is time consuming because of some expensive matrix operations as , e.g. , , and ( is a given positive constant ) .more details can be found in ( * ? ? ?furthermore , the cpu time of the initialization stage of is increased dramatically when the dimension of is increased from to . performs better when the density of is smaller , e.g. , 0.1 , than in the case when it is large , e.g. , 0.2 and 0.4 . the asymptotic convergence speed of is slow , because only one variable is updated in each iteration .  to further evaluate the performance of the proposed algorithm , we test it on the benchmarking platform developed by the optimization group from the department of mathematics at the darmstadt university of technology and compare it with different algorithms in various setups ( data set , problem dimension , etc . ) for the basis pursuit ( bp ) problem : to adapt the bp problem , we use the augmented lagrangian approach : where ( ) , is computed by and this process is repeated until converges .the numerical results summarized in show that , although must be called multiple times before the lagrange multiplier converges , the proposed algorithm for bp based on is very competitive in terms of running time and robust in the sense that it solved all problem instances in the test platform database .in this paper , we have proposed a novel iterative algorithm based on convex approximation .the most critical requirement on the approximate function is that it is pseudo - convex . on the one hand ,the relaxation of the assumptions on the approximate functions can make the approximate problems much easier to solve .we show by a counter - example that the assumption on pseudo - convexity is tight in the sense that when it is violated , the algorithm may not converge . on the another hand , the stepsize based on the exact / successive line search yields notable progress in each iteration .additional structures can be exploited to assist with the selection of the stepsize , so that the algorithm can be further accelerated .the advantages and benefits of the proposed algorithm have been demonstrated using prominent applications in communication networks and signal processing , and they are also numerically consolidated .the proposed algorithm can readily be applied to solve other problems as well , such as portfolio optimization .\i ) firstly , suppose is a stationary point of ( [ eq : original_function ] ) ; it satisfies the first - order optimality condition : using assumption ( a3 ) , we get since is pseudo - convex , the above condition implies that is , and .  secondly ,suppose .we readily get where the equality and inequality comes from assumption ( a3 ) and the first - order optimality condition , respectively , so is a stationary point of ( [ eq : original_function ] ) .\ii ) from the definition of , it is either    or ( [ eq : possibility-1 ] ) is true , then and , as we have just shown , it is a stationary point of ( [ eq : original_function ] ) .so only ( [ eq : possibility-2 ] ) can be true . we know from the pseudo - convexity of in ( cf .assumption ( a1 ) ) and ( [ eq : possibility-2 ] ) that and where the equality comes from assumption ( a3 ) .since is the optimal point of ( [ eq : approximate - problem ] ) , it satisfies the first - order optimality condition :   if ( [ eq : possibility-1 ] ) is true , then and it is a stationary point of ( [ eq : original_function ] ) according to proposition [ prop : descent - property ] ( i ) .besides , it follows from ( [ eq : minimum - principle-1 ] ) ( with and ) that .note that equality is actually achieved , i.e. , because otherwise would be an ascent direction of at and the definition of would be contradicted .then from the definition of the successive line search , we can readily infer that it is easy to see ( [ eq : decreasing-1 ] ) holds for the exact line search as well .if ( [ eq : possibility-2 ] ) is true , is not a stationary point and is a strict descent direction of at according to proposition [ prop : descent - property ] ( ii ) : is strictly decreased compared with if is updated at along the direction . from the definition of the successive line search , there always exists a such that and this strict decreasing property also holds for the exact line search because it is the stepsize that yields the largest decrease , which is always larger than or equal to that of the successive line search .  we know from ( [ eq :decreasing-1 ] ) and ( [ eq : decreasing-2 ] ) that is a monotonically decreasing sequence and it thus converges .besides , for any two ( possibly different ) convergent subsequences and , the following holds : since is a continuous function , we infer from the preceding equation that   now consider any convergent subsequence with limit point , i.e. , . to show that is a stationary point , we first assume the contrary : is not a stationary point . since is continuous in both and by assumption ( a2 ) and is bounded by assumption ( a5 ) , it follows from ( * ? ? ?1 ) that there exists a sequence with such that it converges and . since both and are continuous , applying ( * ? ? ? * th .1 ) again implies there is a such that and converges to defined as : where is the stepsize when either the exact or successive line search is applied to along the direction . since is not a stationary point , it follows from ( [ eq : decreasing-2 ] ) that , but this would contradict ( [ eq : value - convergence ] ) . therefore is a stationary point , and the proof is completed .we first need to show that proposition [ prop : descent - property ] still holds .suppose is a stationary point of ( [ eq : original - function - cartesian ] ) , it satisfies the first - order optimality condition : and it is equivalent to since is pseudo - convex in , the above condition implies for all .suppose for all .the first - order optimality conditions yields adding the above inequality for all yields therefore , is a stationary point of ( [ eq : original - function - cartesian ] ) .firstly , there must exist an index such that otherwise would be a stationary point of ( [ eq : original - function - cartesian ] ) .since is pseudo - convex in for , it follows from ( [ eq : cartesian - proof-2 ] ) that   secondly , for any index such that , minimizes over and for any .setting yields similarly , setting in ( [ eq : cartesian - proof-1 ] ) yields adding ( [ eq : cartesian - proof-4 ] ) and ( [ eq : cartesian - proof-5 ] ) , we can infer that .therefore , we can rewrite ( [ eq : cartesian - proof-5 ] ) as follows and thus adding ( [ eq : cartesian - proof-3 ] ) and ( [ eq : cartesian - proof-6 ] ) over all yields that is , is a descent direction of at the point .the proof of theorem [ thm : convergence ] can then be used verbatim to prove the convergence of the algorithm with the approximate problem ( [ eq : jacobi - approximate - problem - cartesian ] ) and the exact / successive line search .n. jindal , w. rhee , s. vishwanath , s. jafar , and a. goldsmith , `` sum power iterative water - filling for multi - antenna gaussian broadcast channels , '' _ ieee transactions on information theory_ , vol .51 , no . 4 , pp . 15701580 , apr .2005 .q. shi , m. razaviyayn , z .- q .luo , and c. he , `` an iteratively weighted mmse approach to distributed sum - utility maximization for a mimo interfering broadcast channel , '' _ ieee transactions on signal processing _ ,59 , no . 9 , pp .43314340 , sep .g. scutari , f. facchinei , p. song , d. p. palomar , and j .- s .pang , `` decomposition by partial linearization : parallel optimization of multi - agent systems , '' _ ieee transactions on signal processing _62 , no . 3 , pp . 641656 ,y. yang , g. scutari , p. song , and d. p. palomar , `` robust mimo cognitive radio systems under interference temperature constraints , '' _ ieee journal on selected areas in communications _ ,31 , no .24652482 , nov .2013 .kim , k. koh , m. lustig , s. boyd , and d. gorinevsky , `` an interior - point method for large - scale -regularized least squares , '' _ ieee journal of selected topics in signal processing _ ,vol . 1 , no . 4 , pp .606617 , dec . 2007 .s. boyd , n. parikh , e. chu , b. peleato , and j. eckstein , `` distributed optimization and statistical learning via the alternating direction method of multipliers , '' _ foundations and trends in machine learning _ , vol . 3 , no . 1 , 2010 .    p. l. combettes and j .- c .pesquet , `` proximal splitting methods in signal processing , '' in _ fixed - point algorithms for inverse problems in science and engineering _ , ser .springer optimization and its applications , h. h. bauschke , r. s. burachik , p. l. combettes , v. elser , d. r. luke , and h. wolkowicz , eds.1em plus 0.5em minus 0.4emnew york , ny : springer new york , 2011 , vol .185212 .m. razaviyayn , m. hong , and z .- q .luo , `` a unified convergence analysis of block successive minimization methods for nonsmooth optimization , '' _ siam journal on optimization _ , vol .23 , no . 2 ,pp . 11261153 , 2013 .g. scutari , f. facchinei , l. lampariello , , and p. song , `` distributed methods for constrained nonconvex multi - agent optimization - part i : theory , '' oct .2014 , submitted to _ ieee transactions on signal processing_. [ online ] .available : http://arxiv.org/abs/1410.4754      s. m. robinson and r. h. day , `` a sufficient condition for continuity of optimal sets in mathematical programming , '' _ journal of mathematical analysis and applications _ , vol . 45 , no . 2 , pp . 506511 , feb .1974 .r. h. byrd , j. nocedal , and f. oztoprak , `` an inexact successive quadratic approximation method for convex l-1 regularized optimization , '' sep .[ online ] .available : http://arxiv.org/abs/1309.3529      a. zappone , l. sanguinetti , g. bacci , e. jorswieck , and m. debbah , `` energy - efficient power control : a look at 5 g wireless technologies , '' _ ieee transactions on signal processing _64 , no .7 , pp . 16681683 , april 2016 .c. olsson , a. p. eriksson , and f. kahl , `` efficient optimization for -problems using pseudoconvexity , '' in _ 2007 ieee 11th international conference on computer vision_.1em plus 0.5em minus 0.4emieee , 2007 , pp .y. yang , g. scutari , d. p. palomar , and m. pesavento , `` a parallel decomposition method for nonconvex stochastic multi - agent optimization problems , '' dec .2015 , to appear in _ ieee transactions on signal processing_.  r. tibshirani , `` regression shrinkage and selection via the lasso : a retrospective , '' _ journal of the royal statistical society : series b ( statistical methodology ) _ , vol .58 , no . 1 ,267288 , jun . 1996 .m. a. t. figueiredo , r. d. nowak , and s. j. wright , `` gradient projection for sparse reconstruction : application to compressed sensing and other inverse problems , '' _ ieee journal of selected topics in signal processing _ ,vol . 1 , no . 4 , pp .586597 , dec . 2007 .

Abstract: in this paper , we propose a successive pseudo - convex approximation algorithm to efficiently compute stationary points for a large class of possibly nonconvex optimization problems . the stationary points are obtained by solving a sequence of successively refined approximate problems , each of which is much easier to solve than the original problem . to achieve convergence , the approximate problem only needs to exhibit a weak form of convexity , namely , pseudo - convexity . we show that the proposed framework not only includes as special cases a number of existing methods , for example , the gradient method and the jacobi algorithm , but also leads to new algorithms which enjoy easier implementation and faster convergence speed . we also propose a novel line search method for nondifferentiable optimization problems , which is carried out over a properly constructed differentiable function with the benefit of a simplified implementation as compared to state - of - the - art line search techniques that directly operate on the original nondifferentiable objective function . the advantages of the proposed algorithm are shown , both theoretically and numerically , by several example applications , namely , mimo broadcast channel capacity computation , energy efficiency maximization in massive mimo systems and lasso in sparse signal recovery .  energy efficiency , exact line search , lasso , massive mimo , mimo broadcast channel , nonconvex optimization , nondifferentiable optimization , successive convex approximation .


transaction costs preclude perfect replication of contingent claims and introduce a tradeoff between the risk and return to hedging . a standard approach to dealing with this tradeoffis to use utility functions for evaluating hedging errors ( hodges and neuberger 1989 ) and the utility indifference principle for pricing options ( davis et al .in contrast to the elegance of black - scholes delta hedging , these models typically lack explicit solutions and require powerful numerical methods to obtain case - by - case approximations of optimal hedges .  in some situations the particular structure of the optimal hedging strategyis known , which facilitates numerical analysis .one example is the black - scholes model with proportional transaction costs , where the hedging strategy of a trader with cara utility function is defined by a no - trade region ( davis et al .when the hedger s stock holdings are outside this region , she carries out a trade that brings the stock position onto the nearest boundary point .numerical methods that approximate value functions have proved successful in this model , see , e.g. , monoyios ( 2004 ) .a different approach is presented in zakamouline ( 2006 ) who derives analytic approximations of the two boundaries of the no - trade region .motivated by asymptotic results in whalley and wilmott ( 1997 ) , his approach combines a clever guess of a parametric functional form with the estimation of coefficients as functions of model parameters .these estimations , however , require a numerical approximation of the optimal hedging strategy .in this paper we propose an alternative approach to derive explicit hedging strategies for general nonlinear transaction costs .the approach is based on genetic programming which offers several advantages over classical numerical approaches .first , by construction it is constrained to output closed - form expressions that are functions of the model parameters .second , it delivers efficient results at low cost by solving the two tasks of finding an optimal hedging strategy and a closed - form representation for it in one integrated process .third , no assumption is made on the functional form of the approximation .genetic programming therefore avoids the disadvantage of ex post calibration methods and , at the same time , offers a technique that deals with cases in which there is no a priori information on the structure of the optimal solution .the closed - form approximations of hedging strategies derived with this heuristic method can be directly tested in different market situations to verify their efficiency before their integration into an automated trading system .the hard work of obtaining these hedging strategies is left to the computer .once this is done , hedging is without sweat .  in the example considered in this paperwe determine closed - form expressions for the hedging strategy of an option writer with cara utility function under proportional and quadratic transaction costs .the price of the underlying follows a geometric brownian motion .solutions to this problem with a large range of parameter values are derived at various requirements on the simplicity of hedging strategies , ranging from full - fledged computer programs to short analytic functions .parameters include volatility , interest rate , strike price of the european call , and transaction cost parameters .information available to the hedger consists of stock price , black - scholes delta and gamma , time to maturity , and current portfolio holdings , in addition to the parameter values which are random but constant until maturity of an option .our method produces better results than zakamouline ( 2006 ) when transaction costs are proportional . underquadratic transaction costs we find simple near - optimal linear trading strategies that consist of a trade intensity and a no - trade reference solution .the no - trade path can be interpreted as a modified black - scholes delta and the trade intensity as a modified gamma . in summary, the paper demonstrates the practical use of genetic programming in deriving hedges under nonlinear transaction costs .we consider the hedging problem faced by a risk - averse writer of a european call option .the option writer can trade in the underlying whose price follows a geometric brownian motion with constant drift and volatility .she also has access to a money account that pays interest at constant rate .we set to obtain the risk - neutral measure .the option writer can trade only at dates with , and the expiry date of the option .trading incurs transaction costs .we assume that the total transaction cost of buying or selling shares at the current price is given by with and .it is deducted from the trader s money account at the time a transaction takes place .the transaction cost has two components : ( a ) a proportional cost which arises in markets with a bid - ask spread of size and mid point given by the stock price ; and ( b ) a nonlinear cost which arises in markets when a trade can ` walk up ' an order book , i.e. , obtains worse prices the larger the trade .lcl +  + risk aversion & & 0.5 + proportional cost & & 10bp - 200bp + quadratic cost & & 5bp - 100bp + volatility & & uniformly drawn from  ] + strike price & & uniformly drawn from  ] from terminal wealth , where the expectation is taken over the probability measure defined by on the space of sample paths .given two hedging strategies and , we measure their relative performance for an option by the difference in certainty equivalents \right ) - u^{-1}\left({\mathbb{e}}[u(w_{\phi_{\theta}''}(t))]\right)\ ] ] where  ] for a typical trading strategy evolved with the gp algorithm , but in the this case with quadratic transaction costs .the left panel shows that the trade volume is approximately a linear function of the stock position for most stock positions and most periods in time .intersecting the graph with the horizontal plane at zero on the vertical axis , one obtains a curve through combinations of time and stock positions where there is no trade , and such that for other nearby points on the graph , the trade volume on the corresponding day is a constant fraction of the difference between the corresponding no - trade position and stock position .these observations suggest that linear trading strategies will provide good performance under quadratic transaction costs .the right - hand panel of figure [ fig : tradingqc ] shows the projection of the no - trade curve ( grey ) along with the gp hedge ( bold black ) , and the delta hedge ( dotted ) .also shown is the trade intensity ( black ) , defined as trade volume divided by the absolute difference between the current stock position and the no - trade stock position .the latter closely tracks the black - scholes delta .the hedging strategy adjusts the stock position towards the no - trade stock position , but this adjustment is quite slow , as the trade intensity is typically less than 20% .we therefore observe large deviations between actual stock holdings and the no - trade curve .closer to maturity the trade intensity is more volatile but then declines quickly ( resembling the widening of the no - trade region under proportional costs ) .this general pattern of sluggish adjustment leads to substantial savings in transaction costs compared to the black - scholes delta .   to explore the performance of linear hedging strategies under quadratic costs , we apply the gp algorithm to trading strategies of the form \ ] ] where is the no - trade stock position and is the trading intensity , and both are analytic expressions .the functions are constrained to be independent of stock holdings but are allowed to depend on time , greeks , and the parameter vector which are suppressed in .  as in the proportional cost case , making use of information about the ( approximate ) structure of the optimal hedge leads to strategies that are slightly better than those obtained without such information . in the current case , the mean improvement in performance amounts to 0.3 cents per option .this improvement is of the same order of magnitude as in the proportional cost case , where the a priori information was known to be correct .we therefore conjecture that linear strategies of the form ( [ eq : stratqc ] ) are near - optimal for hedgers with cara utility functions and quadratic transaction costs when the initial stock position is not too far from its desired level .the following approximation was obtained by means of the same simplification approach as in section [ sec : prop_notrade ] .it produces hedging decisions that perform as well as the best strategy in its class . for a given date , the no - trade stock position and the trading intensity are given by respectively , where , as before .the paper demonstrates the merits of a gp approach to solving optimal hedging problems under transaction costs .this heuristic method delivers approximations that are analytic functions of various parameters describing the option contract , the dynamic of the underlying , and transaction costs .these explicit approximations to optimal hedging strategies can easily be tested and integrated into an automated trading system .other estimation methods , such as zakamouline s ( 2006 ) , derive closed - form approximations using ad hoc specifications of the functional form and an exact ( numerical ) solution for their calibration .in contrast to such approaches , ours is both simpler and more general , as it requires neither explicit solutions nor assumptions about functional forms .preliminary results indicate that our gp method also works well for exotic options , other nonlinear transaction costs and stochastic volatility models but that is beyond the scope of this short note .= 2em = -2em

Abstract: hedging in the presence of transaction costs leads to complex optimization problems . these problems typically lack closed - form solutions , and their implementation relies on numerical methods that provide hedging strategies for specific parameter values . in this paper we use a genetic programming algorithm to derive explicit formulas for near - optimal hedging strategies under nonlinear transaction costs . the strategies are valid over a large range of parameter values and require no information about the structure of the optimal hedging strategy .  _ keywords : _ hedging , transaction costs , closed - form approximations , genetic programming .   _ jel classification : _ g13 .