the recent development of rf - enabled wet technology provides a new solution to continuously power energy - constrained wireless devices ( wds ) over the air . wireless power in tens to several hundred of microwatts can be effectively transferred to wds within ten meters distance , making self - sustainable network operation truly feasible and efficient for many low - power wireless applications , e.g. , wireless sensor networks and rf identity ( rfid ) systems with a large number of wds .the application of wet to wireless communications spurs a novel networking structure named wireless powered communication network ( wpcn ) , where the wds transmit information using the energy harvested by means of wet .wpcn removes the need of frequent battery replacement / recharging and reduces the probability of energy outage .the network lifetime can thus be largely extended and the communication performance can also be improved with more sustainable power supply .    as shown in fig . [101 ] , we consider a single - cell wpcn where a hybrid access point ( hap ) is responsible for transmitting wireless energy to and receiving wireless information transmission ( wit ) from a set of distributed wds . in practice , wet and wit are desired to operate in the same frequency band to achieve higher spectrum efficiency and cost effectiveness . in this case ,time - division - duplexing ( tdd ) circuit structures are applied at both hap and wds to switch between wet and wit modes to avoid the harmful interference from wet to information decoding . while a major design challenge is transmission scheduling for wet and wit to achieve both efficient communication and energy harvesting .most of the existing studies in wpcn have assumed the hap to centrally coordinate the wet and wit with the wds .for instance , proposes a round - robin based scheduling , where the hap and wds take turns to transmit energy or data .the duration of each wd s transmission is optimized by the hap according to the global instantaneous channel state information ( csi ) and then sent to all the wds . later extends to the case with a multi - antenna hap that enables more efficient energy beamforming technique for wet and sdma ( spatial duplexing multiple access ) for wit .a similar round - robin based scheduling method is considered in , where each energy - harvesting wd can be either active or inactive in a time slot to achieve a balance between communication delay and energy consumption .in addition , considers a polling - based method that the hap periodically inquires the wds about their residual energy levels and performs wet whenever some wds are in low battery state .  in practice , the above centralized methods often incur considerable signaling overhead on channel estimation , control , synchronization , etc. this could be costly in networks with a large number of wds ( e.g. , sensors ) that are randomly deployed and switch on / off over time for energy saving . in this case ,distributed scheduling of wet and wit is of high practical interests .although distributed wireless charging control and data transmissions have been well investigated separately ( e.g. , ) , there are only few studies integrating them in the design of wpcn .for instance , proposes an energy - adaptive csma - type mac ( medium access control ) method , where the access probability of a wd decreases with its energy harvesting rate .however , it assumes that wet is independent of wit , and thus no joint wet and wit scheduling is considered . proposes a rf - mac scheme that multiple haps are divided into groups to perform wet in respond to wds energy request , and the wds use csma - type random access control to coordinate the data exchange among each other .the rf - mac method , however , requires the wds to bear complicated computation and channel estimation tasks .further , considers a simplified version of rf - mac , where the throughput performance of a wpcn using a single hap is evaluated via simulations .nonetheless , the analysis of both works is limited and does not capture the important coupling between energy and information transmissions .  in this paper, we present a practical distributed scheduling protocol for wpcn .similar to the idea of rf - mac , we allow each wd that is about to deplete its battery to broadcast an energy request buzz ( erb ) signal in order to trigger the wet by the hap to recharge its battery .if no erb is sent , the wds then contend to transmit data to the hap based on the conventional -persistent csma .- persistent csma achieves similar performance as the exponential backoff scheme in when the transmit probability of the wds is proportional to the user number . in practice ,the hap is aware of the number of associating wds and thus can calculate and broadcast its value to the wds . ] in particular , we propose an energy queueing model to analyze the throughput performance of the proposed distributed scheduling protocol .simulation results are provided to verify our analysis and show that the proposed method can achieve good throughput performance as compared to a benchmark -persistent csma network assuming always sufficient energy supply .in addition , an interesting _ energy decoupling _property is revealed , which is useful in deriving the throughput and understanding the insight on designing distributed scheduling in wpcn .as shown in fig . 1 , we consider a wpcn consisting of a hap and wds , where all the devices each have one single antenna .we assume that wet and wit are performed over the same frequency band , such that each wd s antenna is used for both energy harvesting and communication in a tdd manner ( see wd1 in fig .[ 101 ] ) . the energy harvesting circuit converts the received rf signal to dc energy and stores in a rechargeable battery . the energy is then used to power the wit .the hap also has a similar tdd circuit structure ( see fig .[ 101 ] ) to switch between energy transfer and communications with the wds .we assume that all the wds are continuously backlogged , i.e. , they always have packets to transmit .besides , the network is fully connected , such that the transmission of one device ( wd or hap ) can be overheard by all the other devices . meanwhile , all devices are assumed to have carrier sensing capability , such that they remain silent when sensing any ongoing energy / information transmission and attempt to transmit only after the channel becomes idle .the proposed distributed scheduling mechanism is illustrated in fig . and explained as follows .     in the proposed distributed scheduling mechanism ,each wd continuously monitors its residual battery level .if it is above a predetermined threshold , the wd waits for the channel to be continuously idle for a difs ( distributed inter - frame spacing ) time and then transmits independently a payload packet with probability to the hap .the duration of difs is much larger than the signal round - time - delay ( rtd ) of the network , such that a wd s data transmission will not interfere with the potential data transmissions of the other wds due to signal prorogation delay .the packet header contains the identity of the transmitting wd , such that the hap can identify the sender if the packet is successfully decoded .otherwise , if a wd finds its residual battery level below the threshold , it waits the channel to be continuously idle for a pifs ( priority inter - frame spacing ) time and then sends a short energy request buzz ( erb ) signal .the duration of the pifs is smaller than that of difs ( but still much larger than the rtd ) , such that a higher priority is assigned to sending an erb than a data payload .  from the hap s perspective, it can identify the current time slot as a wet slot when sensing any erb sent , either by one or multiple wds , and respond by performing wet for amount of time .meanwhile , all the wds sensing the erb signal switch to energy harvesting mode .otherwise , if no erb signal is sensed , the hap identifies the current time slot as a wit slot and switches to information receiving mode . due to the close communication range ( say , within meters ) typically for wpcn, we assume the receiver signal - to - noise ratio ( snr ) is sufficiently high and thus neglect the decoding errors caused by channel fading and receiver noise during data transmissions .accordingly , a wit slot may correspond to one of the following three scenarios :  1 .success : if only one wd transmits data , the transmitted packet can be successfully decoded by the hap , which then responds to the transmitting wd by sending an ack message ( containing the i d of the transmitter ) after sensing the channel to be idle for sifs ( short inter - frame spacing ) time .notice that sifs is shorter than the difs and pifs such that no wd will transmit before the hap sends the ack ; 2 .  collision : if more than one wds transmit data in the same time slot , the multiple packets will collide and none of then can be correctly decoded by the hap . in this case , the hap broadcasts a nak message after sensing the channel to be idle for sifs . by doing so, the transmitting wds can identify collision and schedule data retransmission ; 3 .idle : otherwise , no wd transmits data and all wds keep silent for a mini slot of duration .notice that the wds do not need to wait for another difs time to transmit data after an idle time slot .instead , they can persistently access the channel with probability in the following mini slots until some wd transmits . therefore , we may observe consecutive idle mini slots .  after the current time slot , either for wet or wit ,all the devices continue to sense the wireless medium , and the above iteration repeats itself .we use an example to illustrate the operation of the proposed protocol in fig .3 . initially ,the wds have sufficient energy to transmit and wd1 transmits successfully .after the channel becomes idle for difs , no wd transmits in two consecutive idle time slots each of duration , until they both transmit data in the third attempt and cause a collision .then , after transmitting data , wd is lack of energy and sends an erb signal after a pifs . upon detecting the erb signal ,the hap starts wet for the wds to harvest energy .   for a wet time slot ,we assume channel fading effect is averaged out over the duration of energy transmission such that the received energy by the -th wd is only related to its distance to the hap . due to the broadcasting nature of wireless channels , all the wds can harvest energy in a wet time slot .we also assume that the wds can not harvest energy from wit , as transmit power of wd is significantly lower than that of wet by the hap , e.g. , versus .thus , the received energy by the -th wd in a wet slot is where denotes the energy harvesting efficiency , denotes the antenna gain , denotes the power of wet , denotes the carrier frequency and denotes the path loss exponent , which is assumed equal for all the wds .  for a wit slot, we assume that all the wds transmit with constant power .for the simplicity of analysis , we assume that each wd transmits a payload of fixed duration in a wit slot , such that it consumes amount of energy regardless of that the transmission is successful or results in a collision .nonetheless , our analysis can also be extended to the case that the payload lengths are different in a success and a collision slot , such as a cts / rts - like scheme in 802.11 wlan . with channel error neglected , the decoding failure is only caused by transmit collisions .-node example of the distributed scheduling protocol operation.,scaledwidth=50.0% ]   in this paper , we consider a discrete energy model and assume that the transmission of each fixed - length payload consumes unit of energy and the battery capacity of each wd is units , where is a positive integer .besides , we assume that the -th wd harvests fixed units of energy in each wet slot . here , is a positive integer , , depending on the distance between the wd and the hap .this may correspond to a practical design requirement that , , to avoid frequent energy transmissions , which is achievable through either setting a long enough or preventing ineffective far - away wds from associating with the hap .we denote as the battery level ( in units ) of the -th wd at the end of the -th time slot , and and as the number of units of energy harvested and consumed during the -th time slot , respectively .then , the battery dynamics of the -th wd can be expressed as where and denotes the initial energy level .depending on the type of the -th transmission slot , and can be categorized as follows :  1 .wet slot : and for all the wds ; ; 2 .success / collision wit slot : and for transmitting wds , and otherwise ; 3 .idle wit slot : for all the wds .  without loss of generality , we assume that a wd sends erb signal when .   in this paper ,a key performance metric is the normalized network throughput , defined as the percentage of air time occupied by successful data transmissions expressed as where and denote the probabilities of a successful packet transmission slot , a packet collision slot , an idle slot , and an energy transfer slot , respectively . by assuming the durations of ack and nak are equal, we can see from fig . 3 that in the next section , we analyze the throughput performance of the distributed energy and information scheduling algorithm .we start with modeling the battery dynamic of each wd as a b - d queueing process . as shown in fig . 4, we drop the superscript for simplicity of expression and use to denote the residual energy of the -th wd at the beginning of a time slot .we refer to the wd as in the -th energy state if , .in particular , we use to denote the probability of the -th wd in the -th energy state observing a wet slot , .therefore , we can express the state transition probability , which denotes the probability that the -th wd changes from the -th to the -th energy state , as follows :  [ 7 ]   for intermediate states with .besides , the other two boundary states and satisfy  [ 8 ]   here , ( [ 4])-([5 ] ) correspond to a wet slot , a success / collision wit slot , and an idle slot , respectively .in addition , ( [ 2 ] ) holds because a wd with will immediately send erb and receive energy in the current time slot .( [ 6 ] ) and ( [ 3 ] ) hold because the energy of a fully - charged wd will reduce only when it transmits data , and remain unchanged otherwise .-th wd ( ).,scaledwidth=50.0% ]  it is evident that the occurrence of an energy transfer slot is related to the energy states of all the wds , where a wet slot occurs when for some wd .accordingly , precise system - level analysis requires a high - dimensional markov system that jointly considers the energy states of all the wds .this , however , renders the problem analytically intractable due to the large number of inter - connected states . for tractable analysis, we make in this paper the following _ energy decoupling assumption_.  * energy decoupling assumption ( eda ) : * _ in the considered energy queueing system ( [ 7 ] ) and ( [ 8 ] ) , the limiting probabilities of the wds are independent and each wd observes a constant probability of wet in a time slot independent of its current energy state , i.e. , , . _* remark 1 * : the eda assumption considered in this paper is analogous to the well - known _ mean - field decoupling assumption _ made in the seminal work on performance analysis of 802.11 dcf medium access control , where wds with unlimited energy supply transmit data following a random backoff mechanism .specifically , it assumes that when the number of wds in a 802.11 network is large enough , each wd observes a constant collision probability upon transmission , which is independent of ( but in fact related to ) the current backoff stages of itself and the other wds .  as an initial attempt to investigate the performance of distributed scheduling of wet and wit in wpcn, we leave the proof of the above eda assumption in our future work . for the time being , the eda is verified using simulations later in section v , where we show that this assumption approximately holds when the number of wds is not too small , e.g. , .   with the eda assumption, we can replace s with in ( [ 7 ] ) and ( [ 8 ] ) .we denote the steady - state limiting probabilities of the -th wd as , .for such a birth - death ( b - d ) queueing process in fig . 4, its limiting probabilities satisfy the following equalities by establishing flow conservation " conditions between two adjacent states , i.e. ,  [ 9 ]   the above equations , combined with the total probability condition , can be expressed as where , , with denoting the matrix transpose and here , . because is a full - rank square matrix , we can obtain the steady state limiting probabilities as , with denoting the matrix inverse .in particular , we can infer that {1,c+1},\ ] ] where {i , j} ] .( [ 18 ] ) implies that is a non - decreasing function of each entry in . by stacking the equations in ( [ 13 ] ) and equations in ( [ 18 ] ) , we have where ^t ] . evidently , is a non - increasing function of due to the monotonic property of .for instance , when the wds are homogeneous , i.e. , s are equal for all the wds , we can denote by symmetry that and , . in this case , as is a non - decreasing function , can be obtained using simple bi - section search over until is satisfied within a given precision level . in general , s can be obtained numerically , e.g. , using the quasi - newton method .given , we are ready to derive the throughput performance .specifically , the probability of a wet slot is i.e. , at least one of the wds sends erb .accordingly , the probability of an information transmission slot is .then , the probability of a successful transmission is i.e. , exactly one wd transmits information . besides , the probabilities of an idle slot and a collision slot are respectively by substituting ( [ 21])-([23 ] ) into ( [ 1 ] ) , we can obtain the throughput .we notice that each wd has the equal probability to transmit information in a wit slot .therefore , the wds have the same average data rate . from ( [ 21 ] ) , if some wd has very high probability of energy outage , i.e. , large , the data rates of all the wds can be very low .therefore , our proposed method should be applied to a network with limited wet range , e.g. , the maximum wd - to - hap distance is less than meters to ensure that all wds can be effectively charged by the hap ..simulation parameters [ cols="^,^,^,^",options="header " , ]   [ stat ]in this section , we use simulations to verify the analysis and evaluate the performance of the distributed scheduling protocol proposed . in all simulations, we use the powercast tx91501 - 3w transmitter as the energy transmitter at the hap and p2110 powerharvester as the energy receiver at each wd with energy harvesting efficiency .unless otherwise stated , the simulation parameters are listed in table i , which correspond to a typical outdoor sensor network .we consider two types of wds , where type - i wds are located around meters away from the hap , while type - ii wds are located around meters away . from tablei , each wd consumes around mj energy to transmit a payload . from ( [24 ] ) , type - i and type - ii wds harvest and units of energy , i.e. , and , respectively .unless otherwise stated , we consider wds with type - i wds and type - ii wds . besides, the battery capacity is set as .each point in the figures shown in this section is obtained from simulating the wpcn for time slots .we first verify in figs . [ 105 ] and [ 107 ] the proposed energy decoupling assumption ( eda ) .in particular , for each wd , we calculate by dividing the number of et slots observed at each battery state and the number of occurrences of battery state , . in fig . [ 105 ] , we plot s and the limiting probabilities of battery states ( s ) of type - i wds .we can see that s are approximately constant for , which matches the statement of eda .the only exception is the -st battery boundary state , where is significantly lower than the other s and is much higher than the other states .this is mainly due to its close connection with the -th battery state , where a wd immediately enters the -st battery state when it reaches the -th battery state .we also plot in fig . [ 107 ] the s and s of type - ii wds .interestingly , we can see that s are approximately equal as long as sufficient samples are collected at energy state , e.g. , . for states and , no sample or very few samples are collected due to the extremely low probabilities of the two states , thus the samples collected for the two states are ignored . from the above discussion , we can see that the proposed eda approximately holds , which serves as the basis of our analysis .  .sub - figures ( a ) and ( b ) show and ( in log - scale ) , respectively.,scaledwidth=50.0% ]  .sub - figures ( a ) and ( b ) show and ( in log - scale ) , respectively.,scaledwidth=50.0% ]  in fig .[ 106 ] , we compare the throughput analysis in ( [ 21])-([23 ] ) with simulations when the number of wds changes from to . without loss of generality, we assume wds are type - i and the rest wds are type - ii , and . for brevity, we only present the results for the probability of a wet slot ( in ( [ 21 ] ) ) and that of a successful transmission slot ( in ( [ 22 ] ) ) .we can see that the simulation and analysis match well , which validates our proposed analytical method .besides , decreases with the number of wds , .this is because , by setting , the successful transmission probability keeps almost unchanged but each wd transmits less frequently .this reduces the overall energy consumption and the need for energy transfer .we can also infer from fig . [106 ] that the proposed distributed scheduling method can achieve stable throughput performance against the variation on the number of wds , as long as the transmit probability is set proportionally to the number of wds . in practice , the hap can calculate by counting the number of associating wds and broadcast its value to the wds either periodically or when the number of associating wds varies .  ) and successful transmission slot ( ) , respectively.,scaledwidth=50.0% ]   to the throughput performance .sub - figures ( a ) and ( b ) show and normalized throughput , respectively.,scaledwidth=50.0% ]  at last , we investigate the impact of data transmit probability ( ) to the throughput performance . here , we consider a performance benchmark with unlimited battery supply , i.e. , no need of wet .this may correspond to the conventional -persistent csma wlan network without device energy constraint .evidently , the benchmark method produces a performance upper bound of the energy - constrained scheme considered in this paper .all the points in fig .[ 108 ] are calculated numerically based on the proposed analytical model . in fig .[ 108](a ) , we consider and compare achieved by the two methods when , .we can see that the benchmark method achieves the maximum when .the considered distributed scheduling , however , achieves the maximum at a smaller when .besides , we also plot in fig . [ 108](b ) the throughput performance comparison . similarly , we can see that the maximum throughput of the proposed wireless - powered scheme is achieved at a smaller than that with unlimited energy supply , i.e. , versus . this is because a larger would induce high collision probability in both networks , but in wpcn , it also causes higher device energy consumption , and thus inducing more frequent wet and on average shorter airtime of wit .overall , the throughput of the wpcn is around lower than the case with unlimited energy supply , e.g. , conventional wlan .the performance loss is acceptable considering the additional overhead for wet in wpcn .in this paper , we presented a fully distributed scheduling protocol for energy and information transmissions in rf - enabled wpcn .an energy queueing model was proposed to analyze the throughput performance , which leverages an interesting and novel energy decoupling property in the considered wpcn .simulation results have verified our analysis and showed that the proposed distributed scheduling can achieve sustainable and efficient operation of wpcn ._ proof _ : without loss of generality , we denote when , and when , where and .evidently , we can see from ( [ 14 ] ) that similarly , by substituting into in ( [ 15 ] ) , we have by repeatedly substituting into in either ( [ 15 ] ) or ( [ 16 ] ) , we have therefore , we have which leads to the proof of the desired result.                m. y. naderi and p. nintanavongsa , rf - mac : a medium access control protocol for re - chargeable sensor networks powered by wireless energy harvesting " , _ ieee trans . wireless commun ._ , vol . 13 , no . 7 , pp .3926 - 3937 , jul .

Abstract: wireless powered communication network ( wpcn ) is a novel networking paradigm that uses radio frequency ( rf ) wireless energy transfer ( wet ) technology to power the information transmissions of wireless devices ( wds ) . when energy and information are transferred in the same frequency band , a major design issue is transmission scheduling to avoid interference and achieve high communication performance . commonly used centralized scheduling methods in wpcn may result in high control signaling overhead and thus are not suitable for wireless networks constituting a large number of wds with random locations and dynamic operations . to tackle this issue , we propose in this paper a distributed scheduling protocol for energy and information transmissions in wpcn . specifically , we allow a wd that is about to deplete its battery to broadcast an energy request buzz ( erb ) , which triggers wet from its associated hybrid access point ( hap ) to recharge the battery . if no erb is sent , the wds contend to transmit data to the hap using the conventional -persistent csma ( carrier sensing multiple access ) . in particular , we propose an energy queueing model based on an energy decoupling property to derive the throughput performance . our analysis is verified through simulations under practical network parameters , which demonstrate good throughput performance of the distributed scheduling protocol and reveal some interesting design insights that are different from conventional contention - based communication network assuming the wds are powered with unlimited energy supplies .


there are different kinds of risk in a stock portfolio , for example , a systematic one due to the general trends in the economy affecting the entire market , and an unsystematic one due to events affecting only segments of the whole market under consideration , such as the companies within the same industrial branch or in one country  .all this is borne out in the time series of the stocks . in a more general framework, the term `` stock of a company '' should be replaced by `` risk factor '' .the corresponding portfolios of a bank or an investment company can contain hundreds or thousands of financial instruments , depending on many different risk factors  .however , to be explicit , we will use the term `` stocks '' instead of `` risk factors '' .banks or portfolio managers are particularly interested in the unsystematic correlations  .we refer to them simply as correlations from now on .the elements of the correlation matrix are given through the time average over a product of the properly normalized time series for two stocks .the number of companies considered defines the size of this correlation matrix .we will assume that this number is large enough for a statistical analysis .in recent years , it was shown that the true correlations one is interested in can be buried under noise in the commonly used correlation matrices : laloux et al . studied an empirical correlation matrix and found that the bulk part of its spectral density can be modeled by a purely random matrix .plerou et al . worked out spectral fluctuations and also found that they are compatible with random matrix statistics .random matrices are often used statistical models for quantum chaotic or related spectral problems , for reviews see refs .the presence of random matrix features was coined noise dressing of financial correlations .burda et al . designed a proper model involving random levy matrices . in practice, such noise can appear if the length of the time series employed to construct the correlation matrices is too short .gopikrishnan et al . developed a method to reduce the noise by taking advantage of the fact that large eigenvalues outside the bulk of the spectral density can be associated with industrial branches .these authors  used this for portfolio optimization , see also ref .the identification of branches through stock correlations was also studied by mantegna  .  here , we present a new approach to identify and estimate the noise and , thereby , also the strength of the true correlations . as we do not use the large eigenvalues outside the bulk of the spectral density ,our method is an alternative and a complement to the approach of ref .it will be particularly useful if some of the large eigenvalues are not large enough to lie outside the bulk of the spectral density .our method is based on a power mapping .it does not require any further data processing or any other type of input .to the best of our knowledge , it seems to be a new technique for the analysis of correlation matrices in time series problems .we have three goals : first , we develop an alternative method for noise identification , second , we introduce the power mapping as a new mathematical statistical technique and discuss its features in some detail , third , we present our findings in a self contained way to make possible a transfer to other time series problems and the corresponding correlation matrices .  the article is organized as follows . in sec .[ sec2 ] , we outline the stochastic model , including a review of correlation matrices and a discussion of the noise dressing phenomenon .we study the dependence of the spectral density as a function of the length of the time series in sec .[ sec3 ] . in sec .[ sec4 ] , we introduce the power mapping and use it to identify and estimate the noise. we summarize and conclude in sec .three more detailed discussions are given in the appendix .we formulate the model for financial correlations in sec .[ sec2.1 ] .since we also aim at a heuristic analytical treatment later on , we do that in some detail .correlation matrices and noise dressing are discussed in secs .[ sec2.2 ] and [ sec2.3 ] , respectively .most models for time series of stocks  have a random component , involving a random number and a volatility constant , and a non random component , the drift part , involving a drift constant .geometric brownian motion is particularly popular . the dimensionless quantity is referred to as return . due tothe central limit theorem , geometric brownian motion leads , largely independently of the distribution for the random numbers , to a log normal distribution of the stock prices .this is in fair agreement with the empirical distributions for price changes about one day and greater  .the tails of the empirical distributions are , in general , much fatter  . to describe this , one uses , for example , autoregressive processes which take into account that the volatility is also a fluctuating quantity , see the discussion in ref . . in the present context of correlations , however , it suffices to model the time series in the spirit of geometric brownian motion . in economics, one wishes to measure the correlations independent of the drift .thus , one usually takes the logarithms or the logarithmic differences of the time series to remove the exponential trend in the data due to the drift . moreover , one normalizes the data to zero mean value and unit variance .  in view of these requirements ,noh  suggested a proper model to study financial correlations . from an economics viewpoint , it is of the one factor type and can be interpreted as an application of the arbitrage pricing model due to ross  , see also the discussion in [ app0 ] . from a physics viewpoint , as pointed out in refs . , noh s model has much in common with certain models involving interacting potts spins  . consider a market involving companies , labeled , and industrial branches , labeled .the companies within the same industrial branch are assumed to be correlated .the companies are ordered in such a way that the indices of companies within the same branch follow each other .for example , we have companies and branches and we assume that the first branch with consists of the first companies , the second branch with consists of the next companies , the third branch with consists of the next companies , and , finally , we assume that the remaining companies are in no branch .the branch index is viewed as a function of the company index , i.e. we have . for the companies which are not in any branch , we set .we refer to as to the size of the industrial branch .obviously , we have of course , we assume that . the number can be any non negative integer , including zero .the normalized time series of the returns for the companies are modeled as the sum of two purely random contributions : the first one models the correlations within a given branch and is thus common to this branch , involving random numbers , the second one is specific for the company and involves random numbers , the two contributions are weighted with a parameter , common to all companies in the branch .we assume that the and the are uncorrelated and standard normal distributed with zero mean value .the weights are assumed to be positive with .since the distributions are symmetric , this is the most general form of the weights . in the casethat is not in any branch , i.e. for , we set . here , we use discrete time steps and normalize the time units such that . the time series consist of time values at .the time average of a function over the time series is defined as which depends on the length of the time series . if the time series are infinitely long , , we have here , we simply used , and , , .the correlation coefficient between two companies labeled and is the average over the product of the two normalized time series , if one views the numbers as the entries of a rectangular matrix , one has for the correlation matrix . as these averages depend on the length of the time series , we add the argument to the correlation matrix . within our modeloutlined above , one finds for infinitely long time series  , , thus , the matrix consists of square blocks on the diagonal of dimensions with off diagonal entries for branch , and a unit matrix for the companies which are in no branch .the diagonal entries are all unity .all other entries are zero : the correlation coefficients between companies which are not in any branch , those between companies belonging to different branches and those between a company which is in a branch and another one which is not .some issues related to the normalization of correlation matrices are discussed in [ app0 ] .if the time series have only a finite length , , it is obvious that the averages , and give neither zero nor unity , but some finite numbers .this describes the noise dressing of financial correlation matrices which was found in ref . in the framework of noh s model . due to the finiteness of the time series ,there is a purely random offset to every correlation coefficient , burying the true correlation coefficient which would be found if the time series were sufficiently long .as we aim at a analytical discussion later on , we formulate the noise dressing more quantitatively .we employ a standard result of mathematical statistics  : consider two time series of uncorrelated random numbers and with standard normal distributions and zero mean value .the average is a random number , following a gauss distribution centered at mean value unity with variance , if , and following a gauss distribution centered at mean value zero with variance , if .hence , we can write this average to leading order in as where the are uncorrelated random numbers , independent of , with standard normal distribution and zero mean value . this yields for the correlation coefficients the expression to leading order in .the first two terms stem from the averages and , the last two from the interference averages .there are four sets of random numbers , , , and , respectively .these sets are understood to be different from each other . to avoid a cumbersome notation , we do not add further indices to specify that .for example , the number is different from .as required , the limit of eq .( [ eq2.9 ] ) correctly yields eq .( [ eq2.7 ] ) .as shown in ref . , the spectral density is a well suited observable to study the noise dressing of empirical correlation matrices . in sec .[ sec3.1 ] , we numerically investigate the spectral density of our model as a function of the length of the time series .we support our findings by an analytical discussion in sec .[ sec3.2 ] .as the number of companies in the analysis  of real market data is several hundreds , we choose this number as for our simulation .we assume that there are industrial branches whose sizes and weights are listed in table [ tab_nohp2 ] .since noh  showed that the choice leads to spectral densities which are in agreement with the empirical data  , we use the same parameterization here .in addition , we have companies which are not in any branch .we simulate the correlation matrices for fourdifferent lengths of the time series , calculate the eigenvalues and work out the spectral densities .the results are shown in figure [ fig_cevl_t ] .as already observed by noh  , the first density for the shortest time series with resembles very much the densities of the empirical correlation matrices found in refs .there is a bulk  c|cccccc & 1 & 2 & 3 & 4 & 5 & 6 + & 4 & 8 & 16 & 32 & 64 & 128 + & 0.75 & 0.88 & 0.94 & 0.97 & 0.98 & 0.99 +  spectrum in the interval .moreover , a number of isolated peaks in the interval is seen .each of these large eigenvalues corresponds to an individual industrial branch . for increasing lengths ,the isolated peaks are stable , but the bulk spectrum separates into two groups of eigenvalues .the left group is roughly centered around , the right one around .we will see in the analytical discussion to be performed in sec .[ sec3.2 ] that the left group can clearly be associated with the true correlations due to the industrial branches , while the right group only represents noise around the trivial self correlation of the companies for .   for infinitely long timeseries , , the correlation matrix is block diagonal according to eq .( [ eq2.7 ] ) , and the spectrum of the eigenvalues is easily calculated . for every branch , we have degenerate eigenvalues and one numerically much larger eigenvalue which is for large size roughly proportional to . moreover , there are degenerate eigenvalues which are unity for the companies which are in no branch .thus , the spectral density is given by this formula helps to understand the last density for the longest time series with in figure [ fig_cevl_t ] .the first term corresponds to the left group , representing the true correlations .the second term of the large eigenvalues yield the isolated peaks of the spectra .the last term in the formula represents the noise peaked at unity .the eigenvectors can also be calculated in a straightforward manner for .the density ( [ eq3.1 ] ) is smeared out for finite time series , . to estimate the resulting noisy density to leading order in , we write the correlation matrix in the form   +  +   +   where can be read off from the expansion ( [ eq2.9 ] ) .we also write for the eigenvalues and for the eigenvectors .one quickly finds . since the elements of every are numbers depending on the weights and the sizes , every coefficient is , according to eq .( [ eq2.9 ] ) , a linear combination of the standard normal distributed random numbers , , and .hence , it is itself a gaussian distributed random number with zero mean value and a variance which is some function of the and the .we arrive at for to leading order in .we notice that , in this expansion to order , the gaussian random variables are uncorrelated .the effect of the smearing out , i.e. of the noise in the model , is marginal for the numerically large eigenvalues .their positions change the less , the larger the size .the effect of the noise is much stronger on the numerically smaller eigenvalues .moreover , their degeneracies are lifted and distributions develop around their mean values . for the eigenvalues which belong to the industrial branches and , hence , describe true correlations , the mean value is while it simply reads for the eigenvalues which are unity and which were generated by the time series not belonging to any branch .as the weights satisfy if , we always have this important relation implies that the centers of the distributions generated by the noise from the two set of degenerate eigenvalues are always different from each other .the distribution due to the true correlations will always be left of that which is due to trivial self correlation for . in the numerical simulation, we chose . assuming that the sizes are large , we find to leading order which explains why the distribution due to the true correlations is roughly centered around in figure [ fig_cevl_t ] .  to find an estimate for the noisy density , we argue phenomenologically . some further analytical propertiesare compiled in [ appa ] .we first replace the first term of eq .( [ eq3.1 ] ) with a gaussian distribution centered at , for its variance , we write where is a proper geometric average of those numbers in eq .( [ eq3.2 ] ) which stem from the industrial branches . as a weight factor, we sum over the multiplicities in the degenerate case ( [ eq3.1 ] ) , similarly , we could replace the third term of eq .( [ eq3.1 ] ) with a gaussian centered at . however , it is better to employ the fact the block of the companies which are in no branch belongs to a chiral random matrix ensemble  whose density is known  to have the algebraic form where are the largest and the smallest eigenvalues , respectively , given by both of them converge to for . outside the interval , the square root in eq .( [ eq3.9 ] ) is strictly imaginary .thus , the real part ensures that the function is zero outside the supporting interval . as a weight factor, we take the multiplicity in the degenerate case ( [ eq3.1 ] ) .we notice that , analogous to , there would be in principle a scale entering the density .it would result from a proper average of those numbers in eq .( [ eq3.2 ] ) which do not stem from the industrial branches .however , since we start out from normalized time series and since the weights do not contribute here , we expect . the second term of eq .( [ eq3.1 ] ) we leave unchanged , because we can ignore the shift in the positions of these largest eigenvalues . thus ,collecting everything , we find as the functions and are normalized to unity , the noisy density is correctly normalized to the total number of eigenvalues .we expect the formula ( [ eq3.11 ] ) to work particularly well for large . in figure [ fig_cevl3dp_t ]we fit it to the bulk part of the spectral density for which was presented in figure [ fig_cevl_t ] .the agreement is very good , the fit yields .in sec . [ sec4.1 ] , we introduce the power mapping and discuss its properties numerically .we give a qualitative analytical explanation in sec .[ sec4.2 ] . in sec .[ sec4.3 ] , we demonstrate how the power mapping can detect different correlation structures .finally , we define a measure for the noise in sec .[ sec4.4 ] .the true correlations buried under the noise become visible in the spectral density if the time series are long enough . thus ,if we found a procedure that is equivalent , in some sense , to a prolongation of the time series , we would be able to identify and quantify the noise in a given correlation matrix . in the sequel, we develop such a procedure , the power mapping .we map the correlation matrix to the matrix . here , is a positive number and the elements of are calculated according to the definition thus , the mapping preserves the sign of the matrix element and raises the modulus of it to the power .we notice that is the matrix of the powers of the elements of , but not the power of the matrix .this is crucial , because the spectra of and are , for , related in a simple way : if are the eigenvalues of , are the eigenvalues of .the spectrum of is much more complicated and depends on the eigenvalues and the eigenvectors of .however , depending on the numerical value chosen for , it allows one to suppress , in , certain elements of .this will now be demonstrated through a numerical study of the spectral densities .we simulate correlation matrices for companies from time series of length with the sizes and the weights given in table [ tab_nohp2 ] .we do so to make possible a comparison with figure [ fig_cevl_t ] , where the impact of increasing length for the time series was studied , starting from a correlation matrix with the same sizes and weights and for the same length . to improve the statistical significance and to better illustrate the effect of the power mapping , we simulate an ensemble of 25 such correlation matrices from time series of length .we choose powers and calculate , for a fixed , the 25 power mapped matrices from the 25 matrices of the ensemble .then , the individual spectra are evaluated and the density results as the ensemble average .all densities are shown in figure [ fig_evanp_t ] .the power mapping transforms the original density for into densities which show for intermediate values of two clearly separated peaks .the best separation is obtained near . for values beyond , the two peaks glue together again , and the separation is lost . what do these two peaks in the densities for represent ? comparison with figure [ fig_cevl_t ] shows that , indeed , the left peak in figure [ fig_evanp_t ] corresponds to the one for the true correlations in figure [ fig_cevl_t ] , while the right peak in figure [ fig_evanp_t ] corresponds to the one for the noise around the trivial self correlation of the companies for in figure [ fig_cevl_t ] .hence , we have found the desired procedure which roughly amounts to a prolongation of the time series .   to leading order in the length of the time series , the given by eq .( [ eq2.9 ] ) . to understand the effect of the power mapping , we distinguish three different cases in considering .first , we power map the diagonal elements . as eq .( [ eq2.9 ] ) shows , the vast majority of them matrix elements will be positive if is sufficiently large . thus , to simplify the discussion , we ignore the absolute value sign . to leading order in , we have second , we power map the off diagonal elements in the blocks of the industrial branches where but . for the same reason as in the previous case , we ignore the absolute value sign , and find to leading order in .third , we power map the elements outside the blocks , where and . since all kronecker s in eq .( [ eq2.9 ] ) are zero in this case , we obtain as the leading order in .we notice that the matrix elements in this third case will be positive or negative with equal probability .  in the first two cases ( [ eq4.11 ] ) and ( [ eq4.12 ] ), the powers contain a independent term plus a term which is of order . in the third case, however , there is no independent term , and the leading order of the whole expression is .thus , for , vanishes faster in the third case than in the first two terms . as the case ( [ eq4.13 ] )   +  +   +  +   +  +   +  comprises all elements outside the blocks , where and , we find that , for , the power mapped matrix is block diagonal to leading order .this explains why the power mapping has an effect which is comparable to a prolongation of the time series . at first sight, one would expect that the effect is the stronger , the larger the value of . however , this is not so , because the independent terms are different for the matrix elements in the blocks : in the first case ( [ eq4.11 ] ) of the diagonal elements , it is unity , but a number smaller than unity in the second case ( [ eq4.12 ] ) .the larger the value of , the smaller becomes the latter term .hence , the unity in the diagonal elements dominates more and more , driving the eigenvalues towards unity .the effect discussed above which is comparable to a prolongation of the time series , leads to the separation of the spectral densities into two peaks .if becomes too large , this is counteracted , and the two peaks merge again .consequently , there must be an optimal values for . in the numerical simulation we found that it is roughly .  for infinitely long time series , the power mapped correlation matrix is trivially block diagonal .the eigenvalues and the spectral density are found along lines similar to the ones leading to eq .( [ eq3.1 ] ) .we arrive at this correctly reduces to eq .( [ eq3.1 ] ) for . as already argued , the peaks due to the first term move towards the peak due to the third term if is made large .again , the peaks are smeared out if the length of the time series is finite .we show in [ appb ] that the noisy spectral density of the power mapped correlation matrix can be estimated by here , the parameters and collect the effects due to the smearing out and approximates the center of the gaussian due the true correlations in the first term of eq .( [ eq4.15 ] ) .the third term estimates the peaks due to the noise . as discussed in [ appb ] , these two contributions are affected differently as finite lengths of the time series are concerned .thus , it is useful to scale the variance with in the first term , but with in the third one .we consider three examples for different correlation structures .the parameters are listed in table [ tab_cm ] , and the corresponding correlation matrices are displayed in figure [ fig_cmp_t ] .we refer to the three examples as top , middle and bottom structure , respectively .top structure : the number of industrial branches is , companies are in no branch .c|ccccccccccc & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 & 11 + & 2 & 4 & 7 & 10 & 15 & 20 & 30 & 42 & 64 & 98 & 140 + & 0.5 & 0.75 & 0.85 & 0.9 & 0.93 & 0.95 & 0.96 & 0.97 & 0.984 & 0.989 & 0.99 +  middle structure : the number of industrial branches is , companies are in no branch .c|cccccc & 1 & 2 & 3 & 4 & 5 & 6 + & 4 & 8 & 16 & 32 & 64 & 128 + & 0.008 & 0.01 & 0.03 & 0.07 & 0.2 & 0.99 +  bottom structure : the number of industrial branches is , companies are in no branch .c|cc & 1 & 2 + & 4 & 104 + & 0.75 & 0.99 +  for all three structures , the total number of companies is and the length of the time series is .the top structure has many branches ( ) with relatively strong correlations and little noise , the middle structure has less branches ( ) with weaker correlations and more noise and , finally , the bottom structure has only branches with stronger correlations , and a considerable amount of noise .we notice that the weights are not chosen according to eq .( [ eq3.0 ] ) .rather , we adjusted them in such a way that the spectral densities for the middle and the bottom structure look as similar as possible .this can be seen in the left column of figure [ fig_evep_t ] .the top structure serves as a reference example .as the figure shows , the true correlations are so dominating that the spectral density is already almost separated in two substructures which are separated by a kink around .thus , the top structure is an idealizing example .we now apply the power mapping and calculate matrices according to eq .( [ eq4.1 ] ) .we choose the value which we identified as optimal in sec .[ sec4.1 ] .the resulting spectral densities are shown in the right column of figure [ fig_evep_t ] . for each of the three structures , we see the separation into two peaks , the left one corresponding to the true correlations , the right one to the noise . as expected , the spectral density for the top structure consisting of a big left peak and a small right peak differs considerably from the ones for the middle and the bottom structures where the left peaks are small and the right peaks big .this nicely confirms our expectation that the power mapping is capable of efficiently distinguishing the gross structures in the correlation matrices .it is now interesting to compare the spectral densities for the middle and the bottom structure : although the shape of densities in the left column of figure [ fig_evep_t ] can hardly be distinguished , the densities for the power mapped correlation matrices , shown in the right column , have similar , but distinguishable shapes . for both structures ,the left peak is small , the right one big .however , for the bottom structure , the right peak is narrower and its left shoulder is steeper .hence , the power mapping also gives useful information about the fine structure in the correlation matrices .+  +   +  +   +   using the estimate ( [ eq4.15 ] ) , we can obtain some further understanding of why is a good value for the power mapping .the two gaussian peaks which emerge from the power mapping , i.e. the one due to the true correlations and the one due to the noise , are separated if there is some space between the left side of the former and the right side of the latter . according to eq .( [ eq4.15 ] ) , the peaks should be separated if we have or , equivalently , if the function is positive . here , we used eq .( [ eq4.16 ] ) . in figure[ fig_hpl_t ] , we display the function for different power mapped correlation matrices starting from the original one used in sec .[ sec3.1 ] with .obviously , we may define the optimal value as the point where reaches its maximum which is given by the equation . indeed , this value is close to .the parameters and used in figure [ fig_hpl_t ] were obtained from the fitting procedure to be described in the following .as the dependence of on is very complicated , we do not go into a further analytical discussion .  to fit the spectral density of the power mapped correlation matrices , we use an ansatz motivated by the estimate ( [ eq4.15 ] ) .as we only want to fit the bulk part , we ignore the second term in the estimate ( [ eq4.15 ] ) due to the large eigenvalues and take the prefactors , , the standard deviations , and the mean value are fit parameters .this relatively high number of fit parameters is acceptable , because the data to be fitted have a clear structure .moreover , it should be emphasized that we are only interested in obtaining proper estimates by these fits . from the previous discussion, we expect that the fit should approximately give , first , and for the prefactors and , second , the scaling behavior and for the variances .we employ the parameters and instead of the parameters and , because , in practice , one will mostly have to deal with correlation matrices for one fixed length of the time series . as an example, we fit the ansatz ( [ eq4.21 ] ) to the spectral density of the power mapped correlation matrix discussed in sec .[ sec4.1 ] for .the result is shown in figure [ fig_rmpl12_t ] .we obtain for the prefactors , , for the standard deviations , and for the mean value .thus , we obtain the estimate for the number of companies which are not in any branch .this is in fair agreement with the true value .the sum deviates by about ten percent from the theoretical value .this is not so surprising , considering the fact that our ansatz ( [ eq4.21 ] ) can only be a rough approximation .hence , we suggest to use the ratio to characterize a correlation matrix .the larger , the more noise is present , the smaller , the stronger are the true correlations . in our example , we have , implying that noise and true correlations are more or less equally strong .advantageously , the ratio ( [ eq4.22 ] ) is not so sensitive to the total number of companies , implying that correlation matrices of different sizes can have the same . of course , the number , the dimension of the correlation matrix , is always known in an analysis . in the previous sec .[ sec4.3 ] , we discussed a correlation matrix , labeled `` middle structure '' , also involving companies which are in no branch .most of the correlations within the branches are so weak that the left peak in the middle of the right column in figure [ fig_evep_t ] is considerably smaller than the one in figure [ fig_rmpl12_t ] .the ratio makes the desired distinction : the overall strength of the true correlations is much weaker than in the present case .in addition , the standard deviation and the mean value yield information about the spreading of the weights and about their average .  finally , we test the quality of the scaling behavior and for the standard deviations . to this end , we generate correlation matrices for various lengths of the time series and power map them using , and .we fit the resulting spectral densities using the ansatz ( [ eq4.21 ] ) , extract the standard deviations and , and fit the latter to the expected scaling behavior and , respectively .the expectation is well confirmed for . in the case of ,the general trend is reproduced for the three values . for , however , the most interesting value , the agreement is good .this is encouraging , because the steps that led to the estimate ( [ eq4.15 ] ) involved various approximations . in any case , as already argued , the practical applicability of our ansatz ( [ eq4.21 ] ) is not affected by these scaling questions .+  as figure [ fig_tp1_t ] shows , both standard deviations and become smaller as is made larger while is kept fixed .this is why the power mapping can be viewed as effectively prolonging the time series : to obtain given values for the standard deviations , one can either keep and make the time series longer or make larger and leave unchanged .for example , compared to the correlation matrix with and , the noise is reduced by 60% when one goes to without changing and it is reduced by 75% when one applies the power mapping with while keeping .this is most important for applications , because time series for stocks with , say , are seldom available , while time series with are easily accessible .thus , power mapping can lead to better estimates for the risk and for related quantities such as the value at risk  .we developed a new method to identify and estimate the noise and , hence , also the strength of the true correlations in a financial correlation matrix .the essence of our approach is the power mapping .it suppresses those matrix elements which can be associated with the noise .our approach could be seen as an generalization and extension of shrinking techniques  used in contexts different from the present one .importantly , the spectral density changes drastically due to the suppression .the method itself yields a criterion to choose the optimal power . loosely speaking, our method can be interpreted as an effective prolongation of the time series .this feature makes it particularly suited for problems in which , for whatever reason , only relatively short time series are available .we expect that our approach , which is new to the best of our knowledge , can also be useful in other time series problems .although we presented our method in the framework of stocks , we emphasize that it is applicable to every correlation matrix of time series , independent of what these time series describe : in the context of finance , these can be stocks , interest rates , exchange rates or other risk factors , or , in other fields of science , completely different observables . moreover , advantageously , our approach does not involve further data processing or any other additional input .  since our method does not use the large eigenvalues due to the branches , it is a fundamentally different alternative to the technique of gopikrishnan et al . which is based on such information .our method automatically takes into account all branches , because all of them will contribute to the bulk of the spectral density .thus , even if a large eigenvalue is not large enough to lie outside the bulk , our method will not miss the contribution of the corresponding branch .we also demonstrated , how the power mapping allows one to distinguish different correlation structures .further work is in progress , in particular applications to empirical correlation matrices obtained from real market data .we thank l.a.n . amaral , p. gopikrishnan , v. plerou , b. rosenow , h.e .stanley ( boston university ) , t. rupp ( yale university ) , p. neu ( dresdner bank , frankfurt ) and r. dahlhaus ( universitt heidelberg ) for fruitful discussions . t.g . acknowledges support from the heisenberg foundation and thanks h.a .weidenmller and the max planck institut fr kernphysik ( heidelberg ) for hospitality during the initial stages of the work .empirical correlation matrices for the time series of stocks , say , are often constructed in the following way : to remove the drift in the data , one , first , takes the logarithms and , second , normalizes the resulting time series to zero mean and unit standard deviation , by construction , we have , independently of how the averaging procedure is defined and for all .we notice that this is not the case for our model in eq .( [ eq2.2 ] ) . using this normalization ,the correlation matrix is given by again , by construction , the diagonal elements are unity , , independent of the averaging procedure chosen and also for all .once more , this differs from the diagonal elements of the correlation matrix in our model , as can be seen from eq .( [ eq2.9 ] ) for .we now investigate the off diagonal matrix elements for . to leading order in the length of the time series , we assume where , and , are constants .a straightforward calculation then yields to leading order in .we assume that the variances for infinite are non zero , i.e.  .as required , we have . for , the forms of the expansions to leading order in eqs .( [ eq2.9 ] ) and ( [ eq0.4 ] ) coincide . in the present discussion , we have not explicitly incorporated a structure due to branches .such information would appear , for example , in the parameters .  as compared to the correlation matrices resulting from the model in eq .( [ eq2.2 ] ) , the difference in the normalization of the diagonal elements in eq .( [ eq0.2 ] ) has only a marginal and negligible influence on the eigenvalues and the spectral density , because there are many more off diagonal matrix elements than diagonal ones .this is also clearly borne out in noh s  and in our numerical results .moreover , there is no structural difference for the off diagonal matrix elements .hence , we are convinced that the results of the present work carry over to models involving a normalization of the kind ( [ eq0.1 ] ) .our main reason to work with the normalization in noh s model is the fact that the limiting case for infinitely long time series can conveniently be handled analytically .we mention in passing that practitioners in banks and investment companies commonly work with time series having a typical length of 250 days . these time seriesare often exponentially weighted before the analysis .we do not employ such a weighting procedure , because we are interested in the question , how a information about correlations of longer time series can , to some extent and under proper conditions , be estimated from shorter time series .furthermore , a comment regarding the branches in the model is in order .an international portfolio is likely to contain globally operating , large companies .their activities directly affect various branches , but they are also under different economical influences in different countries . in such cases , it is common to add a further label country to the label , denoted in our case . generally speaking , it is often necessary to go from a one factor model of the type discussed in the present article to a two factor or multi factor model . from the viewpoint of a simulation ,this refined correlation structure implies the need for precise information about the weights which generalize our . on the other hand , not much changes from the viewpoint of data analysis , which was the main motivation of the present workin general , the spectral density for a given length of the time series can be written as the spectral average of the joint probability density function for the eigenvalues of the correlation matrix , p_t(\lambda(t ) )      \sum_{k=1}^k \delta\left(\lambda-\lambda_k(t)\right ) \ , \label{eqa.1}\end{aligned}\ ] ] where $ ] stands for the product of all differentials . in our notation , we distinguish the argument of the density and the eigenvalues of the correlation matrix which are always written with their argument . in particular , eq .( [ eqa.1 ] ) is also valid for and we have p_\infty(\lambda(\infty ) )      \sum_{k=1}^k \delta\left(\lambda-\lambda_k(\infty)\right ) \ .\label{eqa.2}\end{aligned}\ ] ] as a consequence of eq .( [ eq3.2 ] ) , the joint probability densities for finite and infinite can , to leading order , be related according to p_\infty(\lambda(\infty ) )           \int d[a ] \prod_{l=1}^k g(a_l,1 )              \nonumber\\         & & \quad \prod_{k=1}^k              \delta\left(\lambda_k(t)-\lambda_k(\infty )                   -\frac{v_k}{\sqrt{t}}a_k\right )                      \nonumber\\         & = & \int d[\lambda(\infty ) ] p_\infty(\lambda(\infty ) )              \nonumber\\         & & \quad       \prod_{k=1}^k g(\lambda_k(t)-\lambda_k(\infty),v_k^2/t ) \ . \label{eqa.3}\end{aligned}\ ] ] here , is the gaussian depending on the variable , centered at zero , with variance .we plug eq .( [ eqa.3 ] ) into eq .( [ eqa.1 ] ) , do the integrals over and find p_\infty(\lambda(\infty ) )           \sum_{k=1}^k g(\lambda-\lambda_k(\infty),v_k^2/t )                 \nonumber\\         & = & \sum_{k=1}^k            \int_{-\infty}^{+\infty } d\lambda^\prime           g(\lambda^\prime-\lambda , v_k^2/t )                 \nonumber\\         & & \quad           \int d[\lambda(\infty ) ] p_\infty(\lambda(\infty ) )           \delta\left(\lambda^\prime-\lambda_k(\infty)\right ) \ . \label{eqa.4}\end{aligned}\ ] ] in the second step , we inserted the integration over using a function .since we may assume that the joint probability density is invariant under the exchange of its arguments , we can employ eq .( [ eqa.2 ] ) and integrate over the eigenvalues .we arrive at where we defined the average over the gaussians .the parameters are functions of the sizes and the weights .thus , to leading order , we can obtain the spectral density for finite from that for infinite by convoluting the latter with a superposition of gaussians .formula ( [ eqa.5 ] ) is valid for every spectral density . we now apply it to the spectral density ( [ eq3.1 ] ) and obtain as the second term contains the large , widely spaced eigenvalues , we neglect the smearing out there . with this additional approximation ,the result ( [ eqa.6 ] ) is valid to leading order .it gives an analytical motivation for the estimate ( [ eq3.11 ] ) .the crucial effect of the power mapping is the preservation of the block structure in up to order .this is evident from eqs .( [ eq4.11 ] ) and ( [ eq4.12 ] ) .only the inclusion of the order destroys this block structure as seen in eq .( [ eq4.13 ] ) . hence , to understand the noisy spectral density up to order , we can apply the methods of [ appa ] individually to those blocks which contain the companies of one branch .a modification occurs for the block collecting the companies which do not belong to a branch . according to eq .( [ eq4.11 ] ) , this block is up to order still a diagonal matrix , implying that the eigenvalues equal the diagonal elements .since we have , they are given by .on the other hand , employing a line of reasoning similar to the one in sec .[ sec3.2 ] , the eigenvalues of the blocks corresponding to a branch will have have the form where are independent gaussian distributed variables with zero mean and unit variance .the parameters result from a superposition of terms of order unity .thus , the gaussian smearing out will be stronger , roughly by a factor , for the eigenvalues of the blocks corresponding to a branch than for the eigenvalues which do not belong to a branch .if we make the assumption that the sizes are large , we can neglect the smearing out of the latter eigenvalues .we emphasize that this is an additional approximation which is not motivated by the asymptotic expansion in . under this assumption, we obtain from eq .( [ eq4.14 ] ) to leading order the estimate since the second term involves the large eigenvalues outside the bulk whose spacing is large compared to the smearing out , we may assume that the corresponding functions are not affected , either . to find an estimate to leading order , we must apply the methods of [ appa ] to the entire matrix , because the block structure is destroyed .we replace eq .( [ eqb.1 ] ) with as we are only interested in a qualitative discussion , we make the further assumption that the are independent gaussian variables with zero mean and unit variance .the smearing out to order will not affect the first term of eq .( [ eqb.2 ] ) , because it is already of order . however , we can not neglect it in the third term , because the parameters are now of order .we obtain replacing each of the sums over the functions and with one gaussian , we arrive at the estimate ( [ eq4.15 ] ) .

Abstract: financial correlation matrices measure the unsystematic correlations between stocks . such information is important for risk management . the correlation matrices are known to be `` noise dressed '' . we develop a new and alternative method to estimate this noise . to this end , we simulate certain time series and random matrices which can model financial correlations . with our approach , different correlation structures buried under this noise can be detected . moreover , we introduce a measure for the relation between noise and correlations . our method is based on a power mapping which efficiently suppresses the noise . neither further data processing nor additional input is needed .


the empirical analysis of interactions between the elements of a complex system is fundamental to understand both the collective structures and the basic rules inducing the emergent behavior of complex systems .the monitoring of several complex systems nowadays produces large sets of multivariate data .examples of these sets of data are present in physical , biological and economic systems and their analysis is an important and challenging task in the investigation of complex systems .many efforts have been done in the analysis of multivariate data series and most of them focus on the study of pair cross - correlations .the analysis of cross - correlation is precious in order to elicit the emergence of collective structures from multivariate data .classical spectral methods such as the principal component analysis , recent related techniques based on concepts of random matrix theory , hierarchical clustering , factor analysis and graph theory are fruitful approaches to the analysis of correlations among elements of complex systems elicited by multivariate data .cross - correlations estimated from real data are unavoidably affected by the statistical uncertainty due to the finite size of the sample . in most cases ,the length of data is unavoidably limited whereas in other cases the length of data needs to be limited to avoid that sizable non - stationary effects might introduce large errors in the estimation of correlations .cross - correlations might also be affected by noise due to measurement errors and to the interaction of the system with the environment . in order to at least partially overcome these problems , it is advisable to select statistically reliable information from the correlation matrix .we address the selection of the most statistically reliable part of the correlation matrix with the locution _ filtering of the correlation matrix_.  several techniques have been proposed in the literature in order to filter out information from the correlation matrix and therefore it is important to have at hand a method for comparing the performance of such different techniques in a quantitative way .  in this paper , we propose to measure the performance of filtering procedures by using the kullback - leibler distance which is a measure of distance between probability distributions and it is widely used in information theory ( see for instance ) .specifically , for multivariate gaussian variables , we explicitly compute the analytical form of the kullback - leibler distance and we show how it depends on the correlation matrices of the considered sets of data or of filtered versions of them . under the same assumptions we analytically obtain the expected values of the kullback - leibler distance between the correlation matrix of a multivariate model and a sample correlation matrix obtained with the pearson estimator from a finite set of data .one of our key results is that these expected values are model independent .this result shows that the kullback - leibler distance is very good in quantifying the amount of information present in a sample correlation matrix with respect to an hypothetical reference model also in the cases when the specific nature of the model is unknown .we are also able to compute the expected value of the kullback - leibler distance between two distinct samples of the correlation matrix obtained from the same random source .this last quantity is very useful in quantifying the stability associated with any sample estimation and specifically with the stability of the correlation matrices obtained from filtering procedures .we show the effectiveness of the use of the kullback - leibler distance in comparing data and models and in assessing the stability of the estimation of the sample correlation matrix by investigating four different filtering methods .two of them are based on spectral analysis , while the other two are generated by hierarchical clustering procedures .a good filtered correlation matrix is supposed to be informative about the sample correlation matrix and , at the same time , to be statistically more robust than the sample matrix itself with respect to statistical uncertainty . in our investigationwe consider in a quantitative way both these aspects .the paper is organized as follows . in section [ kullbacktheor ]we present the analytical results of the expected values of the kullback - leibler distance and we show how the kullback - leibler distance can be used as an estimator of the goodness of filtering procedures . in section [ methods ] we describe the four filtering procedures that we quantitatively compare in section [ comparison ] both by investigating simulations and real data .finally , in section [ conclusions ] we draw our conclusions .the kullback - leibler distance ( see for instance ) or _ mutual entropy _ is a measure of the distance between two probability densities , say and , which is defined as ,\ ] ] where  ] .the factor and are independent identically distributed random variables with zero mean and unit variance . in our simulations , the factors  and the idiosyncratic noises are gaussian random variables . in the first artificial system that we consider here, elements are grouped in orthogonal clusters . in terms of factor models , this orthogonal grouping of elementsis expressed by the fact that elements belonging to different clusters depend on different ( independent ) factors , i.e. if belongs to the group then .the dimension of groups is heterogeneous to mimic typical conditions observed in some real systems .specifically the number of elements belonging to each group ranges from a minimum of 3 elements to a maximum of 17 .the other artificial system that we have considered is described by a hnfm with factors .this empirically based model has been introduced in ref .we have chosen these two models because they are conceptually very different one from the other .in fact , in the hnfm elements can not be straightforwardly divided in groups because they depend on factors in a nested hierarchical way whereas in the other model the groups of elements are clearly distinguished because elements belonging to different groups depend on different and mutually independent factors .roughly speaking we can say that the block diagonal model describes a `` separable '' system whereas the hnfm represents a `` nested '' system . in a first analysis , both the considered factor models are degenerate models , i.e. the coefficient , which expresses the dependence of the element on the factor in the model of eq .( [ facgen ] ) , is only depending on the factor and not on the element .it is to notice that by applying either the alca or the slca to the correlation matrix of the two considered models one obtains back the correlation matrix of the models .this fact is due to the degeneracy of the models and it gives a certain advantage to hierarchical clustering procedures with respect to spectral techniques in reconstructing the true correlation matrix of these systems .in fact both the considered spectral techniques can not reconstruct the true correlation matrix of the system when applied to itself .this is the first reason why we have decided to perform other simulations of the systems by removing the degeneracy from models .the second reason is that the true correlation matrix of the system is in general unknown for real data : we have only one correlation matrix obtained from a single realization of the system with finite time series length .accordingly , we have decided to perform one single realization , say , with length of data series of each model and we have assumed that the correlation matrix of this single realization of each model represents the true correlation matrix of the corresponding system .this approach removes the degeneracy of the -parameters of models and at the same time allows to treat models in a way more similar to the one used for real data . in order to test the stability of filtering procedures with respect to statistical uncertainty ( as discussed in subsection [ samplesfilt ] ) , we have constructed bootstrap replicas of the single realization of each model .the bootstrap approach has the advantage that it does not require to make assumptions about the data distribution . instead of the bootstrap approach , in order to obtain different realizations of the non degenerate systems .the cholesky decomposition approach allows to construct mutually independent realizations of the system .however results obtained with the cholesky decomposition are in complete agreement with results obtained by using the bootstrap technique that we report in the paper .it is also to notice that by using the cholesky decomposition to perform simulations it is necessary to know the data distribution ( e.g. gaussian or student - t ) , whereas the bootstrap approach does not require to make assumptions about such distribution . ]we have simulated independent sets of data for the artificial systems described by the degenerate models and we have constructed bootstrap replicas of the empirical data .we have also considered 1000 bootstrap replicas of the single realization with series length of both the artificial systems , in order to treat the models more similarly to real data .we have applied all the filtering procedures described above to the correlation matrix of each simulation or replica of the artificial systems and to each replica of the real systems .therefore , we have obtained four filtered correlation matrices that we indicate with associated with each realization or replica of the systems .the label in stands for alca , slca , b and s depending on the filtering procedure .   the first question we want to ask is which filtering procedure performs better in detecting the correlation matrix of the model .we can ask this question only for the simulations where we know the model correlation matrix used to generate the data . in order to evaluate the ability of filtering procedures in reconstructing the correlation matrix of the model , we have evaluated the average kullback - leibler distance between the correlation matrix of the model and the correlation matrix filtered from the samples .averages have been performed over realizations of the models .the smaller the larger is the amount of information about the model that is detected by the filtered matrix .in tables i and ii we distinguish between degenerate models that we indicate with `` block diagonal '' and `` hnfm '' and non degenerate models that we indicate with `` block diagonal ( n.d . ) '' and `` hnfm ( n.d . ) '' . in tablei we report results obtained for all the considered models when the length of simulated normally distributed time series is . in the table , we observe that the alca outperforms all the other filtering procedures both for degenerate and non degenerate models .it is also to notice that the performance of slca is better than both the spectral filtering procedures for all the models with the exception of the non degenerate block diagonal model .such a good performance of hierarchical clustering filtering procedures was expected for the degenerate models .indeed , as we have discussed above , such models give a certain advantage to hierarchical clustering filtering procedures because of the degeneracy of coefficients .the fact that alca outperforms all the other filtering procedures also in the case of non degenerate models can be explained by taking into account both the length of data series and the way in which model degeneracy has been removed .the correlation matrix of the non degenerate models is by construction the correlation matrix of a single realization of the corresponding degenerate models with series length .this fact implies that the dispersion of the non degenerate correlations from the corresponding values in the degenerate model is of the order of . in tablei , the length of simulated data series is also , i.e. .this fact implies that the statistical uncertainty associated with the sample correlations is of the order .this value is equal to , implying that for series length the non degeneracy of model parameters is of the same order of the statistical uncertainty . in other words , details about specific correlation valuescan not be distinguished from statistical uncertainty for such short data series .only the global structure of the correlation matrix is important and hierarchical clustering procedures results to be more capable than spectral techniques in reconstructing the correlation structure of the models . in order to better understand the effect of the non degeneracy of model parameters on the ability of filtering procedures in reconstructing the model , we consider also a case with time series of length longer than in the prevoius case .specifically , in table ii we report results obtained for time series of length , which is ten times the length considered in table i. in the case of , we continue to observe a better performance of hierarchical clustering filtering procedures and in particular of alca with respect to spectral techniques for the degenerate models .this fact was expected because of the degeneracy of the models . however , in table ii we observe that the spectral technique producing as result of the filtering outperforms hierarchical clustering procedures for the non degenerate models .the method producing provides a result which is of the same order than for the block diagonal ( n.d . ) model whereas still underperform with respect to both hierarchical clustering procedures for the hnfm ( n.d . ) .the success of can be explained by the fact that for the statistical uncertainty of sample correlations is of the order which is smaller than .therefore , for the non degeneracy of models becomes relevant as compared with the statistical uncertainty affecting sample correlations and spectral techniques result to be more capable than hierarchical clustering in taking into account such non degeneracy .this aspect is related to the fact that alca and slca are filtering procedures characterized by free parameters whereas spectral methods have a variable number of free parameters which is scaling as when t tends to infinity .  in summary, we have shown that hierarchical clustering procedures better reconstruct the degenerate models both for short and long time series , whereas for the non degenerate models the length of data series becomes relevant in the comparison .specifically , for short time series ( ) , such that the statistical uncertainty of correlations hides the heterogeneity of model parameters , we have observed that hierarchical clustering procedures , and in particular the alca , outperform spectral techniques . on the contrary , for data series long enough ( ) that the heterogeneity of model parameters is relevant with respect to the statistical uncertainty of sample correlations , spectral procedures result typically to be more efficient than hierarchical clustering procedures in reconstructing the correlation matrix of models . & & & + block diagonal & & & & + hnfm & & & & + block diagonal ( n.d . ) & & & & + hnfm ( n.d . ) & & & & +   & & & + block diagonal & & & & + hnfm & & & & + block diagonal ( n.d . ) & & & & + hnfm ( n.d . ) & & & & +   in this subsection we quantify the amount of information that different filtering procedures preserve when applied to sample correlation matrices .this is important in all those real cases when one does not know the model correlation matrix .moreover we investigate the stability of the filtered correlation matrices with respect to different realizations of the process .we use two quantities in order to evaluate the performance of the filtering procedures . the first quantity that we have measured is the kullback - leibler distance between the correlation matrix of the -th sample and the filtered correlation matrix obtained by applying one of the filtering procedure to . is a measure of the information about that is stored in : the smaller , the larger is the amount of information about which is retained in the filtered matrix .the second quantity that we have considered is the kullback - leibler distance between two filtered matrices and obtained by applying the same filtering procedure to two different simulations ( or replicas ) and of the system . measures the statistical robustness of filtered matrices .the smaller , the greater is the stability of the filtering procedure with respect to the statistical uncertainty . in our estimations , we have averaged both and over the independent realizations or replicas of each system .axis ) against the amount of information about the correlation matrix that is retained in the filtered matrix ( axis ) .small values of and correspond to large stability and large amount of information preserved by the filtering respectively .the analysis is performed for a system of 100 elements divided in 12 orthogonal groups , each one depending on a specific gaussian factor , i.e. a block diagonal model .averages have been performed over 1000 independent realizations of the system and error bars correspond to one standard deviation.,scaledwidth=48.0% ]    in fig .[ infostabnohdeg ] , we show the results obtained for the block diagonal model with degenerate coefficients . in the figure we plot versus for all the described filtering procedures .averages that we indicate with the notation are performed over 1000 realizations and the series length is .error bars are one standard deviation . in all the cases presented in this paperwe have verified that the error interval indicated around the mean value of plus and minus one standard deviation includes approximately the 67% of the realizations used to compute the mean value . in the figure we also report the result of an hypothetic perfect filtering procedure , i.e. a filtering techniques which is able to recover exactly the model from each realization . in the figure, we indicate the corresponding correlation matrix with .such a filtering is maximally stable , because it recovers always the correlation matrix of the block diagonal factor model .accordingly , it is .this perfect filtering procedure removes completely the noise due to the finite length of data series and therefore the quantity .instead , it is equal to the expectation value of eq .( [ kullexpecs1sig ] ) , i.e. for and .note that we know the position in the plane of the optimal filtering even if we do not know the underlying model .this is due to the important characteristic that the mean value of the kullback - leibler distance is independent from the model correlation matrix ( at least in the multivariate gaussian case ) . in the figure, we observe that all the filtering procedures , except the slca , retain in average more information about the sample correlation matrix than the true model , i.e. for equal to , and .this fact indicates that these filtering procedures do not discard completely the noise present in the sample correlation matrix as a consequence of the finite length of time series .the slca algorithm is the only one which is retaining less information about the sample correlation matrix than the true model .moreover the slca is more stable than all the other filtering procedures .  in fig .[ infostabhnfmdeg ] , we show the results obtained by applying the considered filtering procedures to the system described by the hnfm with factors and with degenerate coefficients . in this case , only the alca is retaining more information about the sample correlation matrix than the true model .however it is interesting to note that both the spectral techniques are at the same time less informative about the sample correlation matrix and less stable than both hierarchical clustering filtering procedures . in other words , for the degenerate hnfm , hierarchical clustering procedures clearly outperform spectral techniques .this fact is a consequence of the pure hierarchical nature of the hnfm . indeed in ref . , we have shown that when the hierarchical features of a system are prominent with respect to the details of specific correlation values , the spectral procedures have problems in filtering information about the system .such problems do not appear for separable systems , like the block diagonal model considered above .  in summary , for both the considered models we observe that hierarchical clustering techniques produce more stable filtered correlation matrices than spectral procedures .concerning the information about the sample correlation matrix that is stored in the filtering we observe that results obtained for hierarchical clustering procedures are closer to the perfect filtering ( giving as output the true model of the system ) than spectral techniques . finally , it is to notice that the slca is the most stable within the considered filtering procedures .such an excellent performance of hierarchical clustering techniques can be due to the degenerate nature of models as discussed in the first part of this section .in fact when we remove the degeneracy of coefficients from the models we observe a different behavior of filtering procedures . in fig .[ infostabnoh ] we plot versus for the artificial system obtained from a single realization with time series length of the factor model with 12 orthogonal factors .this is equivalent to consider a block model with non degenerate coefficients . in fig .[ infostabhnfm ] , we plot results obtained for the single realization with length of time series of the hnfm with 23 factors . also in this case our investigation is equivalent to consider a hnfm with non degenerate coefficients .mean values and error bars in the figures correspond to the average and the standard deviation respectively both estimated over 1000 bootstrap replicas of the single realization of the models . from figures [ infostabnoh ] and [ infostabhnfm ]we note that in both the figures , we observe that none of the filtering procedures is more informative about the sample correlation matrix than the true correlation matrix of both the models , i.e. \simeq 3.54 ] and \simeq7.81 ] and \simeq27.2 ]  ] .in this appendix , we derive the expectation values of the kullback - leibler distance given in eq.s ( [ kullexpecsigs1]-[kullexpecs1s2 ] ) .we shall use two known results from the theory of wishart matrices .let us consider a multinormally distributed random vector of dimension with correlation matrix .let and be two sample correlation matrices obtained from two independent realizations of the system , and respectively both of length .the first result from the theory of wishart matrices that we shall use hereafter is that , is equal to plus the sum of the logarithms of mutually independent chi - squared random variables with degrees of freedom respectively ( see for instance ) .this fact implies that the expectation value of is .\ ] ] because =\gamma^{\prime}(p/2)/\gamma(p/2)+\log(2) ] as given in eq .( [ kullexpecsigs1 ] ) and the expectation value $ ] as given in eq .( [ kullexpecs1s2 ] ) .finally , by using results of eq.s ( [ logexpecsec ] ) and ( [ expectracesigmas1 ] ) we obtain the expectation value of as given in eq .( [ kullexpecs1sig ] ) .

Abstract: we show that the kullback - leibler distance is a good measure of the statistical uncertainty of correlation matrices estimated by using a finite set of data . for correlation matrices of multivariate gaussian variables we analytically determine the expected values of the kullback - leibler distance of a sample correlation matrix from a reference model and we show that the expected values are known also when the specific model is unknown . we propose to make use of the kullback - leibler distance to estimate the information extracted from a correlation matrix by correlation filtering procedures . we also show how to use this distance to measure the stability of filtering procedures with respect to statistical uncertainty . we explain the effectiveness of our method by comparing four filtering procedures , two of them being based on spectral analysis and the other two on hierarchical clustering . we compare these techniques as applied both to simulations of factor models and empirical data . we investigate the ability of these filtering procedures in recovering the correlation matrix of models from simulations . we discuss such an ability in terms of both the heterogeneity of model parameters and the length of data series . we also show that the two spectral techniques are typically more informative about the sample correlation matrix than techniques based on hierarchical clustering , whereas the latter are more stable with respect to statistical uncertainty .