providing quality of service ( qos ) guarantees in the form of limitations on the queueing delays or buffer violation probabilities is essential in many delay - sensitive wireless systems , e.g. , voice over ip ( voip ) , and wireless interactive and streaming video applications . due to the importance of such qos considerations , it is of significant interest to conduct an analysis and provide predictions for the performance levels of practical systems . in , effective capacity is proposed as a metric that can be employed to measure the performance in the presence of statistical qos limitations .effective capacity formulation uses the large deviations theory and incorporates the statistical qos constraints by capturing the rate of decay of the buffer occupancy probability for large queue lengths .hence , effective capacity can be regarded as the maximum throughput of a system operating under limitations on the buffer violation probability .recently , there has been much interest in the analysis of the effective capacity of fading channels ( see e.g. ,  ) in order to identify the performance of wireless systems operating under statistical queueing constraints .however , in almost all prior studies , the service rates of the queueing model ( or equivalently the instantaneous transmission rates over the wireless channel ) are assumed to be equal to the instantaneous capacity values although channel coding is performed using a finite block of symbols .moreover , transmissions are assumed to be reliable with no decoding errors .however , it is important to note that error - free communication at the rate of channel capacity is generally attained as the codeword length increases without bound .therefore , when finite blocklength codes are employed , transmission is necessarily performed in the presence of decoding errors and possibly at rates less than the channel capacity in order to have high reliability or equivalently low error probability .  in and , negi and goel addressed these considerations .they studied queueing and coding jointly and took explicitly into account decoding errors by considering the random coding exponents of error probabilities for rates less than the instantaneous channel capacity .for instance , in , they analyzed the maximization of the joint exponent of the decoding error and delay violation probability through the appropriate choice of the transmission rate for given delay bound and constant arrival rate .  in this paper , we also depart from the idealistic assumptions of communicating arbitrarily reliably at channel capacity but follow an approach different from that of and .we consider channel coding rates achievable with finite blocklength codes , and incorporate the decoding error probabilities and possible retransmission scenarios into the effective capacity formulation .this analysis is facilitated mainly by the recent results of polyanskiy , poor , and verd in where the authors identified an approximate maximal achievable rate expression for a given error probability in the finite blocklength regime .this expression can be regarded as a second - order asymptotic approximation of the channel coding rate at large but finite blocklength values .we note that and also studied channel coding and achievable error probabilities at finite blocklengths by analyzing the mutual information density and its statistics . in , an outage analysisis performed by using the distribution of the mutual information density . in ,a similar outage formulation is used to determine the optimal physical - layer reliability and to identify the maximum arq throughput . on the other hand ,neither of the above - mentioned papers have investigated the throughput in the finite blocklength regime when the systems operate under buffer constraints .our contributions in this paper can be summarized as follows .we first determine the effective throughput in the finite blocklength regime under constraints on the buffer violation probability .subsequently , we study the performance of different transmission strategies .initially , we consider a scenario in which the transmission rate is varied with the fading realizations while the error probability is kept fixed . the optimal error probability that maximizes the throughputis shown to be unique .we analyze the impact of the power adaptations .then , we investigate the case in which transmission rate is fixed and error probability varies over different transmission blocks . through numerical results , we analyze the interactions between the throughput , queueing constraints , error probabilities , blocklength , signal - to - noise ratio , and different transmission strategies .the remainder of the paper is organized as follows .section [ sec : channelmodel ] describes the fading channel model . in section [ sec : effcap - background ] , we provide preliminaries on the effective capacity as a measure of the throughput under statistical qos constraints .in section [ sec : effcap ] , we provide our results on the effective throughput in the finite blocklength regime .we conclude in section [ sec : conclusion ] . several proofsare relegated to the appendix .we consider a frequency - flat channel model , and assume that the fading coefficients stay fixed for a block of symbols and then change independently for the following block . under this block - fading assumption ,the channel input - output relation in one coherence block can be expressed as where and are the -dimensional , complex , channel input and output vectors , respectively .the input is subject to an average power constraint , i.e. , . is the complex - valued fading coefficient with finite second moment , i.e. , .we assume that both the receiver and transmitter have perfect channel side information ( csi ) and hence perfectly know the instantaneous realizations of the fading coefficients . however , the assumption of perfect csi at the transmitter is relaxed in section [ subsec : fixed - rate ] .finally , represents the gaussian noise vector whose components are independent and identically distributed ( i.i.d . ) , complex , circularly symmetric , gaussian random variables with mean zero and variance , i.e. , where denotes the identity matrix .in , wu and negi defined the effective capacity as the maximum constant arrival rate that a given service process can support in order to guarantee a statistical qos requirement specified by the qos exponent .if we define as the stationary queue length , then is the decay rate of the tail of the distribution of the queue length : therefore , for large , we have the following approximation for the buffer violation probability : hence , while larger corresponds to more strict qos constraints , smaller implies looser qos guarantees .similarly , if denotes the steady - state delay experienced in the buffer , then for large , where is determined by the arrival and service processes .therefore , effective capacity formulation provides the maximum constant arrival rates that can be supported by the time - varying wireless channel under the queue length constraint for large or the delay constraint for large .since the average arrival rate is equal to the average departure rate when the queue is in steady - state , effective capacity can also be seen as the maximum throughput in the presence of such constraints .the effective capacity is given by ( , , ) }\}}\end{gathered}\ ] ] where = \sum_{i=1}^{t}r_i ] denotes the vector formed by concatenating and .since the real and imaginary components are -dimensional vectors , the above channel model is a real gaussian channel with dimensional input and output and with channel gain .note that the real and imaginary noise components and are independent due to the assumption of the circular symmetry of the additive complex gaussian noise .for this channel , the coding rate ( in bits per channel uses ) in the block achieved with block error probability is where denotes the fading coefficient in the block .note that the expression in ( [ eq : codingratef - m - uses ] ) is obtained from that in ( [ eq : codingrateactual ] ) by replacing with , and with , which is the received signal - to - noise ratio in the block .now , the normalized rate in bits per channel use is approximately for large enough for which is negligible .henceforth , we assume that the instantaneous transmission rate in each coherence block of the fading channel is given by the expression in ( [ eq : codingratef ] ) .since the block error rate is , this rate is attained with probability .we assume that the receiver reliably detects the errors , employs a simple arq mechanism and sends a negative acknowledgement requesting the retransmission of the message in case of an erroneous reception .therefore , the data rate is effectively zero when error occurs . under this assumption ,the service rate ( in bits per channel uses ) in each block is with the above service rate characterization , we immediately obtain the following expression for the effective rate .  [prop : effrate ] the effective rate ( in bits per channel use ) at a given , error probability , blocklength , and qos exponent is where is given in ( [ eq : codingratef ] ) and the expectation is with respect to .  _ proof : _ we first note that the service rate is an i.i.d. process due to the facts that the fading process is i.i.d .in different blocks and the noise is an i.i.d .process leading to the independence of error events in different blocks .now , we have }\ } } \\ & = -\lim_{t\rightarrow\infty}\frac{1}{\theta t}\log_e{\mathbb{e}\{e^{-\theta \sum_{i=1}^{t}r_i}\ } } \\ & = -\lim_{t\rightarrow\infty}\frac{1}{\theta t}\log_e{\mathbb{e}\left\{\prod_{i=1}^{t } e^{-\theta r_i}\right\ } } \\ & = -\lim_{t\rightarrow\infty}\frac{1}{\theta t}\log_e{\prod_{i=1}^{t}\mathbb{e}\left\ { e^{-\theta r_i}\right\ } } \label{eq : effthrproof1 } \\ & = -\lim_{t\rightarrow\infty}\frac{1}{\theta t}\log_e{\left(\mathbb{e}\left\ { e^{-\theta r_i}\right\}\right)^t } \label{eq : effthrproof2 } \\ & = -\lim_{t\rightarrow\infty}\frac{1}{\theta t } t\log_e{\mathbb{e}\left\ { e^{-\theta r_i}\right\ } } \label{eq : effthrproof3 } \\ & = -\frac{1}{\theta } \log_e{\mathbb{e}\left\ { e^{-\theta r_i}\right\ } } \label{eq : effthrproof4 } \\ & = -\frac{1}{\theta } \log_e{\mathbb{e}_{|h|^2}\left\ { { \epsilon}+ ( 1-{\epsilon})e^{-\theta m{\bar{r}}_i } \right\ } } \label{eq : effthrproof5}\end{aligned}\ ] ] above , ( [ eq : effthrproof1 ] ) follows from the independence of the service process and ( [ eq : effthrproof2 ] ) is due to its being identically distributed .the expression inside the expectation in ( [ eq : effthrproof5 ] ) is obtained by evaluating the expected value of for fixed . finally ,( [ eq : effthr ] ) is obtained by normalizing ( [ eq : effthrproof5 ] ) by to have the effective rate in the units of bits per channel use , and by dropping the time index .  note that the effective rate is a function of the qos exponent , blocklength , signal - to - noise ratio and error probability . since we assume that coding is performed in each coherence interval , the blocklength is determined by the statistics of the fading process .the value of can be dictated by the application requirements and depends on the power budget .given the values of these parameters , the remaining parameter can be optimized to maximize the throughput .note that large implies that the transmitter attempts to transmit the data at a high rate but at the risk of more frequent errors and hence retransmissions . on the other hand ,if is small , the instantaneous transmission rate is low but the reliability of the transmissions is high .the following result shows that the optimal is unique .[ prop : convexity ] assume that the values of , , and are fixed .then , the function is strictly convex in and therefore the optimal value of that minimizes this function or equivalently maximizes the effective rate in ( [ eq : effthr ] ) is unique ._ proof _ : see appendix [ app : convexity ] .  note that the convexity result indicates that the optimal error probability can be easily found using standard convex optimization methods .the analysis and the resulting provide guidelines on the design of the channel codes and their strength .note further that the above result is shown for the case in which .if there are no qos constraints and hence , then we have the following corollary to proposition [ prop : effrate ] .  when , the effective capacity becomes where is given in ( [ eq : codingratef ] ) .note that the is the average transmission rate averaged over the fading states .below , we show that is a strictly concave function of .[ prop : convexity2 ] assume that the values of , and are fixed . then , the function is strictly concave in and therefore the optimal value of that maximizes this effective rate is unique ._ proof _ : see appendix [ app : convexity2 ] .  next , we provide numerical examples to illustrate the results .although the preceding analysis is applicable to any fading distribution with finite power , we consider a rayleigh fading channel in the numerical analysis , and assume that the fading power is exponentially distributed with unit mean ( i.e. , has the probability density function ) .  in figure[ fig : psi - eps ] , we plot as a function of the error probability in the rayleigh fading channel . in the figure , db and the blocklength .we provide curves for different values of the qos exponent . in all cases, we immediately observe the strict convexity of the curves , confirming the result in proposition [ prop : convexity ] . indeed , the optimal error probabilities that minimize are unique and are equal to for , respectively .  in fig .[ fig : r_e - eps ] , we plot the effective rate in ( [ eq : effthr ] ) as a function of the error probability .the other parameters are the same as in fig .[ fig : psi - eps ] .notice that we have also included in this figure the throughput curve for the case in which . note that if , the system does not have any queueing constraints . in proposition [ prop :convexity2 ] , we have shown that is a strictly concave function of and the optimal that maximizes is unique .the strict concavity is observed in fig .[ fig : r_e - eps ] .the optimal value of the error probability in the case of is . for ,the effective rate curves are not necessarily concave . in fig .[ fig : r_e - eps ] , we observe that these curves are quasiconcave and , as predicted by proposition [ prop : effrate ] , they are maximized at a unique .the optimal error probabilities for the cases in which are equal to the same ones obtained in fig .[ fig : psi - eps ] . at the optimal error probabilities , the maximum effective rate values are bits / channel use for , respectively .note that increasing leads to more stringent qos constraints , and we observe that the effective rate and hence the effective throughput diminishes as increases .this trend is also clearly seen in fig .[ fig : r_e - theta ] where we plot the maximum effective rate values ( i.e. , effective rate at the optimal error probability ) as a function of .another interesting analysis is the behavior of as a function of .this is depicted in fig .[ fig : eps - theta ] . here , we observe that as increases and therefore the qos limitations become more stringent , the value of initially decreases sharply .hence , the transmitter opts for more reliable but low - rate transmissions . on the other hand ,as increases beyond approximately 0.028 , the trend reverses and starts to increase .the transmitter increases the transmission rate at the cost of increased and hence more retransmissions .when exceeds 0.298 , starts decreasing again .note that for high values of , the effective rate is small .this small effective rate can be supported by low - rate transmissions .hence , when is high beyond a threshold , the transmitter chooses to transmit at low rates and keep the error probability and the number of retransmissions low as well .  in fig .[ fig : r_e - n ] , we plot the effective rate as a function of the blocklength for and .the solid - lined curves correspond to the effective rate in ( [ eq : effthr ] ) optimized over .the dashed curves correspond to the effective rate of the ideal model in which the service rate is equal to the instantaneous capacity , i.e. , and the error probability is assumed to be zero , i.e. , . here, we have interesting observations .when and the ideal model is considered , then the effective rate is , which is the ergodic capacity of the fading channel and is clearly independent of the blocklength . on the other hand , if the service rate is given by in ( [ eq : codingratef ] ) , the effective rate increases with blocklength as seen in fig .[ fig : r_e - n ] . in the presence of qos constraints , i.e. , when , we have stark differences . under the idealistic assumption of transmitting at the instantaneous capacity with no errors , we see from the behavior of the dashed curve for that effective rate decreases with increasing .the reason is that since is the coherence duration over which the fading state remains fixed , larger corresponds to slower fading and slow fading is detrimental for buffer - constrained systems . in a slow - fading scenario , deep - fading can be persistent causing long durations of low rate transmissions leading to buffer overflows . in the finite blocklength regime , as seen in the behavior of the solid - lined curve of the case of , there is a certain tradeoff . initially , increasing improves the performance as this allows the system to perform transmissions with longer codewords and to have higher transmission rates .however , if increases beyond a threshold , slowness of the fading starts to degrade the performance .  in all cases in fig .[ fig : r_e - n ] , the gap between the dashed and solid - lined curves diminishes as increases since the idealistic model becomes more accurate . on the other hand , for moderate values of ( e.g. , when ) , the idealistic assumptions lead to significant overestimations of the performance .finally , we provide numerical results for the optimal effective rate and optimal error probability as a function of in figs .[ fig : r_e - snr ] and [ fig : eps - snr ] , respectively , for and .we see that , for fixed , increasing the improves the throughput and also the reliability of the transmissions by lowering the error probabilities .heretofore , we have considered the scenario where the transmitter knows the fading coefficients and performs variable - rate transmission with the same average power in each coherence block of channel uses . in this section , we investigate the gains achieved by varying the transmission power as well with respect to fading .let us denote the power adaptation normalized by the noise power by . with this adaptation policy ,the transmission rate is which is obtained by replacing with in ( [ eq : codingratef ] ) .finding the optimal power adaptation policy that maximizes or the effective rate is in general a difficult task due to the facts that both the first and second terms on the right - hand side of ( [ eq : codingratef - pa ] ) are concave functions .hence , is neither concave or convex . for this reason, we resort to suboptimal strategies .one viable policy , , is the one that maximizes the effective rate when the service process is assumed to be equal to the instantaneous capacity with zero error probability , i.e. ,  is derived in and is given by where and is chosen such that the average long - term signal - to - noise ratio constraint , + is satisfied with equality .note that this policy is close to the optimal one when the blocklength is large and hence is close to and is close to zero .  in fig .[ fig : r_e - theta - var-3 ] , the optimal effective rate is plotted as a function of for both fixed- and variable - power cases . in the fixed power case , db in each coherence block .when power adaptation is employed , signal - to - noise ratio varies in each block while satisfying db . the improved performance with power control is observed in the figure .the analysis above has assumed that the transmitter has perfect knowledge of the fading coefficients and can perform variable - rate and/or variable - power transmissions in each coherence block . on the other hand , it is practically interesting to consider cases in which the transmitter does not know the channel and send the information at a fixed rate . additionally , the transmitter may prefer fixed - rate transmissions , even when it knows the channel , due to complexities in varying the transmission rate for each block . motivated by these considerations , we assume in this section that the transmitter sends the information at the fixed rate . under this assumption ,error probability varies with the fading realizations .the analysis in the previous sections have , on the other hand , considered the scenarios in which the error probability is fixed for all channel states .from ( [ eq : codingratef ] ) , which provides the fundamental tradeoff between the rate and error probability in the finite blocklength regime , we can easily see that the error probability for fixed is note that is a function of the fading magnitude , signal - to - noise ratio , and blocklength .the service rate ( in bits per channel uses ) is now it can also be immediately seen that for given , blocklength , qos exponent , and fixed - rate , the the effective rate in bits per channel use is which is essentially the same as in ( [ eq : effthr ] ) .the only difference is that we now have the rate fixed and error probability varying .similarly , when , we have it is instructive to investigate what is obtained as .we immediately see that leading to - function , i.e. , , and invoking the dominated convergence theorem .additionally , we implicitly assume that the random variable does not have a mass at and hence is a zero - probability event and this event does not affect the expectation . ] therefore , in the limit as , which is defined as the capacity with outage ( * ? ? ?* section 4.2.3 ) .therefore , in ( [ eq : r_e(0)-fixed ] ) can be seen as the outage capacity in the finite blocklength regime .furthermore , in ( [ eq : effth - f ] ) can be regarded as the generalization of such a throughput measure to the scenario with qos limitations .  in figs .[ fig : r_e - r ] [ fig : r_f - theta ] , we illustrate the numerical results . in fig . [fig : r_e - r ] , effective rate is given as a function of the fixed transmission rate .we observe that the effective rate curves are quasiconcave and moreover they are maximized at a unique value of .we also observe that the maximum value of the effective rate diminishes with increasing .this is more clearly seen in fig .[ fig : r_e - theta - fixed ] where the optimal effective rates ( optimized over ) are plotted as a function of . in this figure, we have curves for both fixed - rate and variable - rate transmissions .effective rate for the variable - rate transmission is computed by maximizing ( [ eq : effthr ] ) over .it is interesting to observe that fixed - rate transmissions perform worse than variable - rate transmissions for small values of .however , for , fixed - rate transmissions start outperforming . hence , for high enough values of , fixing the transmission rate and having the error probability vary in each block provide better performance than requiring the error probability to be fixed by varying the rate . finally , in fig .[ fig : r_f - theta ] , we note that as increases , the optimal fixed rate , which maximizes in ( [ eq : effth - f ] ) , diminishes .so far , we have assumed that the transmitter sends a single codeword  ] , is a strictly convex function of .  we now define which is also strictly convex as it can be immediately seen that for and .note that if either or , the coding rate becomes , leading to . since the nonnegative weighted sum of strictly convex functions is strictly convex and since the addition of a constant ( in the case of ) does not have an impact on the strict convexity, we immediately conclude that is strictly convex in , proving proposition [ prop : convexity ] .   the proof is similar to that of proposition [ prop : convexity ] in appendix [ app : convexity ] and will be kept brief .let s first consider the function where we define and .note that if either or , then and for all .next , we consider the case in which and , and therefore and . for already shown in .we provide a similar proof here for the sake of being complete and keep the discussion brief . ]the second derivative of with respect to is using similar arguments as in appendix [ app : convexity ] , we can easily see that for , . for , we can show , employing steps similar to those in ( [ eq : e>1/2_0])([eq : e>1/2_4 ] ) , that where . when , we have .since for all , is a strictly concave function of when and .as argued similarly in appendix [ app : convexity ] , since the nonnegative weighted sum of strictly concave functions is strictly concave and since the addition of a constant ( in the case of ) does not have an impact on the strict concavity , we conclude that is a strictly concave function of .   from the proof of proposition [ prop : convexity ] in appendix [ app : convexity ] , it immediately follows that is a strictly convex function of .then , is strictly convex due to the facts that is a strictly convex and increasing function of and the composition is strictly convex function when is a strictly convex function ( * ? ? ?* section 3.2.4 ) .then , strict convexity of follows from the arguments employed at the end of appendix [ app : convexity ] .       j. tang and x. zhang , quality - of - service driven power and rate adaptation for multichannel communications over wireless links , " _ ieee trans .wireless commun ._ , vol . 6 , no . 12 ,pp.4349 - 4360 , dec .j. tang and x. zhang , cross - layer - model based adaptive resource allocation for statistical qos guarantees in mobile wireless networks , " _ ieee trans .wireless commun ._ , vol . 7 , pp.2318 - 2328 , june 2008 .l. liu , p. parag , j. tang , w .- y .chen and j .- f .chamberland , `` resource allocation and quality of service evaluation for wireless communication systems using fluid models , '' _ ieee trans .inform . theory _ , vol .1767 - 1777 , may 2007    d. qiao , m.c .gursoy , and s. velipasalar , `` the impact of qos constraints on the energy efficiency of fixed - rate wireless transmissions , '' _ ieee trans .wireless commun .5957 - 5969 , dec . 2009 .d. qiao , m.c .gursoy , and s. velipasalar , `` energy efficiency of fixed - rate wireless transmissions under queueing constraints and channel uncertainty , '' proc . of the ieee global communications conference ( globecom ) , dec .2009 .               vs. the blocklength in the rayleigh fading channel . db and the qos exponent is .dashed curves correspond to the effective rate of the ideal model in which the service rate is equal to the instantaneous channel capacity and error probability is zero . ]         vs. in the rayleigh fading channel . and the blocklength is .the dashed curve is the effective rate in ( [ eq : effthr ] ) maximized over and the solid curve is the effective rate in ( [ eq : effthr - parallel ] ) maximized over . ]

Abstract: in this paper , wireless systems operating under queueing constraints in the form of limitations on the buffer violation probabilities are considered . the throughput under such constraints is captured by the effective capacity formulation . it is assumed that finite blocklength codes are employed for transmission . under this assumption , a recent result on the channel coding rate in the finite blocklength regime is incorporated into the analysis and the throughput achieved with such codes in the presence of queueing constraints and decoding errors is identified . performance of different transmission strategies ( e.g. , variable - rate , variable - power , and fixed - rate transmissions ) is studied . interactions between the throughput , queueing constraints , coding blocklength , decoding error probabilities , and signal - to - noise ratio are investigated and several conclusions with important practical implications are drawn . + _ index terms : _ buffer violation probability , coding rate , decoding error probability , effective capacity , fading channels , finite blocklength regime , quality of service constraints , variable - rate / variable - power / fixed - rate transmissions .  1.7


recently , device - to - device ( d2d ) communication as a promising technology of 5 g cellular networks , has been emerged to increase the spectral efficiency by reusing the same radio resources among multiple links .it is considered as a solution to implement proximity services among multiple devices , such as mobile social networks , public safety and location - based advertisement , or for video content delivery .d2d peer discovery is an important procedure to find potential d2d candidates to communicate with each other .several studies have focussed on different aspects of d2d discovery in recent years .particularly , long term evolution - advanced ( lte - a ) infrastructure is widely used in which , there is no need for extra designs and modifications in current cellular networks . in such systems ,peer discovery procedure is under fully control of a central base - station ( enodeb ) and is called centralized network - assisted d2d peer discovery . interference and power control schemes for d2d discovery in inband cellular by considering user densities , addressed in lte - a enhancements .  in current lte - a networks ,energy consumption aspects of d2d discovery mechanisms which are discussed in the third generation partnership project ( 3gpp ) are studied in .a probabilistic model and random access procedure in the lte - a system that discovers pairs of user equipments ( ues ) in a centralized manner have been also proposed in and , respectively .furthermore , a centrally controlled device discovery scheme tailored to the lte system is proposed in .this scheme consists of a comprehensive application layer procedure that enables the device discovery services , and a set of media access control ( mac ) layer enhancements that provides a resource request / allocation procedure for discovery transmissions . by utilizing sounding reference signal ( srs ) channel , which can be accessed by ues that are lte - compliant, a neighbor discovery with d2d channel estimation is proposed in .one approach to allocate radio resources to d2d peer discovery procedures is using lte - a uplink radio resources . in thisregard , physical uplink control channel ( pucch ) of cellular network is used to establish peer discovery signaling messages between bs and potentially d2d pairs .however it can cause inter carrier interference ( ici ) and also transmitted signaling messages need to be strong enough to make sure that the bs and d2d candidates can receive the messages , correctly .boosting transmit power of signaling messages symbols can increase the average transmit power of discovery signal , while maintaining low ici and cellular pucch reception performance .users also may use broadcasting to advertise their presence and service , to discover other devices , autonomously and continuously[11 , 12 ]  in this paper , we propose an uplink underlay network - assisted novel scheme for d2d discovery . a network - controlled signaling algorithm is proposed to exchange discovery messages a ) between user devices in potential d2d pairs , and b ) between d2d pairs and bs . in this scheme ,proximity users feedback their identity and channel information to bs in order to provide accurate estimation of the link quality between d2d pairs candidates .in contrast with the existing works in the literature in which the transmitted discovery messages assumed to be always successful in delivering to the respective receivers without considering cellular users influence , we propose a realistic network model based on the poisson point process ( ppp ) to include channel state information ( csi ) in d2d discovery process by considering imposed interference from cellular users on d2d pairs .we employ stochastic geometry to analyze the system performance in terms of the cumulative distribution function ( cdf ) of the experienced signal - to - interference ratio ( sir ) at d2d receivers and the expected number of needed time slots for d2d discovery processes in a multi - node scenario .the rest of paper is organized as follows . in sectionii , we first delineate our system model and discovery mechanism . then, we present ppp analysis in section iii .the analytic and simulation results are presented in section iv and section v concludes this paper .we consider an infinite cellular network as shown in fig .1 with randomly distributed user devices and bss within the network area .we define as a poisson point process ( ppp ) with density of , which determines the locations of the bss within the network area .we also define as another ppp with density of , which determines the locations of the ues .potential d2d pairs are sharing cellular users resources underlay uplink cellular infrastructure , hence , the mutual interference at d2d receivers is taken into account . the broadband connection is provided for the bss by central scheduler via wired links . for the notational simplicity and mathematical derivations , we focus on a typical random cell , termed representative cell , and analyze the performance of the proposed d2d discovery algorithm in terms of the transmitted signal s success probability and the expected number of required time slots , to satisfy the system minimum constraints. we neglect co - channel interference and neighboring cell users influence , for the sake of simplicity , however , the analysis can be applied for multi - cell scenarios , as well .all d2d communications are under fully control of the bs and they transmit signaling messages at the beginning of each time slot .the transmission scheme is time division multiple access ( tdma ) which implies , given a time for d2d discovery mechanisms , the whole time is divided into small portions called `` time slots '' and each user transmits its message at the beginning of each time slot , according to a transmission probability . for a given time slot, one d2d discovery message is allowed to pass at the beginning of this time slot , hence , if two users send their discovery message at the same time , collision occurs and therefore failed messages need to be retransmitted in the next time slots .we further assume that the system operates under interference - limited regime , therefore , the background noise is negligible in comparison with the experienced interference at the receivers .   in this subsection ,we propose new d2d discovery algorithm by considering the bs as the central management system . technically , to initiate a d2d link between two users ,some signaling messages need to be exchanged , through the network entities . in our proposed scheme ,since the bs is the central management system to exchange required signaling messages for d2d discovery scheme , the proposed scheme is centralized .we focus on a potential d2d pair ( , ) as a representative for all potential d2d pairs and describe the proposed d2d discovery mechanism by considering as transmitter and as its receiver . assuming that the users can detect other users in their proximity ( such a detection can be done by flashlinq mechanism proposed by lte - a for proximity services ), the proposed algorithm can be defined in five signaling steps as follows .  1.  user sends a request message to the bs to initiate a d2d link with user .2 .  after receiving s transmitted signal at bs , the bs forwards discovery message to the user and schedules the user to listen discovery messages transmitted by the user in its vicinity , and sends acknowledgment signal to user .user sends discovery message to user .4 .  after receiving discovery message at user , it measures the experienced signal - to - interference ratio ( sir ) and sends it back to the bs .bs receives measured sir and queues the d2d pair to initiate d2d communication ; if the received sir satisfy the system threshold , bs initiate users and to initiate d2d communication ; otherwise discovery process will fail and the d2d pair need to retransmit their discovery message in the next time slots .  in all the aforementioned steps ,when users send the signaling messages to the bs , the received signal at bs should satisfy the system sir threshold , likewise .in this section we provide the analysis for the proposed scheme by employing stochastic geometry to analyze the transmitted signals success probability . for the proposed system in sectionii , there are two key conditions to successfully deliver a discovery message from the transmitter to receiver .\a ) collision : when two users transmit their discovery message at the beginning of a given time slot at the same time , collision occurs and both users have to retransmit their discovery messages . denoting as the number of potential d2d pairs , the signal from user will successfully be delivered to its respective receiver , if all other pairs are in idle mode .each user transmit its discovery signal with a specific transmission probability .hence we define as the probability of successful signal transmit in a given time slot as where , is the transmission probability .now , we aim to maximize by seeking optimal transmission probability : where is the optimal transmission probability . hence , denoting as the optimal probability of transmitting a signal without collision , we have :   \b ) sir : for a given transmitter and its respective receiver , the received signal strength and consequently the sir level should meet the system design thresholds .this means that a transmitted signal from user can be successfully delivered to receiver , if the experienced sir at receiver is equal or greater than a threshold , i.e. , eq .( 4 ) also delineates the cumulative distribution function ( cdf ) of experienced sir at receiver .since all potential d2d pairs transmit their discovery message at the beginning of a given time slot , the parameter can be considered as the number of messages which are simultaneously transmitted by potential d2d users .now , we define success probability for a transmitted signal from transmitter to its respective receiver , by considering joint collision and sir satisfaction .this means that a transmitted signal from user will successfully be delivered to its respective receiver if a ) there is no collision in transmitting the signal and b ) the received signal at receiver satisfies the system sir threshold , i.e. , since the collision process and sir satisfaction are independent at each d2d pair , we can rewrite the eq .( 5 ) as now we aim to derive the closed form of the cdf for the experienced sir at the receiver of interest , , by employing stochastic geometry approach .the experienced sir at the receiver due to transmitted signal from user is where , is the standard path loss with exponent of and denotes the euclidean distance between transmitter and its respective receiver , . is total interference due to the transmitting nodes in set , which is the set of concurrent transmitting nodes .backslash in eq .( 7 ) implies that the node is excluded from transmitters set . and are the fading power coefficients with exponential distribution of mean one , corresponding to the channel gain between transmitter and its respective receiver , and the interferer , respectively .we consider receiver at origin and transmitting node with a fixed distance of from , which is the nearest user to .hence , all the interferer are in the outside of the circle of radius .now , by denoting as the total interference , i.e. , , and using campbell s theorem , the laplace transform of the interference can be defined as : where , , and is the gamma function .now , the final expression for the cdf of sir at receiver can be defined as by substituting equations ( 3 ) and ( 9 ) in eq .( 6 ) , the final expression for success probability can be derived .now , we define random variable denoting the number of successful transmissions in given time slots .the probability that there is at least one successful discovery signal is now , we define another system design parameter , as the minimum required success probability for requiring at least one time slot for a successful discovery message .we have : by substituting equations ( 3 ) and ( 9 ) in ( 6 ) and then eq . ( 6 ) in ( 10 ) and then ( 10 ) in ( 11 ) ,respectively , and some simple manipulations , the maximum number of required time slots for the proposed d2d discovery mechanism can be defined as where ,  is the requirement of the stochastic geometry approach addressed in .in this section , we provide monte - carlo simulations to evaluate system performance .we focus on an isolated single cell of the voronoi tessellation network ( fig . 1 ) and neglect inter - cell interference .we assume standard microcell channel model by considering log - normal shadow fading effect .the rest of simulation parameters are summarized in table 1 . in the wireless d2d network ,described in section ii , users initiate for d2d communication with the nearest neighbor in the vicinity .we choose potential d2d pairs , according to given distance threshold and label them by ids as .the key parameter in this system is collision detection in simultaneous transmissions by d2d pairs in the beginning of each time slot . as we analyzed in section iii , the optimal transmission probability in the beginning of each time slotis given by ( eq .since the channel access model is tdma , to simulate collision detection , at the beginning of each time slot , each user throws its dice to pick up a number between [ 1 ] , if the chosen number is equal to the respective d2d pair s i d , then , the d2d transmitter will be authorized to transmit its discovery signal to its respective receiver . andif the received signal in the receiver can satisfies system thresholds ( ) , d2d pair succeed in the collision process and will proceed to the next step of the discovery algorithm .  in what follows ,we first describe the analytic results and possible tradeoffs in fig . 1 , 2 , and 3 and then describe the simulation results for delineated system model in section ii .2 demonstrates the impact of on the cdf of sir for different system sir thresholds ( ) . as can be seen in this figure , by increasing the density of users ( ) within the cell area , which corresponds to more potential d2d pairs within the cell area , the probability of successful discovery messages transmissions with respect to system sir threshold , decreases , because , the experienced interference at d2d receivers increases . in other hand, increasing system sir thresholds which corresponds to guaranteeing higher d2d link quality , leads to decreasing the probability of successful discovery messages deliveries .therefore , there is a trade off between d2d link quality and successful discovery messages deliveries .i.e. , guaranteeing higher d2d link quality , the lower is the probability of successful delivery of discovery messages and guaranteeing the lower d2d link quality , the higher is the successful delivery of discovery message .3 shows the impact of system sir threshold on the cdf of sir .similar explanations for fig .2 are valid for fig .3 .  fig .4 demonstrates the main success probability defined in eq .( 6 ) . in this figure, we consider limited number of d2d pairs ( ) with the impact of joint collision detection and sir satisfaction .as can be seen from this figure , collision detection has a major impact on the success probability , due to simultaneous transmission at the beginning of time slots . by increasing the number of d2d pairs in a given time slot , to transmit their desired discovery message ,the impact of collision detection is more accented .5 shows the cdf of the minimum required number of time slots for successful d2d discovery process . as expected , by increasing the number of potential d2d pairs in a specific and , the required time slots increases due to more collisions in channel access . in other hand , by increasing the density of the network , since d2d pairs are sharing uplink resources of the cellular users , there is high interference on d2d receivers , which leads to more fails in d2d discovery messages deliveries .6 demonstrates the number of required time slots for different number of d2d pairs as derived in eq .similar explanations for fig .5 are also valid for fig .in this paper , we proposed a novel scheme for d2d discovery by employing a centralized signaling algorithm for exchanging discovery messages between network entities .we used stochastic geometry to analyze and evaluate the realistic performance of the proposed scheme . for this system , proposing new efficient collision avoidance algorithms in tdma channel access can promises even high reliability to deliver d2d discovery messages .a. prasad , a. kunz , g. velev , k. samdanis , and j. song , `` energy - efficient d2d discovery for proximity services in 3gpp lte - advanced networks : prose discovery mechanisms , '' ieee vehicular technology magazine , vol . 9 , pp . 40 - 50 , 2014 .s. jung and s. chang , `` a discovery scheme for device - to - device communications in synchronous distributed networks , '' _ ieee international conference on advanced communication technology _ , pp .815 - 819 , 2014 .x. wu , s. tavildar , s. shakkottai , t. richardson , j. li , r. laroia and a. jovicic , `` flashlinq : a synchronous distributed scheduler for peer - to - peer ad hoc networks , '' _ ieee / acm trans . on networking _( ton ) , vol .21 , pp.1215 - 1228 , 2013 .h. elsawy ; a. sultan - salem ; m. s. alouini ; m. z. win , `` modeling and analysis of cellular networks using stochastic geometry : a tutorial , '' _ ieee communications surveys and tutorials _ , vol.pp , no.99 , pp.1 - 1 .

Abstract: tremendous growing demand for high data rate services such as video , gaming and social networking in wireless cellular systems , attracted researchers attention to focus on developing proximity services . in this regard , device - to - device ( d2d ) communications as a promising technology for future cellular systems , plays crucial rule . the key factor in d2d communication is providing efficient peer discovery mechanisms in ultra dense networks . in this paper , we propose a centralized d2d discovery scheme by employing a signaling algorithm to exchange d2d discovery messages between network entities . in this system , potential d2d pairs share uplink cellular users resources with collision detection , to initiate a d2d links . stochastic geometry is used to analyze system performance in terms of success probability of the transmitted signal and minimum required time slots for the proposed discovery scheme . extensive simulations are used to evaluate the proposed system performance .   3cm(4.1cm,-12 cm )  d2d , discovery , tdma , stochastic geometry , signaling algorithm .


in wireless communications due to broadcast medium , transmissions by a user cause interference to other users . this reduces the transmission rate and/or increases the transmission error of the other users .therefore each user ( transmitter ) aims to use resources like , power and spectrum efficiently to improve its performance which may conflict with the goals of the other users .thus , in this paper , we present a game theoretic approach to obtain optimal power allocations that achieve an equilibrium among the different transmitters .we consider a single wireless channel which is being shared by multiple transmitter - receiver pairs .it is modeled as an interference channel .its power allocation in non - game theoretic setup has been studied in - and via game theory in - .power allocation for parallel gaussian interference channels is studied in .convergence of iterated best response for computing nash equilibrium under some conditions on the channel gains is studied for single antenna systems in , . in ,convergence of iterative water - filling for parallel gaussian interference channels with multiple antennas is studied .different algorithms are presented in , and to compute a nash equilibrium for parallel gaussian interference channels . in , we presented an algorithm to compute a nash equilibrium for a stochastic game on gaussian interference channels under channel conditions weaker those in .  in general , in a wireless communication system, a user may not have a knowledge about the other users channel states and their power policies .in such a setup , one needs distributed algorithms which each user can use to achieve optimal policies that require less information about the other users .online learning algorithms are such a class of algorithms .some of these algorithms , for example , fictitious play , are partially distributed algorithms that require some knowledge about other users strategies to find an equilibrium of the system . on the other hand , there exist fully distributed learning algorithms which do not need any information about the other users strategies or payoffs to find an equilibrium .we describe the prior work in the literature of wireless communications that has considered learning for optimal power allocation .the problem of minimizing energy consumption in point - to - point communication with delay constraints in stochastic and unknown traffic and channel conditions is considered in .this problem is modeled as a markov decision process and solved using online reinforcement learning .  in ,orthogonal multiple access channels are considered .the problem of power allocation is modeled as a non - cooperative potential game and distributed learning algorithms are proposed .a learning algorithm for finding a nash equilibrium for a multiple - input and multiple - output multiple access channel is proposed in .the problem of minimizing the total transmit power of a parallel gaussian interference channel subject to a minimum signal to interference plus noise ratio ( sinr ) is considered in . fully distributed algorithms based on trial and error are proposed to find a nash equilibrium and a satisfaction equilibrium .learning in wireless networks is also considered in the game theoretic framework in - .we refer to for more information on game theory and learning algorithms for wireless communications .  in , a learning algorithm using regretsis proposed to find a ce in a finite game .unlike fully distributed algorithms , this no - regret algorithm requires the knowledge of actions chosen by the other users from each play of the game .the same authors presented a fully distributed learning algorithm in that leads to a ce when the players are not aware of the functional form of their utility functions .fully distributed algorithms to find a nash equilibrium are developed in - .the algorithms in , are based on trial and error . using this algorithm users approach strategies that play a pure strategy nash equilibrium for a high portion of time . forpotential and dominance solvable games , reinforcement learning algorithms in , converge to a ne .we consider a power allocation game on a wireless interference channel .it is neither a potential game nor dominance solvable .even existence of a pure strategy ne is not guaranteed .therefore , we can not use the algorithms in , to obtain an equilibrium point .thus we propose a variation of the regret matching algorithm to find a ce of the proposed game on the wireless communication system without knowing the strategies chosen by other users .the algorithms proposed in this paper is fully distributed .apart from learning a non - cooperative equilibrium , learning algorithms for finding a pareto point also exist in literature ( , ) .such points can substantially outperform a ce .we make the following contributions in this paper :  * we propose a fully distributed regret matching algorithm that finds a correlated equilibrium ( ce ) of the interference channel .the usual regret matching algorithm is a partially distributed algorithm which requires knowledge of the strategies of the other users .we propose a modification of that algorithm to convert it into a fully distributed algorithm .we also compare the sum rate at the ce obtained by our algorithm with that obtained from the algorithm in and we note that our algorithm converges faster than the algorithm in .* we use a fully distributed no - regret dynamics to compute a cce of our power allocation game . in general, every ce is a cce but the converse may not be true .* we propose a fully distributed learning algorithm to find a pareto point for our game and compare its sum rates with that of a ce . *even though pareto points outperform ce , and cce , fairness among users is not guaranteed at a pareto point . using a minor variation of the proposed algorithm to compute a pareto point, we compute a nash bargaining solution which guarantees fairness among users .* later , we show that we can use the proposed algorithms to compute ce , cce , pareto point and nash bargaining solutions when each transmitter sends at multiple rates rather than at a fixed rate .this paper is organized as follows . in section [ sm ], we describe the system model and define the problem in game theoretic framework .we propose and analyse a learning algorithm to find a ce in section [ l_ce ] .for our game , we find pareto points through a fully distributed learning algorithm in section [ pt ] .we study nash bargaining solutions for our game in section [ nb ] .we use multiplicative weight algorithm to find cce of our game in section [ cce ] . in section [ m_rates ] , we extend the algorithms to the power allocation problem where each transmitter can transmit at multiple rates . in section[ ne ] , we compare the sum of utilities of all the users at a ce and at a pareto point and also with other algorithms also for some numerical examples . section [ concl ] concludes the paper .we consider a wireless channel being shared by independent transmitter - receiver pairs .transmission from each transmitter causes interference at other receivers .the transmitted signal from every transmitter undergoes fading .the fading gain experienced by the intended signal at a receiver from its corresponding transmitter is called the direct link channel gain .similarly , the fading gain experienced by other unintended signals at a receiver is called the cross link channel gain .we model this scenario as a gaussian interference channel with fading , where each receiver perceives the transmitted signal with additive white gaussian noise .let be the direct link channel gain alphabet and be the cross link channel gain alphabet of user .let the random variable denote the channel gain from transmitter to receiver in time slot which is assumed to be a constant during the slot .observe that , and for .we denote a realization of by .we assume that for a fixed the random variables , are independent and identically distributed .we also assume that are statistically independent for any , and .we assume that transmitter knows in the begin of slot but not , ; in fact it does not know the distribution of also .we also assume that transmitter has finite power levels to transmit in a slot .this is a typical wireless scenario .for example if , a receiver is sending an ack / nack to its transmitter and it is a time duplex channel , then a transmitter can estimate its direct link channel gain but will not know the cross link channel gains ; nor will it know the transmit powers used by the other transmitters .user transmits bits in every channel use at a power level which depends on the direct link channel gain .if receiver successfully receives the message sent in that slot , it sends an ack ( positive acknowledgment ) , else the receiver sends back an nack ( negative acknowledgement ) at the end of the slot .we assume that ack messages are small and sent at a low rate such that these are received with negligible probability of error and its transmission overheads are ignored .this is typically assumed .for a gaussian channel , the probability of error is a function of the received sinr and the modulation and coding used . for a given coding and modulation , we can fix a minimum sinr such that the probability of error is negligible above this sinr . to be specific, we assume that if in time slot , user transmitted bits per channel use at a power level  and then , transmitter receives an ack from its corresponding receiver if and only if and the transmitter receives a nack , otherwise ; where is a constant that depends on the modulation and coding used by the receiver . in the followingwe will take for all for convenience .we consider stationary policies , i.e. , the power used by user in slot depends only on the channel gain but is independent of time . thus , we define the feasible action space of user by where is the probability distribution of .we note that user has an average power constraint for each feasible action .as the set of power levels of each user is finite , the number of elements in is also finite .let the cardinality of be .we enumerate the elements of as , i.e. , when we write , we mean is a feasible power policy of user for , and denotes the power used when the direct link channel gain is under the policy .we denote the action space of the set of all users by , and an action profile of all users by .we denote the action space of all users other than user by and the action profile of all users other than by .let indicate the action of user at time .also , . a strategy of user is a probability distribution on , and a pure strategy is a degenerate probability distribution where a certain action is chosen with probability one .  in a given time slot , each user chooses an action that maximizes its probability of successful transmission .the strategy of each user influences the probability of successful transmission of every user and hence we are interested in finding an equilibrium point . to model this as a game , we define the reward of user for a given action profile in time slot with direct link channel gain as this reward of user in a given time slot is random as it depends on the cross link channel gains and the power levels at which the other users transmit in that slot , which in turn depend on the direct link channel gains of those users .the average reward of user for the action profile is by the strong law of large numbers this limit exists and average reward can be interpreted as the probability of successful transmission . the average reward of user given thestrategy is u_i(k^{(i)},k^{(-i)}).\ ] ] each player aims to maximize its own average reward or probability of successful transmission and we model this scenario as a stochastic game and restrict ourselves to stationary policies .then the utility of user can be written as ,\label{ex_ut}\ ] ] where the expectation is with respect to the distribution of random variables for all .thus the stochastic game is equivalent to the one - shot game in which user maximizes its utility defined in ( [ ex_ut ] ) and the set of correlated equilibria ( ce ) , defined below , for both of these games is same .  given a strategic game , a joint probability distribution is said to be a * correlated equilibrium * if for all , and , we have \leq 0.\label{def_ce}\ ] ]  we get a correlated -equilibrium if the zero in the above definition is replaced by .  to find a ce of a one - shot game ,a regret matching algorithm is proposed in .regret for user is defined in terms of utility and it is assumed that the functional form of utility is known to user . in this paper , we assume that user is not aware of functional form of but knows for each at the end of slot .we define regret in terms of and this definition is equivalent to the definition of regret in because , note that as the first summation in ( [ x_bf ] ) is finite , we can exchange summation and limit to get ( [ xchange ] ) .thus , ,\label{reg}\ ] ] where is the utility of user , that it would have received if the action is replaced by whenever it is played .similarly , we write to denote the reward of user , by playing instead of .we use the difference to define regret .thus the regret - matching algorithm in can be described as follows .the regret , where  and denotes the actual interference experienced by user at time and is the upper bound on interference for successful transmission of user , and it is given by   for every pair of pure strategies , the regret of a user is a nonnegative real number that reflects the change in utility received by the user if the choice of pure strategy is replaced with at every time instance that the user chose to play upto time .user chooses a pure strategy according to a probability distribution in which the probability of choosing an action is proportional to its regret .thus , user chooses a pure strategy according to the distribution for a sufficiently large .it is shown in that following the above procedure , the empirical frequencies of actions converge to the set of correlated equilibria .  to implement this algorithm each user not only needs to know its own actions ( transmit powers ) but also of the other users actions and its cross link channel gains ( to compute regret ) , which it does not know .thus , in the next section we modify this algorithm to find a correlated equilibrium where each user updates its strategy based on the rewards it received and the actions it chose in the past .the learning algorithm we propose is fully distributed in the sense that every user updates its strategy based on its own actions and rewards and independently of the other users strategies and rewards .we will show that the joint empirical distribution converges to the set of correlated equilibria with probability 1 for our algorithm .  in our problemas the transmitter is not aware of the interference at the corresponding receiver , it can not find the regrets as in ( [ regret_hm ] ) .therefore , each transmitter estimates regret by estimating the instantaneous reward based on the feedback it has received . the estimated reward is a function of the strategy , with respect to which we want to find the regret for not using instead of .if is the pure strategy that is actually chosen at a time and is the direct link channel gain at that time , then the actual interference perceived by its receiver is less than the threshold whenever the communication is successful and it is greater than whenever the communication is a failure . the user is optimistic in estimating the rewards for using instead of . to formally define the estimated reward we use the following notation . for each and , user finds the instantaneous reward that could have been if user had used action instead of at time , as   for every pair of actions , after slots , the regret is where  and , the actual reward received by the user . if , i.e. , is the action chosen by user at time instant , then an action in time slot is chosen with probability , it should be noted that the quantities are the _ estimated _ values of reward and regret .we define the empirical frequency of strategies chosen upto time as it is shown in that the empirical frequency of strategies converges to the set of the correlated -equilibria if and only if the actual regret converges to zero as .this can be formally stated as  [ prop_ce ] let be a sequence of actions chosen by the users . for any , for each user and every with , if and only if the sequence of empirical frequencies converges to the set of correlated -equilibrium almost surely .  in proposition [ est_act ], we prove that if the estimated regret converges to zero then the actual regret also converges to zero .[ est_act ] let be a sequence of actions chosen by the users . for each user andevery with , if then   to prove the proposition , we consider all such that and prove that for any given  and given channel gain . here, we note that , if then .therefore ( [ step1 ] ) is satisfied and hence in the following we consider only the cases where .we now consider two cases separately : + _ case _ : + in this case , it should be noted that and as can be either or , ( [ step1 ] ) always holds .+ _ case _ : + in this case , if , then by definition and ( [ step1 ] ) always holds .if , then and hence . therefore we have and ( [ step1 ] ) is satisfied with equality .hence , ( [ step1 ] ) always holds and we have .therefore , , and , if converges to zero as approaches infinity , then also converges to zero .  in , authors have extended the regret - matching algorithm of so that one can use a function of the regret in the original procedure instead of regret , where the function satisfies certain conditions .we can not use that result here , as our estimation does not satisfy the conditions on the function .but we can generalize the result in .let the actual regret be defined as in ( [ regret_hm ] ) . let the actual utility in ( [ inst_reg ] ) be replaced by an estimated utility such that then following the regret - matching algorithm ( [ regret_hm])-([strgy1 ] ) with the actualregret replaced by the estimated regret , the estimated regret converges to zero as approaches infinity .following the proof of the main theorem in , we can show that for each and , the estimated regret converges to zeros as approaches infinity. therefore by proposition [ est_act ] , we get the following theorem .if each user chooses strategies in each time slot according to the algorithm ( [ proc_1])-([strgy ] ) , then the empirical frequency converges to the set of correlated for any .  in the proof of the main theorem of , history up to time defined as the actions chosen by all users at time instances . to prove that the estimated regret converges to zero following the regret - matching algorithm, we just need to redefine the history up to time as the actions chosen by all users along with the direct channel states at time instances . with this definition of history ,the entire proof of the main theorem in , carries over and we can conclude that the estimated regret converges to zero.  the performance of the system at a ce may not be very satisfactory from the overall system point of view .therefore , we also provide a distributed algorithm in the next section which achieves a pareto point . the pareto points are socially optimalin this section , we compute a coarse correlated equilibrium which is a generalization of a correlated equilibrium .we present the multiplicative weight ( mw ) algorithm , to compute a cce of our power allocation game .mw algorithm has much less computational complexity per iteration than that of the regret matching algorithm presented in section [ l_ce ] .it also does not require estimation of regret as needed in section [ l_ce ] .also , it has been observed that the price of anarchy ( poa ) of a cce is no worse than that of a ce in a large class of games .however , it is also known that for some other classes of games , e.g. , congestion games , the poa of cce / ce can be larger compared to ne .  from the definition of ce ,condition ( [ def_ce ] ) requires that every user minimizes the conditional expectation of utility where the conditioning is on and . in cce ,user contemplates a deviation knowing only the distribution .let be the cost of user and each user chooses its action to minimize the cost . in our power allocation problem, we can define cost as negative of the utility , i.e. , .we define the cce of a cost minimization game as  a distribution on is said to be a * coarse correlated equilibrium * if \leq { \mathbb{e}}_{k \sim \phi}\left[{c^{(i)}}({\hat{k}^{(i)}},k^{(-i)})\right],\ ] ] for each user , and for all actions .the distribution is called a -coarse correlated equilibrium if \leq { \mathbb{e}}_{k \sim \phi}\left[{c^{(i)}}({\hat{k}^{(i)}},k^{(-i)})\right ] + \epsilon,\ ] ] for each user , for every action .please note that whenever the cost is a random variable , we denote it by rather than . in this definition, cost is a random variable that depends on the randomly chosen actions .every ce is also a cce and thus the set of cce is a larger set than the set of ce .there exist no - regret learning algorithms to compute a cce but the notion of regret used to compute a cce is different from that used to compute a ce . the regret defined in section [ sm ] is known as internal regret and we use external regret to compute a cce which is defined as  the regret of user given the pure strategy sequence with respect to an action is .\label{extn_regret}\ ] ]  an algorithm in which users update their strategies based on the received cost in such a way that the external regret converges to zero is a no - regret algorithm .we now present a no - regret algorithm known as multiplicative weight algorithm to compute a cce .  in the initial iteration ,each user assigns a weight to action .user chooses an action with probability during the iteration , if is the action chosen by user in iteration , then it receives the expected utility $ ] . based on the received utility , user updates the weight of action , as for the iteration , user chooses an action according to ( [ prob_update ] ) with weights replaced by and this process is repeated .we have the following convergence result .following the multiplicative weight update algorithm , there exists a positive integer such that the external regret of user defined in ( [ extn_regret ] ) is less than after iterations .let denote the outcome distribution at time and .then is a -coarse correlated equilibrium .we use this mw algorithm to find a cce of our power allocation game . in general to use the mw algorithm user needs to know the expected utility . in our game user it given the history of actions and rewards as ,\\ & = & \frac{1}{t}\sum_{\tilde{t}=1}^tw^{(i)}_{\tilde{t}}({k^{(i)}},h_{ii}(\tilde{t}))1_{k^{(i)}_{\tilde{t } } = { k^{(i)}}}.\end{aligned}\ ] ] based on , user updates its weights using the mw update and chooses action according to ( [ prob_update ] ) with .unlike in the algorithm of ce , we do not need to evaluate the estimated reward as the mw algorithm does not explicitly depend on the regret defined in ( [ extn_regret ] ) .but the mw algorithm guarantees that the external regret converges to zero .hence we can apply the mw algorithm to our problem to find a cce .an action profile is * pareto optimal * if there does not exist another action profile such that for all with at least one strict inequality .  in this sectionwe present a _ distributed _ algorithm to find a pareto optimal point .the global maximum of is a pareto optimal solution , where are positive constants .we find pareto points by finding a solution of ( [ pareto ] ) .we assume that when a receiver sends an ack / nack to its transmitter , all the other transmitters can also listen to it without error .this is realistic in many wireless systems because an ack / nack message is small and is usually transmitted at low rates with very low probability of error . under this assumption , we present a learning algorithm in which users may or may not choose to experiment and update their strategies in such a way that improves .the algorithm is as follows :  * each user chooses a random action uniformly from .all the users use these randomly chosen actions for a fixed number of time slots .each user follows the procedure below sequentially : * as user receives the feedback of other users , it finds the weighted sum of the utilities at the end of slots , user experiment with probability .when user experiments , with probability , chooses an action randomly with uniform probability from other than , and with probability it chooses an action other than in the following way : * * in the action , a power level has been specified for each value of direct link channel gain .user chooses an action randomly from a subset of , with feasible actions having higher power level than for a channel state with the highest probability of occurrence .if this subset is empty , then it chooses an action with higher power level for the channel gain with second highest probability of occurrence . * * if all the direct link channel gains occur with equal probability , then user chooses an action randomly from a subset of , with feasible actions having higher power level for the maximum value of direct link channel gain .if this subset is empty , it chooses an action with higher power level for the second maximum direct link channel gain . *let this new action be .for the next time slots , user uses action , and user uses actions for .user finds the weighted sum of the utilities of all the users . if , then user replaces its action with , and this new weighted sum of the average rewards is taken as a benchmark .if there is no improvement in the weighted sum of the average rewards , it randomly selects another action following the procedure described above .thus each user may experiment with upto a maximum of max number of actions chosen randomly .  if , each user experiments with randomly chosen actions .but , for small , in our algorithm we are selecting an action from the action space of that user by a _ local search_.the local search often yields a better point , that improves , than a purely random search in the entire action space , and yields a faster rate of convergence as seen in our numerical examples .a user updates its action whenever there is an improvement in the weighted sum of average reward over the benchmark .hence , this benchmark of utility is monotonically increasing and bounded above by . therefore , for a sufficiently large , we find a pareto optimal point with a large probability . by increasing max, this probability can be made arbitrarily close to .our algorithm is a distributed version of a meta heuristic , _ stochastic local search _ , often used for global optimization .we can also obtain a pareto point which satisfies certain minimum probability of success ( e.g. , for voice users ) by including this constraint in the set .pareto points , although they globally maximize , may be _ unfair _ to some users .changing the weights can alleviate some unfairness .otherwise , one can obtain pareto points which are _ nash bargaining solutions _ , which can be obtained via a similar algorithm as explained in the next section .in nash bargaining , we specify a disagreement outcome that specifies utility of each user that it receives by playing the disagreement strategy whenever there is no incentive to play the bargaining outcome . thus , by choosing the disagreement outcomes appropriately , the users can ensure certain fairness .the nash bargaining solutions are pareto optimal and also satisfy certain natural axioms .it is shown in that for a two player game , there exists a unique bargaining solution ( if the feasible region is nonempty ) that satisfies the axioms stated above and it is given by the solution of the optimization problem for an n - user nash bargaining problem , this result can be extended and the solution of an n - user bargaining problem is the solution of the optimization problem   a nash bargaining solution is also related to proportional fairness , another fairness concept commonly used in communication literature .a utility vector is said to be _ proportionally fair _ if for any other feasible vector , for each , the aggregate proportional change is non - positive . if the set is convex , then nash bargaining and proportional fairness are equivalent .proportional fairness is studied in when is non - convex . in our case , is convex and hence nash bargaining solution is also proportionally fair .a major problem in finding a solution of a bargaining problem is choosing the disagreement outcome .it is more common to consider an equilibrium point as a disagreement outcome . in our problemwe can consider the utility vector at a ce as the disagreement outcome .we can also choose for each .if we choose the disagreement outcome to be a ce , each user needs to evaluate a ce first before running the algorithm to find a solution of ( [ nb_n ] ) , which requires more computations .instead , we can choose the disagreement outcome to be the zero vector or by using the following procedure :  * each user chooses an action that gives higher power level to the channel state that has higher probability of occurrence .in other words , among the set of feasible actions , choose a subset of pure strategies that gives the highest power level to the channel state with highest probability of occurrence .we shrink the subset by considering the actions that give higher power level to the second frequently occurring channel state and we repeat this process until we get a single strategy . *if all the channel states occur with equal probability , we follow the above procedure by considering the value of the channel gain instead of the probabilities of occurrence of the channel gains .  let the pure strategy chosen by user be , and assume that the users use these strategies for a fixed number of slots .user finds by averaging the rewards received in the slots , i.e. ,   for our numerical evaluations we have chosen the disagreement outcome following the procedure described above instead of choosing the zero vector . to find the bargaining solution , i.e. , to solve the optimization problem ( [ nb_n ] ) , we use the algorithm of section [ pt ] used to find a pareto optimal point but with objective defined as in section [ ne ] , we present a nash bargaining solution for the numerical examples we consider , and observe that the nash bargaining solution obtained is a pareto optimal point which provides fairness among the users .until now , we have presented learning algorithms to compute a ce , a cce , pareto points and nash bargaining solutions , when a user is transmitting at a fixed rate . in this section ,we generalize the model so that a user can transmit at multiple rates rather than at a fixed rate and show that we can still use the same algorithms to compute equilibria .let be the set of possible transmission rates of user .let be the set of power levels for user , as considered earlier .we denote the new strategy set as the cardinality of is times that of , as every action can be associated with each rate in . we enumerate the elements of as in section [ sm ] . here also we denote an action by , and is the rate of transmission under the action . if is the direct link channel gain of user and the user chooses action , then it transmits at a rate with power .user receives an ack if the interference at receiver satisfies and it receives a nack otherwise .we use the same notation to denote the upper bound on the interference for receiving an ack .we can redefine the estimated reward and estimated regret as in ( [ est_reward ] ) and ( [ est_regret ] ) respectively , but with the threshold redefined as   it can easily be seen even in this case that the estimated reward is greater than or equal to the actual reward .hence , we can use the regret - matching algorithm to compute ce for the game with as the strategy set .we can also use the respective algorithms mentioned earlier to compute pareto points , nash bargaining solution , and cce .we can ensure that these solutions satisfy certain minimum rates by limiting our overall action space to strategies that satisfy these rate constraints .in this section we consider three examples with three transmitter - receiver pairs in the communication system . in the first example , we consider a symmetric scenario where and for each user , and each channel state occurs with equal probability .the set of possible power levels for each player is .each user transmits at a rate of bits per channel use , and receives feedback from its receiver .each user follows the learning algorithm ( [ proc_1])-([strgy ] ) to find a ce , and finds a pareto optimal strategy as described in section [ pt ] . in finding the pareto points ,we choose for all .the sum rate at a ce and at a pareto point are compared in figure [ fig_1 ] .we also compare the sum rate at a ce obtained by using the reinforcement learning ( rl ) algorithm in .we observe that the sum rates at ce obtained via our algorithm and that obtained via the algorithm in almost coincide in this example .even though the sum rates are close for both the algorithms , we observe that our algorithm convergences faster than the rl algorithm . in the rl algorithm, it is required that each pure strategy of each user should be played for a minimum number of time slots to find the regret as defined in .thus the algorithm requires a larger number of iterations to converge to the set of correlated equilibria . in this example , at snr of 15db , our algorithm converges in about iterations , whereas the algorithm in converges in about iterations .  in finding a pareto point ,if we randomly choose a strategy each time , instead of local search , the algorithm runs for about iterations whereas our local search algorithm finds a pareto point in iterations .we also plot in figure [ fig_1 ] the sum rate at a stochastically stable point of the trial and error based algorithm ( te ) in .it is known from that the algorithm therein converges to an efficient ne only if the game under consideration has at least one pure strategy ne . in general, we can not guarantee existence of a pure strategy ne for our game , and hence the stochastically stable point computed by the algorithm in need not be a ne .then the algorithm produces stochastically stable points that maximize for all where , and we refer to for further details of the function .we also plot the sum rate at the cce and at the nash bargaining solution obtained for example 1 in figure [ fig_1 ] .we observe that the sum rate at a cce is better than that at a ce , but that the mw algorithm runs for about iterations which is more than the number of iterations required for computing a ce .the sum rate at the nash bargaining solution is very close to that at the pareto point , but the former provides fairness among users .we present the rates at the pareto point and at the nash bargaining solution in table [ table1 ] to illustrate the fairness provided by the nash bargaining solution for example 1 .the rates of all the three users are mentioned as a triplet , where is the rate of user .it can be seen for several snr values , that the nash bargaining solution provides more fairness than at the pareto point . even though example 1 is symmetric , as the algorithmsare based on stochastic local search , rate allocations need not be symmetric .we note that the sum rate of all the users is higher at the pareto optimal point than at a ce .the improvement is 21.8% at the average transmitsnr constraint of 10db , and 24.6% at the snr of 15db .  in the computation of ce, for each pure strategy , we need to estimate the regret , which requires some calculation of a threshold in advance before starting the running of the algorithm .this requires two multiplications and two additions per action for each user .but , for one iteration in the mw algorithm , each user requires one division per action and two multiplications .hence , even though the regret - matching algorithm requires computation of estimated regret , it may converge faster than the mw algorithm which does not require computation of regret .it is observed from examples that the regret - matching has relatively less running time than the mw algorithm .next we consider an asymmetric scenario , in example 2 . in this examplealso we consider and for each user .the direct link gains from occur with equal probability for each user , but the cross link gains occur with a different probability distribution for each user . for user ,the distribution is , for user 2 it is , and for user 3 , it is .users and transmit at rates bits per channel use respectively . the sum rate at the ce and at the pareto point obtained from our algorithmare compared in figure [ fig_2 ] .we also compare the sum rate at the ce obtained by using the rl algorithm in . in this example also , we observe that our algorithm converges faster than the rl algorithm : at snr of 15db , our algorithm converges in about iterations , whereas the algorithm in converges in about iterations .we also plot the sum rate at a stochastically stable point of the algorithm in which maximizes .we also plot the sum rate at a cce and at a nash bargaining solution for example 2 in figure [ fig_2 ] . in this example also , we observe an improvement in the sum rate at a cce over that at a ce , and the mw algorithm runs for about iterations which is more than the number of iterations required for computing a ce. the sum rate at the nash bargaining solution is very close to that at the pareto point .we observe that the sum of the rates of all the users is higher at the pareto optimal point than at a ce .we observe an improvement of 22.7% at snr 10db and an improvement of 17.5% at snr of 15db .the sum rate obtained via at its stochastically stable point is the lowest .finally , we consider multiple rates of transmission in example 3 .for example 3 , we consider the same parameters as in example 2 , but each user can send data at any rate from the set .we compare the sum rates at a ce , cce , pareto point , and nash bargaining solution , in figure [ fig_3 ] .we also plot the sum rates using the rl algorithm and te algorithm for this example in figure [ fig_3 ] . in this examplealso we observe an improvement in the sum rate at the cce over the sum rate at the ce .sum rate at the nash bargaining solution and at the pareto point almost coincide in this example also . as the cardinality of the strategy set of each useris enlarged by transmitting at multiple rates , the regret - matching algorithm runs for about iterations to compute a ce and the mw algorithm runs for about iterations to compute a cce , at snr of 15db .we have considered a communication system in which transmitter - receiver pairs communicate on a wireless channel .each transmitter sends data at a certain rate at a power level that is a function of the direct link channel gain and the feedback received from its receiver , to maximize the probability of successful transmission .this scenario is modeled as a stochastic game and fully distributed learning algorithms are proposed to find a correlated equilibrium ( ce ) and a pareto point .we have compared the sum of rates of all the users at the ce and the pareto point , and we observe that the pareto optimal power allocations provide higher probability of successful transmission .we have also compared our algorithms with two other recent learning algorithms in literature , .the ce obtained by our algorithm performs as well as the ce obtained via the algorithm in but our algorithm converges much faster . on the other hand the performance of our ce is better than the best point obtained by the algorithm in .  we also note in our examples that we can achieve a higher sum rate by operating at a cce than operating at a ce but at the expense of more number of iterations to compute it .but , in general , it is not guaranteed that a cce yields a better sum rate than a ce . on the other handa nash bargaining solution may be a better operating point than an arbitrary pareto point as it provides fairness among the users .transmitting at multiple rates can significantly improve the sum rate but as the strategy set of each player is enlarged , it requires more number of iterations to converge to either a nash bargaining solution or a cce . in practice , which algorithm to use depends on the nature of the problem , i.e. , if for example , the cardinality of the overall action space is not large , then the users can find a pareto point more quickly than a ce or cce .but if the action space is large and if it requires to converge to an equilibrium quickly , one can use regret - matching to converge to a ce .9 s. deng , t. weber , and a. ahrens , `` capacity optimizing power allocation in interference channels , '' _ aeu international journal of electronics and communications _ ,vol.63 , pp .139 - 147 , feb . 2009 .daniela tuninetti , `` gaussian fading interference channels : power control , '' _ proc . of the 42nd asilomar conference on signals , systems and computers _ , monterey , ca , pp . 701 - 706 , october 2008 .k. a. chaitanya , u. mukherji , and v. sharma , `` power allocation for interference channel , '' _ proc . of national conference on communications _ , new delhi , 2013 .g. scutari , d. p. palomar , and s. barbarossa , `` optimal linear precoding strategies for wideband non - cooperative systems based on game theory - part ii : algorithms , '' _ ieee trans on signal processing _ , vol.56 , no.3 , pp .1250 - 1267 , march 2008 . k. w. shum , k .- k .leung , and c. w. sung , `` convergence of iterative waterfilling algorithm for gaussian interference channels , '' _ ieee journal on selected areas in comm . , _vol.25 , no.6 , pp .1091 - 1100 , august 2007 .g. scutari , d. p. palomar , and s. barbarossa , `` the mimo iterative waterfilling algorithm , '' _ ieee trans on signal processing _ , vol .57 , no.5 , may 2009 .g. scutari , d. p. palomar , and s. barbarossa , `` asynchronous iterative water - filling for gaussian frequency - selective interference channels '' , _ ieee trans on information theory _ , vol.54 , no.7 , july 2008 . l. rose , s. m. perlaza , and m. debbah , `` on the nash equilibria in decentralized parallel interference channels , '' _ proc .of international conference on communications _ , kyoto , 2011 .k. a. chaitanya , u. mukherji , and v. sharma , `` algorithms for stochastic games on interference channels , '' _ proc .of national conference on communications _ , mumbai , 2015 .s. lasaulce , and h. tembine , `` game theory and learning for wireless networks : fundamentals and applications , '' _ elsevier _ , 2011 .s. hart , and a. mas - colell , `` a simple adaptive procedure leading to correlated equilibrium , '' _ econometrica _ , vol .68 , no.5 , pp .1127 - 1150 , sept . 2000d. fudenberg , and d. m. kreps , `` learning mixed equilibria , '' _ games and economic behavior _ , vol .320 - 367 , 1993 .n. mastronarde , and mihaela van der schaar , `` fast reinforcement learning for energy - efficient wireless communication , '' _ ieee tran . on signal processing _ ,p. mertikopoulos , e. v. belmega , a. l. moustakas , and s. lasaulce , `` distributed learning policies for power allocation in multiple access channels , '' _ ieee journal on selected areas in communications _ , vol .96 - 106 , jan . 2012 .e. v. belgama , s. lasaulce , m. debbah , and a. hjorungnes , `` learning distributed power allocation policies in mimo channels , '' _european signal processing conference _ , aalborg , denmark , 2010 .l. rose , s. m. perlaza , m. debbah , and c. j. le martret , `` distributed power allocation with sinr constraints using trial and error learning , '' _ ieee wireless communications and networking conference _ , paris , april 2012 .e. sabir , r. el - azouzi , v. kavitha , y. hayel , and e. bouyakhf , `` stochastic learning solution for constrained nash equilibrium throughput in non saturated wireless collision channels , '' _ proc .the 3rd icst / acm international workshop on game theory in communication networks _ ,pisa , italy , oct .h. jang , s. y. yun , j. shin , and y. yi , `` distributed learning for utility maximization over csma - based wireless multihop networks , '' _ ieee international conference on computer communications _ ,toronto , canada , april - may 2014 .h. saad , a. mohamed , and t. elbatt , `` cooperative q - learning techniques for distributed online power allocation in femtocell networks , '' _ wireless communications and mobile computing _, doi : 10.1002/wcm.2470 , 2014 .s. hart , and a. mas - colell , `` a reinforcement procedure leading to correlated equilibrium , '' _ economics essays _, springer berlin heidelberg , 2001 .k. r. fall , and w. r. stevens , `` tcp / ip illustrated volume 1 , '' addison - wesley , 2012 .b. s. r. pradelski , and h. p. young , `` learning efficient nash equilibria in distributed systems , '' _ games and economic behavior _ , elsevier , 75 , pp.882 - 897 , 2012 .h. p. young ,`` learning by trial and error , '' _ games and economic behavior _ , elsevier , 65 , pp.626 - 643 , 2009 . t. borgers , and r. sarin , `` learning through reinforcement learning and replicator dynamics , '' _ journal of economic theory _, 77 , pp.1 - 14 , 1997 . w. b. arthur , `` on designing economic agents that behave like human agents , '' _ journal of evolutionary economics _ , springer - verlag , pp.1 - 22 , 1993 . m. zuluaga , a. krause , g. sergent , and m. puschel , `` active learning for multi - objective optimization , '' _ proc .30th international conference on machine learning _ ,atlanta , georgia , usa , 2013 .j. r. marden , h. p. young , and l. y. pao , `` achieving pareto optimality through distributed learning , '' _ ieee conference on decision and control _ , maui , hawaii , usa , dec .k. miettinen , `` nonlinear multiobjective optimization , '' kluwer academic publishers , 1999 .z. han , d. niyato , w. saad , t. basar , and a. hjorungnes , `` game theory in wireless and communication networks , '' _ cambridge university press _s. hart , `` adaptive heuristics , '' _ econometrica _ , vol .1401 - 1430 , september , 2005 .h. h. hoos , and t. stutzle , `` stochastic local search : foundations and applications , '' _ morgan kaufmann / elsevier _ , 2004 .n. littlestone , and m. k. warmuth , `` the weighted majority algorithm , '' _ information and computation _ , 108(2 ) , pp 212 - 261 , 1994 .s. arora , e. hazan , and s. kale , `` the multiplicative weights update method : a meta algorithm and applications , '' _ theory of computing _ , 8(1 ) , pp 121 - 164 , 2012 .n. cesa - bianchi , and g. lugosi , `` prediction , learning , and games , '' _ cambridge university press _ , 2006 .t. roughgarden , `` intrinsic robustness of the price of anarchy , '' _ acm symposium on theory of computing _ , pp.513 - 522 , 2009 . j. nash , `` the bargaining problem , '' _ econometrica _ ,18:155 - 162 , 1950 . f. kelly , a. maulloo , and d. tan , `` rate control for communication networks : shadow prices , proportional fairness and stability , '' _ journal of the operations research society _ , vol .237 - 252 , march , 1998 .h. boche , and m. schubert , `` nash bargaining and proportional fairness for wireless systems , '' _ ieee / acm transactions on networking _ , vol .5 , pp . 1453 - 1466 , october , 2009 .

Abstract: we consider a wireless communication system in which transmitter - receiver pairs want to communicate with each other . each transmitter transmits data at a certain rate using a power that depends on the channel gain to its receiver . if a receiver can successfully receive the message , it sends an acknowledgement ( ack ) , else it sends a negative ack ( nack ) . each user aims to maximize its probability of successful transmission . we formulate this problem as a stochastic game and propose a fully distributed learning algorithm to find a correlated equilibrium ( ce ) . in addition , we use a no regret algorithm to find a coarse correlated equilibrium ( cce ) for our power allocation game . we also propose a fully distributed learning algorithm to find a pareto optimal solution . in general pareto points do not guarantee fairness among the users , therefore we also propose an algorithm to compute a nash bargaining solution which is pareto optimal and provides fairness among users . finally , under the same game theoretic setup , we study these equilibria and pareto points when each transmitter sends data at multiple rates rather than at a fixed rate . we compare the sum rate obtained at the ce , cce , nash bargaining solution and the pareto point and also via some other well known recent algorithms .   interference channel , stochastic game , correlated equilibrium , distributed learning , pareto point .