the problem of efficient communication has been extensively studied for a variety of channels and constraints .shannon has found the explicit channel capacity of the additive white gaussian noise ( awgn ) channel subject to average power constraint .this capacity is achieved by a continuous gaussian input . in practice , however , discrete constellations are often used for transmission . in this case , the probability mass function which maximizes the achievable rate subject to a given discrete constellation and average power constraint can be obtained numerically using the modified _ blahut - arimoto _ algorithm ( baa ) . * * the binary symmetric and non - symmetric channels ( bsc and bnsc , respectively ) subject to hamming power ( bounded number of 1 ) constraint are also important examples , with various applications . for bsc capacity and applications , see and . for the capacity of the noise free aperture channel ( free space optical communication ) , modeled by a special case of bnsc ( z - channel ) ,see .all these examples share single letter constraints and memoryless channels .such constraints and channels are treated in this paper .practical shaping methods have been devised over the years to approach channel capacity under constraints .convey and slone used nested lattice codes for constellation shaping .their construction was generalized by forney , subsequently leading to the trellis shaping technique for average power reduction . in this method , a structured large code is formed by nesting the shaping code with the channel code .then , given the coded bits , the free shaping bits creates a set of sequences , i.e. , a coset of the large code , from which one sequence is chosen such that the transmitted codeword has minimal power in the coset .this operation is very similar to choosing a codeword which satisfies a constraint as will be shown in the sequel .the chosen codewords from each coset form a shaped subcode , which achieves shaping gain .another example of such shaped code construction is the ldpc - ldgm scheme , originally designed for dirty paper coding .this scheme creates a shaping code for each transmitted message using a nested ldpc - ldgm structure , which is designed to be both good source code and good channel code , thus shaping is possible .the shaping codes serve as cosets . once again, the minimum energy codeword is chosen from each coset , forming the shaped subcode .the large code , as before , in the union of all the cosets representing each message .all these schemes share a mismatched decoding process .their decoder is only aware of the large codebook , unaware of the constraints and performs optimal decoding as if all codewords might have been transmitted .this decoder is suboptimal , but occasionally more practical since the adaptation to the shaped subcode might be extremely complex , and the decoder for the large code is readily available .we refer to it as the _ mismatched decoder_. the optimal decoder , which is adapted to the shaped subcode , i.e. , familiar with the constraint and is able to repeat the selection process at the receiver , is referred as the_ matched decoder_.  although these works are practical and demonstrate significant shaping gains , the analysis of their performance limit is partial . in ,forney uses `` random shaping '' arguments to suggest why the trellis shaping approaches the ultimate shaping gain so rapidly .these arguments , however , are based upon geometrical volumes analysis , which is more suitable for large constellations and high snr . the achievable rate using the matched decoder and the shaped code distribution are not addressed . for the mismatched decoding , forney andothers rely on the large codebook performance . on first look, mismatch performance analysis seems as a trivial exercise .the transmitted codeword is taken from the large codebook , and therefore the large codebook performance can be used .this analysis is referred to as a _naive approach_. the large code performance analysis , however , assumes the transmission of the typical codewords , while in the shaping system we transmit ( almost ) only non - typical codewords .these codewords may have worse distance spectrum towards their neighbors and error free transmission is not guaranteed at the same noise level .thus , a more precise analysis is necessary .  in this paper , we construct random code ensembles , which satisfy constraints , via codeword selection process .the large codebook and the cosets within are generated at random from an i.i.d .distribution . in this casethe `` cosets '' are random , referred from this point on as random sets .our code construction provides a framework for performance analysis of practical shaping schemes , since both good channels codes and good shaping codes behave approximately as random codes . in this paper, we find the minimal size of a random set needed to ensure that given a set of constraints , at least one codeword satisfies the constraints in each random set with probability tending towards 1 .furthermore , we obtain bounds on the achievable rates of the matched and the mismatched decoders subject to single letter constraints .  more formally , let be the transmission block length and be the transmission rate .let be a random codebook with rate , referred to as the _ large codebook_. let be a design parameter , called _ shaping rate _ , such that .the large codebook is the union of random sets , with codewords in each set , where each codeword is generated according to an i.i.d .probability mass function .a message to be transmitted chooses the random set . out of this randomset a single codeword is selected such that it satisfies a set of single letter constraints . in this casethe constraints are codeword related and can be formulated as where , is the number of constraints , are constraints and are bounded constraints functions .the union of the selected codewords is the shaped code .since each codeword satisfies the constraints , the codebook satisfies the constraints in average .the shaping rate , which determines the random set size , should be chosen such that a random set includes at least one suitable codeword with probability approaching 1 .a simple illustration of the selection process can be seen in fig .[ fig : random - subcode - selection ] , where the large codebook is the union of the three random sets , designated by the different colors and shapes .the constraint is plotted as the circle and one codeword is chosen from each random set such that it is inside the circle , i.e. , satisfies the constraint .these three codewords construct the shaped code .another simple example of a constraint and a selection process is as follows .let the `` constellation '' be binary and the constraint be on the average number of 1 within the codebook , say limited to . to construct the shaped codebook ,let the large codebook be binary and i.i.d . according to .we choose codewords that satisfy this constraint .thus , each codeword in the random set undergoes the following test using .if the average number of 1 in the codeword is limited to , then it is chosen for the shaped codebook .for example , the codeword is typical in the large code , since the number of zeros and ones is equal .however , it does not satisfy the constraint . on the other hand ,the non - typical codeword satisfies the constraint .thus , the questions in hand are how to choose the shaping rate to find such non - typical codewords , and what are the achievable rates using this code construction under the matched and the mismatched decoders.  random subcode construction process.,width=302,height=302 ]  the large deviation theory , which includes the sanov theorem and the conditional limit theorem , addresses the probability of very non - likely events , therefore provides some of the answers .for example , the probability to generate a codeword which satisfies the hamming constraint in ( [ eq : hamming const 1/3 ] ) from a uniform distribution .this probability dictates the minimum size of each random set , and obviously the minimal shaping rate required to ensure that each random set includes at least one such codeword .furthermore , the conditional limit theorem shows that the marginals of the -dimensional distribution of the selected codewords tend toward the _ conditional limit distribution _ , which is an i.i.d .distribution .it is the closest distribution to ( in the sense of divergence ) among all the distributions which satisfy the constraints .further , we derive achievable bounds on the transmission rate under matched and mismatched decoder .starting with the matched decoder , one of the methods to prove achievable rate for a random ensemble of codes , which was generated according to an i.i.d .probability mass function , is the asymptotic equipartition property ( aep ) theorem .since the shaping method involves codeword selection and not random generation from a target distribution , we derive a modified aep theorem .we bound the probability to generate a codeword via this codebook construction using the conditional limit distribution .the requirement that each codeword satisfies the constraints is stronger than the requirement that the whole codebook satisfies the constraints in average ; since a small amount of codewords which do not satisfy the constraints do not effect the average .thus , the codebooks that are shaped using this methods are a subset of the codebooks which satisfy the constraints in average .nevertheless under matched decoding , this code construction may achieve the same rate as a code which was generated i.i.d . according to , i.e. , the mutual information .this property is shown using the modified aep .note that the rate achieved by the matched decoder is usually not the channel capacity since is generally not the distribution that maximizes the mutual information .for example , it can be easily shown that using an i.i.d .uniform distribution for the large code , the conditional limit distribution maximizes the entropy of the input and not the mutual information .hence , achieves only the lower bound on the capacity subject to the constraints .although it is only a lower bound , usually it is quite tight .this conjecture is based of the tightness of this bound for the single letter average power constraint over the awgn channel .this can be verified using the modified baa , since simulations show that the loss in rate due to the suboptimal distribution is insignificant in all snr regions .next , we derive two novel lower bounds on the capacity under mismatched decoding .the first is based upon gallager s random exponent . * * the key here is to perform maximum likelihood decoding using the entire codebook , while taking into account that only a subcode was transmitted and the true distribution of the transmitted codewords . based on this analysis , we redefine the average error probability and present a new error exponent . using the new error exponent , we show achievable rate for the mismatched decoder .our second bound is based upon a modified version of the joint - typicality decoder .this original decoder is not suitable for non - typical codewords , since it declares an error when the input or the output is not typical .thus , we define a modified - joint - typicality ( mjt ) decoder , which decodes solely based upon joint typicality between the input and the output with respect to the large codebook distribution. it might seem surprising that decoding according to a wrong distribution works and indeed this bound is correct in special cases only .one of them is uniform distribution of the large codebook and additive channels .therefore , the awgn channel can be treated using this approach and mjt bound can be obtained .we show result for several examples : pam with power constraints over awgn , bsc and bnsc with hamming constraints .our bounds show that for pam over the awgn channel there is no significant loss due to mismatch decoding .thus , shaping gain can be attained even under mismatched decoding .this conclusion actually justifies the works of forney and others ( unknowingly ) , who used mismatched decoding .the bsc channel is an example where the naive approach works and coincides with our bounds .it is known that the bsc capacity is attained for uniform input probability mass function , and that linear codes achieve the bsc capacity .such codes have the same distance spectrum for all the codewords and therefore the same error probability . in particular, the error probability of the non - typical codewords equals to the error probability of the typical . in this special case ,the large codebook performance is the same as the shaped code .this example provides some insight to why the naive approach is correct for bsc .the paper is organized as follows .section [ sec : preliminaries ] presents some of the preliminary background in large deviations and method of types .section [ sec : subcode - construction ] presents our subcode construction method and the summary of the main results of this paper . including the minimal shaping rate required for this code construction is presented in theorem [ thm : number of nt codewords ] .it also addresses decoder types and examples of related practical shaping schemes .section [ sec : matched - decoder ] analyzes the matched decoder .its significant contribution is the modified aep theorem , which is necessary to establish the achievable rate under matched decoding .section [ sec : mismatched - decoding ] analyzes the mismatched decoder and presents two achievable bounds , the mjt bound and the gallager bound .section [ sec : special - cases ] is dedicated to examples of channels and constraints , such as bsc and awgn subject to average power constraints .let capital letters denote random variables and vectors .let small and calligraphic letters denote their realizations and support , respectively .bold letters represent vectors .let be a random variable with probability mass function and support .let be a discrete memoryless channel and be a random variable with probability mass function and support , such that and are the input and output to the channel , respectively .the joint probability mass function of is . for the sake of completeness, we repeat several basic quantities , which can be found in .thus , the entropy of the variable is the conditional entropy of given is and the mutual information of is the mutual information has also other formulations .furthermore , let be another probability mass function on the same support .the kullback - leibler information divergence ( also relative entropy ) between and ( not symmetric ) is using the conventions that , and .next we review several definitions and theorems from the aep ( * ? ? ?* chapters 3,7 ) .[ a_px]the typical set with respect to the probability mass function is the set of sequences with length and empirical entropies -close to the true entropy :   the aep theorem ( * ? ? ?* theorem 3.1.1 ) , shows that if are sequences of length drawn i.i.d . according to , then : for sufficiently large .[ a_pxy]the jointly typical set with respect to the probability mass function is the set of sequences with length and empirical entropies -close to the true entropies :   the joint aep theorem ( * ? ? ?* theorem 7.6.1 ) , shows that if are sequences of length drawn i.i.d . according to , then : for sufficiently large .we also repeat some definitions and theorems from the method of types given in ( * ? ? ?* chapter 11 ) .the type of a sequence is the relative proportion of occurrences of each symbol of , i.e. , for all , where is the number of times the symbol occurs in the sequence .let be the set of types with denominator ( all possible types when the sequences have length ) .hence , if the type class is the set of sequences of length and type , i.e. ,   the following properties are established in ( * ? ? ?* theorems 11.1.2 - 11.1.4 ) :  1 .  if are drawn i.i.d .according to , the probability of depends only on its type and is given by 2 .  for any type , 3 .  for any type and probability mass function , the probability of type class , where the sequences are drawn according to i.i.d . is given by   the sanov theorem ( * ? ? ?* theorem 11.4.1 ) is part of the large deviation theory , which addresses the probability of very non - likely events .the theorem is formulated as follows ; let be drawn i.i.d . according to .let be a set of probability mass functions .then , the probability that belongs to the set is where is the probability mass function in that is closest to in relative entropy .this is the conditional limit distribution with respect to and .if , in addition , the set is the closure of its interior , then   the conditional limit theorem ( * ? ? ?* theorem 11.6.2 ) strengthens the sanov theorem by showing that with high probability the types of the sequences which belong to are very close to in divergence , and the marginals tend towards when .the theorem is formulated as follows .let be a closed convex subset of and let be a probability mass function not in .let be discrete random variables drawn i.i.d . according to .let achieve . then , where and .as stated in the introduction , the goal is approaching the channel capacity subject to a set of single letter constraints . maximizing the average mutual information subject to these constraintsobtains the maximal achievable transmission rate .let and be the input and output of a memoryless channel with probability mass function .let be a set of probability mass functions subjected to a set of single letter constraints , given by for , where is the closure of its interior , are bounded constraint functions and are constraints .it is well - known that i.i.d .input maximizes the average mutual information subject to single letter constraints .hence , the channel capacity is and the probability mass function which achieves this capacity is   let designate the desirable transmission rate . the first method to create a random codebookis directly generate codewords according to an i.i.d .probability mass function , such that .we refer to this method as _ code construction i _ with respect to .the maximal reliable transmission rate can obviously be achieved using code construction _i _ and .  generating a practical good codebook with a specific probability mass function is not trivial .thus , the shaping method can be used . in this method ,a large codebook is generated according to an i.i.d .uniform probability mass function and then codewords which satisfy a set of constraints are chosen , creating the shaped subcode .we generalize this method , by allowing the large codebook to be generated according to any i.i.d .probability mass function .we refer to this method as _ code construction ii _ with respect to and .we are motivated to analyze code construction _, since good practical codes for uniform probability mass function are readily available , and codeword selection might be a simple procedure for a properly structured code .more formally , let be a random codebook with codewords , referred to as the large codebook .let be the shaping rate .let be the shaped code rate .the large codebook is the union of random sets , with codewords in each random set .each codeword with length is generated according to i.i.d .probability mass function on .a message to be transmitted chooses the random set . out of this random set a single codeword selected such that it satisfies the constraints where is type of .a simple illustration of the selection process can be seen in fig .[ fig : random - subcode - selection ] .the following are examples of constraints and selection metrics .the first is the average power constraint with power limitation of  using .the selected codewords have to satisfy to ensure that the codebook satisfies the constraints in average .another example is the hamming power constraint , where the number of ones ( 1 ) is limited to , i.e. , using . in this casethe selected codewords have to satisfy .  for every constraint and constraint function ,the shaping rate is chosen such that a random set includes at least one suitable codeword with probability approaching 1 .the union of the selected codewords is the shaped code . since each selected codeword satisfies the constraints , the shaped codebook satisfies the constraints in average as required by ( [ eq : e set ] ) .recall that this code construction provides us with a framework for approximating the performance of practical shaping schemes using information theoretic tools .thus , the questions in hand are how to choose the shaping rate to find such non - typical codewords , and what are the achievable rates of this codebook construction . in particular , when using the matched decoder which is aware of the selection process and the constraints , and the mismatched decoders which is aware only of the large codebook .the next section is dedicated to the summary of the main theorems , which provides a full overview of the main results of this paper . after which the reader can safely proceed to section [ sec : special - cases ] for special case and results .let the shaping code construction be code construction _ii_. let be the shaping rate .given the set of constraints , let be the conditional limit distribution .thus , for the probability that a random set of size has at least one codeword which satisfies the constraints approaches 1 .furthermore , the probability that each random set includes at least one such codeword , which is the probability that the shaped codebook exists , also approaches 1 .let be the maximal achievable rate under the constraints in , code construction _ii _ using and , and the matched decoder . then, is the achievable rate .this results from the modified aep theorem ( theorem [ thm : joint - aep - ccii ] ) , developed in this paper with respect to code construction _ii_.  for the mismatched decoder ,let be the achievable rate resulting from the modified development of the gallager error exponent , which suites code construction _ii_. thus , for the rate the average error probability approaches 0 , i.e. , this is an achievable transmission rate .furthermore , let be the achievable rate resulting from the modified joint typicality analysis given in section [ sec : joint - typicality - decoder ] .this rate is achievable in special cases only , in particular when the conditions of lemma [ lem : first error event ] are satisfied .thus in those special cases , is an achievable rate , where the distribution is given in ( [ eq : pstar - mjt ] ) . under the conditions of lemma [ lem : first error event ] , the maximum between the two bounds is the achievable rate .otherwise , only the gallager bound can be used .the remaining of the paper is dedicated to elaborated presentation of these results , formulating theorems and proofs , and analyzing interesting and practical special cases .thus , this section is dedicated to choosing .this choice has to ensure that the large codebook has the right amount of codewords to support the transmission rate , i.e. , that each random set of size has at least one codeword which satisfies the constraints with very high probability . in theorem [ thm : number of nt codewords ] , we show that in the limit , this probability approaches 1 if where .the proof highly relies on the method of types and the sanov theorem .we also prove that ( [ eq : rsplusdelta ] ) guarantees a minimum on , such that the shaped codebook ( using code construction _ ii _ ) exists with very high probability .this is one of the main results of this paper .[ thm : number of nt codewords ] let be the shaping rate in code construction ii . let _ _ be a set of probability mass functions , which is a closure of its interior .let designate the probability of finding at least one codeword which satisfies the constraints ( type belongs to ) in a random set of size , generated according to i.i.d .probability mass function .let be the conditional limit probability mass function with respect to and , which achieves .then :  1 .  _ for any there exists , such that for  _ 2 ._ for any and , such that . 3 ._ for any and any finite , such that ( [ eq : rsplusdelta ] ) is satisfied , for ._  the probability that a codeword satisfies the constraints , is given by the sanov theorem ( [ eq : p(n)e ] ) .let be the set of types with denominator .since is the closure of its interior , it follows that is nonempty for .hence , we can find a sequence of probability mass functions such that and since by definition .this means that for any there exists , such that thus , for  where the inequalities follow from ( [ eq : type probability ] ) and ( [ eq : seq divergence ] ) .it yields that for , the probability that none of the codewords within a random set satisfies the constraints is since , this concludes the first part .  for any and the choice , furthermore , according to ( [ eq : sanov ] ) for every there exists and , such that for  hence for , and for , .thus and .this concludes the second part .furthermore ,  ^{2^{\rho\cdot n}p_{f}}.\end{aligned}\ ] ]  for , since , it yields that . for , this yields that for , and therefore .hence , .thus , we found the shaping rate which ensures that a random set has at least one suitable codeword .since the random sets are drawn independently , the probability that each one of the random sets has at least one suitable codeword is . using the last theorem with , we conclude that , i.e. , the shaped codebook ( code construction _ ii _ ) exists with high probability .in the previous section we have shown that using a large code with code rate and redundancy shaping rate a shaped subcode can be obtained .this subcode consists of codewords which satisfy the constraints with probability tending towards 1 .hence , the maximal number of codewords in the subcode is the last inequality follows since the number of chosen codewords can not exceed the size of the typical set of .this property holds as long as .whereas for , the codewords of the large code satisfy the constraints and .we conclude that the achievable rate would be the minimum between the number of codewords and the rate which achieves average error probability tending towards zero for . to establish this ratewe analyze the performance of several decoders , which are presented next .following our codebook construction method , we consider two types of decoders .the first is the matched decoder , which is aware of the codebook set and performs optimal decoding .the second decoder is the mismatched decoder , which is only aware of the large code and performs optimal decoding as if all codewords of might have been transmitted .this decoder is suboptimal , but occasionally more practical than the matched decoder . in the practical special case where the large codebook is uniform code, the mismatched decoder is readily available while the adaptation to the set might be extremely complex .  the achievable rate of code construction _i _ is known .it is the mutual information related to the probability mass function which generated the codebook and the probability mass function of the channel. the achievable rate of code construction _decoded with the matched decoder , however , is an open question . it is the objective of section [ sec : matched - decoder ] to show that the achievable rate , in this case , is bounded by .  on firstlook , mismatch performance analysis seems as a trivial exercise .the transmitted codeword is part of the large codebook and therefore the large code performance can be used .this is the naive approach .this type of assumption was taken ( without being mentioned ) by forney , using a decoder which is unaware of the selection process and stating that the performance and complexity of their decoder are not affected by shaping .a more careful inspection , however , reveals that the large code performance assumes the transmission of the typical codewords , while we transmit only non - typical codewords .these codewords may have worse distance spectrum towards their neighbors .the average error probability of the large codebook tends towards zero when .thus , the bound on using the naive approach is this bound is achievable in special cases only , for example the bsc channel , as will be shown .this is also correct for codes ( bad codes ) , which operate at working point that justifies the approximation of minimum distance , and this justifies the analysis of forney .another interesting special case is a large code construction in which all words have the same distance spectrum .obviously in such case the selection of specific codewords will not make a difference .for the awgn , geometrically uniform ( gu ) codes ( e.g. linear codes over rings ) are known only to mpsk constellation .this raises an interesting open question : is the mismatched approximation accurate for mpsk ?we know that gu codes generate good codes , and we can use them for the large uniform codebook . if the conditional limit theorem holds for gu codes over mpsk , then we can also know the choice of that will generate enough codewords that satisfy the constraints .the described random shaping code construction method highly resembles existing practical schemes . since well - known good codes( e.g. convolutional codes , ldpc , block codes ) typically generate uniform probability mass function , combined with a selection process the outcome is a shaped subcode .our results are bounds on the performance of random constructions which approximate practical systems , which are approached using uniform i.i.d .probability mass function , the right shaping rate and sufficiently large .next we present two examples of such practical schemes .the trellis shaping , is a sequence oriented approach .bit sign shaping , width=302,height=302 ]  the main idea is to use a trellis shaping code to create equivalence classes of coded sequences and then perform the shaping operation by choosing the minimum power sequence to be transmitted within each equivalence class .sign bit shaping ( sbs ) , is a special case of ts . in sbs ,two bits per symbol are modified by rate 1/2 trellis shaping code , having generating matrix and parity check matrix .let be the codeword length ( symbols ) , be the number of bits represented by each symbol and be a mapping function to the euclidean space .each symbol has unshaped bits , possibly coded by a channel code with rate , and two shaped sign bits , as shown in fig .[ fig : bit - sign - shaping ] .the sign bit sequence is where are information bits .the trellis decoder creates an equivalence class where and represents all binary sequences of length .the bits are mapped into equivalence class of symbols . in this schemethe goal is power minimization ; therefore the trellis decoder transmits the codeword with the minimal energy within the equivalence class , i.e. , or the minimal average energy therefore , the total information rate of this scheme is bits per symbol and the shaping rate is .the decoding process is within the large codebook , which includes all the equivalence classes .once the codeword has been decoded , the demapper extracts the shaped bits and the only additional operation required is reconstruction the shaped bits by this operation is error free in case the entire codeword was decoded correctly ( ) and since .the selection process in this scheme is to choose the codeword with the minimal average power in each equivalence class , as given in ( [ eq : avg eng ] ) .for very large , we claim that this selection is equivalent to selecting a sequence which satisfies the power constraint since the shaping rate is set by the shaping code and the constraint metric is given , there exists minimal such that each equivalence class has at least one such codeword .this is dual to choosing a constraint and finding the minimal for this constraint .the constraint in turn , defines the probability mass function of the selected codewords . recall that these sequences are very close to in distribution .the minimum average energy codeword also satisfies the constraint , thus it is one of the codewords in the previous method .although , the chosen codewords might differ , using the different methods , the outcome is the same .a code which satisfies the constraint and is very close to .thus , choosing the minimum and choosing the codeword satisfying the minimal is asymptotically the same . assuming that the large codebook is uniformly distributed , is the entropy maximizing maxwell - boltzmann probability mass function .note that in case of several constraints the methods are not longer equivalent since several constraints can not be represented by single metric , which can be minimized using the ts decoder at the transmitter .another example to the selection process is the scheme in , which was originally designed for dirty paper coding .this scheme creates a shaping code for each transmitted message of length using where is a sparse generation matrix , and are sparse parity check matrices , and is a fixed binary sequence of length . in this nested ldgm - ldpc schemethe matrix is designed to make the shaping code a good source code and thus suitable for average power shaping .the matrices and embed information and obtain good channel code properties . the encoder , given the information massage , performs a mean square error ( mse ) quantization of a scaled interference ( according to the dirty paper scheme ) using a lattice and transmits the quantization error .a special use of this scheme is the quantization of a zero interference , which results in transmitting a minimum energy codeword in the shaping code for each message .the union of the shaping codes is the large codebook and the subcode is the union of all minimum energy codewords selected from each shaping code .the information rate using this scheme is and the shaping rate can be obtained from the size of the shaping code , i.e. , the selection process is the same as in the sbs scheme , therefore the same conclusions apply to this scheme .in previous sections , we discussed code construction _ ii _ as a practical method to obtain a codebook which satisfies constraints .the objective of this section is showing the achievable rate of this code construction method and the matched decoder , with respect to the large codebook probability mass function and the set of constraints .furthermore , we bound the -dimensional distribution of the codewords , using the conditional limit distribution .these are new results , since the conditional limit theorem treated the distribution of the marginals of the selected codewords , but not the entropy rate nor the average mutual information when transmitted over a channel .let be i.i.d .probability mass function on and let be a closed convex set such that .let be the conditional limit probability mass function , which achieves .let us have a standard random codebook ( code construction _i _ ) which uses this distribution to generate the codewords .thus , is the probability to generate the codeword and is the probability to generate the channel output .let designate the jointly typical set in definition [ a_pxy ] , and let the decoder be the joint typicality decoder with respect to _ _ the i.i.d . probability mass function __ . let , designate the probability to generate the codeword and the channel output , respectively , using code construction _ii_. let .the next two lemmas upper bound , using and .these bounds are later used to derive the modified aep theorem for code construction _ii _ ( theorem [ thm : joint - aep - ccii ] ) .the aep , which defines the average probability of the decoding error , allows to deduct the achievable rate .[ lem : bounding ccii with cci]for every there exists such that for   since is i.i.d .probability mass function on , the probability to generate a codeword from probability mass function is where is the type of .the probability to generate the codeword , using code construction _, is where is the probability of the event , when generating from i.i.d . . since is a closed convex set , ( [ eq : bound on pn(e ) ] ) applies .furthermore , by pythagorean theorem for divergence ( * ? ? ?* theorem 11.6.1 ) , every satisfies if is a closed convex set and .thus , using ( [ eq : bound on pn(e ) ] ) and ( [ eq : pythagorean ] ) .  the probability to generate the codeword with respect to i.i.d .probability mass function using code construction( according to ( [ eq : seq .probability ] ) ) is hence , we conclude that there exists such that for   [ lem : bounding ccii with cci y]for every there exists such that for   the probability of is the sum of the joint probabilities over all the codewords .thus , there exists such that for  where the first inequality follows from lemma [ lem : bounding ccii with cci ] .one of the methods to obtain the achievable rate of a random codebook ensemble which was drawn according to an i.i.d .probability mass function is the channel coding theorem ( * ? ? ?* theorem 7.7.1 ) .it relies on the joint aep theorem ( * ? ? ?* theorem 7.6.1 ) .the aep establishes the probability that a codeword and a channel output are jointly typical with respect to their joint probability mass function in two cases . in casethe output is a noisy version of the input and in case of two independent variables .these two probabilities are essential to analyze the average probability of error and deduct the achievable rate such that the probability of error is tending towards zero when the codewords length tends towards infinity .the decoder is the joint typicality decoder which operates as follows .an error occurs if the transmitted codeword is not jointly typical with the received sequence or when more than one codeword is jointly typical .the aep theorem is not suitable for code construction _ii _ , since the codewords are generated by a selection process and not randomly drawn according to an i.i.d .distribution . nevertheless , we know that their probability mass function converges in a sense to the i.i.d .conditional limit probability mass function .thus , we can use this conditional limit probability mass function to redefine the decoder and derive a modified aep theorem ( theorem [ thm : joint - aep - ccii ] ) .this theorem is used to show one of the main results of this paper , that ( [ eq : matched achievable rate ] ) is the achievable rate under matched decoder .we conclude that the same rate can be achieved either by code construction_ ii _ or by random i.i.d .generation according to , i.e. if the large codebook is uniformly distributed .it follows that code construction _ ii _ does not lose rate under matched decoding in this case .[ thm : joint - aep - ccii](joint aep for code construction ii )let let let designate the jointly typical set in definition [ a_pxy ] , __ .  1 ._ as ._ 2 .  _ for every , the exist such that for  _\1 ) since , it yields that where is the type of .thus , where the last inequality follows from ( * ? ? ?* theorem 17.3.3 ) and is the distance between probability mass functions .the conditional limit theorem ( * ? ? ?* theorem 11.6.2 ) implies that there exists such that for  furthermore , ( * ? ? ?* lemma 11.6.1 ) yields that thus if , the distance since there exists such that from ( [ eq : first aep cond ] ) and ( [ eq : div qx q star ] ) it yields that there exists such that for   in a similar manner , where is the type of .next we bound the divergence where is the joint type of and the inequality follows from the log sum inequality .the typicality of the channel implies that there exists such that for  put and .since there exists such that from ( [ eq : div qx q star]),([eq : sec aep cond ] ) and ( [ eq : div py_x py_x typicality ] ) it yields that there exists such that for  using similar arguments , there exists such that for  thus , there exists , such that this concludes the proof of first property .\2 ) if , are independent , then for _ _ where the first inequality follows from lemmas [ lem : bounding ccii with cci ] and [ lem : bounding ccii with cci y ] , and the second follows from the joint aep theorem ( * ? ? ? * theorem 7.6.1 ) , with respect to the i.i.d . .  hence , theorem [ thm : joint - aep - ccii ] established the joint aep properties of code construction _ii _ with respect to the set .the joint aep properties of code construction _ ii _ are enough to show that for , the average error probability is tending towards zero with the block length .this can be shown by applying the channel coding theorem ( * ? ? ?* theorem 7.7.1 ) and the joint typicality decoder ( ) to code construction _ii_.  according to theorem [ thm : number of nt codewords ] , to ensure that each random set of size includes a codeword which satisfies the constraints with very high probability .therefore , is an achievable rate , taking the maximal rate .since good codes for uniform probability mass function are readily available , the special case where the large codebook is uniformly distributed , i.e. , is of practical importance .it is easy to show that in this case , the conditional limit probability mass function achieves , i.e. , maximize the entropy within .it yields that , since and .this is the achievable rate under matched decoding in this special case .since the selected codewords maximize the entropy of the input and not the mutual information , this is only a lower bound on the capacity subject to the constraints .nevertheless , for the average power constraint on awgn with discrete constellation , the distribution which maximizes the capacity is approximately the distribution which maximizes the entropy .this observation can be verified numerically , using the modified baa .hence , the achievable rate is very close to the maximal rate in this special case .the previous sections treated code construction and the achievable rate under matched decoding . recall that practical schemes , however , use a mismatched decoder from complexity considerations .this decoder is only aware of the large codebook and performs optimal decoding as if all the codewords might have been transmitted .it is suboptimal , but occasionally more practical since the adaptation to the shaped subcode might be extremely complex , and the decoder for the large code is readily available .this section analyzes the mismatched decoder beyond the naive approach , which showed that the non - typical codewords within the large codebook require special treatment .we present two novel lower bounds for the achievable rate using the mismatched decoder .our first lower bound is based upon gallager s error exponent bounding technique .the key here is to perform maximum likelihood decoding using the entire codebook , while taking into account that only a subcode was transmitted and the true distribution of the transmitted codewords .based on this analysis , we redefine the average error probability and present a new error exponent ( [ eq : subcode error prob ] ) .it is important to note that this bound is suitable for any large codebook ensemble , as long as it is generated according to i.i.d .distribution and not necessarily a uniform distribution . using the new error exponent , we show that ( [ eq : achievable transmission rate . ] ) is an achievable rate for the mismatched decoder .our second lower bound is based upon the joint typicality decoder .this decoder is not suitable for non - typical codewords since it declares an error when the input or the output is not typical .thus , we define a modified joint typicality ( mjt ) decoder , which decodes solely based on joint typicality between the input and the output with respect to the large codebook distribution . the mjt achievable rate bound is presented in ( [ eq : achievable rate mtp ] ) .it might seems surprising that decoding according to a wrong distribution might work and indeed this bound is correct in special cases only ( lemma [ lem : first error event ] ) .one of them is uniform distribution of the large codebook and additive channels .thus , the awgn channel can be treated using this approach .more examples are shown in the sequel .   as already claimed , while the matched decoder achieves higher transmission rates , the mismatched decoder is more practical . in this sectionwe use the gallager bounding technique to obtain a lower bound on the maximal achievable rate for the shaped subcode and the mismatched decoder .the mismatched decoder in this section is the maximum likelihood decoder , which uses the entire codebook in the decoding process .the average error probability , however , is redefined ( theorem [ thm : subcode error ] ) to take into account the true distribution of the transmitted codewords .thus , a new error exponent is presented . using this error exponent , theorem [ thm : achievable rate ] shows an achievable rate for the mismatched decoder .it is important to note that this bound is suitable for any large codebook ensemble , as long as it is generated according to i.i.d .distribution and not necessarily the uniform distribution .[ thm : subcode error]let be a memoryless channel probability mass function . for a given number of codewords of block length .ensembleclosed convex setsuch that .let be the probability mass function which achieves and .define a corresponding ensemble of subcodes , generated by code construction ii with respect to .suppose that an arbitrary subcode codeword , , enters the encoder and that maximum - likelihood decoding using the entire code is employed .then there exists such that for , the average probability of decoding error over this ensemble of subcodes , for any choice of , , is bounded by   let designate the average error probability of codeword , when using code construction _ii _ to create the subcode ensemble and maximum - likelihood decoding using the entire code .let denote the probability of codeword within the subcode ensemble .let be the probability of error conditioned on codeword entering the encoder , the particular codeword , the received sequence and .then , the average decoding error probability for the codeword , when maximum - likelihood decoding using the entire code is employed , is following the derivation of the gallager error exponent ( proof of ( * ? ? ?* theorem 5.6.1 ) ) , the probability of error given , and is by substituting ( [ eq : perror givem m xm y ] ) into ( [ eq : avg pem ] ) and recognizing that is a dummy variable we obtain using lemma [ lem : bounding ccii with cci ] , there exists such that for , this yields that , where the equality follows since all the probability mass functions are independent and identically distributed .each subcode codeword is bounded by ( [ eq : avg pem bound ] ) , which does not depend on .thus , the average probability of decoding error is and we obtain the bound .thus , the last theorem showed the modified error exponent for the non - typical codewords .note that the main difference in the derivation lays in ( [ eq : avg pem ] ) , since the averaging is over the actual distribution of the codeword ( ) and not the typical codewords in the large codebook .since the number of codewords exponentially depends on the large code rate , with , the bound can be expressed as where once the error exponent is obtained , the achievable rate can be deducted .the following theorem shows the large code rate , for which the average error probability is tending towards zero .this rate is later translated into an achievable transmission rate , using ( by code construction _ii _ ) .[ thm : achievable rate]suppose that for every there exists such that for every and , let then for every and , the average error probability .according to the limit definition , for every there exists such that taking , this yields thus , for every the exists and the corresponding such that and for  and therefore   theorem [ thm : achievable rate ] showed that when , the average error probability tends towards zero . furthermore , according to theorem [ thm : number of nt codewords ] , .hence , the average error probability , for the shaped code if this is the achievable rate under the mismatched decoding using the gallager exponent bounding technique .the closed form of is given at appendix ( [ eq : iqp ] ) .this yield that , is an achievable transmission rate using the gallager bounding technique .this relies on the average error probability and the maximal number of codewords using code construction _ ii , _ which was discussed in section [ sub : number - of - codewords ] .we use the following lemma to verify that the mismatched decoder performs is worse than the matched .let be memoryless channel and , be input distributions .let , be the corresponding channel outputs , such that .then ,   according to the definition of divergence where the inequality follows from the log - sum inequality .  using and , we conclude that therefore as expected .our second lower bound for mismatched decoding is based on the joint typicality decoder analysis . in the original configuration , however , this decoder is not suitable for non - typical codewords in , since it declares an error when the input or the output is not typical according to .thus , we define a modified joint typicality decoder . the mjt decoder is aware only of the large codebook , and decodes solely based on joint typicality between the input and the output with respect to the large codebook probability mass function .it might seems surprising that decoding according to a wrong probability mass function might work and indeed this bound is correct in special cases only .lemma [ lem : first error event ] , provides a sufficient condition on and the channel , such that the mjt decoder is applicable .we restrict our analysis to this particular case .we bound the average error probability of the shaped subcode , created by code construction _, using the mjt decoder to obtain an achievable rate .suppose that a codeword with index was sent through the channel and a noisy sequence was received .the original joint typicality decoder operates as follows ; it examines all the codewords in the large codebook and establishes which codewords are jointly typical with the received sequence . since , and are typical with respect to and , respectively ; they are not typical with respect to and . therefore , the joint typicality decoder declares an error for each transmitted codeword .for this reason , the typicality of and with respect to , is disregarded .let us define the modified joint typicality set and the modified joint typicality decoder .[ m_pxy]the modified jointly typical set with respect to the probability mass function is the set of sequences with empirical entropies-close to the true entropy :    the modified joint typicality decoder operates as follows ; first it examines all the codewords in the large codebook and establishes which codewords are mjt with the received sequence , i.e. , belong to the set .an error occurs , either when the sent codeword is not mjt with the received sequence or when more that one codeword is mjt with .let us define the event that a codeword is mjt with the output as , .the event ( the complete to ) is the first error event , i.e. , the sent codeword is not mjt with the received sequence .thus , the probability of error is therefore the union of these events .let us now define an event , where the received sequence is typical with respect to the probability mass function , i.e. , and as the complete to . using the aep property for code construction _ for sufficiently large .hence , the probability of error is bounded    where the last inequality follows from the union bound and since .first , we analyze , which is equal to . the following lemma introduces a sufficient condition such that this probability is sufficiently small . in the general case, however , the decoder will not choose the right codeword , since the probability of joint typicality according to a wrong probability mass function ( ) is extremely low .[ lem : first error event]for sufficiently large  if where is a constant .  since , it yields that ,\end{aligned}\ ] ] where is the type of and is the type of the channel , i.e. , of given .hence , \right.+\nonumber \\  & \sum_{x\in\mathcal{x}}q_{\boldsymbol{x}\left(1\right)}\left(x\right)\left[h_{p_{\boldsymbol{y}\mid\boldsymbol{x}\left(1\right)}}\left(y\mid x = x\right)-h_{p}\left(y\mid x = x\right)\right.+\nonumber \\  & \left.\left.d\left(p_{\boldsymbol{y}\mid\boldsymbol{x}\left(1\right)}\left(y\mid x = x\right)\parallel p\left(y\mid x = x\right)\right)\right]\right|\leq\nonumber \\  & \sum_{x\in\mathcal{x}}q_{\boldsymbol{x}\left(1\right)}\left(x\right)\left[\left|h_{p_{\boldsymbol{y}\mid\boldsymbol{x}\left(1\right)}}\left(y\mid x = x\right)-h_{p}\left(y\mid x = x\right)\right|\right.+\nonumber \\  & \left.d\left(p_{\boldsymbol{y}\mid\boldsymbol{x}\left(1\right)}\left(y\mid x = x\right)\parallel p\left(y\mid x = x\right)\right)\right],\end{aligned}\ ] ] where the last derivation is correct since for every . using the typicality of the channel and the same arguments as in theorem [ thm : joint - aep - ccii ] , for every there exists such that for  for every with probability larger than .we conclude that for  i.e. , .  a special case where condition ( [ eq : entropy equality ] ) is satisfied is uniform probability mass function and a channel which satisfies . binary symmetric channel ( bsc ) and additive noise channels , are examples for which this property is satisfied .hence , for these channels and uniformly distributed large codebook for sufficiently large .next we bound the probability , .the event is equivalent to . since by the code generation process , and are independent for , so are and for . since was generated according to i.i.d . and was generated from code construction _ thus , using lemma [ lem : bounding ccii with cci y ] the event is easier to analyze .it is equivalent to the event , where are drawn according to i.i.d . , is the type of and according to sanov s theorem ( * ? ? ?* theorem 11.4.1 ) , the probability that has type is bounded by where to obtain the minimal divergence in , we construct the following lagrangian and its derivative the solution is given in a parametric form where the lagrange multipliers should satisfy the constraints in ( [ eq : pi constraints ] ) .  finally , by substituting ( [ eq : first error event ] ) and ( [ eq : second error event ] ) into ( [ eq : error prob bound ] ) , the error probability can be further bounded hence , for the error probability tend towards zero for . according to theorem [ thm : number of nt codewords ] , to ensure that each random set of size includes a codeword which satisfies the constraints with very high probability .it yields that for , the minimum between and the maximal amount of codewords ( [ eq : maximal number of code words ] ) is an achievable rate using this bounding technique .in the previous sections we discussed code construction _ ii _ , as a method to obtain shaped codebooks . we have shown the maximal amount of the available codewords and achievable rates for the matched and the mismatched decoders . in this sectionwe analyze several special cases .the first is the case where the large codebook is generated according to an i.i.d .uniform distribution and the channel satisfies the condition in lemma [ lem : first error event ] . in this case , all the bounds are applicable .the more general case , where the large codebook is non - uniform , is later discusses in context of several special cases : the bnsc and awgn with gaussian large codebook .the details are given next .the special case where the large codebook is uniformly distributed on the support is of practical importance .furthermore , although the gallager bound can be obtained for any i.i.d . , the mjt bound can be obtained only when condition ( [ eq : entropy equality ] ) is satisfied . in order to compare all the bounds ,the following examples focus on a special case of ( [ eq : entropy equality ] ) , where and the channel satisfies for any .in particular , we analyze the awgn and bsc channels .our first observation is regarding the achievable rate for the matched decoder .it is easy to show that for uniform , the conditional limit probability mass function does not maximize the mutual information , but the entropy within the set , i.e. , thus , code construction _ii _ can not achieve the maximal rate even for the matched decoder , since generally different probability mass functions maximize the mutual information and the entropy subject to the constraints .nevertheless , for the average power constraint over awgn with discrete input constellation , the probability mass function which maximizes the mutual information is approximately the probability mass function which maximizes the entropy .hence , the achievable rate is very close to the maximal rate .this can be verified using the modified baa .  for the mismatched decoder ,the gallager bound takes the form of since the parametric form of the probability mass function of the mjt bound in ( [ eq : opt p ] ) is also simplified into where , and ( represents all the constants ) satisfy the constraints in ( [ eq : pi constraints ] ) .there is no closed form solution in the general case and the lagrange multipliers should be obtained numerically .once these parameters are found , the mjt bound takes the form of   the maximal amount of codewords is not a limitation in this case , since the entropy of the large codebook is . thus taking the maximal , we can obtain the typical set of ( up to the ) , since .for more details recall section [ sub : number - of - codewords ] .the first example that we analyze is the memoryless awgn channel with uniformly distributed large codebook , which is a special case of ( [ eq : symmetry cond ] ) .let be a discrete constellation with cardinality .* * let be a random vector of length , representing the transmitted symbols , where .we allow the amplitude levels of the constellation points to change to model an amplifier at the transmitter .this is done by multiplying the constellation by a real and positive scaling parameter .let be an i.i.d .gaussian noise vector , where are i.i.d .gaussian noise samples , normalized to unit variance .let be the random vector at the receiver , such that the transmission is subject to an average power constraint , where the maximum average power allowed is .since the scaling of the constellation is a degree of freedom , for each scaling the power constraint can be formulated as using .the corresponding decision rule for choosing codewords is therefore which is equivalent to for example , let the constellation be ( 4-pam ) , and the power constraint be . for each scaling , is a set of probability mass functions which satisfy the constraint . in particular , lets consider and the corresponding sets , and .probability mass functions corresponding to and .,width=302,height=302 ]  the probability mass functions  in fig .[ fig : probability - mass - functions ] satisfy the constraints in , and , respectively .it can be seen that larger scaling is required as the distribution becomes more `` gaussian '' .thus , there is a tradeoff between the scaling parameter and how much the distribution is `` gaussian '' or non - uniform .thus , we have proved that ( [ eq : rm_uniform ] ) , ( [ eq : gallager rate uniform ] ) and ( [ eq : mjt rate uniform ] ) are achievable rates for the matched and mismatched decoders , corresponding to large code distribution and the constraints sets . since is a degree of freedom , we still need to establish the optimal scaling parameter , which attains the maximum of each bound .this also dictates the optimal decision rule to choose the codewords in the shaped code construction process .hence , the achievable rate using the matched decoder is it is also known that the maxwell boltzmann distribution achieves the maximal entropy subject an average power constraint , therefore the conditional limit probability mass function is the parameter is chosen such that the constraint is satisfied with equality , i.e. , which follows from the concavity of the entropy .the maximal rates for the mismatched gallager and mjt bounds are also calculated with respect to the optimal for each bound , such that and using ( [ eq : mb dist ] ) .achievable rates 16-pam , awgn , width=302,height=302 ]  achievable rates 16-pam , awgn , low snr.,width=302,height=302 ]  achievable rates 16-pam , awgn , high snr.,width=302,height=302 ]  figs .[ fig : awgn16pam]-[fig : awgn16pam - high snr ] present the maximal achievable rates using the matched decoder and our two lower bounds for the mismatched decoder versus snr . the scaling parameter is optimized for each bound at each snr pointwe keep the noise variance constant ( ) and change the transmission power to control the snr .the large codebook is uniformly distributed over 16-pam .these rates are compared to the capacity of the power constrained continuous awgn channel , which is denoted by `` capacity '' .they are further compared to the uniformly distributed 16-pam constellation , which is denoted by `` uniform '' .it can be seen that the matched decoder shaping curve is very close to the capacity curve up to the rate of 3 bps .the gap to capacity in high snr ( around 3 bps ) is approximately 0.1 db , 0.2 db for the matched and the mismatched decoders , respectively . in low snrthe gap is approximately 0 db , 0.3 db for the matched and the mismatched decoders , respectively .thus , the main conclusion from the graphs is that for pam over awgn there is no significant loss due to mismatched decoding , and that shaping gain can be attained .this conclusion actually justifies the works of forney and others ( unknowingly ) , who used mismatched decoding .our second example is the binary symmetric channel , in which the input is inverted with probability .the mutual information of this channel is where is the input probability mass function and .bsc : matched and the mismatched decoder versus , for and uniform large codebook.,width=302,height=302 ]  this channel is another special case of ( [ eq : symmetry cond ] ) , for ( uniform large codebook ) . the hamming power constraint is very common for this channel .thus , let be the constrained probability mass functions set where is the maximal number of 1 s allowed , and let be the conditional limit probability mass function which attains minimum in . using the convexity of the divergence in , it implies that the constraint is satisfied with an equality .thus , .the achievable rate for the matched decoder is therefore .next we obtain the achievable rates and for this channel , using the gallager and the mjt bounding techniques .the gallager bound is simplified to the rate may be obtained numerically , using the parametric form in ( [ eq : opt p uniform ] ) .alternatively , let us guess that the lagrange multipliers are , with .subsequently , satisfies the constraints in .this can be easily verified by and which assures that the second constraint is satisfied .hence , the rate is achievable and it coincides with the gallager bound .the large code has error probability tending towards zero if and therefore using ( [ eq : uniform rs ] ) , and .it follows that the three bounds ( gallager , mjt and the naive approach ) coincide in this particular case .it is known that the bsc capacity is attained for uniform input probability mass function , and that linear codes achieve the bsc capacity .such codes have the same distance spectrum for all the codewords and therefore the same error probability . in particular, the error probability of the non - typical codewords equals to the error probability of the typical .this example provides some insight to why the naive approach is achievable in this case ( as shown using gallager and mjt ) .the rate loss is comparing to the matched decoder .results are illustrated in fig .[ fig : bsc ] , using .the third example is the binary non - symmetric channel .this channel is an example where condition ( [ eq : symmetry cond ] ) is not satisfied .thus , for the practical case of uniform large codebook , only the gallager bound can be obtained .let the input be .the input is complemented with probability if and if . the achievable rate of this channel is where is the input probability mass .bnsc : matched and the mismatched decoder versus , for , and .,width=302,height=302 ]  let the hamming power constraint be the same as for the bsc channel ( [ eq : hamming power const ] ) . fig .[ fig : bnsc_uniform-1 ] shows the achievable rates for the matched and mismatched decoders for and .the matched decoder outperforms the mismatched as expected . at ,the typical codewords are transmitted since the large codebook is uniform , thus the two decoders are the same . the naive approach describes the performance of the typical codewords . in this case , the mismatched performs better than the naive approach , thus the non - typical codewords have lower average probability of error than the typical codewords .the non - uniform large codebook is the more general case .although it is not common to use non - uniform distributions in practical system , we present several toy examples for the sake of completeness .the first example is the bnsc which satisfies the general condition ( [ eq : entropy equality ] ) ( required for the mjt bound to be valid ) .otherwise , we can not compare all the bounds .the second example is generating a gaussian codebook with power from a larger gaussian codebook with average power , where .   in this sectionwe consider the binary non - symmetric channel , such that the general condition ( [ eq : entropy equality ] ) is satisfied . in order to be able to demonstrate all our bounds ( i.e. , satisfy ( [ eq : entropy equality ] ) ) we have to choose a large codebook which is not uniformly distributed .in particular , the large codebook is according to which solves i.e. , the hamming power constraint is used once again , limiting the average number of ones in a codeword to be lower than in the large codebook , i.e. .  bnsc : matched and the mismatched decoder versus , for , and .,width=302,height=302 ]  fig .[ fig : bnsc ] shows the achievable rates for and , using the matched and the mismatched decoders .the probability mass function of the large codebook ( according to ( [ eq : pbnsc ] ) ) is . in this example , the gallager and the mjt are also above the naive approach .furthermore , the gallager and the mjt bounds are very close to each other .all bounds coincide at , since the typical codewords are transmitted .   in the following example we consider the situation where the large codebook has codewords with average power , according to gaussian distribution . whereas , the transmitted codewords have much smaller power .it is easy to show that the conditional distribution in this case is also gaussian with average power .thus , the capacity of the gaussian channel is achieved for the matched decoder .the mismatched decoder derivation is less trivial .let us first derive the gallager bound using where is gaussian with variance and zero mean , and is gaussian with variance and zero mean . in a similar manner ,  assuming that the channel is awgn with variance and zero mean .in the scenario where and , the gallager bound is deduced to this example is related to nested lattice shaping .the nested lattice scheme consists of fine lattice and coarse lattice .the fine lattice is the `` large codebook '' ( having infinite number of codewords ) .this lattice represents all the cosets .the coarse lattice provides the shaping region , such that it bounds the coset representatives ( the minimum power codewords in each coset ) , which construct the shaped codebook .a practical decoding scheme for nested lattices is lattice decoding , which is in fact very similar to the mismatched decoder described in this paper .the large gaussian codebook can be viewed as a region in the infinite lattice , which is large enough such that decoding in that codebook is the same as decoding in infinite lattice for codewords transmitted from the small codebook .it is interesting though to compare our bound ( [ eq : gallagergaussian ] ) to the performance of lattice decoding , and indeed the result is identical .in this paper , we proposed a method to analyze practical shaping schemes , in which a shaped subcode is chosen from a larger code ( in most cases uniformly distributed large codebook ) .the codewords in the shaped codebook satisfy as set of constraints .we provide a theoretical framework for such analysis , using random coding and a selection process subject to constraints .in particular , we have found the achievable rates of the matched and the mismatched decoders , where the last is unaware of the selection process at the transmitter .this decoder is suboptimal , but occasionally more practical since it does not have to repeat the selection process at the receiver . in general , we show that the large code performance does not dictate the subcode performance , and that the non - typical codewords require different analysis .hence , we obtained two novel achievable bounds for the mismatched decoding using a modification on the gallager and the joint typicality bounding techniques .we examine several special cases in detail . in the special case of bsc , however , the large code analysis is actually valid and can be justified using capacity achieving linear codes . for m - pam over awgn the loss due to mismatcheddecoding is not very significant .we show other examples where there is a larger difference between the matched and mismatched decoding .  to obtain in closed form, we calculate the derivative of with respect to .let us define substituting ( [ eq : a])-([eq : d ] ) into ( [ eq : e0qp ] ) differentiating with respect to , yields such that  finally , using these terms at , yields subsequently , using the definitions of entropy ( [ eq : entropy ] ) , conditional entropy ( [ eq : conditional entropy ] ) , mutual information ( [ eq : mutual information ] ) and the kullback - leibler information divergence ( [ eq : divergence ] ) we conclude that this gives in closed form .j. barron , b. chen , and g. w. wornell , `` the duality between information embedding and source coding with side information and some applications , '' _ ieee trans .inform . theory _49 , pp . 1159 - 1180 , may 2003 .

Abstract: shaping gain is attained in schemes where a shaped subcode is chosen from a larger codebook by a codeword selection process . this includes the popular method of trellis shaping ( ts ) , originally proposed by forney for average power reduction . the decoding process of such schemes is mismatched , since it is aware of only the large codebook . this study models such schemes by a random code construction and derives achievable bounds on the transmission rate under matched and mismatched decoding . for matched decoding the bound is obtained using a modified asymptotic equipartition property ( aep ) theorem derived to suit this particular code construction . for mismatched decoding , relying on the large codebook performance is generally wrong , since the performance of the non - typical codewords within the large codebook may differ substantially from the typical ones . hence , we present two novel lower bounds on the capacity under mismatched decoding . the first is based upon gallager s random exponent , whereas the second on a modified version of the joint - typicality decoder .   constellation shaping , aep , mismatched decoding , gallager random exponent .