
where, for user \(k\) , \({\mathcal {D}}_k := \lbrace {\mathcal {U}}\ | \ {\mathcal {U}}\subseteq \hat{{\mathcal {K}}}, |{\mathcal {U}}| = \hat{t}+1, {\mathcal {U}}\ni k\rbrace \)  and \({\mathcal {I}}_k := \lbrace {\mathcal {V}}\ | \ {\mathcal {V}}\subseteq \hat{{\mathcal {K}}} \setminus k, |{\mathcal {V}}| = \hat{t}+1\rbrace \)  are the set of all desired and interfering message indices in \({\mathbf {x}}_{\hat{{\mathcal {K}}}}\) , respectively. Also, \(Y_{{\mathcal {U}},k}\)  represents the data term/packet for user \(k\)  in the nested codeword \(x_{{\mathcal {U}}}\)  (explained in more details shortly after), and \(|Y_{{\mathcal {U}},k}|\)  is the length of \(Y_{{\mathcal {U}},k}\)  in bits used for relative weighting. Note that (REF ) is a modified version of max-min problem in [1]} (equations (37)-(41)) with additional weights to reflect the multi-rate modulation transmission. Hence, problem (REF ) is non-convex and NP-hard, which can be sub-optimally solved using the successive convex approximation method [1]}. However, a detailed discussion is left for the extended version of this paper due to lack of space. Finally, problem (REF ) is also solved for power loading (without inter-stream interference terms) when ZF beamformers in (REF ) are used.
