Recently, the use of NNs to emulate the end-to-end signal transmission in
communication systems has drawn considerable attention [1]}, [2]}, [3]}, [4]}, [5]}, [6]}. In this work, based on the NN structure developed in [2]}, we employ a composite structure comprised of 4 CNNs for the hybrid AD transceiver design, as shown in Fig. REF . Specifically, the two CNNs labeled “P_NN” and
“\(\theta _F\) _NN” in the top dotted rectangle at the transmitter,
implement the digital transmit beamforming matrices \(\mathbf {P}_k\)  and
the analog transmit beamforming phase matrix \(\mathbf {\theta }_F\) ,
respectively, where the original symbol vectors \(\mathbf {b}_k\)  are
converted into the precoded signals. The CNNs labeled “W_NN” and
“\(\theta _U\) _NN” in the bottom dotted rectangle at the receiver
implement the digital receive beamforming matrices \(\mathbf {W}_k\)  and
the analog receive beamforming phase matrices \(\mathbf {\theta }_{U_k}\) ,
respectively, which use the received signals \(\mathbf {y}_k\)  to
generate \(\tilde{\mathbf {b}}_k\) . In the training stage, the SER is
obtained by comparing the original symbol vector \(\mathbf {b}_k\)  with
the detected symbol vector \(\hat{\mathbf {b}}_k\)  and the SER function
(REF ) is defined as the loss function. Since
standard CNN implementations (e.g. Tensorflow) cannot process
complex-number matrices directly, the channel matrix \(\mathbf {H}_k \in \mathbb {C}^{N_r \times N_t}\)  is transformed into a \(2 \! \times \! N_r\! \times \! N_t\)  dimensional real-number tensor
\(\tilde{\mathbf {H}}_k\) . The inputs of “\(\theta _F\) _NN” and
“\(\theta _U\) _NN” are \(\tilde{\mathbf {H}}_k\)  and \(\mathbf {b}_k\) , and
the outputs are used to generate a low-dimensional equivalent channel
\(\mathbf {H}_{eq}\) , which are served as the inputs of “P_NN” and
“W_NN”.
<FIGURE>