Artificial Intelligence Empowered Multiple Access for Ultra Reliable and Low Latency THz Wireless Networks

Terahertz (THz) wireless networks are expected to catalyze the beyond fifth generation (B5G) era. However, due to the directional nature and the line-of-sight demand of THz links, as well as the ultra-dense deployment of THz networks, a number of challenges that the medium access control (MAC) layer needs to face are created. In more detail, the need of rethinking user association and resource allocation strategies by incorporating artificial intelligence (AI) capable of providing"real-time"solutions in complex and frequently changing environments becomes evident. Moreover, to satisfy the ultra-reliability and low-latency demands of several B5G applications, novel mobility management approaches are required. Motivated by this, this article presents a holistic MAC layer approach that enables intelligent user association and resource allocation, as well as flexible and adaptive mobility management, while maximizing systems' reliability through blockage minimization. In more detail, a fast and centralized joint user association, radio resource allocation, and blockage avoidance by means of a novel metaheuristic-machine learning framework is documented, that maximizes the THz networks performance, while minimizing the association latency by approximately three orders of magnitude. To support, within the access point (AP) coverage area, mobility management and blockage avoidance, a deep reinforcement learning (DRL) approach for beam-selection is discussed. Finally, to support user mobility between coverage areas of neighbor APs, a proactive hand-over mechanism based on AI-assisted fast channel prediction is~reported.

Abstract-Terahertz (THz) wireless networks are expected to catalyze the beyond fifth generation (B5G) era. However, due to the directional nature and the line-of-sight demand of THz links, as well as the ultra-dense deployment of THz networks, a number of challenges that the medium access control (MAC) layer needs to face are created. In more detail, the need of rethinking user association and resource allocation strategies by incorporating artificial intelligence (AI) capable of providing "real-time" solutions in complex and frequently changing environments becomes evident. Moreover, to satisfy the ultrareliability and low-latency demands of several B5G applications, novel mobility management approaches are required. Motivated by this, this article presents a holistic MAC layer approach that enables intelligent user association and resource allocation, as well as flexible and adaptive mobility management, while maximizing systems' reliability through blockage minimization. In more detail, a fast and centralized joint user association, radio resource allocation, and blockage avoidance by means of a novel metaheuristic-machine learning framework is documented, that maximizes the THz networks performance, while minimizing the association latency by approximately three orders of magnitude. To support, within the access point (AP) coverage area, mobility management and blockage avoidance, a deep reinforcement learning (DRL) approach for beam-selection is discussed. Finally, to support user mobility between coverage areas of neighbor APs, a proactive hand-over mechanism based on AI-assisted fast channel prediction is reported.
Index Terms-Artificial intelligence, machine learning, medium access control, metaheuristic optimization, reinforcement learning, terahertz wireless networks, ultra reliable and low latency communications.

I. INTRODUCTION
Bringing the fiber quality of experience (QoE) into the wireless world is a pillar promise of beyond fifth generation (B5G) networks [1]. In this direction, the commercialization of terahertz (THz) wireless networks capable of supporting bandwidth-hungry applications, like holographic reality, collaborative robots, and self-driving vehicles, without compromising reliability or increasing latency, is a key objective [2]. However, THz wireless systems are susceptible to line-of-sight (LoS) obstructions [3]. Moreover, due to the shorter ranges of THz frequencies, THz wireless networks are expected to be ultra-dense [4], [5]. Thus, the congestion of static or dynamic objects/users would lead to LoS blockages.
This poses a number of important challenges, including user equipment (UE) to access point (AP) association, radio resource (RR) allocation, and mobility management (MM) that the THz wireless systems medium access control (MAC) need to address. In particular, a joint UE-AP association and RR allocation strategy capable of minimizing the LoS blockage, while maximizing the network performance, under an ultralow latency demand is required. This inspired the design of joint UE-AP association and RR allocation strategies, via the implementation of metaheurestic [6] or machine learning (ML) approaches [7], [8]. However, the aforementioned contributions overlook an important factor, i.e., the negative impact of blockage [9].
From the MM point of view, two MAC-related research directions, namely beam selection and pro-active hand-over, have been followed [10]. In more detail, in [11], a random forest (RF) classification-based beam-selection scheme was reported that alleviates the computational intensity of conventional approaches, while boosting performance in terms of data-rate maximization. In [12], a deep learning (DL)based, proactive hand-over and beam-selection approach for THz drone communications was presented. Both the aforementioned approaches, require a sufficient number of training samples and lack the ability to adapt to continuously changing wireless environments.
A MAC protocol for ultra-reliable and low-latency (URLL) THz wireless networks should, by-design, countermeasure against the effects of blockage, provide fast adaptation to changing environments, and support mobile UE (MUE). However, no holistic approaches that jointly address these challenges have been proposed. Motivated by this, this article aims to set the stage for the development of an intelligent MAC protocol that enables centralized UE-AP association, local (i.e., w.r.t. individual APs) and adaptable beam-selection, as well as pro-active hand-over. Specifically, the contribution of this work is as follows: • We present a joint UE-AP association and RR allocation with blockage avoidance approach, based on a novel, hybrid metaheuristic-ML framework, capable of reducing the association latency by more than 3 orders of magnitude. • To support mobility management and blockage avoidance within the AP coverage area, we propose a DRL approach that enables flexible and fast adaptation. • Finally, to reliably support MUE that moves between AP coverage areas, we present a pro-active hand-over approach, built upon a novel AI-assisted fast channel prediction framework.
II. AI-ASSISTED JOINT USER ASSOCIATION AND RR ALLOCATION FOR BLOCKAGE AVOIDANCE Problem description: The association and blockage minimization problem aims at providing URLL connectivity by quickly associating UEs to APs, while satisfying a number of hard and soft constraints. The difference between hard and soft constraints is that the first ones determine the feasibility of the solution, while the latter the quality of the solution. Three hard constraints have been considered, namely: i) unique assignment, ii) grouped allocation, and iii) RR allocation efficiency maximization. The first constraint ensures that each UE is associated to a single AP. Note that the association process requires that the AP has enough available RR to cover the UE data-rate demands. The second constraint enables the association of multiple UEs to a single AP. The final hard constraint aims to maximize the RR usage at APs, without exceeding their resource availability, while using the minimal number of APs. On the other hand, one soft constraint need to be satisfied, namely LoS maintenance. This is translated to LoS blockages avoidance or minimization.
Why AI is needed: Efficiently solving the aforementioned problem in dense and changing environments is expected to play a vital role in optimal RR management towards delivering high QoE. However, conventional approaches based on constraint solving and combinatorial optimization techniques cannot meet the latency requirements for connection establishment, given the scale, density, coverage, and RR management requirements. For instance, the association strategy in THz wireless systems requires LoS blockage avoidance/minimization. Even in networks of moderate size, this problem quickly becomes intractable for classical constraint solving techniques, due to the combinatorial explosion in the space of possible assignments. An alternative is to use ML-based techniques to train models that learn to approximately solve such optimization problems. Such techniques trade optimality for efficiency, since the solutions are typically sub-optimal. On the other hand, once trained, such a model may predict solutions on unseen instances of the optimization problem in a tiny fraction of the time that a constraint solver would need, thus fostering efficiency in tasks that require realtime responsiveness and adaptivity to change, such as the tasks that we address here.
A hybrid metaheuristic-ML framework: We propose such an AI/ML-based approach to solve the UE-AP association problem, that combines classical search with ML. In particular, after formalizing the problem as a constraint satisfaction one and given a number of different instances thereof (snapshots), classical constraint solving techniques, assisted by metaheuristics to select among a rich library of sophisticated search algorithms, are used to derive optimal solutions to these snapshots. These solutions are then used to train and validate a classifier that learns to mimic the solver's behaviour by generalizing from the training snapshots, towards approximating solutions to unseen ones. As indicated above, the approach is generic enough to be applicable to any type of optimization problem, while, as we detail below, the it is capable of adapting to change, by retraining the model, on-the-fly, on new snapshots it exhibits low predictive performance on.
As illustrated in Fig. 1, the approach has an offline learning phase, which is used for training and validating the ML model responsible for UE-AP association and RR allocation, and an online learning phase that ensures adaptation to the dynamic wireless network. In the offline learning phase, a dataset that contains different instances of the network topology, i.e., UE, AP, and blockers positions, each AP RRs and each UE data-rate demands, is input to a metaheuristic optimizer. The optimizer employs a number of state-of-the-art algorithms, like tabu search (TS), simulated annealing (SA), late acceptance (LA), and strategic oscillation (SO) to extract the optimal UE-AP association, where the RR allocation and blockage avoidance constraints are satisfied for each instance.
The metaheuristic optimizer's output is split into training and validation data. The training data are forwarded into a relative labeling (RLB) scheme, which categorizes all APs in the network. The categorization creates three sub-classes that represent essential aspects of APs. These sub-classes are concatenated together to create a singular class, resulting in a multi-class label attribute. These sub-classes are created as follows: • Geographic Location: Using APs' coordinates, we divide each AP among 4 quadrants, namely: North-East, North-West, South-East, and South-West. • Resource Capacity: Considering the available RR among all APs, we create 10 sub-classes that quantize the RR capacity of each AP. • LoS Blockages: Considering the number of partial blockages that the i−th UE would face when establishing a direct connection with the j−th AP, we create 9 subclasses that quantize the level of partial blockage. Next, by treating the AP that was associated by metaheuristic optimization to a UE as ground truth, we append the corresponding RLB of this AP to complete the training dataset. The ML model is trained to predict this RLB, which are mapped back to an AP in a post-prediction step. Several ML models including generalized linear models (GLM), naive Bayes (NB), random forests (RF), gradient boosted trees (GBT), and deep learning (DL) can be employed. The model is validated using the validation data produced by the metaheuristic optimizer.
In the online learning phase, the trained ML model is used to perform UE-AP association and RR allocation on unseen Indicative results: We use the following scenario To demonstrate the performance of our approach and highlight some of the design decisions involved. A wireless network that operates in the 100 GHz band and consists of 741 UEs, 125 APs in a 1 km 2 disk-shaped area. The positions of UEs and APs are determined by independent Poisson point processes. The available RR at each AP, as well as the UE requested data-rates are modeled as independent uniform random variables. Both the APs and UEs locations are described by polar coordinates. A third element of the model is a tuple that describes an initial minimum-distance-basted UE-AP clustering. The cluster is obtained using k-means without accounting for the LoS blockage. This clustering reveals UEs that potentially block each others' LoS when facing a particular AP.
In the simulated scenario the theoretic upper bound of possible AP-UE combinations is 10 1553 . This quantifies the association complexity. The average blockage density gives the number of blockers a single UE would face on average when a direct link with an AP is considered. Finally, the pairs of UE-AP that have at least 1 UE blocking their LoS is 62.5%. Hence, the dataset presents a formidable challenge for the metaheuristic algorithm(s) to optimize the specified constraints. Figure 2 quantifies the performance of the different types of metaheuristic optimizers. Figure 2.a illustrates how the search algorithms' improvement over time (w.r.t. the optimization score), while Fig. 2.b depicts the traversal of the search space in terms of solutions processed per second. These results provide insights on how different algorithms perform given a specific dataset and optimization constraints. The two figures reveal that SA and LA dominate TS and SO.
After labeling the optimizer's outputs, we perform a 95-5 split, i.e., 95% data was split (using stratified sampling) for training and 5% kept for validation and we test the performance of GLM, NB, RF, GBT, and DL. Overall, the results of GBT are superior.  UEs. This shows that both methods have a good capability to provide network coverage. Moreover, the balance of both approaches is quantified. Note that "balance" shows how the resource use was balanced by using a certain number of APs. If lesser than total APs are used effectively, then energy consumption can also be reduced, which reduces the carbon footprint. We observe that the metaheuristic solution is more efficient as it employs 95 APs, while the ML solution uses 98 APs to satisfy the resources required by all the UEs. The QoE is determined by the provisioning of required RR and avoidance of LoS blockages. RR utilization can measured by overload or underload of capacity among the used APs. We observe that the metaheuristic solution has slightly overloaded 25 APs with overall capacity overloaded by 4.4% on average. The ML solution has overloaded only 20 APs, but the overall capacity has overloaded by 14.5% on average. This shows that although the ML model exhibits that it has recognized and learned to respect the resource satisfaction constraint, yet further training is needed. Moreover, the QoE is also influenced by the number of LoS blockers that are avoided or minimized. The metaheuristic optimizer was able to discover APs for all 741 UEs such that no partial blockers existed in the direct LoS link between them. In case of the ML, a vast majority of 643 UEs were associated with APs with no LoS blocker and 98 UEs were associated with APs, where the LoS blockers were minimized to 1. This result further confirms that the model learnt the pattern to avoid LoS blockages or minimizing them.
Latency is a non-functional criterion, which was not intended to evaluate the optimization performance, because the purpose of the offline optimization is merely to discover optimal UE-AP associations for the generation of training data. The ML model trained using this data is the actual artifact. Hence, the inference efficiency of the ML model for a given data-point is of actual interest to conclude whether the overall connection establishment latency meets the B5G level requirement. A data-point is a vector of UE features and the network snapshot at a particular point in time. The scoring (or inference) time is dependent on the hardware, where the ML model is deployed. In our case, the scoring time was 0.75 ms for 1 data-point, which shows a promising connection latency. However, if the GBT model is further pruned and feature engineering can further reduce the dimensions of data, then this time can be further reduced. We also recognize that it would be better to obtain predictions for multiple data-points, which means that a total of data-points that is equal to the number of UEs multiplied by the number of APs, needs to be predicted. The relative label predicted most often can then be reliably used to identify the optimal AP. In our case, for the vast majority of UEs, the same relative label was predicted even with fewer data-points. When scoring against all APs, predictions took 94.5 ms on average. However, domain-specific knowledge can be applied to limit the number of APs e.g., to the nearest 3 APs, in which case, the AP association can be predicted in 2.26 ms.

III. DRL FOR ADAPTIVE BEAM ALLOCATION
Problem description: Within each AP serving area we consider a dynamic wireless network, where UEs and blockers are allowed to changed positions. An AP needs to be equipped with an intelligent mechanism to solve RR management/allocation problems. This motivates the idea of having the AP interact with the environment by using deep reinforcement learning (DRL). In particular, DRL can be used as a tool for sum-rate maximization through beamforming optimization in multi-UE THz wireless networks.
A deep reinforcement learning approach: Both DRL and reinforcement learning (RL) are methods for creating software agents that can learn to interact with environments in an unsupervised manner. Instead of learning from a labeled dataset, DRL receives feedback in the form of a reward. By changing its behavior to maximize the cumulative reward over multiple steps, the agent improves its performance [13]. At each step, the agent receives a representation of the environment's state and it predicts an optimal action to take given this state. The action is in turn used to update the state and to compute a reward value. This process continues until the agent reaches some signified end state or until some other stopping criterion (e.g. a maximum number of iterations) is met [13].
In our case, an agent corresponds to the AP and the actions correspond to codebook selection. By learning how to take actions, the AP can select the codebook beam from the predefined codebook beamset, while serving the UEs at the same time. The objective is to maximize the cumulative reward, which is defined as the sum of the UEs data rates. The environment (observation space) is modelled by information related to the signal-to-interference-plus-noise-ratio (SINR) of the UEs and the UEs' positions over time. Environmental states generated over time by the agent's interaction with the environment, as outlined above, are input to an LSTM network (this particular type of network is capable of encoding in its weights long-range temporal relations between states), which learns to sequentially predict actions guided by the sequence of received rewards over a training session. The overall approach and information flow is illustrated in Fig. 3.a.
Implementation based on Proximal Policy Gradient (PPO): We adopt the state-of-the-art PPO framework [14] for implement our DRL approach. In policy gradient methods, a neural network is used to represent a policy function, which maps an environment state directly into an action. The network parameters are then updated during training to maximize the expected cumulative reward produced by the actions. When computing the gradients for policy updates, the value function can be used together with the sampled rewards to improve the quality of the updates. The combination of policy and value functions into one RL agent is called the "actor-critic" model. While training the network, the actor performs actions according to its current policy, which is used to select the actions and the estimated value function. After each action selection, the critic provides feedback based on how good or bad it considers those actions to be.
PPO uses the actor-critic model to learn the policy. The PPO architecture enables the learner to be accelerated using graphic processing units and actors to be distributed across several machines [15]. Then, we use RLiib, which is a module in ray distribution package with implementations of RL algorithms. Results: Figure 3.b illustrates the maximum reward versus number of episodes of the system. We consider two main scenarios which are (i) when the agent (AP) selects different codebook beam to serve each UE and (ii) when the AP allows more than one UE to use the same beam. In this system, we consider that the AP allocates codebook beam to serve 5 UEs and 10 UEs at the same time. We assume that the AP deploys 8 antennas with 16 predefined beams, and in the other set-up, the AP deploys 16 antennas with 32 predefined beams. The UEs are randomly located near by the AP, i.e., within 100 metres and the UEs are near to each other. We can see that when some UEs can be served by the same beam, the reward is higher than when the AP allocates different beam to each UE. The reason is that some UEs locate close to each other want to use the same beam; therefore, this leads to higher reward.

IV. AI-ASSISTED FAST-CHANNEL PREDICTION FOR
PRO-ACTIVE HAND-OVER Problem description: In an ultra-dense wireless network, where several APs are capable of serving the same MUE, in order to maximize the system's reliability, efficiency, and throughput, it is important to enable pro-active hand-over based on accurate channel characteristics predictions. The following features have been identified as key inputs of most hand-over policies in the THz band: i) signal strength, ii) directionality, and iii) existence of LoS path. Conventional approaches that have been extensively employed in lower frequency bands are rendered infeasible due to inherent complexity that is originated from the high number of degrees of freedom of massive multiple-input multiple-output (MIMO) systems.
ML-based approach: To overcome this barrier, ML-based models that extract the relationship between the environment and the aforementioned channel characteristics, are expected to become key enablers of URLL proactive hand-over approaches. The first step towards creating such an ML model is to acquire training and testing data. In this direction, we collected the baseband channel coefficients, delays, azimuth angle of departure (AAoD), elevation angle of departure(EAoD), azimumth angle of arrival (AAoA), elevation angle of arrival (EAoA), delay spread angle (DSA), and azimuth angle (AA) as well as the transmitter and receiver characteristics of a MUE that performed a number of different routes. These data were fed in a pre-processing unit that returned their statistical characteristics, namely expectations and extension.
Since the available dataset has a limited feature set and observed error margins, the approach relies on investigating the existence of patterns within the available features, as well as taking into account the geographic location, in order to examine the density according to the user mobility for predicting LOS connectivity. Initial results reveal a certain degree of correlation among attributes, when location information is included, but the predictive modeling aspects require additional feature engineering towards achieving various KPIs related to accuracy, fading statistics, correlations, complexity and versatility, which can find the hidden non-linear relationships among different features and can indeed improve the feasibility and viability of the model. This inspires breaking the channel prediction problem into two sub-problems. The first one focuses on the behavior of raw or fine granular form of individual path data, while the second one aims to aggregate (coarse granular) data, subject to feature engineering. The former can be used to predict the presence of the LOS or non-LOS path, path-loss for higher gain, or higher power, delay spread, etc. The latter is capable of predicting the link level behaviors, e.g. the properties of a link proportional to the received power for a given location. Both sub-problems follow similar solution approaches that include pre-processing, simulation execution, and postprocessing. Pre-processing involves model settings and input data preparation for further training so that the data can be used easily and effectively. This steps often ends when a simulation is ready to launch. The simulation phase will allow interactive visualization to be performed at each time step. In more detail, based on user mobility across a certain route in specific environment, it will show whether the route on which the user is moving falls under LOS or non-LOS connectivity. Moreover, the simulations tailor down how the model performs with respect to reality. Post-processing extracts the results and put them in usable form.
The dataset was split into training and testing sets. We performed Feature selection and parameter tuning and trained various ML algorithms to assess which types of model achieve superior performance. In general, ML helps to extract significant features from the data to deliver insightful predictions. Various ML models are built and validated on testing data . The ML, such as decision tree, RF, NB for classifying LoS vs NLoS identification, are compared to evaluate the performance.
Results: Decision tree achieves the best performance in terms of precision. Table II depicts the prediction performance in form of confusion matrix, which shows the models compares resulting classification outcome with actual values of the given observation. There are four potential outcomes here: true positive (TP) value 3072 indicates the model predicted an outcome of true, and the actual observation was true. False positive (FP) value 55 indicates the model predicted a true outcome, but the actual observation was false. False negative (FN) value 5673 indicates the model predicted a false outcome, while the actual observation was true. Finally, we have the true negative (TN) value 14076, which indicates the model predicted an outcome of false, while the actual outcome is also false. Accuracy is calculated, which is generally used to judge the model performance. However, in this particular balanced dataset, the accuracy of 75% was achieved with decision tree.
The results shows room for improvements, which can be explicitly made by using optimizing parameters to further fine tune the parameters in order to extract and learn more patterns for the model. The complexity of these tasks is frequently beyond the capabilities of non-domain experts; hence, the quick development of ML applications has increased the demand for ready-to-use ML techniques. It is the ensuing field of study that focuses on progressive automation of ML, which compares multiple models, set the default parameters all at once and outputs the accuracy, run time error for different algorithms.  Figure 4 illustrates the performance of different models in terms of accuracy, along with its respected classification run time error for each model. As the execution time error decreases, the model improves. Hence, uncovering the influence of environmental characteristics and other micro-and macrocellular scenarios that are essential for evaluating the potential of two polarized MIMO systems and the efficacy of LOS connectivity will be the goals worth pursuing to see if they outperform the conventional methods.

V. CONCLUSIONS
In this article, we presented a holistic and intelligent MAC layer approach that enhances THz wireless systems reliability, while minimizes the association latency, through a metaheuristic-ML based joint UE-AP association and RR allocation with blockage avoidance mechanism, support UE mobility and blockage avoidance within the AP coverage area through a DRL beam selection scheme, and enables ultra-reliable and pro-active hand-over via AI-assisted fastchannel prediction. The results provided useful guidelines for the design of intelligent MAC for URLL THz wireless systems and highlighted the key role of ML in these networks. Edwin Yaqub is a Lead Data Scientist working with RapidMiner since 2016. He has expertise in artificial intelligence, machine learning, constraint solving optimizations, data enrichment and product prototyping, which he has successfully applied to solve a variety of industrial and scientific problems. He amassed 13 years of experience in research and commercial projects from diverse sectors including finance, automotive, healthcare, eCommerce and telecommunication. Edwin received his MSc. degree in Computer Science in 2008 from RWTH Aachen University, Germany. His Master thesis at Philips Research Labs developed a framework for automated health decision support using ontologies and rules-based reasoning. In 2015, he earned Ph.D. in Computer Science from the University of Göttingen, Germany. In his doctoral research, he investigated decision-making algorithms to automatically negotiate and optimize provisioning of cloud infrastructure resources. His recent work addresses resource management using optimizations, predictive and prescriptive machine learning in B5G and 6G wireless networks.
Rachana Desai , Senior Data scientist at RapidMiner with extensive experience from the field of Bioinformatics. She completed her M.Sc. in General Engineering with a focus on bioinformatics at the San Jose University(California, USA) with International publications demonstrating expertise in the field of data science as part of agile development teams. During her Master's degree she worked on optimization and dimension reduction of 3D-QSAR validation metrics and linear SVM cross-validation metrics to facilitate by comparing the best pre-prediction bio-statistical models where she developed interest in machine learning and AI. Since 2019, she is working with RapidMiner on data science projects covering vivid industries like electronics and telecommunication. She worked previously with Vizzario, Inc. and Cisco Systems,Inc. as a data scientist in USA with expertise in Implementing, designing and developing algorithms on Machine Learning, Optimization Algorithms, Deep learning and AI by solving customer problems in areas such as wireless network troubleshooting, physical surveillance, security, IoT visual sensing and retail analytics.