Sim2real for Reinforcement Learning Driven Next Generation Networks
The next generation of networks will actively embrace artificial intelligence
(AI) and machine learning (ML) technologies for automation networks and optimal
network operation strategies. The emerging network structure represented by
Open RAN (O-RAN) conforms to this trend, and the radio intelligent controller
(RIC) at the centre of its specification serves as an ML applications host.
Various ML models, especially Reinforcement Learning (RL) models, are regarded
as the key to solving RAN-related multi-objective optimization problems.
However, it should be recognized that most of the current RL successes are
confined to abstract and simplified simulation environments, which may not
directly translate to high performance in complex real environments. One of the
main reasons is the modelling gap between the simulation and the real
environment, which could make the RL agent trained by simulation ill-equipped
for the real environment. This issue is termed as the sim2real gap. This
article brings to the fore the sim2real challenge within the context of O-RAN.
Specifically, it emphasizes the characteristics, and benefits that the digital
twins (DT) could have as a place for model development and verification.
Several use cases are presented to exemplify and demonstrate failure modes of
the simulations trained RL model in real environments. The effectiveness of DT
in assisting the development of RL algorithms is discussed. Then the current
state of the art learning-based methods commonly used to overcome the sim2real
challenge are presented. Finally, the development and deployment concerns for
the RL applications realisation in O-RAN are discussed from the view of the
potential issues like data interaction, environment bottlenecks, and algorithm
design.