We are facing an online learning problem constituted by the well-known exploration versus exploitation dilemma. To remedy this, we propose a novel Distributed Stable Strategy Learning (DSSL) algorithm for solving the problem. Since the FSMCs are unknown, the rate means must be learned by accessing all channels via exploration phases. This results in increasing the regret, since the stable allocation is not performed. Thus, the exploration time must be minimized, while guaranteeing efficient learning. Roughly speaking, each channel can be learned by different exploration times, depending on its unknown parameters (see more details in Section REF ). The algorithm design in this paper contributes to both tackling the more general model, as well as improving the learning efficiency in a fully-distributed manner. Specifically, in existing algorithms [1]}, [2]}, [3]}, [4]}, [5]}, the exploration phase of all channels is determined by the channel that requires the largest exploration time. This results in oversampling the channels and significantly increases the regret. By contrast, the DSSL algorithm estimates online the desired (unknown) exploration rate of each channel. Thus, by sampling the channels according to the desired exploration rate, it avoids oversampling the channels, and thus reduces the regret scaling significantly as compared to existing algorithms.
