In our A2C algorithm, the actor has two DNNs, one for the discrete offloading policy with two hidden layers of sizes {64, 32} and one for the continuous allocation parameter policy with two hidden layers of sizes {128, 64} with ReLU as the activation function. For the output layers, we used Softmax to generate offloading decisions for MDs and adopted Tanh to produce allocation parameters given the discrete offloading policy.
The KL divergence constraint \( \epsilon _{KL}\)  is set to 0.01 for the TRPO-based actor policy optimizer [1]}. The critic was also built by a DNN that contains two hidden layers of sizes {200, 100} blackto train the state-value function of our A2C scheme with the Adam optimizer. To implement our parameterized A2C algorithm, we set up a virtual agent that is allowed to interact with a pre-defined BFL environment to learn the parameterized offloading policy for all MDs and observe the return, i.e., system utility, after each iteration. We trained the agent over 10000 episode with 100 timesteps per each episode. All simulations were implemented in Python 3.8 with TensorFlow 2.0 using a computer with 4.7GHz CPU Intel Core i7 and 512 GB memory. All numerical results are averaged over 10 independent simulation runs.
<FIGURE><FIGURE>