Here, we formulate (1) as a long-run average cost MDP with state space \(\mathcal {S} \doteq \lbrace 0,1,\cdots ,B\rbrace  \times \mathbb {Z}_+\)  and an intermediate state space \(\mathcal {V}=\lbrace 0, 1,…..,B\rbrace \times \mathbb {Z}_{+} \times \lbrace C_{1}, C_{2},\cdots , C_{m}\rbrace \) . A generic state \(s = (E,T)\)  means that the energy buffer has \(E\)  energy packets, and the source was last activated \(T\)  slots ago. A generic intermediate state \(v = (E,T,C)\)  additionally means that the current channel state obtained via probing is \(C\) . The action space is \(\mathcal {A}=\lbrace \lbrace 0,0\rbrace \cup \lbrace 1\times \lbrace 0,1 \rbrace \rbrace \rbrace \)  with \(a(t), b(t) \in \lbrace 0,1\rbrace \) . At each time, if the source node decides not to probe the channel (\(b(t) = 0\) ) and consequently does not sample (\(a(t) = 0\) ), the expected single-stage AoI cost is \(T\) . However, if \(b(t)=1\) , then the expected single-stage AoI cost is \(T\)  for \(a(t) = 0\)  and \(T(1-p(C))\)  for \(a(t) = 1\) , where the expectation of the single stage cost is taken over packet success probability \(p(C)\) . We first formulate the average-cost MDP problem as an \(\alpha \) -discounted cost MDP problem with \(\alpha \in (0, 1)\) , and derive the optimal policy, from which the solution of the average cost minimization problem can be obtained by taking \(\alpha \rightarrow 1\)  [1]}.
