A large body of applications that involve monitoring, decision making, and forecasting require timely status updates for their efficient operation. Age of Information (AoI) is a newly proposed metric that effectively captures this requirement. Recent research on the subject has derived AoI optimal policies for the generation of status updates and AoI optimal packet queueing disciplines. Unlike previous research we focus on low-end devices that typically support monitoring applications in the context of the Internet of Things. We acknowledge that these devices host a diverse set of applications some of which are AoI sensitive while others are not. Furthermore, due to their limited computational resources they typically utilize a simple First-In First-Out (FIFO) queueing discipline. We consider the problem of optimally controlling the status update generation process for a system with a source-destination pair that communicates via a wireless link, whereby the source node is comprised of a FIFO queue and two applications, one that is AoI sensitive and one that is not. We formulate this problem as a dynamic programming problem and utilize the framework of Markov Decision Processes to derive optimal policies for the generation of status update packets. Due to the lack of comparable methods in the literature, we compare the derived optimal policies against baseline policies, such as the zero-wait policy, and investigate the performance of all policies for a variety of network configurations. Results indicate that existing status update policies fail to capture the trade-off between frequent generation of status updates and queueing delay and thus perform poorly.
Age of Information (AoI) has emerged as a performance metric capturing the freshness of data for status-update based applications (e.g., remote monitoring) as a more suitable alternative to classical network performance indicators such as throughput or delay. Optimizing AoI often requires distinctly novel and sometimes counter-intuitive networking policies that adapt the rate of update transmissions to the randomness in network resources. However, almost all previous work on AoI to data has been theoretical, assuming idealized networking models, and known delay and service time distributions. It is difficult to obtain these statistics and optimize for them in a real-life network as there are many interacting phenomena in different networking layers (e.g., consider an end-to-end IoT application running over the Internet). With this work we introduce a deep reinforcement learning-based approach that can learn to minimize the AoI with no prior assumptions about network topology. After evaluating the learning model on an emulated network, we have shown that the method can be scaled up to any realistic network with unknown delay distribution. This paper considers a sequential sensor scheduling and remote estimation problem with multiple communication channels. Departing from the classical remote estimation paradigm, which involves one communication channel (noiseless or noisy), we consider here the more realistic setting of two channels with different characteristics (one is cheap but noisy, the other one is costly but noiseless). We first show, via a counter-example, that the common folklore of applying symmetric threshold-based policy, which is well known to be optimal (for unimodal state densities) in the classical remote estimation problem, can no longer be optimal in our setting. In view of that, and in order to make the problem tractable, we introduce a side channel which signals to the receiver the sign of the underlying state. We show, under some technical assumptions, that a threshold-in-threshold based communication scheduling is optimal. The impact of the results is analyzed numerically based on dynamic programming. This numerical analysis reveals some rather surprising results inheriting known properties from the single channel settings, such as not exhausting all the opportunities available for the noisy channel. Scheduling of the transmission of status updates over an error-prone communication channel is studied in order to minimize the long-term average age of information (AoI) at the destination, under an average resource constraint at the source node, which limits the average number of transmissions. After each transmission, the source receives an instantaneous ACK NACK feedback, and decides on the next update, without a priori knowledge on the success of the future transmissions. The optimal scheduling policy is studied under different feedback mechanisms; in particular, standard automatic repeat request (ARQ) and hybrid ARQ (HARQ) protocols are considered. Average-cost reinforcement learning algorithms are proposed when the error probabilities for the HARQ system are unknown.