Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints
Many real-world settings involve costs for performing actions; transaction
costs in financial systems and fuel costs being common examples. In these
settings, performing actions at each time step quickly accumulates costs
leading to vastly suboptimal outcomes. Additionally, repeatedly acting produces
wear and tear and ultimately, damage. Determining when to act is crucial for
achieving successful outcomes and yet, the challenge of efficiently learning to
behave optimally when actions incur minimally bounded costs remains unresolved.
In this paper, we introduce a reinforcement learning (RL) framework named
Learnable Impulse Control Reinforcement Algorithm (LICRA), for learning to
optimally select both when to act and which actions to take when actions incur
costs. At the core of LICRA is a nested structure that combines RL and a form
of policy known as impulse control which learns to maximise objectives when
actions incur costs. We prove that LICRA, which seamlessly adopts any RL
method, converges to policies that optimally select when to perform actions and
their optimal magnitudes. We then augment LICRA to handle problems in which the
agent can perform at most $k<\infty$ actions and more generally, faces a budget
constraint. We show LICRA learns the optimal value function and ensures budget
constraints are satisfied almost surely. We demonstrate empirically LICRA's
superior performance against benchmark RL methods in OpenAI gym's Lunar Lander
and in Highway environments and a variant of the Merton portfolio problem
within finance.