Similar patterns arise in other domains like precision agriculture (fields need time to replenish), online education (students need time before repeating the same exercise), or music recommendations (a song becomes “fresh” again after a break). The appeal of studying recharging bandits is that they combine the explore-exploit dilemma of classical bandits with an added temporal planning aspect: we must decide not just which arm to play, but when to play it for maximum effect.
While theory gives us insight, real-world applications of recovering bandits can be far more complex than the idealized models (e.g. recovery functions might not be perfectly concave or even known, and the number of arms could be very large). This is where deep learning comes into play as a powerful tool to learn and generalize in ways classical algorithms might struggle with. Deep neural networks can approximate the relationship between an arm’s state (e.g. how long since last pulled, and possibly other contextual features) and its future reward. Instead of assuming a specific parametric form for the recovery function, a neural network could learn the shape directly from data. This is useful if the recovery dynamics are complicated or not known upfront. This project is not only academically enriching but also highly relevant, as many real-world systems (from recommendation engines to autonomous resource allocation) can be modeled as recovering bandits. The end result for the master student is a deeper understanding of both reinforcement learning and sequential decision optimization, delivered through the lens of an attractive, gamified scenario of “when to pull the trigger, when to hold off”.
References:
Simchi-Levi, David, Zeyu Zheng, and Feng Zhu. "Dynamic planning and learning under recovering rewards." International Conference on Machine Learning. PMLR, 2021.
Kleinberg, R., & Immorlica, N. (2018, October). Recharging bandits. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS) (pp. 309-319). IEEE.
Pike-Burke, C., & Grunewalder, S. (2019). Recovering bandits. Advances in Neural Information Processing Systems, 32.
Tang, W., Ho, C. J., & Liu, Y. (2021). Bandit learning with delayed impact of actions. Advances in Neural Information Processing Systems, 34, 26804-26817.
Nakhleh, Khaled, et al. "NeurWIN: Neural Whittle index network for restless bandits via deep RL." Advances in Neural Information Processing Systems 34 (2021): 828-839..