More Like this

1. Online Maintenance Prioritization Via Monte Carlo Tree Search and Case-Based Reasoning

   https://doi.org/10.1115/1.4053408  

   Hoffman, Michael; Song, Eunhye; Brundage, Michael; Kumara, Soundar (August 2022, Journal of Computing and Information Science in Engineering)

   Abstract When maintenance resources in a manufacturing system are limited, a challenge arises in determining how to allocate these resources among multiple competing maintenance jobs. This work formulates an online prioritization problem to tackle this challenge using a Markov decision process (MDP) to model the system behavior and Monte Carlo tree search (MCTS) to seek optimal maintenance actions in various states of the system. Further, case-based reasoning (CBR) is adopted to retain and reuse search experience gathered from MCTS to reduce the computational effort needed over time and to improve decision-making efficiency. The proposed method results in increased system throughput when compared to existing methods of maintenance prioritization while also reducing the computation time needed to identify optimal maintenance actions as more information is gathered. This is especially beneficial in manufacturing settings where maintenance decisions must be made quickly to minimize the negative performance impact of machine downtime. 

   more » « less
2. Component-wise Markov decision process for solving condition-based maintenance of large multi-component systems with economic dependence

   https://doi.org/10.1080/24725854.2023.2295376  

   Bansal, Vipul; Chen, Yong; Zhou, Shiyu (February 2024, IISE Transactions)

   Condition-based maintenance of multi-component systems is a prevalent engineering problem due to its effectiveness in reducing the operational and maintenance costs of the system. However, developing the exact optimal maintenance decisions for the large multi-component system is computationally challenging, even not feasible, due to the exponential growth in system state and action space size with the number of components in the system. To address the scalability issue in CBM of large multi-component systems, we propose a Component-Wise Markov Decision Process(CW-MDP) and an Adjusted Component-Wise Markov Decision Process (ACW-MDP) to obtain an approximation of the optimal system-level CBM decision policy for large systems with heterogeneous components. We propose using an extended single-component action space to model the impact of system-level setup cost on a component-level solution. The theoretical gap between the proposed approach and system-level optima is also derived. Additionally, theoretical convergence and the relationship between ACW-MDP and CW-MDP are derived. The study further shows extensive numerical studies to demonstrate the effectiveness of component-wise solutions for solving large multi-component systems. 

   more » « less
3. Approximate Dynamic Programming for Selective Maintenance in Series–Parallel Systems

   https://doi.org/10.1109/TR.2019.2916898  

   Ahadi, Khatereh; Sullivan, Kelly M. (June 2019, IEEE Transactions on Reliability)

   The problem of allocating limited resources to maintain components of a multicomponent system, known as selective maintenance, is naturally formulated as a high-dimensional Markov decision process (MDP). Unfortunately, these problems are difficult to solve exactly for realistically sized systems. With this motivation, we contribute an approximate dynamic programming (ADP) algorithm for solving the selective maintenance problem for a series–parallel system with binary-state components. To the best of our knowledge, this paper describes the first application of ADP to maintain multicomponent systems. Our ADP is compared, using a numerical example from the literature, against exact solutions to the corresponding MDP. We then summarize the results of a more comprehensive set of experiments that demonstrate the ADP’s favorable performance on larger instances in comparison to both the exact (but computationally intensive) MDP approach and the heuristic (but computationally faster) one-step-lookahead approach. Finally, we demonstrate that the ADP is capable of solving an extension of the basic selective maintenance problem in which maintenance resources are permitted to be shared across stages. 

   more » « less
4. A rollout approach for condition-based maintenance of large multi-unit systems with economic dependence

   https://doi.org/10.1177/1748006X251341025  

   Bansal, Vipul; Chen, Yong; Zhou, Shiyu (July 2025, Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability)

   With advancements in sensor technology, real-time monitoring of machine health conditions allows us to perform condition-based maintenance (CBM) for multi-unit systems. The maintenance decision of a unit is usually dependent on other units in a multi-unit system, inducing an exponentially large state space, which makes CBM of large multi-unit systems a very challenging engineering problem. In this work, we first propose two heuristic decision policies for multi-unit systems, namely the binary action policy and the $(n,N_1,N_2)$-policy. Then we propose a multi-step lookahead rollout approach using the two heuristic policies to solve the challenging CBM problem. By applying the binary action policy, we can effectively reduce the action space and thus reduce the computational load in the rollout, while the $(n,N_1,N_2)$-policy can be an excellent base policy for the rollout to improve upon. The theoretical gap between the proposed rollout approach and the optimal policy is also derived. The study further shows extensive experimentation to demonstrate the effectiveness of the proposed lookahead rollout approach for solving the CBM problem for small (3 and 5 units), medium (10 and 15 units), and large (20, 30, 40, and 50 units) scale systems. 

   more » « less
5. Decision Making in Non-Stationary Environments with Policy-Augmented Monte Carlo Tree Search

   Pettet, Geoffrey; Mukhopadhyay, Ayan; Dubey, Abhishek (January 2022, The Multi-disciplinary Conference on Reinforcement Learning and Decision Making)

   Decision-making under uncertainty (DMU) is present in many important problems. An open challenge is DMU in non-stationary environments, where the dynamics of the environment can change over time. Reinforcement Learning (RL), a popular approach for DMU problems, learns a policy by interacting with a model of the environment offline. Unfortunately, if the environment changes the policy can become stale and take sub-optimal actions, and relearning the policy for the updated environment takes time and computational effort. An alternative is online planning approaches such as Monte Carlo Tree Search (MCTS), which perform their computation at decision time. Given the current environment, MCTS plans using high-fidelity models to determine promising action trajectories. These models can be updated as soon as environmental changes are detected to immediately incorporate them into decision making. However, MCTS’s convergence can be slow for domains with large state-action spaces. In this paper, we present a novel hybrid decision-making approach that combines the strengths of RL and planning while mitigating their weaknesses. Our approach, called Policy Augmented MCTS (PA-MCTS), integrates a policy’s actin-value estimates into MCTS, using the estimates to seed the action trajectories favored by the search. We hypothesize that PA-MCTS will converge more quickly than standard MCTS while making better decisions than the policy can make on its own when faced with nonstationary environments. We test our hypothesis by comparing PA-MCTS with pure MCTS and an RL agent applied to the classical CartPole environment. We find that PC-MCTS can achieve higher cumulative rewards than the policy in isolation under several environmental shifts while converging in significantly fewer iterations than pure MCTS. 

   more » « less