| INTRODUCTION

In recent years, the use of machine learning (ML) has been on the rise in the field of materials, from material discovery [1][2][3][4] to predictive tasks. [5][6][7] This surge in ML applications has significantly contributed to the accelerated pace of materials research and design, offering innovative solutions to challenges that were once considered insurmountable. Among the materials garnering attention, metal-organic frameworks (MOFs) have emerged as a particularly promising class of materials due to their tunable porous structures and versatile chemical composition. [8][9][10] MOFs, characterized by their crystalline structures comprising metal nodes interconnected by organic ligands, can exhibit exceptional surface areas and tailorable functionalities. These attributes make MOFs highly desirable for applications in gas adsorption, separation, and storage. 10,11 However, the extensive database of MOFs [12][13][14][15][16] calls for the adoption of more advanced computational methods to efficiently screen these materials for their potential applications.

In the realm of ML, various methods have been utilized to screen MOFs and discern their adsorption behavior. These methods encompass support vector machines, 17,18 neural networks, [19][20][21][22][23][24] random forests, 17,25,26 among others. However, this paper directs its focus towards one particularly potent approach-reinforcement learning (RL). RL stands out by enabling an agent to learn from sequential experiences and adjust its strategies through trial and error, showcasing distinct capabilities for effectively exploring the extensive landscape of MOF-adsorption studies.

Optimizing the prediction of adsorption in MOFs through an RL framework offers a compelling approach to be studied within the context of active learning (AL) strategies: AL makes requests for data to be labeled as a surrogate model is being developed. Unlike conventional models, RL provides a dynamic method for selecting the training data set, which is vital in scenarios where the data set size is vast.

Compared to other models, RL adaptively navigates through extensive data sets, iteratively identifying training points. Recently, RL has been used as a tool for the inverse design of MOFs for direct air capture of carbon dioxide. 27 In this study, the goal shifts from exploring structural configurations to determining optimal training points for a Gaussian process (GP) model, which, in turn, predicts the adsorption isotherms of MOFs.

GPs provide a flexible and probabilistic framework for capturing complex relationships within data sets, taking features as inputs and have mean predictions and uncertainties as outputs. 28,29 In the context of predicting adsorption behaviors in MOFs, GPs offer a versatile approach to model adsorption isotherms, resulting in predictions of adsorption loading and uncertainties. [30][31][32][33] The integration of GPs into the proposed framework enhances the robustness and reliability of the predictive modeling, making it a key component in predicting the adsorption in MOFs.

Q-learning, also known as Quality learning, treats the selection of training points as a dynamic decision-making process. 34,35 Within this framework, the RL agent, representing the exploration strategy, learns to iteratively choose training points based on a reward metric, thereby optimizing its predictions over the agent's experience. This approach not only enhances the efficiency of predicting adsorption isotherms within MOFs but also offers a fresh perspective on leveraging RL in data-driven materials science. The integration of Q-learning into the RL-driven design framework signifies a departure from traditional methodologies, highlighting the model's adaptability to the complexities inherent in MOF adsorption studies. This research demonstrates the potential of Q-learning to redefine our approach to training predictive models for intricate phenomena like adsorption in MOFs.

In this study, we initially apply the Q-learning framework across eight distinct systems involving methane (CH 4 ) and carbon dioxide (CO 2 ) within two MOFs: Cu-BTC and IRMOF-1, at a temperature of 298K. This approach enables us to explore the applicability of the Q-learning framework across different systems, providing insights into its versatility and effectiveness. Subsequently, we extend our analysis by applying RL to a broader set of MOFs from the CoRE MOF database 16 for both gases, further expanding the scope of our investigation and evaluating the generalizability of our approach.

| METHODS

| Adsorption ground truth generation for Cu-BTC and IRMOF-1

The adsorption isotherms were generated through grand canonical Monte Carlo (GCMC) simulations with the software RASPA 36 at 298K. The non-bonded interactions in the MOFs were modeled using the Universal Forcefield (UFF), while the adsorbate molecules were modeled using the Transferable Potentials for Phase Equilibria (TraPPE) and using Lorentz Berthelot mixing rule. [37][38][39] MOF atoms were held fixed at their crystallographic positions and the charges for these two MOFs were not considered. The Monte Carlo (MC) moves were translation, rotation, reinsertion, and swap. A total of 200,000 cycles were executed, preceded by 100,000 initialization cycles.

| Adsorption ground truth generation for CoRE MOFs

The ground truth for carbon dioxide (CO 2 ) and methane (CH 4 ) was established through an AL protocol, consistent with our previous work, where we evaluated various initial training data selection schemes to predict full isotherms across 11 MOFs. 30 Building upon this methodology, we applied AL to generate complete adsorption points (64 data points) for both CO 2 and CH 4 within each MOF from the CoRE MOFs database. 16 The AL process began with a log-spaced initial data, comprising pressures ranging from 1eÀ5 to 100 bar, and corresponding adsorption values generated from RASPA at 298K. This initial data set, consisting of 19 pressure points, served as the training data for the Gaussian process regression (GPR) model. Subsequently, predictions were made on an unlabeled data set, comprising the remaining pressure data points required for the full isotherms.

During the AL iterations, uncertainties from the GP predictions were collected and utilized to compute the GP relative error. This error metric guided the selection of the next data point to be added to the training data set. Specifically, the pressure data point corresponding to the highest relative error was passed to RASPA to compute the actual adsorption value, which was then incorporated into the training data set. The MOF atoms were also held fixed at their crystallographic positions and the charges were considered. The charges were taken from a study on partial charge assignment by Raza et al. 40 The MC moves and cycles are the same as in the previous section. This iterative process continued until the maximum GP relative error reached 0.01. By following this protocol, we successfully generated complete adsorption isotherms for both gases across MOFs in the CoRE MOF database.

| Q-learning

In this study, we utilize the Q-learning framework within an AL context to optimize the selection of training data points for adsorption studies. The RL agent begins with an initial limited data set and iteratively selects additional data points based on its learned policy. Only the labels of the selected data points are revealed and used to update the GP model, ensuring that the remaining data points' labels remain unknown to the GP model until selected by the agent. The reward metrics used in this study are the improvements in MRE or R 2 which directly measure the enhancement in predictive performance. This approach allows us to demonstrate the potential of RL in driving the AL process and optimizing model accuracy.

In this study, we utilize Q-learning as a strategic methodology to optimize the selection of training data points for predicting adsorption isotherms within MOFs. Q-learning, rooted in the principles of RL, operates within the framework of a Markov decision process, where initial and updated training data serve as states (s), and the selection of training data points represents actions (a). The fundamental objective is to adapt the Q-learning algorithm iteratively to improve the efficiency of predicting adsorption loading in MOFs.

The Q-learning algorithm updates Q-values based on the Bellman equation:

The learning rate (α), a critical parameter in Q-learning, dictates the magnitude of Q-value updates. A higher α facilitates rapid adaptation to new information but may risk overshooting optimal values, while a lower α provides stability but slower convergence.

The immediate reward (r) denotes the feedback for a specific action in each state and guides the Q-value update process. In MOF adsorption, rewards are associated with the accuracy of predictions for selected training data points. The discount factor (γ) determines the influence of future rewards on the Q-value update. A higher γ emphasizes long-term rewards, encouraging the agent to consider future consequences, while a lower γ prioritizes immediate rewards. The next state (s) reflects the updated training data after the agent's action, guiding the learning process. States (s) in the MOF adsorption context correspond to the available training data set, while actions (a) involve the selection of specific training data points for updating. The Q-learning agent systematically explores the total available data set by iteratively choosing actions that maximize the accuracy of predicted adsorption properties. 2.4 | States, actions, rewards, and states 2.4.1 | States In the initial state (state 0) of the Q-learning process, the training data set is chosen to lay the foundation for subsequent exploration within the pressure-adsorption space study. The configuration of this initial training data set involves the selection of lower and upper bounds for pressures which are set at 1eÀ5 and 100 bar, respectively. This choice results in an initial training data set comprising just two data points.

| Actions

The actions in the Q-learning context refer to the selection of specific training data points. Each action influences the subsequent state and the agent's understanding of the environment. In the described meth- During exploration and exploitation, the agent relies on the Q-table to inform its decision-making. The action selected by the agent is often the one with the highest Q-value in each state. The dynamics of the Q-table are shaped by the Q-learning update rule, which incorporates immediate rewards and anticipated future rewards. The agent learns from sequential experiences, adjusting its strategies through trial and error. By maximizing the Q-values, the agent hones its decisionmaking skills, progressively selecting actions that lead to higher rewards and, consequently, more accurate predictions of adsorption isotherms.

| Rewards

In the context of RL, rewards play a pivotal role in guiding the learning process of an agent. Rewards in RL encapsulate the desirability or utility associated with the outcomes of an agent's actions. They act as a form of reinforcement, providing feedback to the agent and influencing its decision-making processes. In the Q-learning approach tailored for MOFs, the purpose of rewards is to encourage the agent to learn a policy that maximizes cumulative rewards over time.

In this methodology, rewards are evaluated using two metrics: In summary, the methodology incorporates a reward-driven approach within the Q-learning framework, aligning the agent's learning objectives with the goal of iteratively enhancing its decisionmaking strategies. By prioritizing actions that contribute positively to the model's accuracy, the methodology aims to achieve refined predictions of adsorption isotherms in MOFs.

| States

In the completion of each episode, as outlined above, the addition of a training data point triggers the formation of a new state within the Q-learning framework. The concept of episodes, crucial to the iterative learning process, encapsulates a sequence of actions taken by the agent as it dynamically explores and exploits the MOF adsorption space.

| Gaussian process regression

GPs provide a probabilistic and nonparametric approach to regression, allowing for the modeling of complex relationships between input parameters and the corresponding output values. In the GP framework, functions are considered random variables, and the joint distribution over functions is governed by a multivariate normal distribution. The choice of mean function (m) and the covariance function (K) defines the GP's prior distribution, reflecting beliefs about the functions before observing any data. This can be mathematically described in Equation (1).

For this study, the covariance function (kernel) implemented was the rational quadratic kernel 28,41,42 as described by the mathematical form in Equation (2):

The kernel is defined by x À x 0 representing the Euclidean distance between x and x 0 , while l denotes the length scale parameter, which defines the characteristic length over which variations in the function occur. Additionally, α influences the balance between large-scale and small-scale fluctuations within the function by adjusting their respective weights. The Scikit-learn library was used for the GP implementation. 43 In the Q-learning methodology, a surrogate model is used in To ensure the efficiency and accuracy of the RL-GP method, we implemented two key adjustments. First, initial data points used for training are excluded from the list of possible actions at the start of the training process. This prevents the RL agent from selecting these points again, ensuring that each selected point contributes new information to the model. Second, we implemented a mechanism to track evaluated data points and remove them sequentially from the action list. This prevents repeated selection and ensures continuous exploration of new areas in the data space.

In Figure 1 below, we show the sequence of the RL algorithm involving the states, actions, rewards, and the GP in use.

In our modeling approach, it is important to acknowledge the role of random seeds in influencing the outcomes of the exploration process. The random seed affects the selection of exploration points in the data space. This initial setting can lead to variability in which points are selected for exploration, potentially impacting the learning trajectory and the efficiency with which the model converges to an optimal solution. Recognizing this, we used a random seed of 42, enabling reproducibility. However, in Section 3, we offer a comprehensive analysis of different random seeds.

| RESULTS AND DISCUSSIONS

In this section of the study, our attention was directed toward the utilization of Q-learning to scrutinize the adsorption characteristics of two gases, CH 4 and CO 2 , within two distinct MOFs: Cu-BTC and IRMOF-1. Our investigation encompassed the exploration of various hyperparameters, including the learning rate, the tradeoff between exploration and exploitation (ε), and discount factor, to discern their impact on the learning process. Table 1 offers an overview of the hyperparameters employed in the Q-learning exercises, facilitating a systematic evaluation of their efficacy across different adsorbateadsorbent pairs. The primary objective was to identify the optimal combination of parameters tailored to each adsorbate-adsorbent pair in this paper.

In this investigation, the Q-learning algorithm was capped at a maximum of 10 episodes. We evaluated two reward metrics: the difference in MRE between consecutive states, and the difference in R 2

values between consecutive states. The MRE reward metric demonstrated superior performance compared to the R 2 reward metric. Consequently, we primarily focus on the MRE metric in this paper, while presenting results for the R 2 metric in the Supporting Information (SI).

The determination of the optimal hyperparameter combination relied on the achieved final MRE for the MRE reward metric and the final R 2 values for the R 2 reward metric.

| MRE as reward metric

In this section, rewards are based on MRE as calculated between predicted adsorption and actual adsorption data. The reward (r) is calculated as the difference in MRE between states t and t + 1. Once the reward has been calculated, it is fed into Equation (1) to construct the Q-table.

The MRE can be calculated as described by the equation:

| Q-learning on two pure gases within two MOFs

For CH₄ adsorption in Cu-BTC, the optimal hyperparameter combination was determined to be a learning rate (α) of 0.001, a discount factor (γ) of 0.1, and an exploration/exploitation (ϵ) value of 0.9, with 10 episodes.

Figure 2 illustrates the outcomes for this specific adsorbateadsorbent pair based on the identified best hyperparameter combinations, as stated above. In Figure 2A, the evolution of the MRE between the GP-predicted adsorption and ground truth adsorption is depicted as Q-learning episodes progress. Notably, the algorithm achieved a final MRE of 0.043. Figure 2B presents a comparative analysis between the predicted and ground truth adsorption, demonstrating the model's predictive performance, where an R 2 of 0.99 and MAE of 0.237 cm 3 STP/g was achieved. Additionally, Figure 2C visualizes the pressure space sampled by the RL agent. Finally, Figure 2D shows a good comparison between the simulated isotherm and the GPpredicted isotherm, as directed by Q-learning.

For the case of CH 4 in IRMOF-1, a final MRE of 0.044 was achieved by the Q-learning agent after the maximum episodes of 10.

The best Q-learning parameters were found to be α of 0.001, γ of 0.5, and ϵ of 0.9. In Figure 3A, we show the evolution of MRE with the episodes, with a significant rise in the MRE in the second episode.

Additionally, Figure 3B displays the resulting comparison between the GCMC and GP-predicted adsorptions, showing good agreements and a R 2 of 0.99 and MAE of 0.763 cm 3 STP/g was achieved. In Figure 3C, we observe the pressure-sampled regions by the agents showing a distribution across the pressure space. Finally, Figure 3D also shows a good comparison between the simulated isotherm and the RL-based GP-predicted isotherm.

CO 2 adsorption prediction in both Cu-BTC and IRMOF-1 were optimized in the same number of Q-learning episodes of 10, to achieve desirable outcomes. Specifically, in the case of Cu-BTC, the optimal hyperparameter combination was determined to be a α of 0.001, γ of 0.9, and an ϵ value of 0.9, with 10 episodes. In Figure 4A, a final MRE value of 0.046 was achieved after a consistent increase in MRE from the second to fourth episodes, followed by a drop in MRE from the fifth episode, indicating the dynamic nature of the learning process. Furthermore, Figure 4B showcases a comparison between GP-predicted and GCMC adsorption in Cu-BTC, demonstrating commendable agreement between the two. An R 2 of 0.99 and MAE of 0.764 cm 3 STP/g. Figure 4C provides insights into the pressure space sampled by the agent, highlighting the agent's ability to determine the optimal points. In Figure 4D, we show a good prediction of the isotherm as compared to the GCMC-simulated isotherm, indicating that the RL-based GP can predict the isotherm.

Similarly, for IRMOF-1, the optimal hyperparameter combination was determined to be a α of 0.001, γ of 0.5, and ϵ value of 0.9, with with failure in the highest-pressure regions. This failure is evident in Figure 5C, where the algorithm fails to explore high-pressure regions.

In Figure 5C, the pressure space sampled by the agent in IRMOF-1

indicates an exploration of the diverse pressure regions. In Figure 5D, we show a good agreement between the simulated and RL-GP predicted isotherms. An R 2 of 0.99 and MAE of 4.232 cm 3 STP/g was achieved.

| Q-learning on CH 4 gas within CoRE MOFs

The optimal Q-learning parameters established for CH

| Q-learning on CO 2 gas within CoRE MOFs

Given the established Q-learning parameters for CO 2 from IRMOF-1 and Cu-BTC, the RL scheme is applied to the same CoRE MOFs as in the previous sections. Figure 8 below shows the isotherms of some selected MOFs for CO 2 . The first four frames indicate the best performance in terms of lowest MRE(s) and the next four frames are the least performing cases in terms of highest MRE(s).

In the case of CO 2 , our analysis of the combined actual and RLbased GP predictions across all MOFs yielded an R 2 value of 0.99, as depicted in Figure 9A below. The MAE stood at 0.4667 cm 3 STP/g. The differences in R 2 and MAE values between CO 2 and CH 4 can be attributed to the stronger interactions of CO 2 , leading to more complex adsorption isotherms with sharper transitions.

This complexity poses a challenge for accurate prediction, potentially resulting in higher MAE value, compared to CH 4 . Upon closer examination, as depicted in Figure 9B, we analyzed the critical points sampled by the RL agent, revealing a distribution across the pressure input space, particularly in the lower-pressure regions.

The convergence in sampling behavior underscores the robustness of the RL-guided exploration approach in identifying critical regions within the pressure space with significant implications for adsorption behavior. In the SI, we show the results of using an R 2 reward metric in the sequence used in this paper. First, we applied the RL framework to CH 4 both in Cu-BTC and IRMOF-1 and used the optimal Q-learning parameters (based on highest R 2 ) in the case of the CoRE MOFs.

For CH₄ adsorption in Cu-BTC (Figure S1), we determined the optimal combination of hyperparameters to be α = 0.001, γ = 0.9, and ϵ = 0.9. For the case of CH₄ in IRMOF-1 (Figure S2), the parameters α = 0.001, γ = 0.1, and ϵ = 0.9, yielded the best results. For CH 4 , the combined actual and RL-based GP predictions yielded an R 2 of 0.99, with an MAE of 0.278 cm 3 STP/g (Figure S4A), which is higher than the MAE value realized using the MRE reward metric. For CO 2 (Figure S8A), R 2 of 0.99 and MAE of 0.914 cm 3 STP/g. T A B L E 2 Comparison between average metrics from 1000 different seeds to a random seed of 42. Mean MAE (cm 3 STP/g) MAE for random seed of 42 (cm 3 STP/g) Mean MRE MRE for random seed of 42 Mean R 2 R 2 for random seed for 42 CH 4 in CuBTC 0.3301 0.237 0.048 0.043 0.99 0.99 CH 4 in IRMOF-1 0.761 0.763 0.040 0.044 0.99 0.99 CO 2 in CuBTC 0.791 0.764 0.049 0.046 0.99 0.99 CO 2 in IRMOF-1 4.3 4.232 0.1 0.1 0.99 0.99 Abbreviations: MAE, mean absolute error; MRE, mean relative error. Although RL outperformed AL in three of the studied cases, this enhanced performance can be attributed to RL having access to all labels (data) upfront, whereas AL does not have this advantage.

CH4 in Cu-BTC

Figure 10A shows the pressure points collected by AL for the final GP model, while Figure 10B shows the RL sampled point by the Q-learning agent. For this case, we observe that the RL approach with a lesser MAE of 0.237 cm 3 STP/g outperforms that of AL with an MAE of 0.538 cm 3 STP/g, as shown in Figure 10C.

CH4 in IRMOF-1

Figure 11A displays the pressure points collected by AL for the final GP model, and Figure 11B shows the RL sampled point by the Q-learning agent. In this scenario, it was observed that the AL approach with a lesser MAE of 0.461 cm 3 STP/g outperforms that of RL with an MAE of 0.763 cm 3 STP/g, as shown in Figure 11C.

CO 2 in Cu-BTC

Figure 12A shows the pressure points collected by AL for the final GP model, and Figure 12B shows the RL sampled point by the Q-learning agent. We observed that the RL approach with a lesser MAE of 0.764 cm 3 STP/g outperforms that of AL with an MAE of 5.815 cm 3 STP/g, as shown in Figure 12C.

CO 2 in IRMOF-1

Figure 13A shows the pressure points collected by AL for the final GP model, while Figure 13B shows the RL sampled point by the Q-learning agent. For this case, we observe that the RL approach with a lesser MAE of 4.232 cm 3 STP/g outperforms that of AL with an MAE of 5.377 cm 3 STP/g, as shown in Figure 13C.

| CONCLUSION AND RECOMMENDATIONS

In this study, we introduced an approach that integrates Q-learning with GPs to optimize predictive modeling in MOF adsorption studies. A pivotal aspect of our approach lies in the strategic sampling of critical points by the RL agent within the pressure input space. By systematically exploring the pressure space, our methodology shows that there are informative pressure regions to develop an accurate adsorption model across all MOFs. Our findings underscore the significance of systematically probing these regions. The general RL code for both the MRE and R 2 reward metrics, and the isotherms generated through AL for both CH 4 , and CO 2 can all be found via the GitHub repository.

In this study, we acknowledge that RL is computationally expensive compared to previous AL schemes or random sampling and requires access to all labels for its learning process. While fitting all the data or randomly selected points into a GP model can be sufficient, RL distinguishes itself by learning a policy that balances exploration and exploitation of the data points.

The adoption of RL for modeling adsorption isotherms as an AL method is justified by its capabilities in managing the explorationexploitation tradeoff. This exploration is guided by an algorithm that not only seeks to uncover potentially more efficient adsorption points but also that each iteration contributes maximally to the accuracy of the resulting model. Hence, the integration of RL in this context enhances the accuracy of adsorption isotherm surrogate models.

DATA AVAILABILITY STATEMENT

The RL codes for both the MRE and R 2 reward metrics can be found in the GitHub repository for this project. Also, the tabulated data for