More Like this

1. Hindsight logging for model training

   https://doi.org/10.14778/3436905.3436925  

   Garcia, Rolando; Liu, Eric; Sreekanti, Vikram; Yan, Bobby; Dandamudi, Anusha; Gonzalez, Joseph E.; Hellerstein, Joseph M.; Sen, Koushik (December 2020, Proceedings of the VLDB Endowment)

   null (Ed.)

   In modern Machine Learning, model training is an iterative, experimental process that can consume enormous computation resources and developer time. To aid in that process, experienced model developers log and visualize program variables during training runs. Exhaustive logging of all variables is infeasible, so developers are left to choose between slowing down training via extensive conservative logging, or letting training run fast via minimalist optimistic logging that may omit key information. As a compromise, optimistic logging can be accompanied by program checkpoints; this allows developers to add log statements post-hoc, and "replay" desired log statements from checkpoint---a process we refer to as hindsight logging. Unfortunately, hindsight logging raises tricky problems in data management and software engineering. Done poorly, hindsight logging can waste resources and generate technical debt embodied in multiple variants of training code. In this paper, we present methodologies for efficient and effective logging practices for model training, with a focus on techniques for hindsight logging. Our goal is for experienced model developers to learn and adopt these practices. To make this easier, we provide an open-source suite of tools for Fast Low-Overhead Recovery (flor) that embodies our design across three tasks: (i) efficient background logging in Python, (ii) adaptive periodic checkpointing, and (iii) an instrumentation library that codifies hindsight logging for efficient and automatic record-replay of model-training. Model developers can use each flor tool separately as they see fit, or they can use flor in hands-free mode, entrusting it to instrument their code end-to-end for efficient record-replay. Our solutions leverage techniques from physiological transaction logs and recovery in database systems. Evaluations on modern ML benchmarks demonstrate that flor can produce fast checkpointing with small user-specifiable overheads (e.g. 7%), and still provide hindsight log replay times orders of magnitude faster than restarting training from scratch. 

   more » « less
2. Hour-Ahead Solar Irradiance Forecasting Using Multivariate Gated Recurrent Units

   https://doi.org/10.3390/en12214055  

   Wojtkiewicz, Jessica; Hosseini, Matin; Gottumukkala, Raju; Chambers, Terrence Lynn (November 2019, Energies)

   Variation in solar irradiance causes power generation fluctuations in solar power plants. Power grid operators need accurate irradiance forecasts to manage this variability. Many factors affect irradiance, including the time of year, weather and time of day. Cloud cover is one of the most important variables that affects solar power generation, but is also characterized by a high degree of variability and uncertainty. Deep learning methods have the ability to learn long-term dependencies within sequential data. We investigate the application of Gated Recurrent Units (GRU) to forecast solar irradiance and present the results of applying multivariate GRU to forecast hourly solar irradiance in Phoenix, Arizona. We compare and evaluate the performance of GRU against Long Short-Term Memory (LSTM) using strictly historical solar irradiance data as well as the addition of exogenous weather variables and cloud cover data. Based on our results, we found that the addition of exogenous weather variables and cloud cover data in both GRU and LSTM significantly improved forecasting accuracy, performing better than univariate and statistical models. 

   more » « less
3. A Computational Framework for Solving Nonlinear Binary Optimization Problems in Robust Causal Inference

   https://doi.org/10.1287/ijoc.2022.1226  

   Islam, Md Saiful; Morshed, Md Sarowar; Noor-E-Alam, Md. (November 2022, INFORMS Journal on Computing)

   Identifying cause-effect relations among variables is a key step in the decision-making process. Whereas causal inference requires randomized experiments, researchers and policy makers are increasingly using observational studies to test causal hypotheses due to the wide availability of data and the infeasibility of experiments. The matching method is the most used technique to make causal inference from observational data. However, the pair assignment process in one-to-one matching creates uncertainty in the inference because of different choices made by the experimenter. Recently, discrete optimization models have been proposed to tackle such uncertainty; however, they produce 0-1 nonlinear problems and lack scalability. In this work, we investigate this emerging data science problem and develop a unique computational framework to solve the robust causal inference test instances from observational data with continuous outcomes. In the proposed framework, we first reformulate the nonlinear binary optimization problems as feasibility problems. By leveraging the structure of the feasibility formulation, we develop greedy schemes that are efficient in solving robust test problems. In many cases, the proposed algorithms achieve a globally optimal solution. We perform experiments on real-world data sets to demonstrate the effectiveness of the proposed algorithms and compare our results with the state-of-the-art solver. Our experiments show that the proposed algorithms significantly outperform the exact method in terms of computation time while achieving the same conclusion for causal tests. Both numerical experiments and complexity analysis demonstrate that the proposed algorithms ensure the scalability required for harnessing the power of big data in the decision-making process. Finally, the proposed framework not only facilitates robust decision making through big-data causal inference, but it can also be utilized in developing efficient algorithms for other nonlinear optimization problems such as quadratic assignment problems. History: Accepted by Ram Ramesh, Area Editor for Data Science and Machine Learning. Funding: This work was supported by the Division of Civil, Mechanical and Manufacturing Innovation of the National Science Foundation [Grant 2047094]. Supplemental Material: The online supplements are available at https://doi.org/10.1287/ijoc.2022.1226 . 

   more » « less
4. Stateful Offline Contextual Policy Evaluation and Learning

   Nathan Kallus, Angela Zhou (January 2022, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics)

   We study off-policy evaluation and learning from sequential data in a structured class of Markov decision processes that arise from repeated interactions with an exogenous sequence of arrivals with contexts, which generate unknown individual-level responses to agent actions. This model can be thought of as an offline generalization of contextual bandits with resource constraints. We formalize the relevant causal structure of problems such as dynamic personalized pricing and other operations management problems in the presence of potentially high-dimensional user types. The key insight is that an individual-level response is often not causally affected by the state variable and can therefore easily be generalized across timesteps and states. When this is true, we study implications for (doubly robust) off-policy evaluation and learning by instead leveraging single time-step evaluation, estimating the expectation over a single arrival via data from a population, for fitted-value iteration in a marginal MDP. We study sample complexity and analyze error amplification that leads to the persistence, rather than attenuation, of confounding error over time. In simulations of dynamic and capacitated pricing, we show improved out-of-sample policy performance in this class of relevant problems. 

   more » « less
5. Efficient optimization with higher-order ising machines

   https://doi.org/10.1038/s41467-023-41214-9  

   Bybee, Connor; Kleyko, Denis; Nikonov, Dmitri E.; Khosrowshahi, Amir; Olshausen, Bruno A.; Sommer, Friedrich T. (December 2023, Nature Communications)

   Abstract A prominent approach to solving combinatorial optimization problems on parallel hardware is Ising machines, i.e., hardware implementations of networks of interacting binary spin variables. Most Ising machines leverage second-order interactions although important classes of optimization problems, such as satisfiability problems, map more seamlessly to Ising networks with higher-order interactions. Here, we demonstrate that higher-order Ising machines can solve satisfiability problems more resource-efficiently in terms of the number of spin variables and their connections when compared to traditional second-order Ising machines. Further, our results show on a benchmark dataset of Booleank-satisfiability problems that higher-order Ising machines implemented with coupled oscillators rapidly find solutions that are better than second-order Ising machines, thus, improving the current state-of-the-art for Ising machines. 

   more » « less