An official website of the United States government
Accurate wait-time prediction for HPC jobs contributes to a positive user experience but has historically been a challenging task. Previous models lack the accuracy needed for confident predictions, and many were developed before the rise of deep learning. In this work, we investigate and develop TROUT, a neural network-based model to accurately predict wait times for jobs submitted to the Anvil HPC cluster. Data was taken from the Slurm Workload Manager on the cluster and transformed before performing additional feature engineering from jobs’ priorities, partitions, and states. We developed a hierarchical model that classifies job queue times into bins before applying regression, outperforming traditional methods. The model was then integrated into a CLI tool for queue time prediction. This study explores which queue time prediction methods are most applicable for modern HPC systems and shows that deep learning-based prediction models are viable solutions.
more »
« less
Predicting the future is hard and other lessons from a population time series data science competition.
Population forecasting, in which past dynamics are used to make predictions of future state, has many real-world applications. While time series of animal abundance are often modeled in ways that aim to capture the underlying biological processes involved, doing so is neither necessary nor sufficient for making good predictions. Here we report on a data science competition focused on modelling time series of Antarctic penguin abundance. We describe the best performing submitted models and compare them to a Bayesian model previously developed by domain experts and build an ensemble model that outperforms the individual component models in prediction accuracy. The top performing models varied tremendously in model complexity, ranging from very simple forward extrapolations of average growth rate to ensembles of models integrating recently developed machine learning techniques. Despite the short time frame for the competition, four of the submitted models outperformed the model previously created by the team of domain experts. We discuss the structure of the best performing models and components therein that might be useful for other ecological applications, the benefit of creating ensembles of models for ecological prediction, and the costs and benefits of including detailed domain expertise in ecological modelling. Additionally, we discuss the benefits of data science competitions, among which are increased visibility for challenging science questions, the generation of new techniques not yet adopted within the ecological community, and the ability to generate ensemble model forecasts that directly address model uncertainty.
more »
« less
- Award ID(s):
- 1633299
- PAR ID:
- 10100635
- Date Published:
- Journal Name:
- Ecological informatics
- Volume:
- 48
- ISSN:
- 1574-9541
- Page Range / eLocation ID:
- 1-11
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract Projecting species’ responses to future climate conditions is critical for anticipating conservation challenges and informing proactive policy and management decisions. However, best practices for choosing climate models for projection ensembles are currently in flux. We compared including a maximum number of models against trimming ensembles based on model validation. This was done within the emerging practice of ensemble building using an increasingly larger number of global climate models (GCMs) for future projections. We used recently reported estimates on primary drivers of population fluctuations for the migratory monarch butterfly (Danaus plexippus) to examine how multiple sources of uncertainty impact population forecasts for a well‐studied species. We compared mean spring temperature and precipitation observed in Texas from 1980 to 2005 with predictions from 16 GCMs to determine which of the models performed best. We then built tailored climate projections accumulating both temperature (in the form of growing degree days) and rainfall using both “complete” (all 16) and “trimmed” (best‐performing) ensembles based on three emission scenarios. We built the tailored projections of spring growing conditions to assess the range of possible climate outcomes and their potential impacts on monarch development. Results were similar when mean predictions were compared between trimmed and complete ensembles. However, when daily projections and uncertainty were accumulated over the entire spring season, we showed substantial differences between ensembles in terms of possible ecological outcomes. Ensembles that used all 16 GCMs included so much uncertainty that projections for future spring conditions ranged from being too cold to too hot for monarch development. GCMs based on best‐performing metrics provided much more useful information, projecting higher spring temperatures for developing monarch larvae in the future which could lead to larger summer populations but also suggesting risk from excessive heat. When there is a strong basis for identifying mechanistic drivers of population dynamics, our results support using a smaller subset of validated GCMs to bracket a range of the most defensible future environmental conditions tailored to the species of interest. Yet understating uncertainty remains a risk, and we recommend clearly articulating the rationale and consequences of selecting GCMs for long‐term projections.more » « less
-
Ensemble perception is a process by which we summarize complex scenes. Despite the importance of ensemble perception to everyday cognition, there are few computational models that provide a formal account of this process. Here we develop and test a model in which ensemble representations reflect the global sum of activation signals across all individual items. We leverage this set of minimal assumptions to formally connect a model of memory for individual items to ensembles. We compare our ensemble model against a set of alternative models in five experiments. Our approach uses performance on a visual memory task for individual items to generate zero-free-parameter predictions of interindividual and intraindividual differences in performance on an ensemble continuous-report task. Our top-down modelling approach formally unifies models of memory for individual items and ensembles and opens a venue for building and comparing models of distinct memory processes and representations.more » « less
-
null (Ed.)Launched in 2013, LivDet-Iris is an international competition series open to academia and industry with the aim to assess and report advances in iris Presentation Attack Detection (PAD). This paper presents results from the fourth competition of the series: LivDet-Iris 2020. This year's competition introduced several novel elements: (a) incorporated new types of attacks (samples displayed on a screen, cadaver eyes and prosthetic eyes), (b) initiated LivDet-Iris as an on-going effort, with a testing protocol available now to everyone via the Biometrics Evaluation and Testing (BEAT)* open-source platform to facilitate reproducibility and benchmarking of new algorithms continuously, and (c) performance comparison of the submitted entries with three baseline methods (offered by the University of Notre Dame and Michigan State University), and three open-source iris PAD methods available in the public domain. The best performing entry to the competition reported a weighted average APCER of 59.10% and a BPCER of 0.46% over all five attack types. This paper serves as the latest evaluation of iris PAD on a large spectrum of presentation attack instruments.more » « less
-
In this study, nine different statistical models are constructed using different combinations of predictors, including models with and without projected predictors. Multiple machine learning (ML) techniques are employed to optimize the ensemble predictions by selecting the top performing ensemble members and determining the weights for each ensemble member. The ML-Optimized Ensemble (ML-OE) forecasts are evaluated against the Simple-Averaging Ensemble (SAE) forecasts. The results show that for the response variables that are predicted with significant skill by individual ensemble members and SAE, such as Atlantic tropical cyclone counts, the performance of SAE is comparable to the best ML-OE results. However, for response variables that are poorly modeled by individual ensemble members, such as Atlantic and Gulf of Mexico major hurricane counts, ML-OE predictions often show higher skill score than individual model forecasts and the SAE predictions. However, neither SAE nor ML-OE was able to improve the forecasts of the response variables when all models show consistent bias. The results also show that increasing the number of ensemble members does not necessarily lead to better ensemble forecasts. The best ensemble forecasts are from the optimally combined subset of models.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
