An official website of the United States government
Abstract Cross-survey generalization is a critical challenge in stellar spectral analysis, particularly in cases such as transferring from low- to moderate-resolution surveys. We investigate this problem using pretrained models, focusing on simple neural networks such as multilayer perceptrons (MLPs), with a case study transferring from LAMOST low-resolution spectra (LRS) to DESI medium-resolution spectra. Specifically, we pretrain MLPs on either LRS or their embeddings and fine-tune them for application to DESI stellar spectra. We compare MLPs trained directly on spectra with those trained on embeddings derived from transformer-based models (self-supervised foundation models pretrained for multiple downstream tasks). We also evaluate different fine-tuning strategies, including residual-head fine-tuning, low-rank adaptation, and full fine-tuning. We find that MLPs pretrained on LAMOST LRS achieve strong performance, even without fine-tuning, and that modest fine-tuning with DESI spectra further improves the results. For iron abundance, embeddings from a transformer-based model yield advantages in the metal-rich ([Fe/H] > −1.0) regime, but underperform in the metal-poor regime compared to MLPs trained directly on LRS. We also show that the optimal fine-tuning strategy depends on the specific stellar parameter under consideration. These results highlight that simple pretrained MLPs can provide competitive cross-survey generalization, while the role of spectral foundation models for cross-survey stellar parameter estimation requires further exploration.
more »
« less
TransformerPayne: Enhancing Spectral Emulation Accuracy and Data Efficiency by Capturing Long-range Correlations
Abstract Stellar spectra emulators often rely on large grids and tend to reach a plateau in emulation accuracy, leading to significant systematic errors when inferring stellar properties. Our study explores the use of Transformer models to capture long-range information in spectra, comparing their performance to the Payne emulator (a fully connected multilayer perceptron), an expanded version of The Payne, and a convolutional-based emulator. We tested these models on synthetic spectral grids, evaluating their performance by analyzing emulation residuals and assessing the quality of spectral parameter inference. The newly introduced TransformerPayne emulator outperformed all other tested models, achieving a mean absolute error (MAE) of approximately 0.15% when trained on the full grid. The most significant improvements were observed in grids containing between 1000 and 10,000 spectra, with TransformerPayne showing 2–5 times better performance than the scaled-up version of The Payne. Additionally, TransformerPayne demonstrated superior fine-tuning capabilities, allowing for pretraining on one spectral model grid before transferring to another. This fine-tuning approach enabled up to a 10-fold reduction in training grid size compared to models trained from scratch. Analysis of TransformerPayne's attention maps revealed that they encode interpretable features common across many spectral lines of chosen elements. While scaling up The Payne to a larger network reduced its MAE from 1.2% to 0.3% when trained on the full data set, TransformerPayne consistently achieved the lowest MAE across all tests. The inductive biases of the TransformerPayne emulator enhance accuracy, data efficiency, and interpretability for spectral emulation compared to existing methods.
more »
« less
- Award ID(s):
- 2406729
- PAR ID:
- 10704940
- Publisher / Repository:
- The Astrophysical Journal
- Date Published:
- Journal Name:
- The Astrophysical Journal
- Volume:
- 980
- Issue:
- 1
- ISSN:
- 0004-637X
- Page Range / eLocation ID:
- 66
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract The Sloan Digital Sky Survey (SDSS) has recently initiated its fifth survey generation (SDSS-V), with a central focus on stellar spectroscopy. In particular, SDSS-V's Milky Way Mapper program will deliver multiepoch optical and near-infrared spectra for more than 5 × 10 6 stars across the entire sky, covering a large range in stellar mass, surface temperature, evolutionary stage, and age. About 10% of those spectra will be of hot stars of OBAF spectral types, for whose analysis no established survey pipelines exist. Here we present the spectral analysis algorithm, ZETA-PAYNE, developed specifically to obtain stellar labels from SDSS-V spectra of stars with these spectral types and drawing on machine-learning tools. We provide details of the algorithm training, its test on artificial spectra, and its validation on two control samples of real stars. Analysis with ZETA-PAYNE leads to only modest internal uncertainties in the near-IR with APOGEE (optical with BOSS): 3%–10% (1%–2%) for T eff , 5%–30% (5%–25%) for v sin i , 1.7–6.3 km s −1 (0.7–2.2 km s −1 ) for radial velocity, <0.1 dex (<0.05 dex) for log g , and 0.4–0.5 dex (0.1 dex) for [M/H] of the star, respectively. We find a good agreement between atmospheric parameters of OBAF-type stars when inferred from their high- and low-resolution optical spectra. For most stellar labels, the APOGEE spectra are (far) less informative than the BOSS spectra of these stars, while log g , v sin i , and [M/H] are in most cases too uncertain for meaningful astrophysical interpretation. This makes BOSS low-resolution optical spectra better for stellar labels of OBAF-type stars, unless the latter are subject to high levels of extinction.more » « less
-
Fine-tuning is essential to adapting pre-trained large language models to downstream applications. With the increasing popularity of LLM-enabled applications, fine-tuning has been performed intensively worldwide, incurring a tremendous amount of computing costs that correspond to big carbon footprint and environmental impact. Mitigating such environmental impact directly correlates to reducing the fine-tuning FLOPs. Existing fine-tuning schemes focus on either saving memory or reducing the overhead of computing weight updates, but cannot achieve sufficient FLOPs reduction due to their ignorance of the training cost in backpropagation. To address this limitation, in this paper we present GreenTrainer, a new technique that minimizes the FLOPs of LLM fine-tuning via adaptive backpropagation, which adaptively selects the most appropriate set of LLM tensors for fine-tuning based on their importance and backpropagation cost in training. Experiment results show that GreenTrainer can save up to 64% training FLOPs compared to full fine-tuning, without any noticeable accuracy loss. Compared to the existing schemes such as Prefix Tuning and LoRA, GreenTrainer can achieve up to 4% improvement of model accuracy, with on-par FLOPs reduction.more » « less
-
Integrated hydrologic models solve coupled mathematical equations that represent natural processes, including groundwater, unsaturated, and overland flow. However, these models are computationally expensive. It has been recently shown that machine leaning (ML) and deep learning (DL) in particular could be used to emulate complex physical processes in the earth system. In this study, we demonstrate how a DL model can emulate transient, three-dimensional integrated hydrologic model simulations at a fraction of the computational expense. This emulator is based on a DL model previously used for modeling video dynamics, PredRNN. The emulator is trained based on physical parameters used in the original model, inputs such as hydraulic conductivity and topography, and produces spatially distributed outputs (e.g., pressure head) from which quantities such as streamflow and water table depth can be calculated. Simulation results from the emulator and ParFlow agree well with average relative biases of 0.070, 0.092, and 0.032 for streamflow, water table depth, and total water storage, respectively. Moreover, the emulator is up to 42 times faster than ParFlow. Given this promising proof of concept, our results open the door to future applications of full hydrologic model emulation, particularly at larger scales.more » « less
-
The integration of renewable energy resources has made power system management increasingly complex. DRL is a potential solution to optimize power system operations, but it requires significant time and resources during training. The control policies developed using DRL are specific to a single grid and require retraining from scratch for other grids. Training the DRL model from scratch is computationally expensive. This paper proposes a novel TL with a DRL framework to optimize VV C across different grids. This framework significantly reduces training time and improves VVC control performance by fine-tuning pre-trained DRL models for various grids. We developed a policy reuse classifier that transfers the knowledge from the IEEE-123 Bus system to the IEEE-13 Bus system. We performed an impact analysis to determine the effectiveness of TL. Our results show that TL improves the VVC control policy by 69.51 %, achieves faster convergence, and reduces the training time by 98.14%.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Journal Article:
- https://doi.org/10.3847/1538-4357/ad9b99
