An official website of the United States government
Personalized recommender systems play a crucial role in modern society, especially in e-commerce, news, and ads areas. Correctly evaluating and comparing candidate recommendation models is as essential as constructing ones. The common offline evaluation strategy is holding out some user-interacted items from training data and evaluating the performance of recommendation models based on how many items they can retrieve. Specifically, for any hold-out item or so-called target item for a user, the recommendation models try to predict the probability that the user would interact with the item and rank it among overall items, which is calledglobal evaluation. Intuitively, a good recommendation model would assign high probabilities to such hold-out/target items. Based on the specific ranks, some metrics likeRecall@KandNDCG@Kcan be calculated to further quantify the quality of the recommender model. Instead of ranking the target items among all items, Koren first proposed to rank them among a smallsampled set of items, then quantified the performance of the models, which is calledsampling evaluation. Ever since then, there has been a large amount of work adopting sampling evaluation due to its efficiency and frugality. In recent work, Rendle and Krichene argued that the sampling evaluation is “inconsistent” with respect to a global evaluation in terms of offline top-Kmetrics. In this work, we first investigate the “inconsistent” phenomenon by taking a glance at the connections between sampling evaluation and global evaluation. We reveal the approximately linear relationship between sampling with respect to its global counterpart in terms of the top-KRecall metric. Second, we propose a new statistical perspective of the sampling evaluation—to estimate the global rank distribution of the entire population. After the estimated rank distribution is obtained, the approximation of the global metric can be further derived. Third, we extend the work of Krichene and Rendle, directly optimizing the error with ground truth, providing not only a comprehensive empirical study but also a rigorous theoretical understanding of the proposed metric estimators. To address the “blind spot” issue, where accurately estimating metrics for small top-Kvalues in sampling evaluation is challenging, we propose a novel adaptive sampling method that generalizes the expectation-maximization algorithm to this setting. Last but not least, we also study the user sampling evaluation effect. This series of works outlines a clear roadmap for sampling evaluation and establishes a foundational theoretical framework. Extensive empirical studies validate the reliability of the sampling methods presented.
more »
« less
Using projection operators with maximum overlap methods to simplify challenging self‐consistent field optimization
Abstract Maximum overlap methods are effective tools for optimizing challenging ground‐ and excited‐state wave functions using self‐consistent field models such as Hartree‐Fock and Kohn‐Sham density functional theory. Nevertheless, such models have shown significant sensitivity to the user‐defined initial guess of the target wave function. In this work, a projection operator framework is defined and used to provide a metric for non‐aufbau orbital selection in maximum‐overlap‐methods. The resulting algorithms, termed the Projection‐based Maximum Overlap Method (PMOM) and Projection‐based Initial Maximum Overlap Method (PIMOM), are shown to perform exceptionally well when using simple user‐defined target solutions based on occupied/virtual molecular orbital permutations. This work also presents a new metric that provides a simple and conceptually convenient measure of agreement between the desired target and the current or final SCF results during a calculation employing a maximum‐overlap method.
more »
« less
- PAR ID:
- 10363254
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Journal of Computational Chemistry
- Volume:
- 43
- Issue:
- 6
- ISSN:
- 0192-8651
- Page Range / eLocation ID:
- p. 382-390
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract MotivationCell function is regulated by gene regulatory networks (GRNs) defined by protein-mediated interaction between constituent genes. Despite advances in experimental techniques, we can still measure only a fraction of the processes that govern GRN dynamics. To infer the properties of GRNs using partial observation, unobserved sequential processes can be replaced with distributed time delays, yielding non-Markovian models. Inference methods based on the resulting model suffer from the curse of dimensionality. ResultsWe develop a simulation-based Bayesian MCMC method employing an approximate likelihood for the efficient and accurate inference of GRN parameters when only some of their products are observed. We illustrate our approach using a two-step activation model: an activation signal leads to the accumulation of an unobserved regulatory protein, which triggers the expression of observed fluorescent proteins. With prior information about observed fluorescent protein synthesis, our method successfully infers the dynamics of the unobserved regulatory protein. We can estimate the delay and kinetic parameters characterizing target regulation including transcription, translation, and target searching of an unobserved protein from experimental measurements of the products of its target gene. Our method is scalable and can be used to analyze non-Markovian models with hidden components. Availability and implementationOur code is implemented in R and is freely available with a simple example data at https://github.com/Mathbiomed/SimMCMC.more » « less
-
Fast solution of wave propagation in periodic structures usually relies on simplified approaches, such as analytical methods, transmission line models, scattering matrix approaches, plane wave methods, etc. For complex multi-dimensional problems, computationally intensive direct numerical simulation (DNS) is always needed. This study demonstrates a fast and accurate simulation methodology enabled by a physics-based learning methodology, derived from proper orthogonal decomposition (POD) and Galerkin projection, for periodic quantum nanostructure and photonic crystals. POD is a projection-based method that generates optimal basis functions (or POD modes) via solution data collected from DNSs. This process trains the POD modes to adapt parametric variations of the system and offers the best least squares (LS) fit to the solution using the smallest number of modes. This is very different from other projection approaches, e.g., Fourier, Legendre, Bessel, Airy functions, etc., that adopt assumed basis functions selected for the problem based on the solution form. After generating the optimal POD modes, Galerkin projection of the wave equation onto each of the POD modes is performed to close the model and incorporate physical principles guided by the wave equation. Such a rigorous approach offers efficient simulations with high accuracy and exhibits the extrapolation ability in cases reasonably beyond the training bounds. The POD-Galerkin methodology is applied in this study to predict band structures and wave solutions for 2D periodic quantum-dot and photonic-lattice structures. The plane-wave approach is also included in a periodic quantum-dot structure to illustrate the superior performance of the POD-Galerkin methodology. The POD-Galerkin approach offers a 2-order computing speedup for both nanostructure and optical superlattices, compared to DNS, when solving both the wave solution and band structure. If the band structure is the only concern, a 4-order improvement in computational efficiency can be achieved. Fig. 1(a) shows the optical superlattice in a demonstration, where a unit cell includes 22 discs with diagonally symmetrical refractive indices and the background index n = 1. The POD modes for this case are trained by TE mode electric field data collected from DNSs with variation of diagonally symmetrical refractive indices. The LS error of the predicted electric field wave solution from the POD-Galerkin approach, shown in Fig. 1(b) compared to DNS, is below 1% with just 8 POD modes that offer a more than 4-order reduction in the degrees of freedom, compared to DNS. In addition, an extremely accurate prediction of band structure is illustrated in Fig. 1(c) with a maximum error below 0.1% in the entire Brillouin zone.more » « less
-
Accurate simulation of electronic excited states of large chromophores is often difficult due to the computationally expensive nature of existing methods. Common approximations such as fragmentation methods that are routinely applied to ground-state calculations of large molecules are not easily applicable to excited states due to the delocalized nature of electronic excitations in most practical chromophores. Thus, special techniques specific to excited states are needed. Δ-SCF methods are one such approximation that treats excited states in a manner analogous to that for ground-state calculations, accelerating the simulation of excited states. In this work, we employed the popular initial maximum overlap method (IMOM) to avoid the variational collapse of the electronic excited state orbitals to the ground state. We demonstrate that it is possible to obtain emission energies from the first singlet (S1) excited state of many thousands of dye molecules without any external intervention. Spin correction was found to be necessary to obtain accurate excitation and emission energies. Using thousands of dye-like chromophores and various solvents (12,318 combinations), we show that the spin-corrected initial maximum overlap method accurately predicts emission maxima with a mean absolute error of only 0.27 eV. We further improved the predictive accuracy using linear fit-based corrections from individual dye classes to achieve an impressive performance of 0.17 eV. Additionally, we demonstrate that IMOM spin density can be used to identify the dye class of chromophores, enabling improved prediction accuracy for complex dye molecules, such as dyads (chromophores containing moieties from two different dye classes). Finally, the convergence behavior of IMOM excited state SCF calculations is analyzed briefly to identify the chemical space, where IMOM is more likely to fail.more » « less
-
Abstract We propose a novel method for sampling and optimization tasks based on a stochastic interacting particle system. We explain how this method can be used for the following two goals: (i) generating approximate samples from a given target distribution and (ii) optimizing a given objective function. The approach is derivative‐free and affine invariant, and is therefore well‐suited for solving inverse problems defined by complex forward models: (i) allows generation of samples from the Bayesian posterior and (ii) allows determination of the maximum a posteriori estimator. We investigate the properties of the proposed family of methods in terms of various parameter choices, both analytically and by means of numerical simulations. The analysis and numerical simulation establish that the method has potential for general purpose optimization tasks over Euclidean space; contraction properties of the algorithm are established under suitable conditions, and computational experiments demonstrate wide basins of attraction for various specific problems. The analysis and experiments also demonstrate the potential for the sampling methodology in regimes in which the target distribution is unimodal and close to Gaussian; indeed we prove that the method recovers a Laplace approximation to the measure in certain parametric regimes and provide numerical evidence that this Laplace approximation attracts a large set of initial conditions in a number of examples.more » « less
