An official website of the United States government
Uncertainty arises naturally in many application domains due to, e.g., data entry errors and ambiguity in data cleaning. Prior work in incomplete and probabilistic databases has investigated the semantics and efficient evaluation of ranking and top-k queries over uncertain data. However, most approaches deal with top-k and ranking in isolation and do represent uncertain input data and query results using separate, incompatible data models. We present an efficient approach for under- and over-approximating results of ranking, top-k, and window queries over uncertain data. Our approach integrates well with existing techniques for querying uncertain data, is efficient, and is to the best of our knowledge the first to support windowed aggregation. We design algorithms for physical operators for uncertain sorting and windowed aggregation, and implement them in PostgreSQL. We evaluated our approach on synthetic and real world datasets, demonstrating that it outperforms all competitors, and often produces more accurate results.
more »
« less
SQEE: A Machine Perception Approach to Sensing Quality Evaluation at the Edge by Uncertainty Quantification
Cyber-physical systems are starting to adopt neural network (NN) models for a variety of smart sensing applications. While several efforts seek better NN architectures for system performance improvement, few attempts have been made to study the deployment of these systems in the field. Proper deployment of these systems is critical to achieving ideal performance, but the current practice is largely empirical via trials and errors, lacking a measure of quality. Sensing quality should reflect the impact on the performance of NN models that drive machine perception tasks. However, traditional approaches either evaluate statistical difference that exists objectively, or model the quality subjectively via human perception. In this work, we propose an efficient sensing quality measure requiring limited data samples using smart voice sensing system as an example. We adopt recent techniques in uncertainty evaluation for NN to estimate audio sensing quality. Intuitively, a deployment at better sensing location should lead to less uncertainty in NN predictions. We design SQEE, Sensing Quality Evaluation at the Edge for NN models, which constructs a model ensemble through Monte-Carlo dropout and estimates posterior total uncertainty via average conditional entropy. We collected data from three indoor environments, with a total of 148 transmitting-receiving (t-r) locations experimented and more than 7,000 examples tested. SQEE achieves the best performance in terms of the top-1 ranking accuracy---whether the measure finds the best spot for deployment, in comparison with other uncertainty strategies. We implemented SQEE on a ReSpeaker to study SQEE's real-world efficacy. Experimental result shows that SQEE can effectively evaluate the data collected from each t-r location pair within 30 seconds and achieve an average top-3 ranking accuracy of over 94%. We further discuss generalization of our framework to other sensing schemes.
more »
« less
- Award ID(s):
- 2040727
- PAR ID:
- 10403518
- Date Published:
- Journal Name:
- Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems
- Page Range / eLocation ID:
- 277 to 290
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Rankings and scores are two common data types used by judges to express preferences and/or perceptions of quality in a collection of objects. Numerous models exist to study data of each type separately, but no unified statistical model captures both data types simultaneously without first performing data conversion. We propose the Mallows-Binomial model to close this gap, which combines a Mallows $$\phi$$ ranking model with Binomial score models through shared parameters that quantify object quality, a consensus ranking, and the level of consensus among judges. We propose an efficient tree-search algorithm to calculate the exact MLE of model parameters, study statistical properties of the model both analytically and through simulation, and apply our model to real data from an instance of grant panel review that collected both scores and partial rankings. Furthermore, we demonstrate how model outputs can be used to rank objects with confidence. The proposed model is shown to sensibly combine information from both scores and rankings to quantify object quality and measure consensus with appropriate levels of statistical uncertainty.more » « less
-
Context. Terrestrial exoplanets in the habitable zone are likely a common occurrence. The long-term goal is to characterize the atmospheres of dozens of such objects. The Large Interferometer For Exoplanets (LIFE) initiative aims to develop a space-based mid-infrared (MIR) nulling interferometer to measure the thermal emission spectra of such exoplanets. Aims. We investigate how well LIFE could characterize a cloudy Venus-twin exoplanet. This allows us to: (1) test our atmospheric retrieval routine on a realistic non-Earth-like MIR emission spectrum of a known planet, (2) investigate how clouds impact retrievals, and (3) further refine the LIFE requirements derived in previous Earth-centered studies. Methods. We ran Bayesian atmospheric retrievals for simulated LIFE observations of a Venus-twin exoplanet orbiting a Sun-like star located 10 pc from the observer. The LIFE SIM noise model accounted for all major astrophysical noise sources. We ran retrievals using different models (cloudy and cloud-free) and analyzed the performance as a function of the quality of the LIFE observation. This allowed us to determine how well the atmosphere and clouds are characterizable depending on the quality of the spectrum. Results. At the current minimal resolution ( R = 50) and signal-to-noise ( S / N = 10 at 11.2 μ m) requirements for LIFE, all tested models suggest a CO 2 -rich atmosphere (≥30% in mass fraction). Further, we successfully constrain the atmospheric pressure-temperature ( P–T ) structure above the cloud deck ( P–T uncertainty ≤ ± 15 K). However, we struggle to infer the main cloud properties. Further, the retrieved planetary radius ( R pl ), equilibrium temperature ( T eq ), and Bond albedo ( A B ) depend on the model. Generally, a cloud-free model performs best at the current minimal quality and accurately estimates R pl , T eq , and A B . If we consider higher quality spectra (especially S / N = 20), we can infer the presence of clouds and pose first constraints on their structure. Conclusions. Our study shows that the minimal R and S/N requirements for LIFE suffice to characterize the structure and composition of a Venus-like atmosphere above the cloud deck if an adequate model is chosen. Crucially, the cloud-free model is preferred by the retrieval for low spectral qualities. We thus find no direct evidence for clouds at the minimal R and S / N requirements and cannot infer the thickness of the atmosphere. Clouds are only constrainable in MIR retrievals of spectra with S / N ≥ 20. The model dependence of our retrieval results emphasizes the importance of developing a community-wide best-practice for atmospheric retrieval studies.more » « less
-
In practice, it is essential to compare and rank candidate policies offline before real-world deployment for safety and reliability. Prior work seeks to solve this offline policy ranking (OPR) problem through value-based methods, such as Off-policy evaluation (OPE). However, they fail to analyze special case performance (e.g., worst or best cases), due to the lack of holistic characterization of policies’ performance. It is even more difficult to estimate precise policy values when the reward is not fully accessible under sparse settings. In this paper, we present Probabilistic Offline Policy Ranking (POPR), a framework to address OPR problems by leveraging expert data to characterize the probability of a candidate policy behaving like experts, and approximating its entire performance posterior distribution to help with ranking. POPR does not rely on value estimation, and the derived performance posterior can be used to distinguish candidates in worst-, best-, and average-cases. To estimate the posterior, we propose POPR-EABC, an Energy-based Approximate Bayesian Computation (ABC) method conducting likelihood-free inference. POPR-EABC reduces the heuristic nature of ABC by a smooth energy function, and improves the sampling efficiency by a pseudo-likelihood. We empirically demonstrate that POPR-EABC is adequate for evaluating policies in both discrete and continuous action spaces across various experiment environments, and facilitates probabilistic comparisons of candidate policies before deployment.more » « less
-
In recent years, significant work has been done in technological enhancements for mobility aids (smart walkers). However, most of this work does not cover the millions of people who have both mobility and visual impairments. In this paper, we design and study four different configurations of smart walkers that are specifically targeted to the needs of this population. We investigated different sensing technologies (ultrasound-based, infrared depth cameras and RGB cameras with advanced computer vision processing), software configurations, and user interface modalities (haptic and audio signal based). Our experiments show that there are several engineering choices that can be used in the design of such assistive devices. Furthermore, we found that a holistic evaluation of the end-to-end performance of the systems is necessary, as the quality of the user interface often has a larger impact on the overall performance than increases in the sensing accuracy beyond a certain point.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3560905.3568534
