More Like this

1. Navigating Uncertainty: Ambiguity Quantification in Fingerprinting-based Indoor Localization

   https://doi.org/10.1109/AIoT63253.2024.00033  

   Ma, Junwei; Wang, Xiangyu; Zhang, Jian; Mao, Shiwen; Periaswamy, Senthilkumar_C G; Patton, Justin (July 2024, IEEE)

   In this paper, we present a conformal prediction (CP) based method to evaluate the performance of a finger-printing localization system through uncertainty quantification. The proposed method emphasizes a standalone module that is compatible with any well-trained fingerprint classifier without incurring extra training costs. It provides rigorous statistical guarantees for revealing true labels in the fingerprinting multi-class classification problems with high efficiency. Uncertainty quantification of the predictions is accomplished by leveraging a small calibration dataset and a given error tolerance level. Three specific metrics are introduced to quantify the uncertainty of the CP-based method from the perspective of efficiency, adaptivity, and accuracy, respectively. The proposed method allows developers to track the model state with minimal effort and evaluate the reliability of their model and measurements, such as in a dynamic environment. The proposed technique, therefore, prevents the intrinsic label inaccuracy and the additional labor cost of ground truth collection. We evaluate the proposed method and metrics in two representative indoor environments using vanilla fingerprint-based localization models with extensive experiments. Our experimental results show that the proposed method can successfully quantify the uncertainty of predictions. 

   more » « less
2. Leave-One-Out Stable Conformal Prediction

   Lee, Kiljae; Zhang, Yuan (January 2025, The Thirteenth International Conference on Learning Representations)

   Conformal prediction (CP) is an important tool for distribution-free predictive uncertainty quantification. Yet, a major challenge is to balance computational efficiency and prediction accuracy, particularly for multiple predictions. We propose Leave-One-Out Stable Conformal Prediction (LOO-StabCP), a novel method to speed up full conformal using algorithmic stability without sample splitting. By leveraging leave-one-out stability, our method is much faster in handling a large number of prediction requests compared to existing method RO-StabCP based on replace-one stability. We derived stability bounds for several popular machine learning tools: regularized loss minimization (RLM) and stochastic gradient descent (SGD), as well as kernel method, neural networks and bagging. Our method is theoretically justified and demonstrates superior numerical performance on synthetic and real-world data. We applied our method to a screening problem, where its effective exploitation of training data led to improved test power compared to state-of-the-art method based on split conformal. 

   more » « less
3. Uncertainty Quantification over Graph with Conformalized Graph Neural Networks

   Huang, Kexin; Jin, Ying; Candes, Emmanuel; Leskovec, Jure (December 2023, Advances in neural information processing systems)

   Graph Neural Networks (GNNs) are powerful machine learning prediction models on graph-structured data. However, GNNs lack rigorous uncertainty estimates, limiting their reliable deployment in settings where the cost of errors is significant. We propose conformalized GNN (CF-GNN), extending conformal prediction (CP) to graph-based models for guaranteed uncertainty estimates. Given an entity in the graph, CF-GNN produces a prediction set/interval that provably contains the true label with pre-defined coverage probability (e.g. 90%). We establish a permutation invariance condition that enables the validity of CP on graph data and provide an exact characterization of the test-time coverage. Besides valid coverage, it is crucial to reduce the prediction set size/interval length for practical use. We observe a key connection between non-conformity scores and network structures, which motivates us to develop a topology-aware output correction model that learns to update the prediction and produces more efficient prediction sets/intervals. Extensive experiments show that CF-GNN achieves any pre-defined target marginal coverage while significantly reducing the prediction set/interval size by up to 74% over the baselines. It also empirically achieves satisfactory conditional coverage over various raw and network features. 

   more » « less
4. IHSR: A Framework Enables Robots to Learn Novel Hand Signals From a Few Samples

   https://doi.org/10.1109/AIM55361.2024.10637093  

   Podder, Kanchon Kanti; Zhang, Jian; Wu, Yongshuai (July 2024, IEEE)

   This project introduces a framework to enable robots to recognize human hand signals, a reliable and feasible device-free means of communication in many noisy environments such as construction sites and airport ramps, to facilitate efficient human-robot collaboration. Various hand signal systems are accepted in many small groups for specific purposes, such as Marshalling on airport ramps and construction site crane operations. Robots must be robust to unpredictable conditions, including various backgrounds and human appearances, an extreme challenge imposed by open environments. To address these challenges, we propose Instant Hand Signal Recognition (IHSR), a learning-based framework with world knowledge of human gestures embedded, for robots to learn novel hand signals in a few samples. It also offers robust zero-shot generalization to recognize learned signals in novel scenarios. Extensive experiments show that our IHSR can learn a novel hand signal in only 50 samples, which is 30+ times more efficient than the state-of-the-art method. It also demonstrates a robust zero-shot generalization for deploying a learned model in unseen environments to recognize hand signals from unseen human users. 

   more » « less
5. Data-Efficient Conformalized Interval Prediction of Minimum Operating Voltage Capturing Process Variations

   Yin, Yuxuan; Chen, Rebecca; He, Chen; Li, Peng (June 2024, 61st ACM/IEEE Design Automation Conference (DAC ’24))

   Accurate minimum operating voltage Vmin prediction is a critical element in manufacturing tests. Conventional methods lack coverage guarantees in interval predictions. Conformal Prediction (CP), a distribution-free machine learning approach, excels in providing rigorous coverage guarantees for interval predictions. However, standard CP predictors may fail due to a lack of knowledge of process variations. We address this challenge by providing principled conformalized interval prediction in the presence of process variations with high data efficiency, where the data from a few additional chips is utilized for calibration. We demonstrate the superiority of the proposed method on industrial 16nm chip data. 

   more » « less