Search for: All records

Long-horizon tasks in unstructured environments are notoriously challenging for robots because they require the prediction of extensive action plans with thousands of steps while adapting to ever-changing conditions by reasoning among multimodal sensing spaces. Humans can efficiently tackle such compound problems by breaking them down into easily reachable abstract sub-goals, significantly reducing complexity. Inspired by this ability, we explore how we can enable robots to acquire sub-goal formulation skills for long-horizon tasks and generalize them to novel situations and environments. To address these challenges, we propose the Zero-shot Abstract Sub-goal Framework (ZAS-F), which empowers robots to decompose overarching action plans into transferable abstract sub-goals, thereby providing zero-shot capability in new task conditions. ZAS-F is an imitation-learning-based method that efficiently learns a task policy from a few demonstrations. The learned policy extracts abstract features from multimodal and extensive temporal observations and subsequently uses these features to predict task-agnostic sub-goals by reasoning about their latent relations. We evaluated ZAS-F in radio frequency identification (RFID) inventory tasks across various dynamic environments, a typical long-horizon task requiring robots to handle unpredictable conditions, including unseen objects and structural layouts. Ourexperiments demonstrated that ZAS-F achieves a learning efficiency 30 times higher than previous methods, requiring only 8k demonstrations. Compared to prior approaches, ZAS-F achieves a 98.3% scanning accuracy while significantly reducing the training data requirement. Further, ZAS-F demonstrated strong generalization, maintaining a scan success rate of 99.4% in real-world deployment without additional finetuning. In long-term operations spanning 100 rooms, ZAS-F maintained consistent performance compared to short-term tasks, highlighting its robustness against compounding errors. These results establish ZAS-F as an efficient and adaptable solution for long-horizon robotic tasks in unstructured environments. 

Dense RFID environments pose critical challenges such as Reader-to-Reader Interference (RRI), Reader-to-Tag Collisions (RTC), and inefficient resource utilization, which degrade system performance and scalability. Traditional Media Access Control (MAC) protocols, including CSMA and TDMA, struggle to address these issues effectively, particularly in dynamic and large-scale deployments. This paper introduces MCSMARA (Markov Decision Process (MDP)-based Carrier Sense Multiple Access with Reader Arbitration), a novel MAC protocol designed to optimize reader coordination in dense RFID networks. By leveraging an MDP framework, MCSMARA models reader state transitions and employs a utility-based arbitration mechanism to dynamically allocate frequencies and time slots. The protocol incorporates adaptive backoff and decentralized neighborhood discovery for efficient resource management without centralized control. Simulation results demonstrate that MCSMARA reduces collisions by up to 30%, improves throughput by 25%, and ensures superior scalability, supporting a large amount of readers with minimal computational overhead. These findings establish MCSMARA as a transformative solution for RFID networks in logistics, retail, and industrial IoT, with potential for extension to mobile and heterogeneous environments. 

This paper presents the MAE model that uses a Masked AutoEncoder (MAE) to enhance the observations from commercial passive Radio-Frequency Identification (RFID) devices. It is crucial to address the common issue of RFID readers failing to collect observations from all their hop channels and antennas due to environmental effects and device limitations. The proposed method examines the inner rationale among observations from various channels and antennas to reconstruct the missing observations, which can significantly improve the performance of downstream applications. The experiment results show that when we collect more than 70% observation in all antennas at all channels, our MAE model can restore 90% of the missing phase with an error of less than 0.1 radians, which is even less than the error caused by thermal noise in an RFID system. Our MAE model's accuracy in restoring missing data provides a promising future to improve the performance of various RFID applications like localization and motion tracking by providing more complete observations. 

In dense RFID systems, efficient coordination of multiple readers is crucial to prevent reader-to-reader interference (RRI) and ensure optimal system performance. As the number of readers and tags increases, static frequency and time-slot assignment become insufficient to handle dynamic network conditions, leading to collisions, missed tag reads, and degraded throughput. In this paper, we propose a decentralized neigh-borhood discovery and management scheme for RFID systems operating in high-density environments. Our approach minimizes interference and improves tag read accuracy by dynamically adjusting communication parameters like frequency and time slots based on current system conditions, which are updated by periodic information exchanges among readers. Experimental results demonstrate that the proposed method significantly improves system scalability, throughput, and reliability. The proposed framework offers a scalable and adaptive solution for dense reader environments. 

In dense RFID systems, power control provides an effective means for maintaining communication efficiency and preventing reader-to-reader and reader-to-tag interference. Traditional RFID systems often operate at fixed power levels, which can lead to communication bottlenecks and inefficient tag reads in dynamic environments. This paper proposes an adaptive power control technique to improve the system performance by dynamically adjusting the transmit power based on environmental conditions, tag distance, and network congestion. Simulations and experimental results demonstrate that the proposed approach improves tag read rates, reduces interference, and enhances system robustness in dense environments. 

Sign language is a priceless means of communication for deaf and hard-of-hearing people to fully enable them to participate in society and interact with others. This study introduces a novel universal sign language system that uses the Gesture-script to generate a detailed description of gestures in videos, which involve continuous movement of hands, arms, heads, and body language. Subsequently, we input this description into a Large Language Model (LLM) to interpret sign language. We deployed a few-shot prompting technique for LLM, enabling it to precisely transfer the sign videos into corresponding sentences in natural language. Furthermore, the Few-shot prompting technique enables our system to interpret multiple types of sign language without pre-training or fine-tuning. 

Hand signals are the most widely used, feasible, and device-free communication method in manufacturing plants, airport ramps, and other noisy or voice-prohibiting environments. Enabling IoT agents, such as robots, to recognize and communicate by hand signals will facilitate human-machine collaboration for the emerging “Industry 5.0.” While many prior works succeed in hand signal recognition, few can rigorously guarantee the accuracy of their predictions. This project proposes a method that builds on the theory of conformal prediction (CP) to provide statistical guarantees on hand signal recognition accuracy and, based on it, measure the uncertainty in this communication process. It utilizes a calibration set with a few representative samples to ensure that trained models provide a conformal prediction set that reaches or exceeds the truth worth and trustworthiness at a user-specified level. Subsequently, the uncertainty in the recognition process can be detected by measuring the length of the conformal prediction set. Furthermore, the proposed CP-based method can be used with IoT models without fine-tuning as an out-of-the-box and promising lightweight approach to modeling uncertainty. Our experiments show that the proposed conformal recognition method can achieve accurate hand signal prediction in novel scenarios. When selecting an error level α = 0.10, it provided 100% accuracy for out-of-distribution test sets. 

In this paper, we present a conformal prediction (CP) based method to evaluate the performance of a finger-printing localization system through uncertainty quantification. The proposed method emphasizes a standalone module that is compatible with any well-trained fingerprint classifier without incurring extra training costs. It provides rigorous statistical guarantees for revealing true labels in the fingerprinting multi-class classification problems with high efficiency. Uncertainty quantification of the predictions is accomplished by leveraging a small calibration dataset and a given error tolerance level. Three specific metrics are introduced to quantify the uncertainty of the CP-based method from the perspective of efficiency, adaptivity, and accuracy, respectively. The proposed method allows developers to track the model state with minimal effort and evaluate the reliability of their model and measurements, such as in a dynamic environment. The proposed technique, therefore, prevents the intrinsic label inaccuracy and the additional labor cost of ground truth collection. We evaluate the proposed method and metrics in two representative indoor environments using vanilla fingerprint-based localization models with extensive experiments. Our experimental results show that the proposed method can successfully quantify the uncertainty of predictions. 

This project introduces a framework to enable robots to recognize human hand signals, a reliable and feasible device-free means of communication in many noisy environments such as construction sites and airport ramps, to facilitate efficient human-robot collaboration. Various hand signal systems are accepted in many small groups for specific purposes, such as Marshalling on airport ramps and construction site crane operations. Robots must be robust to unpredictable conditions, including various backgrounds and human appearances, an extreme challenge imposed by open environments. To address these challenges, we propose Instant Hand Signal Recognition (IHSR), a learning-based framework with world knowledge of human gestures embedded, for robots to learn novel hand signals in a few samples. It also offers robust zero-shot generalization to recognize learned signals in novel scenarios. Extensive experiments show that our IHSR can learn a novel hand signal in only 50 samples, which is 30+ times more efficient than the state-of-the-art method. It also demonstrates a robust zero-shot generalization for deploying a learned model in unseen environments to recognize hand signals from unseen human users. 

In this work, we integrate digital twin technology with RFID localization to achieve real-time monitoring of physical items in a large-scale complex environment, such as warehouses and retail stores. To map the item-level realities into a digital environment, we proposed a sensor fusion technique that merges a 3D map created by RGB-D and tracking cameras with real-time RFID tag location estimation derived from our novel Bayesian filter approach. Unlike mainstream localization methods, which rely on phase or RSSI measurements, our proposed method leverages a fixed RF transmission power model. This approach extends localization capabilities to all existing RFID devices, offering a significant advancement over conventional techniques. As a result, the proposed method transforms any RFID device into a digital twin scanner with the support of RGB-D cameras. To evaluate the performance of the proposed method, we prototype the system with commercial off-the-shelf (COTS) equipment in two representative retail scenarios. The overall performance of the system is demonstrated in a mock retail apparel store covering an area of 207 m2, while the quantitative experimental results are examined in a small-scale testbed to showcase the accuracy of item-level tag localization.