An official website of the United States government
Mechanical search is a robotic problem where a robot needs to retrieve a target item that is partially or fully occluded from its camera. State-of-the-art approaches for mechanical search either require an expensive search process to find the target item, or they require the item to be tagged with a radio frequency identification tag (e.g., RFID), making their approach beneficial only to tagged items in the environment. We present FuseBot, the first robotic system for RF-Visual mechanical search that enables efficient retrieval of both RFtagged and untagged items in a pile. Rather than requiring all target items in a pile to be RF-tagged, FuseBot leverages the mere existence of an RF-tagged item in the pile to benefit both tagged and untagged items. Our design introduces two key innovations. The first is RF-Visual Mapping, a technique that identifies and locates RF-tagged items in a pile and uses this information to construct an RF-Visual occupancy distribution map. The second is RF-Visual Extraction, a policy formulated as an optimization problem that minimizes the number of actions required to extract the target object by accounting for the probabilistic occupancy distribution, the expected grasp quality, and the expected information gain from future actions. We built a real-time end-to-end prototype of our system on a UR5e robotic arm with in-hand vision and RF perception modules. We conducted over 180 real-world experimental trials to evaluate FuseBot and compare its performance to a of-the-art vision-based system named X-Ray. Our experimental results demonstrate that FuseBot outperforms X-Ray’s efficiency by more than 40% in terms of the number of actions required for successful mechanical search. Furthermore, in comparison to X-Ray’s success rate of 84%, FuseBot achieves a success rate of 95% in retrieving untagged items, demonstrating for the first time that the benefits of RF perception extend beyond tagged objects in the mechanical search problem.
more »
« less
Exploiting Synergies between Augmented Reality and RFIDs for Item Localization and Retrieval
Locating RFID-tagged items in the environment and guiding humans to retrieve the tagged items is an important problem in the RFID community. This paper explores how to exploit synergies between Augmented Reality (AR) headsets and RFID localization to help solve this problem by improving both user experience and localization accuracy. Using fundamental mathematical formulations for RFID localization, we derive confidence metrics and display guidance to the user to improve their experience and enable them to retrieve items faster. We build our primitives into an end - to-end system, RF - AR, and show that it achieves 8.6 cm median localization accuracy within 76 seconds and enables 55% faster retrieval than state-of-the-art past systems. Our results demonstrate that AR-based “human-in-the-loop” designs can make the localization task more accurate and efficient, and thus holds the potential to improve processes where items need to be retrieved quickly, such as in manufacturing, retail, and warehousing.
more »
« less
- Award ID(s):
- 1844280
- PAR ID:
- 10502569
- Publisher / Repository:
- IEEE
- Date Published:
- ISBN:
- 979-8-3503-3551-4
- Page Range / eLocation ID:
- 30 to 35
- Format(s):
- Medium: X
- Location:
- Seattle, WA, USA
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
We present the design, implementation, and evaluation of RFusion, a robotic system that can search for and retrieve RFID-tagged items in line-of-sight, non-line-of-sight, and fully-occluded settings. RFusion consists of a robotic arm that has a camera and antenna strapped around its gripper. Our design introduces two key innovations: the first is a method that geometrically fuses RF and visual information to reduce uncertainty about the target object's location, even when the item is fully occluded. The second is a novel reinforcement-learning network that uses the fused RF-visual information to efficiently localize, maneuver toward, and grasp target items. We built an end-to-end prototype of RFusion and tested it in challenging real-world environments. Our evaluation demonstrates that RFusion localizes target items with centimeter-scale accuracy and achieves 96% success rate in retrieving fully occluded objects, even if they are under a pile. The system paves the way for novel robotic retrieval tasks in complex environments such as warehouses, manufacturing plants, and smart homes.more » « less
-
We present RL2, a robotic system for efficient and accurate localization of UHF RFID tags. In contrast to past robotic RFID localization systems, which have mostly focused on location accuracy, RL2 learns how to jointly optimize the accuracy and speed of localization. To do so, it introduces a reinforcement learning-based (RL) trajectory optimization network that learns the next best trajectory for a robot-mounted reader antenna. Our algorithm encodes the aperture length and location confidence (using a synthetic-aperture-radar formulation) from multiple RFID tags into the state observations and uses them to learn the optimal trajectory. We built an end-to-end prototype of RL2 with an antenna moving on a ceiling-mounted 2D robotic track. We evaluated RL2 and demonstrated that with the median 3D localization accuracy of 0.55m, it locates multiple RFID tags 2.13x faster compared to a baseline strategy. Our results show the potential for RL-based RFID localization to enhance the efficiency of RFID inventory processes in areas spanning manufacturing, retail, and logistics.more » « less
-
In this work, we integrate digital twin technology with RFID localization to achieve real-time monitoring of physical items in a large-scale complex environment, such as warehouses and retail stores. To map the item-level realities into a digital environment, we proposed a sensor fusion technique that merges a 3D map created by RGB-D and tracking cameras with real-time RFID tag location estimation derived from our novel Bayesian filter approach. Unlike mainstream localization methods, which rely on phase or RSSI measurements, our proposed method leverages a fixed RF transmission power model. This approach extends localization capabilities to all existing RFID devices, offering a significant advancement over conventional techniques. As a result, the proposed method transforms any RFID device into a digital twin scanner with the support of RGB-D cameras. To evaluate the performance of the proposed method, we prototype the system with commercial off-the-shelf (COTS) equipment in two representative retail scenarios. The overall performance of the system is demonstrated in a mock retail apparel store covering an area of 207 m2, while the quantitative experimental results are examined in a small-scale testbed to showcase the accuracy of item-level tag localization.more » « less
-
There is much interest in fine-grained RFID localization systems. Existing systems for accurate localization typically require infrastructure, either in the form of extensive reference tags or many antennas (e.g., antenna arrays) to localize RFID tags within their radio range. Yet, there remains a need for fine-grained RFID localization solutions that are in a compact, portable, mobile form, that can be held by users as they walk around areas to map them, such as in retail stores, warehouses, or manufacturing plants. We present the design, implementation, and evaluation of POLAR, a portable handheld system for fine-grained RFID localization. Our design introduces two key innovations that enable robust, accurate, and real-time localization of RFID tags. The first is complex-controlled polarization (CCP), a mechanism for localizing RFIDs at all orientations through software-controlled polarization of two linearly polarized antennas. The second is joint tag discovery and localization (JTDL), a method for simultaneously localizing and reading tags with zero-overhead regardless of tag orientation. Building on these two techniques, we develop an end-to-end handheld system that addresses a number of practical challenges in self-interference, efficient inventorying, and self-localization. Our evaluation demonstrates that POLAR achieves a median accuracy of a few centimeters in each of the x/y/z dimensions in practical indoor environments.more » « less
