skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: Vision‐Assisted Avocado Harvesting with Aerial Bimanual Manipulation
Robotic fruit harvesting holds potential in precision agriculture to improve harvesting efficiency. While ground mobile robots are mostly employed in fruit harvesting, certain crops, like avocado trees, cannot be harvested efficiently from the ground alone. This is because of unstructured ground and planting arrangement and high‐to‐reach fruits. In such cases, aerial robots integrated with manipulation capabilities can pave new ways in robotic harvesting. This paper outlines the design and implementation of a bimanual aerial robot that employs visual perception and learning to detect avocados, reach, and harvest them autonomously. The dual‐arm system comprises a gripper and a fixer arm, to address a key challenge when harvesting avocados: once grasped, applying a rotational motion is the most mechanically efficient way to detach the avocado from the peduncle; however, the peduncle may store elastic energy preventing the avocado from being harvested. The fixer arm aims to stabilize the peduncle, allowing the gripper arm to harvest. The integrated visual perception process enables the detection of avocados and the determination of their pose; the latter is then used to determine target points for a bimanual manipulation planner. Several experiments are conducted in controlled indoor and outdoor settings to assess the efficacy of each component individually. Further, an integrated experiment in outdoor semicontrolled settings is used for feasibility assessment of the overall system. Results demonstrate that all different components can work synergistically to enable robotic avocado harvesting in (semi‐)controlled settings. Results also highlight limitations of an airborne harvesting solution and reveal tradeoffs to be considered in the selection of a harvesting robot.  more » « less
Award ID(s):
2326309
PAR ID:
10640271
Author(s) / Creator(s):
 ;  ;  ;  
Publisher / Repository:
Wiley Blackwell (John Wiley & Sons)
Date Published:
Journal Name:
Advanced Robotics Research
ISSN:
2943-9973
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    This paper addresses the problem of autonomously deploying an unmanned aerial vehicle in non-trivial settings, by leveraging a manipulator arm mounted on a ground robot, acting as a versatile mobile launch platform. As real-world deployment scenarios for micro aerial vehicles such as searchand- rescue operations often entail exploration and navigation of challenging environments including uneven terrain, cluttered spaces, or even constrained openings and passageways, an often arising problem is that of ensuring a safe take-off location, or safely fitting through narrow openings while in flight. By facilitating launching from the manipulator end-effector, a 6- DoF controllable take-off pose within the arm workspace can be achieved, which allows to properly position and orient the aerial vehicle to initialize the autonomous flight portion of a mission. To accomplish this, we propose a sampling-based planner that respects a) the kinematic constraints of the ground robot / manipulator / aerial robot combination, b) the geometry of the environment as autonomously mapped by the ground robots perception systems, and c) accounts for the aerial robot expected dynamic motion during takeoff. The goal of the proposed planner is to ensure autonomous collision-free initialization of an aerial robotic exploration mission, even within a cluttered constrained environment. At the same time, the ground robot with the mounted manipulator can be used to appropriately position the take-off workspace into areas of interest, effectively acting as a carrier launch platform. We experimentally demonstrate this novel robotic capability through a sequence of experiments that encompass a micro aerial vehicle platform carried and launched from a 6-DoF manipulator arm mounted on a four-wheel robot base. 
    more » « less
  2. Shi, Haitao (Ed.)
    Avocados are an important economic crop of Hawaii, contributing to approximately 3% of all avocados grown in the United States. To export Hawaii-grown avocados, growers must follow strict United States Department of Agriculture Animal and Plant Health Inspection Service (USDA-APHIS) regulations. Currently, only the Sharwil variety can be exported relying on a systems approach, which allows fruit to be exported without quarantine treatment; treatments that can negatively impact the quality of avocados. However, for the systems approach to be applied, Hawaii avocado growers must positively identify the avocados variety as Sharwil with APHIS prior to export. Currently, variety identification relies on physical characteristics, which can be erroneous and subjective, and has been disputed by growers. Once the fruit is harvested, variety identification is difficult. While molecular markers can be used through DNA extraction from the skin, the process leaves the fruit unmarketable. This study evaluated the feasibility of using near-infrared spectroscopy to non-destructively discriminate between different Hawaii-grown avocado varieties, such as Sharwil, Beshore, and Yamagata, Nishikawa, and Greengold, and to positively identify Sharwil from the other varieties mentioned above. The classifiers built using a bench-top system achieved 95% total classification rates for both discriminating the varieties from one another and positively identifying Sharwil while the classifier built using a handheld spectrometer achieved 96% and 96.7% total classification rates for discriminating the varieties from one another and positively identifying Sharwil, respectively. Results from chemometric methods and chemical analysis suggested that water and lipid were key contributors to the performance of classifiers. The positive results demonstrate the feasibility of NIR spectroscopy for discriminating different avocado varieties as well as authenticating Sharwil. To develop robust and stable models for the growers, distributors, and regulators in Hawaii, more varieties and additional seasons should continue to be added. 
    more » « less
  3. Abstract Soft robots have attracted great attention in the past decades owing to their unique flexibility and adaptability for accomplishing tasks via simple control strategies, as well as their inherent safety for interactions with humans and environments. Here, a soft robotic manipulation system capable of stiffness variation and dexterous operations through a remotely controlled manner is reported. The smart manipulation system consists of a soft omnidirectional arm, a dexterous multimaterial gripper, and a self‐powered human–machine interface (HMI) for teleoperation. The cable‐driven soft arm is made of soft elastomers and embedded with low melting point alloy as a stiffness‐tuning mechanism. The self‐powered HMI patches are designed based on the triboelectric nanogenerator that utilizes a sliding mode of tribo‐layers made of copper and polytetrafluoroethylene. The novel soft manipulation system has great potential for soft and remote manipulation and human machine interactions in a variety of applications from elderly care to surgical operation to agriculture harvesting. 
    more » « less
  4. null (Ed.)
    Although general purpose robotic manipulators are becoming more capable at manipulating various objects, their ability to manipulate millimeter-scale objects are usually limited. On the other hand, ultrasonic levitation devices have been shown to levitate a large range of small objects, from polystyrene balls to living organisms. By controlling the acoustic force fields, ultrasonic levitation devices can compensate for robot manipulator positioning uncertainty and control the grasping force exerted on the target object. The material agnostic nature of acoustic levitation devices and their ability to dexterously manipulate millimeter-scale objects make them appealing as a grasping mode for general purpose robots. In this work, we present an ultrasonic, contact-less manipulation device that can be attached to or picked up by any general purpose robotic arm, enabling millimeter-scale manipulation with little to no modification to the robot itself. This device is capable of performing the very first phase-controlled picking action on acoustically reflective surfaces. With the manipulator placed around the target object, the manipulator can grasp objects smaller in size than the robot's positioning uncertainty, trap the object to resist air currents during robot movement, and dexterously hold a small and fragile object, like a flower bud. Due to the contact-less nature of the ultrasound-based gripper, a camera positioned to look into the cylinder can inspect the object without occlusion, facilitating accurate visual feature extraction. 
    more » « less
  5. null (Ed.)
    Localizing and tracking the pose of robotic grippers are necessary skills for manipulation tasks. However, the manipulators with imprecise kinematic models (e.g. low-cost arms) or manipulators with unknown world coordinates (e.g. poor camera-arm calibration) cannot locate the gripper with respect to the world. In these circumstances, we can leverage tactile feedback between the gripper and the environment. In this paper, we present learnable Bayes filter models that can localize robotic grippers using tactile feedback. We propose a novel observation model that conditions the tactile feedback on visual maps of the environment along with a motion model to recursively estimate the gripper's location. Our models are trained in simulation with self-supervision and transferred to the real world. Our method is evaluated on a tabletop localization task in which the gripper interacts with objects. We report results in simulation and on a real robot, generalizing over different sizes, shapes, and configurations of the objects. 
    more » « less