skip to main content


Title: Learned Visual Navigation for Under-Canopy Agricultural Robots
This paper describes a system for visually guided autonomous navigation of under-canopy farm robots. Low-cost under-canopy robots can drive between crop rows under the plant canopy and accomplish tasks that are infeasible for over-the-canopy drones or larger agricultural equipment. However, autonomously navigating them under the canopy presents a number of challenges: unreliable GPS and LiDAR, high cost of sensing, challenging farm terrain, clutter due to leaves and weeds, and large variability in appearance over the season and across crop types. We address these challenges by building a modular system that leverages machine learning for robust and generalizable perception from monocular RGB images from low-cost cameras, and model predictive control for accurate control in challenging terrain. Our system, CropFollow, is able to autonomously drive 485 meters per intervention on average, outperforming a state-of-the-art LiDAR based system (286 meters per intervention) in extensive field testing spanning over 25 km.  more » « less
Award ID(s):
2007035
NSF-PAR ID:
10416327
Author(s) / Creator(s):
; ; ; ; ; ;
Date Published:
Journal Name:
Robotics: Science and Systems (RSS)
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Site description. This data package consists of data obtained from sampling surface soil (the 0-7.6 cm depth profile) in black mangrove (Avicennia germinans) dominated forest and black needlerush (Juncus roemerianus) saltmarsh along the Gulf of Mexico coastline in peninsular west-central Florida, USA. This location has a subtropical climate with mean daily temperatures ranging from 15.4 °C in January to 27.8 °C in August, and annual precipitation of 1336 mm. Precipitation falls as rain primarily between June and September. Tides are semi-diurnal, with 0.57 m median amplitudes during the year preceding sampling (U.S. NOAA National Ocean Service, Clearwater Beach, Florida, station 8726724). Sea-level rise is 4.0 ± 0.6 mm per year (1973-2020 trend, mean ± 95 % confidence interval, NOAA NOS Clearwater Beach station). The A. germinans mangrove zone is either adjacent to water or fringed on the seaward side by a narrow band of red mangrove (Rhizophora mangle). A near-monoculture of J. roemerianus is often adjacent to and immediately landward of the A. germinans zone. The transition from the mangrove to the J. roemerianus zone is variable in our study area. An abrupt edge between closed-canopy mangrove and J. roemerianus monoculture may extend for up to several hundred meters in some locations, while other stretches of ecotone present a gradual transition where smaller, widely spaced trees are interspersed into the herbaceous marsh. Juncus roemerianus then extends landward to a high marsh patchwork of succulent halophytes (including Salicornia bigellovi, Sesuvium sp., and Batis maritima), scattered dwarf mangrove, and salt pans, followed in turn by upland vegetation that includes Pinus sp. and Serenoa repens. Field design and sample collection. We established three study sites spaced at approximately 5 km intervals along the western coastline of the central Florida peninsula. The sites consisted of the Salt Springs (28.3298°, -82.7274°), Energy Marine Center (28.2903°, -82.7278°), and Green Key (28.2530°, -82.7496°) sites on the Gulf of Mexico coastline in Pasco County, Florida, USA. At each site, we established three plot pairs, each consisting of one saltmarsh plot and one mangrove plot. Plots were 50 m^2 in size. Plots pairs within a site were separated by 230-1070 m, and the mangrove and saltmarsh plots composing a pair were 70-170 m apart. All plot pairs consisted of directly adjacent patches of mangrove forest and J. roemerianus saltmarsh, with the mangrove forests exhibiting a closed canopy and a tree architecture (height 4-6 m, crown width 1.5-3 m). Mangrove plots were located at approximately the midpoint between the seaward edge (water-mangrove interface) and landward edge (mangrove-marsh interface) of the mangrove zone. Saltmarsh plots were located 20-25 m away from any mangrove trees and into the J. roemerianus zone (i.e., landward from the mangrove-marsh interface). Plot pairs were coarsely similar in geomorphic setting, as all were located on the Gulf of Mexico coastline, rather than within major sheltering formations like Tampa Bay, and all plot pairs fit the tide-dominated domain of the Woodroffe classification (Woodroffe, 2002, "Coasts: Form, Process and Evolution", Cambridge University Press), given their conspicuous semi-diurnal tides. There was nevertheless some geomorphic variation, as some plot pairs were directly open to the Gulf of Mexico while others sat behind keys and spits or along small tidal creeks. Our use of a plot-pair approach is intended to control for this geomorphic variation. Plot center elevations (cm above mean sea level, NAVD 88) were estimated by overlaying the plot locations determined with a global positioning system (Garmin GPS 60, Olathe, KS, USA) on a LiDAR-derived bare-earth digital elevation model (Dewberry, Inc., 2019). The digital elevation model had a vertical accuracy of ± 10 cm (95 % CI) and a horizontal accuracy of ± 116 cm (95 % CI). Soil samples were collected via coring at low tide in June 2011. From each plot, we collected a composite soil sample consisting of three discrete 5.1 cm diameter soil cores taken at equidistant points to 7.6 cm depth. Cores were taken by tapping a sleeve into the soil until its top was flush with the soil surface, sliding a hand under the core, and lifting it up. Cores were then capped and transferred on ice to our laboratory at the University of South Florida (Tampa, Florida, USA), where they were combined in plastic zipper bags, and homogenized by hand into plot-level composite samples on the day they were collected. A damp soil subsample was immediately taken from each composite sample to initiate 1 y incubations for determination of active C and N (see below). The remainder of each composite sample was then placed in a drying oven (60 °C) for 1 week with frequent mixing of the soil to prevent aggregation and liberate water. Organic wetland soils are sometimes dried at 70 °C, however high drying temperatures can volatilize non-water liquids and oxidize and decompose organic matter, so 50 °C is also a common drying temperature for organic soils (Gardner 1986, "Methods of Soil Analysis: Part 1", Soil Science Society of America); we accordingly chose 60 °C as a compromise between sufficient water removal and avoidance of non-water mass loss. Bulk density was determined as soil dry mass per core volume (adding back the dry mass equivalent of the damp subsample removed prior to drying). Dried subsamples were obtained for determination of soil organic matter (SOM), mineral texture composition, and extractable and total carbon (C) and nitrogen (N) within the following week. Sample analyses. A dried subsample was apportioned from each composite sample to determine SOM as mass loss on ignition at 550 °C for 4 h. After organic matter was removed from soil via ignition, mineral particle size composition was determined using a combination of wet sieving and density separation in 49 mM (3 %) sodium hexametaphosphate ((NaPO_3)_6) following procedures in Kettler et al. (2001, Soil Science Society of America Journal 65, 849-852). The percentage of dry soil mass composed of silt and clay particles (hereafter, fines) was calculated as the mass lost from dispersed mineral soil after sieving (0.053 mm mesh sieve). Fines could have been slightly underestimated if any clay particles were burned off during the preceding ignition of soil. An additional subsample was taken from each composite sample to determine extractable N and organic C concentrations via 0.5 M potassium sulfate (K_2SO_4) extractions. We combined soil and extractant (ratio of 1 g dry soil:5 mL extractant) in plastic bottles, reciprocally shook the slurry for 1 h at 120 rpm, and then gravity filtered it through Fisher G6 (1.6 μm pore size) glass fiber filters, followed by colorimetric detection of nitrite (NO_2^-) + nitrate (NO_3^-) and ammonium (NH_4^+) in the filtrate (Hood Nowotny et al., 2010,Soil Science Society of America Journal 74, 1018-1027) using a microplate spectrophotometer (Biotek Epoch, Winooski, VT, USA). Filtrate was also analyzed for dissolved organic C (referred to hereafter as extractable organic C) and total dissolved N via combustion and oxidation followed by detection of the evolved CO_2 and N oxide gases on a Formacs HT TOC/TN analyzer (Skalar, Breda, The Netherlands). Extractable organic N was then computed as total dissolved N in filtrate minus extractable mineral N (itself the sum of extractable NH_4-N and NO_2-N + NO_3-N). We determined soil total C and N from dried, milled subsamples subjected to elemental analysis (ECS 4010, Costech, Inc., Valencia, CA, USA) at the University of South Florida Stable Isotope Laboratory. Median concentration of inorganic C in unvegetated surface soil at our sites is 0.5 % of soil mass (Anderson, 2019, Univ. of South Florida M.S. thesis via methods in Wang et al., 2011, Environmental Monitoring and Assessment 174, 241-257). Inorganic C concentrations are likely even lower in our samples from under vegetation, where organic matter would dilute the contribution of inorganic C to soil mass. Nevertheless, the presence of a small inorganic C pool in our soils may be counted in the total C values we report. Extractable organic C is necessarily of organic C origin given the method (sparging with HCl) used in detection. Active C and N represent the fractions of organic C and N that are mineralizable by soil microorganisms under aerobic conditions in long-term soil incubations. To quantify active C and N, 60 g of field-moist soil were apportioned from each composite sample, placed in a filtration apparatus, and incubated in the dark at 25 °C and field capacity moisture for 365 d (as in Lewis et al., 2014, Ecosphere 5, art59). Moisture levels were maintained by frequently weighing incubated soil and wetting them up to target mass. Daily CO_2 flux was quantified on 29 occasions at 0.5-3 week intervals during the incubation period (with shorter intervals earlier in the incubation), and these per day flux rates were integrated over the 365 d period to compute an estimate of active C. Observations of per day flux were made by sealing samples overnight in airtight chambers fitted with septa and quantifying headspace CO_2 accumulation by injecting headspace samples (obtained through the septa via needle and syringe) into an infrared gas analyzer (PP Systems EGM 4, Amesbury, MA, USA). To estimate active N, each incubated sample was leached with a C and N free, 35 psu solution containing micronutrients (Nadelhoffer, 1990, Soil Science Society of America Journal 54, 411-415) on 19 occasions at increasing 1-6 week intervals during the 365 d incubation, and then extracted in 0.5 M K_2SO_4 at the end of the incubation in order to remove any residual mineral N. Active N was then quantified as the total mass of mineral N leached and extracted. Mineral N in leached and extracted solutions was detected as NH_4-N and NO_2-N + NO_3-N via colorimetry as above. This incubation technique precludes new C and N inputs and persistently leaches mineral N, forcing microorganisms to meet demand by mineralizing existing pools, and thereby directly assays the potential activity of soil organic C and N pools present at the time of soil sampling. Because this analysis commences with disrupting soil physical structure, it is biased toward higher estimates of active fractions. Calculations. Non-mobile C and N fractions were computed as total C and N concentrations minus the extractable and active fractions of each element. This data package reports surface-soil constituents (moisture, fines, SOM, and C and N pools and fractions) in both gravimetric units (mass constituent / mass soil) and areal units (mass constituent / soil surface area integrated through 7.6 cm soil depth, the depth of sampling). Areal concentrations were computed as X × D × 7.6, where X is the gravimetric concentration of a soil constituent, D is soil bulk density (g dry soil / cm^3), and 7.6 is the sampling depth in cm. 
    more » « less
  2. The University of Alaska Fairbanks T Field is a legacy farm field that is part of the National Science Foundation (NSF) Funded Permafrost Grown project. We are studying the long-term effects of permafrost thaw following initial clearing for cultivation purposes. In this regard, we have acquired very high resolution light detection and ranging (LiDAR) data and digital photography from a DJI M300 drone using a Zenmuse L1 and a MicaSense RedEdge-P camera. The Zenmuse L1 integrates a Livox Lidar module, a high-accuracy inertial measurement units (IMU), and a camera with a 1-inch CMOS on a 3-axis stabilized gimbal. The MicaSense RedEdge-P camera has five multispectral bands and a high-resolution panchromatic band. The drone was configured to fly in real-time kinematic (RTK) mode at an altitude of 60 meters above ground level using the DJI D-RTK 2 base station. Data was acquired using a 50% sidelap and a 70% frontlap for the Zenmuse L1 and an 80% sidelap and a 75% frontlap for the MicaSense. Additional ground control was established with a Leica GS18 global navigation satellite system (GNSS) and all data have been post-processed to World Geodetic System 1984 (WGS84) universal transverse mercator (UTM) Zone 6 North using ellipsoid heights. Data outputs include a two-class-classified LiDAR point cloud, digital surface model, digital terrain model, an orthophoto mosaic, and a multispectral orthoimage consisting of five bands. Image acquisition occurred on 18 August 2023. 
    more » « less
  3. The Midnight Sun Golf Course in Fairbanks, Alaska is a legacy farm field that is part of the National Science Foundation (NSF) Funded Permafrost Grown project. This 65 hectare (ha) parcel was initially cleared for agriculture purposes but changed land-use practices to a golf course around 25 years ago. The land-use conversion was in part due to ice-rich permafrost thaw following clearing. We are studying the long-term effects of permafrost thaw following initial clearing for cultivation purposes. We are working with the current landowners to provide information regarding ongoing thermokarst development on the property and to conduct studies in reforested portions of the land area to understand land clearing and reforestation on permafrost-affected soils. In this regard, we have acquired very high resolution light detection and ranging (LiDAR) data and digital photography from a DJI M300 drone using a Zenmuse L1. The Zenmuse L1 integrates a Livox Lidar module, a high-accuracy inertial measurement units (IMU), and a camera with a 1-inch CMOS on a 3-axis stabilized gimbal. The drone was configured to fly in real-time kinematic (RTK) mode at an altitude of 60 meters above ground level using the DJI D-RTK 2 base station. Data was acquired using a 50% sidelap and a 70% frontlap. Additional ground control was established with a Leica GS18 global navigation satellite system (GNSS) and all data have been post-processed to World Geodetic System 1984 (WGS84) universal transverse mercator (UTM) Zone 6 North using ellipsoid heights. Data outputs include a two-class classified LiDAR point cloud, digital surface model, digital terrain model, and an orthophoto mosaic. Image acquisition occurred on 10 September 2023. The input images are available for download at http://arcticdata.io/data/10.18739/A2PC2TB1T. 
    more » « less
  4. null (Ed.)
    Swarms of ground-based robots are presently limited to relatively simple environments, which we attribute in part to the lack of locomotor capabilities needed to traverse complex terrain. To advance the field of terradynamically capable swarming robotics, inspired by the capabilities of multilegged organisms, we hypothesize that legged robots consisting of reversibly chainable modular units with appropriate passive perturbation management mechanisms can perform diverse tasks in variable terrain without complex control and sensing. Here, we report a reconfigurable swarm of identical low-cost quadruped robots (with directionally flexible legs and tail) that can be linked on demand and autonomously. When tasks become terradynamically challenging for individuals to perform alone, the individuals suffer performance degradation. A systematic study of performance of linked units leads to new discoveries of the emergent obstacle navigation capabilities of multilegged robots. We also demonstrate the swarm capabilities through multirobot object transport. In summary, we argue that improvement capabilities of terrestrial swarms of robots can be achieved via the judicious interaction of relatively simple units. 
    more » « less
  5. Abstract

    To feed the world population while mitigating pressing nitrogen (N) pollution problems, tremendous efforts have been devoted to developing and implementing N‐efficient technologies in crop or livestock production, but limited progress has been made. The N management improvement on a farm does not necessarily translate to N pollution reduction on a broader scale due to complex responses of natural and human systems and lack of coordination among stakeholders. Consequently, it is imperative to develop an N management framework that encompasses the complex N dynamics across systems and spatial scales, yet simple enough to guide policies and actions of various stakeholders. Here, we propose a new framework,CAFE, that defines four N management systems (Cropping,Animal‐crop,Food, andEcosystem) in a hierarchical manner, and apply it to 13 representative countries to partition N surpluses across systems in a simple and consistent manner, thereby facilitating the identification and prioritization of systems‐based intervention strategies. Surprisingly, theCropping system contributes less than half of the total N surplus within itsEcosystem for most countries, highlighting the importance of N management beyond croplands. This framework reveals that the relevant priorities and key stakeholders for enhanced N management vary among countries, such as improving theCropping‐system efficiencies in China, adjusting the animal‐crop portfolio in the Netherlands, reducing food wastage in the U.S., and lowering crop storage losses and increasing overall production capacities in African countries. As N surplus increases along theCAFEhierarchy, systems‐based intervention strategies are revealed: (a) coupling chemical fertilizers with other N sources by maintaining half of the N from manure and biological N fixation; (b) coupling animal‐crop production by reducing animal density to lower than 1.2 livestock units per hectare, and increasing self‐sufficiency of animal feed to above 50%; (c) coupling food trade with domestic demand and production; and (d) coupling population needs for economic opportunities with environmental capacity of the region. This novel framework can help unpack the “wicked” N management challenges across systems to provide new insights and tools for improving N management on and beyond farms.

     
    more » « less