An official website of the United States government
The ability to automatize the analysis of video for monitoring animals and insects is of great interest for behavior science and ecology. In particular, honeybees play a crucial role in agriculture as natural pollinators. However, recent studies has shown that phenomena such as colony collapse disorder are causing the loss of many colonies. Due to the high number of interacting factors to explain these events, a multi-faceted analysis of the bees in their environment is required. We focus in our work in developing tools to help model and understand their behavior as individuals, in relation with the health and performance of the colony. In this paper, we report the development of a new system for the detection, localization and tracking of honeybee body parts from video on the entrance ramp of the colony. The proposed system builds on the recent advances in Convolutional Neural Networks (CNN) for Human pose estimation and evaluates the suitability for the detection of honeybee pose as shown in Figure 1. This opens the door for novel animal behavior analysis systems that take advantage of the precise detection and tracking of the insect pose.
more »
« less
SLEAP: A deep learning system for multi-animal pose tracking
Abstract The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation in tools to quantify natural animal behavior. While advances in deep learning and computer vision have enabled markerless pose estimation in individual animals, extending these to multiple animals presents unique challenges for studies of social behaviors or animals in their natural environments. Here we present Social LEAP Estimates Animal Poses (SLEAP), a machine learning system for multi-animal pose tracking. This system enables versatile workflows for data labeling, model training and inference on previously unseen data. SLEAP features an accessible graphical user interface, a standardized data model, a reproducible configuration system, over 30 model architectures, two approaches to part grouping and two approaches to identity tracking. We applied SLEAP to seven datasets across flies, bees, mice and gerbils to systematically evaluate each approach and architecture, and we compare it with other existing approaches. SLEAP achieves greater accuracy and speeds of more than 800 frames per second, with latencies of less than 3.5 ms at full 1,024 × 1,024 image resolution. This makes SLEAP usable for real-time applications, which we demonstrate by controlling the behavior of one animal on the basis of the tracking and detection of social interactions with another animal.
more »
« less
- PAR ID:
- 10366517
- Author(s) / Creator(s):
- ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; more »
- Publisher / Repository:
- Nature Publishing Group
- Date Published:
- Journal Name:
- Nature Methods
- Volume:
- 19
- Issue:
- 4
- ISSN:
- 1548-7091
- Page Range / eLocation ID:
- p. 486-495
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Quantitative descriptions of animal behavior are essential to study the neural substrates of cognitive and emotional processes. Analyses of naturalistic behaviors are often performed by hand or with expensive, inflexible commercial software. Recently, machine learning methods for markerless pose estimation enabled automated tracking of freely moving animals, including in labs with limited coding expertise. However, classifying specific behaviors based on pose data requires additional computational analyses and remains a significant challenge for many groups. We developed BehaviorDEPOT (DEcoding behavior based on POsitional Tracking), a simple, flexible software program that can detect behavior from video timeseries and can analyze the results of experimental assays. BehaviorDEPOT calculates kinematic and postural statistics from keypoint tracking data and creates heuristics that reliably detect behaviors. It requires no programming experience and is applicable to a wide range of behaviors and experimental designs. We provide several hard-coded heuristics. Our freezing detection heuristic achieves above 90% accuracy in videos of mice and rats, including those wearing tethered head-mounts. BehaviorDEPOT also helps researchers develop their own heuristics and incorporate them into the software’s graphical interface. Behavioral data is stored framewise for easy alignment with neural data. We demonstrate the immediate utility and flexibility of BehaviorDEPOT using popular assays including fear conditioning, decision-making in a T-maze, open field, elevated plus maze, and novel object exploration.more » « less
-
Global Positioning Systems (GPSs) can collect tracking data to remotely monitor livestock well-being and pasture use. Supervised machine learning requires behavioral observations of monitored animals to identify changes in behavior, which is labor-intensive. Our goal was to identify animal behaviors automatically without using human observations. We designed a novel framework using unsupervised learning techniques. The framework contains two steps. The first step segments cattle tracking data using state-of-the-art time series segmentation algorithms, and the second step groups segments into clusters and then labels the clusters. To evaluate the applicability of our proposed framework, we utilized GPS tracking data collected from five cows in a 1096 ha rangeland pasture. Cow movement pathways were grouped into six behavior clusters based on velocity (m/min) and distance from water. Again, using velocity, these six clusters were classified into walking, grazing, and resting behaviors. The mean velocity for predicted walking and grazing and resting behavior was 44, 13 and 2 min/min, respectively, which is similar to other research. Predicted diurnal behavior patterns showed two primary grazing bouts during early morning and evening, like in other studies. Our study demonstrates that the proposed two-step framework can use unlabeled GPS tracking data to predict cattle behavior without human observations.more » « less
-
Abstract Significant advances in computational ethology have allowed the quantification of behaviour in unprecedented detail. Tracking animals in social groups, however, remains challenging as most existing methods can either capture pose or robustly retain individual identity over time but not both.To capture finely resolved behaviours while maintaining individual identity, we built NAPS (NAPS is ArUco Plus SLEAP), a hybrid tracking framework that combines state‐of‐the‐art, deep learning‐based methods for pose estimation (SLEAP) with unique markers for identity persistence (ArUco). We show that this framework allows the exploration of the social dynamics of the common eastern bumblebee (Bombus impatiens).We provide a stand‐alone Python package for implementing this framework along with detailed documentation to allow for easy utilization and expansion. We show that NAPS can scale to long timescale experiments at a high frame rate and that it enables the investigation of detailed behavioural variation within individuals in a group.Expanding the toolkit for capturing the constituent behaviours of social groups is essential for understanding the structure and dynamics of social networks. NAPS provides a key tool for capturing these behaviours and can provide critical data for understanding how individual variation influences collective dynamics.more » « less
-
Zebrafish is a widely used animal model in behavioral neuroscience. However, zebrafish learning capabilities are not completely understood. Technological advancements in robotics promise fine behavioral control of artificial conspecifics to study complex aspects of social behavior. In this work, we developed a training system aimed at investigating individual and social learning of zebrafish. The system consists of a shallow water tank, a 2- dimensional robotic platform, and a real-time tracking software. In the tank, a focal individual is separated from a shoal of conspecifics by a one-way glass and a transparent partition, allowing the focal fish to see the shoal. In the transparent partition are two doors, one that automatically opens when the focal individual spends a predetermined amount of time in front of it and another that remains closed regardless of the fish behavior. We tested the system by training one na¨ıve fish in individual learning and one fish in social learning over 20 sessions. Test results show that the fish can learn to open the door and also validated the effectiveness of the developed system applying on individual and social learning.more » « less
