More Like this

1. PLM-Net: Perception Latency Mitigation Network for Vision-Based Lateral Control of Autonomous Vehicles

   https://doi.org/10.3390/s26061798  

   Khalil, Aws; Kwon, Jaerock (March 2026, Sensors)

   This study introduces the Perception Latency Mitigation Network (PLM-Net), a modular deep learning framework designed to mitigate perception latency in vision-based imitation-learning lane-keeping systems. Perception latency, defined as the delay between visual sensing and steering actuation, can degrade lateral tracking performance and steering stability. While delay compensation has been extensively studied in classical predictive control systems, its treatment within vision-based imitation-learning architectures under constant and time-varying perception latency remains limited. Rather than reducing latency itself, PLM-Net mitigates its effect on control performance through a plug-in architecture that preserves the original control pipeline. The framework consists of a frozen Base Model (BM), representing an existing lane-keeping controller, and a Timed Action Prediction Model (TAPM), which predicts future steering actions corresponding to discrete latency conditions. Real-time mitigation is achieved by interpolating between model outputs according to the measured latency value, enabling adaptation to both constant and time-varying latency. The framework is evaluated in a closed-loop deterministic simulation environment under fixed-speed conditions to isolate the impact of perception latency. Results demonstrate significant reductions in steering error under multiple latency settings, achieving up to 62% and 78% reductions in Mean Absolute Error (MAE) for constant and time-varying latency cases, respectively. These findings demonstrate the architectural feasibility of modular latency mitigation for vision-based lateral control under controlled simulation settings. The project page including video demonstrations, code, and dataset is publicly released. 

   more » « less
2. Efficient Kernels for Real-Time Position Decoding from In Vivo Calcium Images

   Zhe Chen, Jim Zhou (January 2022, The 2022 IEEE Symposium on Circuits and Systems (ISCAS))

   Recent studies have found that the position of mice or rats can be decoded from calcium imaging of brain activity offline. However, given the complex analysis pipeline, real-time position decoding remains a challenging task, especially considering strict requirements on hardware usage and energy cost for closed-loop feedback applications. In this paper, we propose two neural network based methods and corresponding hardware designs for real-time position decoding from calcium images. Our methods are based on: 1) convolutional neural network (CNN), 2) spiking neural network (SNN) converted from the CNN. We implemented quantized CNN and SNN models on FPGA. Evaluation results show that the CNN and the SNN methods achieve 56.3%/83.1% and 56.0%/82.8% Hit-1/Hit-3 accuracy for the position decoding across different rats, respectively. We also observed an accuracy-latency tradeoff of the SNN method in decoding positions under various time steps. Finally, we present our SNN implementation on the neuromorphic chip Loihi. Index Terms—calcium image, decoding, neural network. 

   more » « less
3. Cloud Is Closer Than It Appears: Revisiting the Tradeoffs of Distributed Real-Time Inference

   Sharma, Pragya; Qiu, Hang; Srivastava, Mani (August 2025, IEEE)

   The increasing deployment of deep neural networks (DNNs) in cyber-physical systems (CPS) enhances perception fidelity, but imposes substantial computational demands on execution platforms, posing challenges to real-time control deadlines. Traditional distributed CPS architectures typically favor on-device inference to avoid network variability and contention-induced delays on remote platforms. However, this design choice places significant energy and computational demands on the local hardware. In this work, we revisit the assumption that cloud-based inference is intrinsically unsuitable for latency-sensitive control tasks. We demonstrate that, when provisioned with high-throughput compute resources, cloud platforms can effectively amortize network and queueing delays, enabling them to match or surpass on-device performance for real-time decision-making. Specifically, we develop a formal analytical model that characterizes distributed inference latency as a function of the sensing frequency, platform throughput, network delay, and task-specific safety constraints. We instantiate this model in the context of emergency braking for autonomous driving and validate it through extensive simulations using real-time vehicular dynamics. Our empirical results identify concrete conditions under which cloud-based inference adheres to safety margins more reliably than its on-device counterpart. These findings challenge prevailing design strategies and suggest that the cloud is not merely a feasible option, but often the preferred inference location for distributed CPS architectures. In this light, the cloud is not as distant as traditionally perceived; in fact, it is closer than it appears. 

   more » « less
4. Multi-Modal Vision at the Edge: Toward Low-Latency Perception for Autonomous Systems

   https://doi.org/10.1109/COINS65080.2025.11125752  

   Lamb, Lily; Mohammadi, Mohammadreza; Zand, Ramtin (August 2025, IEEE)

   Multi-modal perception is critical for robust and reliable object detection in autonomous systems, particularly under challenging environmental conditions. This paper presents an extended YOLOv8 framework that integrates RGB, LiDAR-based depth, and grayscale modalities using early, mid, and late fusion strategies, specifically tailored for deployment on edge AI platforms. To enable real-time processing, we propose a parallelized depth generation pipeline that efficiently converts sparse LiDAR point clouds into dense depth maps. We evaluate the proposed models on the KITTI dataset and benchmark their accuracy and latency across three edge devices: Raspberry Pi 5, NVIDIA Jetson Nano, and Google Coral Edge TPU. Experimental results demonstrate that while late fusion with all three modalities yields the highest detection accuracy (mAP@50: 92.7%, mAP@50-95: 70.6%), early fusion offers the lowest inference latency. Our findings underscore the trade-off between accuracy and latency, highlighting the need to choose a fusion strategy that aligns with the application requirements. 

   more » « less
5. Edge-Enabled Collaborative Object Detection for Real-Time Multi-Vehicle Perception

   https://doi.org/10.1109/EDGE67623.2025.00011  

   Richards, Everett; Thapa, Bipul; Mashayekhy, Lena (August 2025, IEEE)

   Accurate and reliable object detection is critical for ensuring the safety and efficiency of Connected Autonomous Vehicles (CAVs). Traditional on-board perception systems have limited accuracy due to occlusions and blind spots, while cloud-based solutions introduce significant latency, making them unsuitable for real-time processing demands required for autonomous driving in dynamic environments. To address these challenges, we introduce an innovative framework, Edge-Enabled Collaborative Object Detection (ECOD) for CAVs, that leverages edge computing and multi-CAV collaboration for real-time, multi-perspective object detection. Our ECOD framework integrates two key algorithms: Perceptive Aggregation and Collaborative Estimation (PACE) and Variable Object Tally and Evaluation (VOTE). PACE aggregates detection data from multiple CAVs on an edge server to enhance perception in scenarios where individual CAVs have limited visibility. VOTE utilizes a consensus-based voting mechanism to improve the accuracy of object classification by integrating data from multiple CAVs. Both algorithms are designed at the edge to operate in real-time, ensuring low-latency and reliable decision-making for CAVs. We develop a hardware-based controlled testbed consisting of camera-equipped robotic CAVs and an edge server to evaluate the efficacy of our framework. Our experimental results demonstrate the significant benefits of ECOD in terms of improved object classification accuracy, outperforming traditional single-perspective onboard approaches by up to 75%, while ensuring low-latency, edge-driven real-time processing. This research highlights the potential of edge computing to enhance collaborative perception for latency-sensitive autonomous systems. 

   more » « less