Search for: All records

The ubiquity of surveillance networks in public infrastructure presents a significant, yet underutilized, opportunity to assist vulnerable populations, particularly individuals who are blind or have low vision (BLV). However, transforming these passive video feeds into active guidance systems requires accurate camera calibration, a process that is traditionally labor-intensive and unscalable in large facilities. This paper introduces a novel, automated framework that leverages a mobile quadruped robot (Boston Dynamics Spot) as a dynamic calibration agent. We propose a single-pass approach with two planar calibration algorithms, which mines unstructured robotic trajectories to construct distinct geometric features on the ground plane. By synthesizing “virtual rectangles” from the robot’s odometry, our method first recovers camera focal length through vanishing point estimation by constructing virtual rectangles, and then solves for extrinsic 6-DoF pose using two algorithms: our proposed Virtual Rectangle (ViR) algorithm and a standard planar Perspective-n-Point (PnP) algorithm. Experimental validation using real-world data demonstrates the system’s robustness to sensor noise, maintaining focal length errors around 5%, rotational errors around 5° and relative translation errors of approximately 10%, while synthetic simulations indicate focal length errors generally under 10%, rotational errors under 1.5°, and translation errors also around 10% despite heavy image and object disturbances. This work eliminates the need for manual calibration targets for dynamic camera calibration, effectively converting static security infrastructure into a metric sensing network capable of supporting high-fidelity social robotics applications. 

Over 2 [...] 

Standard training for Multi-modal Large Language Models (MLLMs) involves concatenating non-textual information, like vision or audio, with a text prompt. This approach may not encourage deep integration of modalities, limiting the model's ability to leverage the core language model's reasoning capabilities. This work examined the impact of interleaved instruction tuning in an audio MLLM, where audio tokens are interleaved within the prompt. Using the Listen, Think, and Understand (LTU) model as a testbed, we conduct an experiment using the Synonym and Hypernym Audio Reasoning Dataset (SHARD), our newly created reasoning benchmark for audio-based semantic reasoning focusing on synonym and hypernym recognition. Our findings show that while even zero-shot interleaved prompting improves performance on our reasoning tasks, a small amount of fine-tuning using interleaved training prompts improves the results further, however, at the expense of the MLLM's audio labeling ability.