An official website of the United States government
The ability to accurately interpret complex visual information is a crucial topic of multimodal large language models (MLLMs). Recent work indicates that enhanced visual perception significantly reduces hallucinations and improves performance on resolution-sensitive tasks, such as optical character recognition and document analysis. A number of recent MLLMs achieve this goal using a mixture of vision encoders. Despite their success, there is a lack of systematic comparisons and detailed ablation studies addressing critical aspects, such as expert selection and the integration of multiple vision experts. This study provides an extensive exploration of the design space for MLLMs using a mixture of vision encoders and resolutions. Our findings reveal several underlying principles common to various existing strategies, leading to a streamlined yet effective design approach. We discover that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies. We additionally introduce Pre-Alignment to bridge the gap between vision-focused encoders and language tokens, enhancing model coherence. The resulting family of MLLMs, Eagle, surpasses other leading open-source models on major MLLM benchmarks.
more »
« less
Can Multimodal Large Language Models Be Guided to Improve Industrial Anomaly Detection?
Abstract Industrial environments demand accurate detection of anomalies to maintain product quality and ensure operational safety. Traditional industrial anomaly detection (IAD) methods often lack the flexibility and adaptability needed in dynamic production settings, where new defect types and operational changes continually emerge. Recent advancements in multimodal large language models (MLLMs) have shown promise by combining visual and textual processing capabilities, yet they are often limited by their lack of domain-specific expertise, particularly regarding industry-standard defect tolerances. To overcome limitations, we introduce Echo, a novel multi-expert framework designed to enhance MLLM performance for IAD. Echo integrates four specialized modules: the Reference Extractor retrieves similar normal images to establish contextual baselines; the Knowledge Guide provides critical, industry-specific insights; the Reasoning Expert enables structured, stepwise analysis for complex queries; and the Decision Maker synthesizes information from the preceding modules to deliver precise, context-aware responses. Evaluations on the MMAD benchmark reveal that Echo significantly improves adaptability, precision, and robustness compared to conventional approaches. Our results demonstrate that guided MLLMs, when augmented with expert modules, can effectively bridge the gap between general visual understanding and the specialized requirements of industrial anomaly detection, paving the way for more reliable and interpretable inspection systems.
more »
« less
- PAR ID:
- 10650988
- Publisher / Repository:
- American Society of Mechanical Engineers
- Date Published:
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract With the continuous modernization of water plants, the risk of cyberattacks on them potentially endangers public health and the economic efficiency of water treatment and distribution. This article signifies the importance of developing improved techniques to support cyber risk management for critical water infrastructure, given an evolving threat environment. In particular, we propose a method that uniquely combines machine learning, the theory of belief functions, operational performance metrics, and dynamic visualization to provide the required granularity for attack inference, localization, and impact estimation. We illustrate how the focus on visual domain‐aware anomaly exploration leads to performance improvement, more precise anomaly localization, and effective risk prioritization. Proposed elements of the method can be used independently, supporting the exploration of various anomaly detection methods. It thus can facilitate the effective management of operational risk by providing rich context information and bridging the interpretation gap.more » « less
-
Abstract This review paper examines the application and challenges of machine learning (ML) in intelligent welding processes within the automotive industry, focusing on resistance spot welding (RSW) and laser welding. RSW is predominant in body-in-white assembly, while laser welding is critical for electric vehicle battery packs due to its precision and compatibility with dissimilar materials. The paper categorizes ML applications into three key areas: sensing, in-process decision-making, and post-process optimization. It reviews supervised learning models for defect detection and weld quality prediction, unsupervised learning for feature extraction and data clustering, and emerging generalizable ML approaches like transfer learning and federated learning that enhance adaptability across different manufacturing conditions. Additionally, the paper highlights the limitations of current ML models, particularly regarding generalizability when moving from lab environments to real-world production, and discusses the importance of adaptive learning techniques to address dynamically changing conditions. Case studies like virtual sensing, defect detection in RSW, and optimization in laser welding illustrate practical applications. The paper concludes by identifying future research directions to improve ML adaptability and robustness in high-variability manufacturing environments, aiming to bridge the gap between experimental ML models and real-world implementation in automotive welding.more » « less
-
Iris presentation attack detection (PAD) is critical for secure biometric deployments, yet developing specialized models faces significant practical barriers: collecting data representing future unknown attacks is impossible, and collecting diverse-enough data, yet still limited in terms of its predictive power, is expensive. Additionally, sharing biometric data raises privacy concerns. Due to rapid emergence of new attack vectors demanding adaptable solutions, we thus investigate in this paper whether general-purpose multimodal large language models (MLLMs) can perform iris PAD when augmented with human expert knowledge, operating under strict privacy constraints that prohibit sending biometric data to public cloud MLLM services. Through analysis of vision encoder embeddings applied to our dataset, we demonstrate that pre-trained vision transformers in MLLMs inherently cluster many iris attack types despite never being explicitly trained for this task. However, where clustering shows overlap between attack classes, we find that structured prompts incorporating human salience (verbal descriptions from subjects identifying attack indicators) enable these models to resolve ambiguities. Testing on an IRB-restricted dataset of 224 iris images spanning seven attack types, using only university-approved services (Gemini 2.5 Pro) or locally-hosted models (e.g., Llama 3.2-Vision), we show that Gemini with expert-informed prompts outperforms both a specialized convolutional neural networks (CNN)-based baseline and human examiners, while the locally-deployable Llama achieves near-human performance. Our results establish that MLLMs deployable within institutional privacy constraints offer a viable path for iris PAD.more » « less
-
The Internet of Things (IoT) has revolutionized built environments by enabling seamless data exchange among devices such as sensors, actuators, and computers. However, IoT devices often lack robust security mechanisms, making them vulnerable to cyberattacks, privacy breaches, and operational anomalies caused by environmental factors or device faults. While anomaly detection techniques are critical for securing IoT systems, the role of testbeds in evaluating these techniques has been largely overlooked. This systematic review addresses this gap by treating testbeds as first-class entities essential for the standardized evaluation and validation of anomaly detection methods in built environments. We analyze testbed characteristics, including infrastructure configurations, device selection, user-interaction models, and methods for anomaly generation. We also examine evaluation frameworks, highlighting key metrics and integrating emerging technologies such as edge computing and 5G networks into testbed design. By providing a structured and comprehensive approach to testbed development and evaluation, this paper offers valuable guidance to researchers and practitioners in enhancing the reliability and effectiveness of anomaly detection systems. Our findings contribute to the development of more secure, adaptable, and scalable IoT systems, ultimately improving the security, resilience, and efficiency of built environments.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Conference Paper:
- https://doi.org/10.1115/DETC2025-168875
