An official website of the United States government
Existing building recognition methods, exemplified by BRAILS, utilize supervised learning to extract information from satellite and street-view images for classification and segmentation. However, each task module requires human-annotated data, hindering the scalability and robustness to regional variations and annotation imbalances. In response, we propose a new zero-shot workflow for building attribute extraction that utilizes large-scale vision and language models to mitigate reliance on external annotations. The proposed workflow contains two key components: image-level captioning and segment-level captioning for the building images based on the vocabularies pertinent to structural and civil engineering. These two components generate descriptive captions by computing feature representations of the image and the vocabularies, and facilitating a semantic match between the visual and textual representations. Consequently, our framework offers a promising avenue to enhance AI-driven captioning for building attribute extraction in the structural and civil engineering domains, ultimately reducing reliance on human annotations while bolstering performance and adaptability.
more »
« less
This content will become publicly available on June 19, 2026
Adaptive Segmentation of Thermal Anomalies in Buildings Using Human-in-the-Loop Active Learning Capsule Networks
Buildings account for 40% of energy consumption in the U.S., with significant energy losses stemming from poor insulation and leaks. To meet decarbonization goals by 2050, efficient methods for diagnosing and retrofitting thermal anomalies are essential. Infrared thermography (IRT), combined with emerging technologies like computer vision and deep learning, offers the potential for automating thermal anomaly detection and segmentation. However, challenges such as building diversity, scenario variations, and labor-intensive image annotation hinder model reliability and robustness. This study proposes a human-in-the-loop active learning approach to fine-tune the pretrained Capsule-based network (CapsLab) to enhance adaptability to new building scenarios efficiently. Using a Query-by-Committee (QBC) strategy, the method selects the most informative thermal images from a target dataset for expert verification and iterative model refinement. To streamline the expert annotation verification process, a weakly annotation strategy is introduced supporting human-in-the-loop training by Simple Linear Iterative Clustering (SLIC)-based superpixel segmentation and scribble prompts for efficient labeling. The pretrained CapsLab model is iteratively fine-tuned using these verified annotations to enhance segmentation performance. Three fine-tuning methods, including parameter freezing strategies, are evaluated for optimal results. This iterative workflow reduces annotation effort, improves model adaptability, and improves the precision of thermal anomaly segmentation, facilitating energy-efficient building retrofitting and maintenance.
more »
« less
- Award ID(s):
- 2431468
- PAR ID:
- 10651775
- Publisher / Repository:
- Purdue e-Pubs
- Date Published:
- Journal Name:
- CIB Conferences
- Volume:
- 1
- Issue:
- 1
- ISSN:
- 3067-4883
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Training deep learning models for unbiased stereology requires a large data set with associated ground truth. However manual ground truth annotation is tedious, time-consuming, and expert dependent. We propose an active deep learning method for automatic stereology counts using a snapshot ensemble approach. The method provides a confidence score for each mask in an unlabeled pool that reduces user verification to only images with high information content for training the deep learning model. The proposed method reduces the error rate to less than 1% for unbiased stereology cell counts on immunostained brain cells compared to manual stereology and requires ~25% less expert verification time compared to a previously proposed iterative deep learning approach.more » « less
-
Expert-layman text style transfer technologies have the potential to improve communication between members of scientific communities and the general public. High-quality information produced by experts is often filled with difficult jargon laypeople struggle to understand. This is a particularly notable issue in the medical domain, where layman are often confused by medical text online. At present, two bottlenecks interfere with the goal of building high-quality medical expert-layman style transfer systems: a dearth of pretrained medical-domain language models spanning both expert and layman terminologies and a lack of parallel corpora for training the transfer task itself. To mitigate the first issue, we propose a novel language model (LM) pretraining task, Knowledge Base Assimilation, to synthesize pretraining data from the edges of a graph of expert- and layman-style medical terminology terms into an LM during self-supervised learning. To mitigate the second issue, we build a large-scale parallel corpus in the medical expert-layman domain using a margin-based criterion. Our experiments show that transformer-based models pretrained on knowledge base assimilation and other well-established pretraining tasks fine-tuning on our new parallel corpus leads to considerable improvement against expert-layman transfer benchmarks, gaining an average relative improvement of our human evaluation, the Overall Success Rate (OSR), by 106%.more » « less
-
In this work, we present recEnergy, a recommender system for reducing energy consumption in commercial buildings with human-in-the-loop. We formulate the building energy optimization problem as a Markov Decision Process, show how deep reinforcement learning can be used to learn energy saving recommendations, and effectively engage occupants in energy-saving actions. is a recommender system that learns actions with high energy saving potential, actively distribute recommendations to occupants in a commercial building, and utilize feedback from the occupants to learn better energy saving recommendations. Over a four week user study, four different types of energy saving recommendations were trained and learned. improves building energy reduction from a baseline saving (passive-only strategy) of 19% to 26%.more » « less
-
Lack of enough labeled data is a major problem in building machine learning based models when the manual annotation (labeling) is error-prone, expensive, tedious, and time-consuming. In this paper, we introduce an iterative deep learning based method to improve segmentation and counting of cells based on unbiased stereology applied to regions of interest of extended depth of field (EDF) images. This method uses an existing machine learning algorithm called the adaptive segmentation algorithm (ASA) to generate masks (verified by a user) for EDF images to train deep learning models. Then an iterative deep learning approach is used to feed newly predicted and accepted deep learning masks/images (verified by a user) to the training set of the deep learning model. The error rate in unbiased stereology count of cells on an unseen test set reduced from about 3 % to less than 1 % after 5 iterations of the iterative deep learning based unbiased stereology process.more » « less
