More Like this

1. Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification

   Yang, Yue and (June 2023, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR))

   Concept Bottleneck Models (CBM) are inherently interpretable models that factor model decisions into human-readable concepts. They allow people to easily understand why a model is failing, a critical feature for high-stakes applications. CBMs require manually specified concepts and often under-perform their black box counterparts, preventing their broad adoption. We address these shortcomings and are first to show how to construct high-performance CBMs without manual specification of similar accuracy to black box models. Our approach, Language Guided Bottlenecks (LaBo), leverages a language model, GPT-3, to define a large space of possible bottlenecks. Given a problem domain, LaBo uses GPT-3 to produce factual sentences about categories to form candidate concepts. LaBo efficiently searches possible bottlenecks through a novel submodular utility that promotes the selection of discriminative and diverse information. Ultimately, GPT-3's sentential concepts can be aligned to images using CLIP, to form a bottleneck layer. Experiments demonstrate that LaBo is a highly effective prior for concepts important to visual recognition. In the evaluation with 11 diverse datasets, LaBo bottlenecks excel at few-shot classification: they are 11.7% more accurate than black box linear probes at 1 shot and comparable with more data. Overall, LaBo demonstrates that inherently interpretable models can be widely applied at similar, or better, performance than black box approaches. 

   more » « less
2. Explainable Deep Learning for Insights in El Nino and River Flows

   https://doi.org/10.48550/arXiv.2201.02596  

   Liu, Yumin; Duffy, Kate; Dy, Jennifer G.; Ganguly, Auroop R. (January 2022, ArXivorg)

   The El Nino Southern Oscillation (ENSO) is a semi-periodic fluctuation in sea surface temperature (SST) over the tropical central and eastern Pacific Ocean that influences interannual variability in regional hydrology across the world through long-range dependence or teleconnections. Recent research has demonstrated the value of Deep Learning (DL) methods for improving ENSO prediction as well as Complex Networks (CN) for understanding teleconnections. However, gaps in predictive understanding of ENSO-driven river flows include the black box nature of DL, the use of simple ENSO indices to describe a complex phenomenon and translating DL-based ENSO predictions to river flow predictions. Here we show that eXplainable DL (XDL) methods, based on saliency maps, can extract interpretable predictive information contained in global SST and discover novel SST information regions and dependence structures relevant for river flows which, in tandem with climate network constructions, enable improved predictive understanding. Our results reveal additional information content in global SST beyond ENSO indices, develop new understanding of how SSTs influence river flows, and generate improved river flow predictions with uncertainties. Observations, reanalysis data, and earth system model simulations are used to demonstrate the value of the XDL-CN based methods for future interannual and decadal scale climate projections. 

   more » « less
3. Weakly Supervised Spatial Deep Learning for Earth Image Segmentation Based on Imperfect Polyline Labels

   https://doi.org/10.1145/3480970  

   Jiang, Zhe; He, Wenchong; Kirby, Marcus Stephen; Sainju, Arpan Man; Wang, Shaowen; Stanislawski, Lawrence V.; Shavers, Ethan J.; Usery, E. Lynn (April 2022, ACM Transactions on Intelligent Systems and Technology)

   In recent years, deep learning has achieved tremendous success in image segmentation for computer vision applications. The performance of these models heavily relies on the availability of large-scale high-quality training labels (e.g., PASCAL VOC 2012). Unfortunately, such large-scale high-quality training data are often unavailable in many real-world spatial or spatiotemporal problems in earth science and remote sensing (e.g., mapping the nationwide river streams for water resource management). Although extensive efforts have been made to reduce the reliance on labeled data (e.g., semi-supervised or unsupervised learning, few-shot learning), the complex nature of geographic data such as spatial heterogeneity still requires sufficient training labels when transferring a pre-trained model from one region to another. On the other hand, it is often much easier to collect lower-quality training labels with imperfect alignment with earth imagery pixels (e.g., through interpreting coarse imagery by non-expert volunteers). However, directly training a deep neural network on imperfect labels with geometric annotation errors could significantly impact model performance. Existing research that overcomes imperfect training labels either focuses on errors in label class semantics or characterizes label location errors at the pixel level. These methods do not fully incorporate the geometric properties of label location errors in the vector representation. To fill the gap, this article proposes a weakly supervised learning framework to simultaneously update deep learning model parameters and infer hidden true vector label locations. Specifically, we model label location errors in the vector representation to partially reserve geometric properties (e.g., spatial contiguity within line segments). Evaluations on real-world datasets in the National Hydrography Dataset (NHD) refinement application illustrate that the proposed framework outperforms baseline methods in classification accuracy. 

   more » « less
4. A New Approach for Pedestrian Density Estimation Using Moving Sensors and Computer Vision

   https://doi.org/10.1145/3397575  

   Tokuda, Eric; Lockerman, Yitzchak; Ferreira, Gabriel; Sorrelgreen, Ethan; Boyle, David; Cesar Junior, Roberto Marcondes; Silva, Claudio (September 2020, ACM transactions on spatial algorithms and systems)

   An understanding of person dynamics is indispensable for numerous urban applications, including the design of transportation networks and planning for business development. Pedestrian counting often requires utilizing manual or technical means to count individuals in each location of interest. However, such methods do not scale to the size of a city and a new approach to fill this gap is here proposed. In this project, we used a large dense dataset of images of New York City along with computer vision techniques to construct a spatio-temporal map of relative person density. Due to the limitations of state-of-the-art computer vision methods, such automatic detection of person is inherently subject to errors. We model these errors as a probabilistic process, for which we provide theoretical analysis and thorough numerical simulations. We demonstrate that, within our assumptions, our methodology can supply a reasonable estimate of person densities and provide theoretical bounds for the resulting error. 

   more » « less
5. Visualization and ecohydrologic models: Opening the box

   https://doi.org/10.1002/hyp.13991  

   Tague, Christina; Frew, James (December 2020, Hydrological Processes)

   Abstract Earth system models synthesize the science of interactions amongst multiple biophysical and, increasingly, human processes across a wide range of scales. Ecohydrologic models are a subset of earth system models that focus particularly on the complex interactions between ecosystem processes and the storage and flux of water. Ecohydrologic models often focus at scales where direct observations occur: plots, hillslopes, streams, and watersheds, as well as where land and resource management decisions are implemented. These models complement field‐based and data‐driven science by combining theory, empirical relationships derived from observation and new data to create virtual laboratories. Ecohydrologic models are tools that managers can use to ask “what if” questions and domain scientists can use to explore the implications of new theory or measurements. Recent decades have seen substantial advances in ecohydrologic models, building on both new domain science and advances in software engineering and data availability. The increasing sophistication of ecohydrologic models however, presents a barrier to their widespread use and credibility. Their complexity, often encoding 100s of relationships, means that they are effectively “black boxes,” at least for most users, sometimes even to the teams of researchers that contribute to their design. This opacity complicates the interpretation of model results. For models to effectively advance our understanding of how plants and water interact, we must improve how we visualize not only model outputs, but also the underlying theories that are encoded within the models. In this paper, we outline a framework for increasing the usefulness of ecohydrologic models through better visualization. We outline four complementary approaches, ranging from simple best practices that leverage existing technologies, to ideas that would engage novel software engineering and cutting edge human–computer interface design. Our goal is to open the ecohydrologic model black box in ways that will engage multiple audiences, from novices to model developers, and support learning, new discovery, and environmental problem solving. 

   more » « less