More Like this

1. Communicating Uncertain Information from Deep Learning Models in Human Machine Teams

   Subramanian, H. V.; Canfield, C; Shank, D. B.; Andrews, L.; Dagli, C. (October 2020, Proceedings of the American Society for Engineering Management 2020 International Annual Conference)

   Keathley, H.; Enos, J.; Parrish, M. (Ed.)

   The role of human-machine teams in society is increasing, as big data and computing power explode. One popular approach to AI is deep learning, which is useful for classification, feature identification, and predictive modeling. However, deep learning models often suffer from inadequate transparency and poor explainability. One aspect of human systems integration is the design of interfaces that support human decision-making. AI models have multiple types of uncertainty embedded, which may be difficult for users to understand. Humans that use these tools need to understand how much they should trust the AI. This study evaluates one simple approach for communicating uncertainty, a visual confidence bar ranging from 0-100%. We perform a human-subject online experiment using an existing image recognition deep learning model to test the effect of (1) providing single vs. multiple recommendations from the AI and (2) including uncertainty information. For each image, participants described the subject in an open textbox and rated their confidence in their answers. Performance was evaluated at four levels of accuracy ranging from the same as the image label to the correct category of the image. The results suggest that AI recommendations increase accuracy, even if the human and AI have different definitions of accuracy. In addition, providing multiple ranked recommendations, with or without the confidence bar, increases operator confidence and reduces perceived task difficulty. More research is needed to determine how people approach uncertain information from an AI system and develop effective visualizations for communicating uncertainty. 

   more » « less
2. Generalized Structure for Adaptable Immersive Learning Environments

   https://doi.org/10.1109/IRI51335.2021.00047  

   Coltey, Erik; Tao, Yudong; Wang, Tianyi; Vassigh, Shahin; Chen, Shu-Ching; Shyu, Mei-Ling (August 2021, IEEE 22nd International Conference on Information Reuse and Integration for Data Science (IRI))

   Immersive Learning Environments (ILEs) developed in Virtual and Augmented Reality (VR/AR) are a novel pro- fessional training platform. An ILE can facilitate an Adaptive Learning System (ALS), which has proven beneficial to the learning process. However, there is no existing AI-ready ILE that facilitates collecting multimedia multimodal data from the environment and users for training AI models, nor allows for the learning contents and complex learning process to be dynamically adapted by an ALS. This paper proposes a novel multimedia system in VR/AR to dynamically build ILEs for a wide range of use-cases, based on a description language for the generalizable ILE structure. It will detail users’ paths and conditions for completing learning activities, and a content adaptation algorithm to update the ILE at runtime. Human and AI systems can customize the environment based on user learning metrics. Results show that this framework is efficient and low- overhead, suggesting a path to simplifying and democratizing the ILE development without introducing bloat. Index Terms—virtual reality, augmented reality, content generation, immersive learning, 3D environments 

   more » « less
3. A User-based Visual Analytics Workflow for Exploratory Model Analysis

   https://doi.org/10.1111/cgf.13681  

   Cashman, Dylan; Humayoun, Shah Rukh; Heimerl, Florian; Park, Kendall; Das, Subhajit; Thompson, John; Saket, Bahador; Mosca, Abigail; Stasko, John and; Gleicher, Michael; et al (June 2019, Computer graphics forum)

   Many visual analytics systems allow users to interact with machine learning models towards the goals of data exploration and insight generation on a given dataset. However, in some situations, insights may be less important than the production of an accurate predictive model for future use. In that case, users are more interested in generating of diverse and robust predictive models, verifying their performance on holdout data, and selecting the most suitable model for their usage scenario. In this paper, we consider the concept of Exploratory Model Analysis (EMA), which is defined as the process of discovering and selecting relevant models that can be used to make predictions on a data source. We delineate the differences between EMA and the well‐known term exploratory data analysis in terms of the desired outcome of the analytic process: insights into the data or a set of deployable models. The contributions of this work are a visual analytics system workflow for EMA, a user study, and two use cases validating the effectiveness of the workflow. We found that our system workflow enabled users to generate complex models, to assess them for various qualities, and to select the most relevant model for their task. 

   more » « less
4. SAI: AI-Enabled Speech Assistant Interface for Science Gateways in HPC

   Pouya Kousha, Arpan Jain (May 2023, International Conference on High Performance Computing)

   High-Performance Computing (HPC) is increasingly being used in traditional scientific domains as well as emerging areas like Deep Learning (DL). This has led to a diverse set of professionals who interact with state-of-the-art HPC systems. The deployment of Science Gateways for HPC systems like Open On-Demand has a significant positive impact on these users in migrating their workflows to HPC systems. Although computing capabilities are ubiquitously available (as on-premises or in the cloud HPC infrastructure), significant effort and expertise are required to use them effectively. This is particularly challenging for domain scientists and other users whose primary expertise lies outside of computer science. In this paper, we seek to minimize the steep learning curve and associated complexities of using state-of-the-art high-performance systems by creating SAI: an AI-Enabled Speech Assistant Interface for Science Gateways in High Performance Computing. We use state-of-the-art AI models for speech and text and fine-tune them for the HPC arena by retraining them on a new HPC dataset we create. We use ontologies and knowledge graphs to capture the complex relationships between various components of the HPC ecosystem. We finally show how one can integrate and deploy SAI in Open OnDemand and evaluate its functionality and performance on real HPC systems. To the best of our knowledge, this is the first effort aimed at designing and developing an AI-powered speech-assisted interface for science gateways in HPC. 

   more » « less
5. NAS-Navigator: Visual Steering for Explainable One-Shot Deep Neural Network Synthesis

   https://doi.org/10.1109/TVCG.2022.3209361  

   Tyagi, Anjul; Xie, Cong; Mueller, Klaus (January 2023, IEEE Transactions on Visualization and Computer Graphics)

   The success of DL can be attributed to hours of parameter and architecture tuning by human experts. Neural Architecture Search (NAS) techniques aim to solve this problem by automating the search procedure for DNN architectures making it possible for non-experts to work with DNNs. Specifically, One-shot NAS techniques have recently gained popularity as they are known to reduce the search time for NAS techniques. One-Shot NAS works by training a large template network through parameter sharing which includes all the candidate NNs. This is followed by applying a procedure to rank its components through evaluating the possible candidate architectures chosen randomly. However, as these search models become increasingly powerful and diverse, they become harder to understand. Consequently, even though the search results work well, it is hard to identify search biases and control the search progression, hence a need for explainability and human-in-the-loop (HIL) One-Shot NAS. To alleviate these problems, we present NAS-Navigator, a visual analytics (VA) system aiming to solve three problems with One-Shot NAS; explainability, HIL design, and performance improvements compared to existing state-of-the-art (SOTA) techniques. NAS-Navigator gives full control of NAS back in the hands of the users while still keeping the perks of automated search, thus assisting non-expert users. Analysts can use their domain knowledge aided by cues from the interface to guide the search. Evaluation results confirm the performance of our improved One-Shot NAS algorithm is comparable to other SOTA techniques. While adding Visual Analytics (VA) using NAS-Navigator shows further improvements in search time and performance. We designed our interface in collaboration with several deep learning researchers and evaluated NAS-Navigator through a control experiment and expert interviews. 

   more » « less