More Like this

1. (QA)^2: Question Answering with Questionable Assumptions

   Kim, Najoung; Htut, Phu Mon; Bowman, Samuel R.; Petty, Jackson (July 2023, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics)

   Naturally-occurring information-seeking questions often contain questionable assumptions -- assumptions that are false or unverifiable. Questions containing questionable assumptions are challenging because they require a distinct answer strategy that deviates from typical answers to information-seeking questions. For instance, the question "When did Marie Curie discover Uranium?" cannot be answered as a typical when question without addressing the false assumption "Marie Curie discovered Uranium". In this work, we propose (QA)2 (Question Answering with Questionable Assumptions), an open-domain evaluation dataset consisting of naturally-occurring search engine queries that may or may not contain questionable assumptions. To be successful on (QA)2, systems must be able to detect questionable assumptions and also be able to produce adequate responses for both typical information-seeking questions and ones with questionable assumptions. We find that current models do struggle with handling questionable assumptions -- the best performing model achieves 59% human rater acceptability on abstractive QA with (QA)2 questions, leaving substantial headroom for progress. 

   more » « less
2. Multi-Target Embodied Question Answering

   Yu, Licheng; Chen, Xinlei; Gkioxari, Georgia; Bansal, Mohit; Berg, Tamara L; Batra, Dhruv (January 2019, IEEE Conference on Computer Vision and Pattern Recognition)

   Embodied Question Answering (EQA) is a relatively new task where an agent is asked to answer questions about its environment from egocentric perception. EQA as introduced in [8] makes the fundamental assumption that every question, e.g. “what color is the car?”, has exactly one target (“car”) being inquired about. This assumption puts a direct limitation on the abilities of the agent. We present a generalization of EQA – Multi-Target EQA (MT-EQA). Specifically, we study questions that have multiple targets in them, such as “Is the dresser in the bedroom bigger than the oven in the kitchen?”, where the agent has to navigate to multiple locations (“dresser in bedroom”, “oven in kitchen”) and perform comparative reasoning (“dresser” bigger than “oven”) before it can answer a question. Such questions require the development of entirely new modules or components in the agent. To address this, we propose a modular architecture composed of a program generator, a controller, a navigator, and a VQA module. The program generator converts the given question into sequential executable sub-programs; the navigator guides the agent to multiple locations pertinent to the navigation-related sub-programs; and the controller learns to select relevant observations along its path. These observations are then fed to the VQA module to predict the answer. We perform detailed analysis for each of the model components and show that our joint model can outperform previous methods and strong baselines by a significant margin. Project page: https://embodiedqa.org. 

   more » « less
3. Can a Cybersecurity Question Answering Assistant Help Change User Behavior? An In Situ Study

   Duesterwald, Lea; Yang, Ian; Sadeh, Norman (February 2025, NDSS Symposium on Usable Security and Privacy (USEC) 2025)

   Human actions or lack thereof contribute to a large majority of cybersecurity incidents. Traditionally, when looking for advice on cybersecurity questions, people have turned to search engines or social sites like Reddit. The rapid adoption of chatbot technologies is offering a potentially more direct way of getting similar advice. Initial research suggests, however, that while chatbot answers to common cybersecurity questions tend to be fairly accurate, they may not be very effective as they often fall short on other desired qualities such as understandability, actionability, or motivational power. Research in this area thus far has been limited to the evaluation by researchers themselves on a small number of synthetic questions. This article reports on what we believe to be the first in situ evaluation of a cybersecurity Question Answering (QA) assistant. We also evaluate a prompt engineered to help the cybersecurity QA assistant generate more effective answers. The study involved a 10-day deployment of a cybersecurity QA assistant in the form of a Chrome extension. Collectively, participants (N=51) evaluated answers generated by the assistant to over 1,000 cybersecurity questions they submitted as part of their regular day-to-day activities. The results suggest that a majority of participants found the assistant useful and often took actions based on the answers they received. In particular, the study indicates that prompting successfully improved the effectiveness of answers and, in particular, the likelihood that users follow their recommendations (fraction of participants who actually followed the advice was 0.514 with prompting vs. 0.402 without prompting, p=4.61E-04), an impact on people’s actual behavior. We provide a detailed analysis of data collected in this study, discuss their implications, and outline next steps in the development and deployment of effective cybersecurity QA assistants that offer the promise of changing actual user behavior and of reducing human-related security incidents. 

   more » « less
4. Can a Cybersecurity Question Answering Assistant Help Change User Behavior? An In Situ Study

   Duesterwald, Lea; Yang, Ian; Sadeh, Norman (February 2025, Symposium on Usable Security and Privacy (USEC) 2025 - NDSS Symposium)

   Human actions or lack thereof contribute to a large majority of cybersecurity incidents. Traditionally, when looking for advice on cybersecurity questions, people have turned to search engines or social sites like Reddit. The rapid adoption of chatbot technologies is offering a potentially more direct way of getting similar advice. Initial research suggests, however, that while chatbot answers to common cybersecurity questions tend to be fairly accurate, they may not be very effective as they often fall short on other desired qualities such as understandability, actionability, or motivational power. Research in this area thus far has been limited to the evaluation by researchers themselves on a small number of synthetic questions. This article reports on what we believe to be the first in situ evaluation of a cybersecurity Question Answering (QA) assistant. We also evaluate a prompt engineered to help the cybersecurity QA assistant generate more effective answers. The study involved a 10-day deployment of a cybersecurity QA assistant in the form of a Chrome extension. Collectively, participants (N=51) evaluated answers generated by the assistant to over 1,000 cybersecurity questions they submitted as part of their regular day-to-day activities. The results suggest that a majority of participants found the assistant useful and often took actions based on the answers they received. In particular, the study indicates that prompting successfully improved the effectiveness of answers and, in particular, the likelihood that users follow their recommendations (fraction ofparticipants who actually followed the advice was 0.514 with prompting vs. 0.402 without prompting, p=4.61E-04), an impacton people’s actual behavior. We provide a detailed analysis of data collected in this study, discuss their implications, and outline next steps in the development and deployment of effective cybersecurity QA assistants that offer the promise of changing actual user behavior and of reducing human-related security incidents. 

   more » « less
5. What Makes an Inquisitive Engineer? An Exploration of Question-Asking, Self-Efficacy, and Outcome Expectations among Engineering Students.

   Pink, S.; Rieken, B.; Bjorklund, T.; Sheppard, S. (January 2017, Proceedings of the American Society for Engineering Education Annual Conference, June 25-28. Columbus, OH.)

   In order be successful, engineers must ask their clients, coworkers, and bosses questions. Asking questions can improve work quality and make the asker appear smarter. However, people often hesitate to ask questions for fear of seeming incompetent or inferior. This study investigates: what characteristics and experiences are connected to engineering students’ perceptions of asking questions? We analyzed data from a survey of over a thousand engineering undergraduates across a nationally representative sample of 27 U.S. engineering schools. We focused on three dependent variables: question-asking self-efficacy (how confident students are in their ability to ask a lot of questions), social outcome expectations around asking questions (whether students believe if they ask a lot of questions, they will earn the respect of their colleagues), and career outcome expectations (whether they believe asking a lot of questions will hurt their chances for getting ahead at work). We were surprised to find that question-asking self-efficacy or outcome expectations did not significantly vary by gender, under-represented minority status, and school size. However, students with high question-asking self-efficacy and outcome expectations were more likely to have engaged in four extracurricular experiences: participating in an internship or co-op, conducting research with a faculty member, participating in a student group, and holding a leadership role in an organization or student group. The number of different types of these extracurricular activities a student engaged in correlated with question-asking self-efficacy and positive outcome expectations around asking questions. The results illustrate the relationship between extracurricular activities and students’ self-efficacy and behavior outcome expectations. The college experience is more than just formal academic classes. Students learn from experiences that occur after class or during the summer, and ideally these experiences complement class-derived skills and confidence in asking questions. 

   more » « less