skip to main content

Attention:

The NSF Public Access Repository (NSF-PAR) system and access will be unavailable from 11:00 PM ET on Thursday, October 10 until 2:00 AM ET on Friday, October 11 due to maintenance. We apologize for the inconvenience.

Explore Research Products in the NSF-PAR

Title: Investigating User Perceptions of Conversational Agents for Software-related Exploratory Web Search
Conversational agents that respond to user information requests through a natural conversation have the potential to revolutionize how we acquire new information on the Web (i.e., perform exploratory Web searches). Recent advances to conversational search agents use popular Web search engines as a back-end and sophisticated AI algorithms to maintain context, automatically generate search queries, and summarize results into utterances. While showing impressive results on general topics, the potential of this technology for software engineering is unclear. In this paper, we study the potential of conversational search agents to aid software developers as they acquire new knowledge. We also obtain user perceptions of how far the most recent generation of such systems (e.g., Facebook's BlenderBot2) has come in its ability to serve software developers. Our study indicates that users find conversational agents helpful in gaining useful information for software-related exploratory search; however, their perceptions also indicate a large gap between expectations and current state of the art tools, especially in providing high-quality information. Participant responses provide directions for future work.  more » « less
Award ID(s):
1813253
NSF-PAR ID:
10376704
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
Proceedings of the 44th International Conference on Software Engineering (ICSE) NIER Track
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ; ( , Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering)
    null (Ed.)
    Mobile apps are one of the most widely used types of software systems in existence today and more programmers and students learn how to develop them everyday. One of the most popular resources for learning mobile programming are videos hosted on social platforms such as YouTube. While useful, this type of resource has also its limitations, especially when developers are looking for user interface (UI) designs for mobile applications, since these are hard to search for and locate in videos. We propose UIScreens, a web-based analysis and search engine that analyzes the visual contents of mobile programming video tutorials, then identifies and extracts the UI screens displayed in the videos. Our tool offers features such as searching for UI screens in videos, displaying an overview of the UI screens identified in a video under each search result, and navigating to the part of a video where a particular UI screen is being displayed and discussed. In a user study, participants agreed that UIScreens is usable and useful to quickly skim through videos, while the UI screens it extracts can help developers further determine the relevance of videos to a search topic. 
    more » « less
  2. ; ( , Journal of the Association for Information Science and Technology)
    Abstract

    Understanding the roles ofsearch gainandcostin users' search decision‐making is a key topic in interactive information retrieval (IIR). While previous research has developed user models based onsimulatedgains and costs, it is unclear how users' actualperceptions of search gains and costsform and change during search interactions. To address this gap, our study adopted expectation‐confirmation theory (ECT) to investigate users' perceptions of gains and costs. We re‐analyzed data from our previous study, examining how contextual and search features affect users' perceptions and how their expectation‐confirmation states impact their following searches. Our findings include: (1) The point where users' actual dwell time meets their constant expectation may serve as a reference point in evaluating perceived gain and cost; (2) these perceptions are associated with in situ experience represented by usefulness labels, browsing behaviors, and queries; (3) users' current confirmation states affect their perceptions of Web page usefulness in the subsequent query. Our findings demonstrate possible effects of expectation‐confirmation, prospect theory, and information foraging theory, highlighting the complex relationships among gain/cost, expectations, and dwell time at the query level, and the reference‐dependent expectation at the session level. These insights enrich user modeling and evaluation in human‐centered IR.

     
    more » « less
  3. ; ; ; ; ; ( , Proceedings of the 41st International ACM SIGIR Conference onResearch and Development in Information Retrieval)
    Understanding and characterizing how people interact in information-seeking conversations will be a crucial component in developing effective conversational search systems. In this paper, we introduce a new dataset designed for this purpose and use it to analyze information-seeking conversations by user intent distribution, co-occurrence, and flow patterns. The MSDialog dataset is a labeled conversation dataset of question answering (QA) interactions between information seekers and providers from an online forum on Microsoft products. The dataset contains more than 2,000 multi-turn QA dialogs with 10,000 utterances that are annotated with user intents on the utterance level. Annotations were done using crowdsourcing. With MSDialog, we find some highly recurring patterns in user intent during an information-seeking process. They could be useful for designing conversational search systems. We will make our dataset freely available to encourage exploration of information-seeking conversation models. 
    more » « less
  4. ; ; ; ; ( , Proceedings of the 27th ACM International Conference on Information and Knowledge Management)
    Recent years have witnessed the emerging of conversational systems, including both physical devices and mobile-based applications, such as Amazon Echo, Google Now, Microsoft Cortana, Apple Siri, and many others. Both the research community and industry believe that conversational systems will have a major impact on human-computer interaction, and specifically, the IR community has begun to focus on Conversational Search. Conversational search based on user-system dialog exhibits major differences from conventional search in that 1) the user and system can interact for multiple semantically coherent rounds on a task through natural language dialog, and 2) it becomes possible for the system to understand user needs or to help users clarify their needs by asking appropriate questions from the users directly. In this paper, we propose and evaluate a unified conversational search framework. Specifically, we define the major components for conversational search, assemble them into a unified framework, and test an implementation of the framework using a conversational product search scenario in Amazon. To accomplish this, we propose the Multi-Memory Network (MMN) architecture, which is end-to-end trainable based on large-scale collections of user reviews in e-commerce. The system is capable of asking aspect-based questions in the right order so as to understand user needs, while (personalized) search is conducted during the conversation and results are provided when the system feels confident. Experiments on real-world user purchasing data verified the advantages of conversational search against conventional search algorithms in terms of standard evaluation measures such as NDCG. 
    more » « less
  5. ; ; ; ; ; ( , Proceedings of the HCI International 2019 Conference on Human-Computer Interaction)
    To improve the academic and professional achievement of underrepresented minorities in computing, a newfound interest in innovative mentoring practices has captivated STEM education researchers. Studies suggest that virtual mentoring conversational agents can be leveraged across multiple platforms to provide supplemental mentorship, offsetting the lack of access to in-person mentorship in disadvantaged communities. A within-subjects mixed-method experiment was carried out to assess the usability of a mentoring conversational agent. Mobile interfaces (Twitter and SMS) were compared to each other and against a web-based embodied conversational agent (ECA). Results suggest that mobile interfaces are more usable than the web-based ECA. The findings from this study help to identify areas for improvement in virtual learning alternatives and other potential applications for pervasive conversational interfaces. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email