An official website of the United States government
Recent advances in Large Language Models (LLMs) show the potential to significantly augment or even replace complex human writing activities. However, for complex tasks where people need to make decisions as well as write a justification, the trade offs between making work efficient and hindering decisions remain unclear. In this paper, we explore this question in the context of designing intelligent scaffolding for writing meta-reviews for an academic peer review process. We prototyped a system called MetaWriter'' trained on five years of open peer review data to support meta-reviewing. The system highlights common topics in the original peer reviews, extracts key points by each reviewer, and on request, provides a preliminary draft of a meta-review that can be further edited. To understand how novice and experienced meta-reviewers use MetaWriter, we conducted a within-subject study with 32 participants. Each participant wrote meta-reviews for two papers: one with and one without MetaWriter. We found that MetaWriter significantly expedited the authoring process and improved the coverage of meta-reviews, as rated by experts, compared to the baseline. While participants recognized the efficiency benefits, they raised concerns around trust, over-reliance, and agency. We also interviewed six paper authors to understand their opinions of using machine intelligence to support the peer review process and reported critical reflections. We discuss implications for future interactive AI writing tools to support complex synthesis work.
more »
« less
SearchLens: composing and capturing complex user interests for exploratory search
Whether figuring out where to eat in an unfamiliar city or deciding which apartment to live in, consumer generated data (ie reviews and forum posts) are often an important influence in online decision making. To make sense of these rich repositories of diverse opinions, searchers need to sift through a large number of reviews to characterize each item based on aspects that they care about. We introduce a novel system, SearchLens, where searchers build up a collection of “Lenses” that reflect their different latent interests, and compose the Lenses to find relevant items across different contexts. Based on the Lenses, SearchLens generates personalized interfaces with visual explanations that promotes transparency and enables deeper exploration. While prior work found searchers may not wish to put in effort specifying their goals without immediate and sufficient benefits, results from a controlled lab study suggest that our approach incentivized participants to express their interests more richly than in a baseline condition, and a field study showed that participants found benefits in SearchLens while conducting their own tasks.
more »
« less
- Award ID(s):
- 1701005
- PAR ID:
- 10101442
- Date Published:
- Journal Name:
- ACM IUI
- Page Range / eLocation ID:
- 498 to 509
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Automatic Text Simplification (ATS) software aims at automatically rewrite complex text to make it simpler to read. Prior research has explored the use of ATS as a reading assistance technology, identifying benefits from providing these technologies to different groups of users, including Deaf and Hard-of-hearing (DHH) adults. However, little work has investigated the interests and requirements of specific groups of potential users of this technology. Considering prior work establishing that computing professionals often need to read about new technologies in order to stay current in their profession, in this study, we investigated the reading experiences and interests of DHH individuals with work experience in the computing industry in ATS-based reading assistance tools, as well as their perspective on the social accessibility of those tools. Through a survey and two sets of interviews, we found that these users read relatively often, especially in support of their work, and were interested in tools to assist them with complicated texts; but misperceptions arising from public use of these tools may conflict with participants’ desired image in a professional context. This empirical contribution motivates further research into ATS-based reading assistance tools for these users, prioritizing which reading activities users are most interested in seeing the application of this technology, and highlighting design considerations for creating ATS tools for DHH adults, including considerations for social accessibility.more » « less
-
An Empirical Study Comparing Unobtrusive Physiological Sensors for Stress Detection in Computer WorkSeveral unobtrusive sensors have been tested in studies to capture physiological reactions to stress in workplace settings. Lab studies tend to focus on assessing sensors during a specific computer task, while in situ studies tend to offer a generalized view of sensors’ efficacy for workplace stress monitoring, without discriminating different tasks. Given the variation in workplace computer activities, this study investigates the efficacy of unobtrusive sensors for stress measurement across a variety of tasks. We present a comparison of five physiological measurements obtained in a lab experiment, where participants completed six different computer tasks, while we measured their stress levels using a chest-band (ECG, respiration), a wristband (PPG and EDA), and an emerging thermal imaging method (perinasal perspiration). We found that thermal imaging can detect increased stress for most participants across all tasks, while wrist and chest sensors were less generalizable across tasks and participants. We summarize the costs and benefits of each sensor stream, and show how some computer use scenarios present usability and reliability challenges for stress monitoring with certain physiological sensors. We provide recommendations for researchers and system builders for measuring stress with physiological sensors during workplace computer use.more » « less
-
Visualization tools facilitate exploratory data analysis, but fall short at supporting hypothesis-based reasoning. We conducted an exploratory study to investigate how visualizations might support a concept-driven analysis style, where users can optionally share their hypotheses and conceptual models in natural language, and receive customized plots depicting the fit of their models to the data. We report on how participants leveraged these unique affordances for visual analysis. We found that a majority of participants articulated meaningful models and predictions, utilizing them as entry points to sensemaking. We contribute an abstract typology representing the types of models participants held and externalized as data expectations. Our findings suggest ways for rearchitecting visual analytics tools to better support hypothesis- and model-based reasoning, in addition to their traditional role in exploratory analysis. We discuss the design implications and reflect on the potential benefits and challenges involved.more » « less
-
This study aimed to find how IT-related hobbies and interests impacted students’ educational and career decision-making. Methods: As part of a six-year-long, ATE-funded study of IT students at Ivy Tech Community College, our team conducted semi-structured, phenomenological interviews. These interviews were analyzed using keyword searches and a combined inductive-deductive approach to coding to explore how IT-related hobbies and interests interacted with other personal characteristics to inform student decisionmaking. Findings: Our team identified a potential link between early IT interest, IT-related hobbies, and persistence in IT education and careers. Many participants in the study had a moment of clarity where they realized that their IT hobby could become their career, the “hobby-to-career reckoning.” Contributions: This piece explores the potential connection between IT interests/hobbies and student outcomes within the field of IT while exploring the different social factors that may impact student decision-making and the role of the hobby-to-career reckoning in the decision-making process. This piece will give practitioners and researchers insight into how early interest in IT and IT-related hobbies may impact student decision-making about IT educational programs and careers.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3301275.3302321
