<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>Toward a Human-centered Evaluation Framework for Trustworthy LLM-powered GUI Agents</title></titleStmt>
			<publicationStmt>
				<publisher>The ACM Conference on Human Factors in Computing Systems - HEAL Workshop (HEAL@CHI)</publisher>
				<date>06/17/2025</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10637684</idno>
					<idno type="doi"></idno>
					<title level='j'>CHI  conference proceedingsCHI Conference</title>
<idno>2159-6468</idno>
<biblScope unit="volume"></biblScope>
<biblScope unit="issue"></biblScope>					

					<author>Chaoran Chen</author><author>Zhiping Zhang</author><author>Ibrahim Khalilov</author><author>Bingcan Guo</author><author>Simret Gebreegziabher</author><author>Yanfang Ye</author><author>Ziang Xiao</author><author>Yaxing Yao</author><author>Tianshi Li</author><author>Toby Jia-Jun Li</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[The rise of Large Language Models (LLMs) has revolutionized Graphical User Interface (GUI) automation through LLM-powered GUI agents, yet their ability to process sensitive data with limited human oversight raises significant privacy and security risks. This position paper identifies three key risks of GUI agents and examines how they differ from traditional GUI automation and general autonomous agents. Despite these risks, existing evaluations focus primarily on performance, leaving privacy and security assessments largely unexplored. We review current evaluation metrics for both GUI and general LLM agents and outline five key challenges in integrating human evaluators for GUI agent assessments. To address these gaps, we advocate for a human-centered evaluation framework that incorporates risk assessments, enhances user awareness through in-context consent, and embeds privacy and security considerations into GUI agent design and evaluation.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1">Introduction</head><p>The rise of large language models (LLMs) has transformed Graphical User Interface (GUI) automation across web applications <ref type="bibr">[59]</ref>, mobile devices <ref type="bibr">[52]</ref>, and operating systems <ref type="bibr">[51]</ref>. Traditional automation frameworks, such as Selenium <ref type="bibr">[5]</ref>, rely on static action scripts and predefined rules to automate specific workflows. While effective in predefined tasks, these tools lack flexibility and require manual scripting or rule-based logic, making them struggle with the complexities of modern, dynamic, and context-sensitive interfaces <ref type="bibr">[50]</ref>. Recent advancements in LLMs have led to the development of LLM-powered GUI agents, offering unique capabilities to overcome these challenges. An LLM-powered GUI agent (hereafter referred to as a GUI agent) is a specialized autonomous system that perceives and interprets UI elements by either analyzing screenshots or GUI source files, translates user commands into sequential actions using LLMs, and interacts with GUIs by executing actions such as clicking, typing, and tapping to fulfill user requirements <ref type="bibr">[32]</ref>. Unlike traditional GUI automation, GUI agents further enhance automation by interpreting natural language commands, processing multi-modal content, and dynamically simulating user actions <ref type="bibr">[32]</ref>. For example, OpenAI's Operator <ref type="bibr">[34]</ref> and Claude's Computer Use <ref type="bibr">[2]</ref> can assist users by automatically filling out complex web forms and navigating dynamic websites based on verbal instructions of only high-level objectives, eliminating the need for pre-programmed scripts or step-by-step instructions.</p><p>Beyond enhancing automation capabilities, GUI agents also make automation more accessible to a wider range of non-technical users. By enabling interaction through natural language prompts, these agents reduce the complexity of workflow automation, eliminating repetitive manual inputs and intricate navigation. In Claude's Computer Use demo, for instance, the agent automates data retrieval and entry, easing the cognitive burden on users who would otherwise have to manually search spreadsheets and customer relationship management systems. Instead of remembering data locations, switching between applications, and copying details into forms, users can rely on the agent to automatically gather, extract, and integrate information. When integrated with assistive technologies such as screen readers and speech-to-text systems, GUI agents further enhance accessibility for users with disabilities. From automating price comparisons to generating email responses, these agents extend automation beyond software testing to everyday tasks, making technology more accessible and boosting productivity across diverse domains.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1.1">Privacy and Security Risks in GUI Agents</head><p>As GUI agents continue to enhance their automation capabilities and expand their user base, emerging privacy concerns have backfired, resulting in a significant trust issue for GUI agents <ref type="bibr">[50]</ref>. Research <ref type="bibr">[31]</ref> shows that even commercial models like GPT-4 and ChatGPT struggle with privacy reasoning, sometimes exposing private information in ways that humans would not. Shao et al. <ref type="bibr">[37]</ref> illustrate this with an LLM agent disclosing John's job search to his manager without consent.</p><p>Building on prior work, we organize GUI agents' privacy and security risks into three key categories: (1) amplified data leaks from their need for direct access to sensitive data and frequent third-party interactions, (2) diminished privacy and security control as GUI agents autonomously handle data, limiting human oversight, and (3) insufficient guardrails, making GUI agents susceptible to data breach and adversarial attacks.</p><p>Amplified Data Leaks: The nature of GUI agents and the contexts in which they are used often necessitate access to sensitive user data. Unlike direct LLM prompting, where it is possible for users to redact sensitive details in the information provided, GUI agents typically need unfiltered sensitive data to complete tasks. For example, to book a flight, the user needs to provide actual travel details, payment credentials, and account information to be filled in by the agent in its automated interactions with the underlying system. As a result, many privacy-enhancing prompting techniques <ref type="bibr">[13]</ref> designed to modify sensitive information in LLM prompts become ineffective in the context of GUI agents.</p><p>Another way of how the use of GUI agents amplifies data leaks is its high frequency of accessing and possibly disclosing sensitive data. For example, while a user manually searching for a medical device may visit only a few websites, an automated agent may be configured to query dozens within minutes, or in some cases, periodically, embedding the user's medical interests into multiple tracking systems. If these queries or form submissions are shared with malicious websites, they could expose sensitive health information. Similarly, an agent repeatedly checking flight prices may unknowingly broadcast location data across multiple services, increasing surveillance risks.</p><p>Beyond immediate exposure, the automated interactions of GUI agents create long-term privacy concerns. Frequent engagements with third-party services contribute to detailed behavioral profiles, which, if retained, leaked, or misused, can lead to data exploitation and unauthorized inference of personal habits. Unlike one-time LLM interactions, GUI agents operate continuously across platforms, increasing the persistence and exposure of private data.</p><p>Diminished Privacy and Security Control: As intermediaries between users and online services, GUI agents improve interaction efficiency but reduce users' control, making privacy and security risks harder to assess. Unlike direct interactions, where users can pause, reflect on the context, and adjust their inputs, GUI agents operate autonomously, requiring users' reliance in their decisionmaking and data handling prior to the interaction.</p><p>For instance, when authorizing a GUI agent to automate tax filing, users may provide credentials to access financial platforms, upload sensitive documents, and share personal financial data. While the agent executes these tasks, users may be unaware that their data could be stored, retained, or even exposed within the agent's backend systems. Similarly, an agent assisting with account recovery on a social media platform might input security questions or recovery codes without user oversight. If these details are stored insecurely or misused, they could lead to unauthorized account access. This potential over-reliance on GUI leads to the lack of visibility into how their data is processed, stored, or shared. Unlike direct interactions, where users retain control and can adjust their behaviors reflectively based on the information they receive from the process of interaction, GUI agents abstract these processes, making risk assessment and proactive mitigation difficult. Their opacity, coupled with complex data use policies, further erodes user agency, increasing the likelihood of unintended data exposure and misuse.</p><p>Insufficient Guardrails: Privacy and security safeguards are often overlooked in the training and prompting of GUI agents, leaving them vulnerable to adversarial attacks. As shown in Fig. <ref type="figure">1</ref>, Claude's Computer Use agent unknowingly shared a (fake) driver's license number with a phishing website we created. Following the user's instruction to obtain a discount, the agent failed to recognize the fraudulent site or question the unusual request to submit a driver's license number for a discount. The risks extend beyond screenshot-based agents like the Computer Use agent.</p><p>GUI agents processing structured files, such as HTML or APK, are equally vulnerable. Liao et al. <ref type="bibr">[28]</ref> introduced the Environmental Injection Attack (EIA), which exploits this weakness by injecting malicious content that dynamically adapts to the agent's environment. Their study demonstrated EIA on a real website, where a web agent processing HTML was tricked into entering personally identifiable information into an invisible, injected field containing malicious instructions. The agent unknowingly leaked the data and continued executing its task, unaware of the breach.</p><p>These examples illustrate how both screenshot-based and fileprocessing agents can be manipulated to expose sensitive information. When GUI agents lack proper training or guardrails for handling adversarial scenarios, they become easy targets for exploitation. The failure to integrate privacy and security safeguards into their development leaves users increasingly vulnerable to data leaks and security breaches.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1.2">Challenges in Evaluations</head><p>Despite growing privacy and security concerns, GUI agent evaluations primarily focus on performance. Existing metrics typically assess effectiveness (e.g., task completion rates) and efficiency (e.g., speed and resource use). While some studies incorporate safety metrics to evaluate risk management, policy adherence, and safeguard mechanisms, these mainly address immediate security risks and compliance rather than nuanced, individual concerns. Priva-cyLens <ref type="bibr">[37]</ref> introduced a safety-helpfulness tradeoff, showing that models with lower leakage rates often perform worse in helpfulness. This suggests that some agents prioritize responsiveness and task success at the expense of privacy, potentially exposing sensitive data. To address this issue, evaluation frameworks must explicitly consider this tradeoff, promoting the development of GUI/LLM agents that balance privacy and effectiveness rather than treating them as conflicting objectives.</p><p>A major challenge in assessing privacy risks for GUI agents is their strong dependence on context, which can be understood through two key theoretical frameworks: privacy calculus <ref type="bibr">[10]</ref> and contextual integrity <ref type="bibr">[33]</ref>. Privacy calculus theory suggests that users weigh the risks and benefits of sharing sensitive information based on perceived rewards, task relevance, and trust in the system. Meanwhile, contextual integrity theory highlights that privacy decisions are shaped by the specific context in which data is shared, including the type of information, the situation, and the user-system relationship. Together, these theories emphasize that privacy risks are not uniform, but vary based on individual privacy value judgments and circumstances. For example, users may readily share data for routine tasks like shopping, but hesitate when handling financial or personal information. This variability complicates standardized risk assessments, as what one user finds an acceptable trade-off may not apply to another. Thus, evaluating privacy risks in GUI agents requires a context-aware approach that accounts for individual risk-reward considerations.</p><p>To bridge this gap, we advocate for a human-centered evaluation framework for trustworthy GUI agents. Unlike traditional GUI automation, which operates within predefined workflows, GUI agents leverage LLMs to dynamically interpret and interact with user interfaces, enabling flexible and adaptive task execution. As GUI agents advance, ensuring both performance and privacy safeguards becomes essential. We propose three key actions to enhance privacy and trust: (1) human-centered evaluation for privacy and security risk assessment, (2) integrating privacy measures into agent development, and (3) enhancing users' awareness of these issues through in-context consent mechanisms.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2">GUI Agents vs. Traditional GUI Automation</head><p>Traditional GUI automation relies on rule-based frameworks that execute predefined sequences of user interactions, such as button clicks, text inputs, and navigation commands. Common tools like Selenium<ref type="foot">foot_0</ref> , AutoIt<ref type="foot">foot_1</ref> , and Robot Framework<ref type="foot">foot_2</ref> , script interactions based on explicitly defined rules. While effective for testing and automating repetitive tasks, traditional GUI automation lacks adaptability, requiring extensive reconfiguration when UI elements change or when unforeseen interaction scenarios arise.</p><p>Recent advancements in artificial intelligence (AI) and LLMs have facilitated the emergence of GUI agents, which represent a paradigm shift in GUI automation. Unlike traditional methods, GUI agents leverage multi-modal AI models, reinforcement learning, and dynamic reasoning to interact with interfaces more flexibly and autonomously, without relying on predefined or rule-based scripts. These agents interpret UI components in real time, dynamically adapting to interface modifications such as layout changes, content updates, or element repositioning based on user interactions and system responses. For instance, the Test-Agent framework proposed by Li et al. <ref type="bibr">[27]</ref> introduced an LLM-powered GUI automation system that significantly enhances testing flexibility by enabling AI to interpret and adapt to new UI configurations without explicit reprogramming.</p><p>Additionally, the incorporation of semantic analysis and symbolic reasoning enables GUI agents to perform complex automation tasks beyond rule-based scripting. Judson et al. <ref type="bibr">[20]</ref> discuss the role of automated decision-making frameworks that incorporate symbolic reasoning and machine learning to enhance GUI interactions, particularly in legal accountability scenarios. This approach highlights how GUI agents can operate in domains requiring higher reasoning and compliance with contextual constraints.</p><p>The key distinctions between traditional GUI automation and GUI agents can be summarized as follows: While GUI agents offer greater adaptability and automation capabilities, they also introduce heightened privacy risks compared to traditional rule-based automation. Unlike predefined scripts that execute specific tasks with minimal data access, GUI agents dynamically generate data processing strategies without human review, increasing uncertainty about how sensitive data is handled. This lack of oversight raises the risk of unintended data exposure, as agents may access sensitive on-screen content, retain interaction logs, or transmit data externally, potentially leading to privacy leaks or unauthorized data sharing. Wen et al. <ref type="bibr">[48]</ref> highlight that realtime UI access may inadvertently expose sensitive information such as passwords, financial data, or personal messages. Risks escalate when GUI agents interact with unsecured or phishing websites, misinterpret UI elements containing confidential data, or store interaction logs without proper safeguards, increasing unauthorized access or data leaks.</p><p>Another major concern is data persistence and external processing. Traditional automation tools execute tasks without retaining user information, whereas GUI agents may store interaction logs or transmit data to cloud-based models for inference, increasing the risk of unauthorized access or third-party interception <ref type="bibr">[54]</ref>. Moreover, the lack of granular permission controls in AI-driven automation makes it difficult to restrict access, leading to unintended data retrieval or misuse <ref type="bibr">[32]</ref>.</p><p>Additionally, adversarial attacks and prompt injection vulnerabilities pose unique threats to GUI agents. Unlike static scripts, GUI agents interpret and generate responses dynamically, which increases their vulnerability to manipulated inputs. Unlike rule-based scripts that follow predefined workflows, these agents process and act upon real-time user inputs, making them susceptible to adversarial attacks such as UI dark patterns, phishing attempts, or prompt injections <ref type="bibr">[4]</ref>. Maliciously crafted UI elements or deceptive prompts can mislead the agent into exposing private information, executing unintended actions, or interacting with fraudulent interfaces.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3">GUI Agents as a Specialized Class of LLM-powered Autonomous Agents</head><p>A GUI agent is a specialized type of autonomous agent designed to interact with digital platforms through their graphical interfaces. These agents translate natural language commands into concrete actions such as clicking, typing, and scrolling, mimicking human interaction patterns. While GUI agents and other LLM-powered autonomous agents, such as AutoGPT <ref type="bibr">[49]</ref> and AutoGLM <ref type="bibr">[29]</ref>, both extend LLMs' intelligence to sequential action execution, they differ in the degree of autonomy and user oversight they provide. LLM-powered autonomous agents, particularly those emphasizing full autonomy, often function as black-box systems that generate and execute multi-step plans without user validation. These agents leverage external APIs and other automation tools to solve complex tasks independently. In contrast, GUI agents integrate LLM-driven automation with user-interactive workflows, providing explainable action steps and opportunities for human oversight. Users can monitor each proposed action and intervene when necessary, ensuring greater control over the automation process.</p><p>However, the automation capabilities of GUI agents introduce a double-edged sword. By reducing friction in user interactions, they streamline tasks and improve efficiency, yet they may also limit user reflection and error correction. Unlike conversational LLMs, which operate solely in the text token space, GUI agents operate in both the text token space and action space, enabling interactions such as clicking, text entry, and scrolling. This expanded action space allows GUI agents to translate user intents into real-world interactions through automation techniques (e.g., Selenium WebDriver and Android Debug Bridge). While GUI agents incorporate human oversight, their automation model can sometimes make unintended actions harder to detect and correct, amplifying potential privacy and security risks.</p><p>Because GUI agents operate within users' digital environments, they may inadvertently access and process sensitive on-screen information. Unauthorized interactions-such as unintended form submissions or exposure of private data during automation-raise concerns about data security and user trust. However, their step-bystep execution model also presents a unique opportunity for humancentered privacy evaluations. Unlike fully autonomous agents that execute entire workflows without user intervention, GUI agents allow users to dynamically assess and mitigate privacy risks in context. This balancing act between automation and oversight introduces a novel paradigm where users can actively engage in privacy-aware decision-making rather than relying solely on predefined safeguards.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4">Evaluation Metrics of GUI Agents</head><p>Building on surveys for GUI agents <ref type="bibr">[32,</ref><ref type="bibr">50]</ref>, we categorize their evaluation metrics into three key areas: effectiveness, efficiency, and safety. Effectiveness measures how well the GUI agent achieves its intended objectives at task level or step level. Efficiency evaluates the agent's speed and resource usage, considering factors such as task completion time, latency, and computational overhead. Safety ensures the agent minimizes unintended actions and compliance with safety policies. In the following subsections, we explore each of these evaluation metrics in detail.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.1">Effectiveness</head><p>4.1.1 Task-wise metrics. Task-wise evaluation assesses an agent's ability to complete an entire task successfully. The Task Completion Rate (TCR) is a key measure of reliability, indicating the proportion of assigned tasks completed successfully. A high TCR is critical for automation applications, where seamless task execution is necessary to reduce human intervention. Beyond completion, the Success Rate refines this evaluation by measuring how often an agent completes a task without external assistance, offering insights into its autonomy and robustness. Zhang et al. <ref type="bibr">[55]</ref> found that a GUI agent achieved an 88% task completion rate in structured environments but exhibited decreased performance in unstructured workflows. This highlights the challenge of ensuring adaptability across diverse task settings. Additionally, Task Progress serves as a complementary metric, quantifying how far an agent progresses toward task completion on average, even when full completion is not achieved.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.1.2">Step-wise metrics.</head><p>Step-wise evaluation focuses on the accuracy and reliability of individual actions within a task. The Step Success Rate measures the proportion of correctly executed steps out of the total steps required for a task. A high step success rate indicates precise action execution, which is critical for tasks requiring multiple sequential interactions. Since steps collectively form a trajectory representing a complete task, accuracy at this level directly impacts overall task success. Step-wise evaluation often employs macro-averaging, where scores are first averaged within a trajectory and then across tasks, ensuring that each task contributes proportionally to the final metric. Additionally, the Error Rate highlights unintended or incorrect actions, providing insight into failure points that require model improvement. Another crucial step-wise metric is Adaptability, which measures how well an agent generalizes across different UI environments without explicit reconfiguration. Poor adaptability often results in increased error rates when transitioning between structured and unstructured workflows. Evaluating adaptability is essential to improving real-world usability, as GUI agents must handle varying interface designs and dynamic user interactions.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.2">Efficiency</head><p>4.2.1 Speed. Speed is a critical aspect of efficiency, as it directly impacts the responsiveness and practicality of a GUI agent. Two key factors in measuring speed are Time Cost and Step Cost. Time cost refers to the total latency required for task completion, reflecting how quickly an agent can execute an instruction. Step cost, on the other hand, quantifies the number of steps taken to reach task completion, where fewer steps often indicate a more optimized execution strategy. A lower step cost typically correlates with reduced time cost, as efficient step execution leads to faster task resolution.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.2.2">Resource.</head><p>Resource efficiency focuses on minimizing computational and financial overhead while maintaining reliable performance. Two key aspects are Internal Resource Cost and External Resource Cost. Internal resource cost measures the internal computational resources consumed, including memory, CPU, and GPU usage, which directly affect an agent's scalability and deployment feasibility. In contrast, external resource cost accounts for external computational expenses, such as the number of LLM calls made during task execution, which impacts both processing load and financial costs in cloud-based systems. For example, Song et al. <ref type="bibr">[39]</ref> optimized API calls by reducing unnecessary API interactions and optimizing model queries, so that agents can achieve a balance between performance and cost-effectiveness.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.3">Safety</head><p>To enhance security and user trust, agents must recognize and mitigate potentially harmful actions through safeguard mechanisms, policy compliance, and risk assessment. Safeguard mechanisms require user confirmation before executing critical operations, such as file deletions or system modifications, ensuring that unintended or harmful actions are prevented. Zhang et al. <ref type="bibr">[53]</ref> introduce the Safeguard Rate as a metric to assess how effectively an agent detects sensitive actions and prompts verification, with a high safeguard rate indicating stronger protective measures. Additionally, policy compliance ensures that agents operate within predefined rules and constraints, preventing automation from violating security protocols, privacy regulations, or ethical boundaries. The Completion Under Policy metric evaluates the percentage of tasks successfully executed while adhering to these guidelines, which is crucial in regulatory-sensitive environments. However, even with safeguards and compliance measures in place, agents may still pose risks due to incorrect predictions or unintended actions. The Risk Ratio quantifies the likelihood of security vulnerabilities, errors, or violations arising from agent behavior, with a lower ratio indicating greater reliability. Continuous monitoring and optimization of these metrics are essential for deploying agents in high-stakes applications, ensuring secure and trustworthy interactions.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5">Evaluating Privacy in GUI/LLM Agents</head><p>Privacy evaluation for GUI agents remains unexplored. Most relevant studies focus on evaluating web-based LLM agents and their ability to resist specific malicious attacks, yet no systematic evaluation frameworks or benchmarks have been established <ref type="bibr">[9,</ref><ref type="bibr">28,</ref><ref type="bibr">56]</ref>. For example, Liao et al. <ref type="bibr">[28]</ref> propose an environmental injection attack (EIA) that aims to steal users' personally identifiable information (PII) during web interactions to evaluate the privacy protection capabilities of LLM-powered web agents.</p><p>Current studies primarily focus on model-level evaluation or auditing of privacy risks under different attacks. For example, several benchmarks have been proposed to assess LLMs' vulnerability to various attacks, including membership inference attacks (MIA) <ref type="bibr">[12,</ref><ref type="bibr">35,</ref><ref type="bibr">36]</ref>, data extraction <ref type="bibr">[1,</ref><ref type="bibr">45]</ref>, and intentional retrieval of sensitive information during model inference <ref type="bibr">[45]</ref>. Li et al. <ref type="bibr">[25]</ref> proposed LLM-PBE, a toolkit that systematically evaluates privacy risks in LLMs through attacks (MIA, data extraction, prompt leakage, and jailbreak attacks). A few studies used prompt engineering to conduct a privacy audit on LLMs to evaluate the extent to which these models align with the privacy requirements outlined in the compliance <ref type="bibr">[7,</ref><ref type="bibr">16,</ref><ref type="bibr">30]</ref>. Some researchers have also explored how well LLMs can understand and reason about privacy based on contextual integrity theory (CI) <ref type="bibr">[18,</ref><ref type="bibr">31,</ref><ref type="bibr">37]</ref>.</p><p>Recent efforts have begun exploring agent-level privacy evaluation. The Agent Security Bench (ASB) provides a structured approach to formalize, benchmark, and evaluate both security attacks and defenses relevant to LLM-based agents across diverse scenarios but does not specifically focus on the privacy aspect <ref type="bibr">[53]</ref>. Shao et al. <ref type="bibr">[37]</ref> developed a pipeline and benchmark to assess LLM-based agents' privacy awareness through privacy leakage in the agent's actions. Interestingly, their results reveal a discrepancy between model performance in answering probing questions and their actual behavior when executing user instructions in an agent setup <ref type="bibr">[37]</ref>.</p><p>These findings also suggest that model-level privacy evaluations alone are insufficient for fully understanding LLM-based agents' privacy-related capabilities, emphasizing the need for more agentlevel evaluations. Moreover, while most studies focus on text-based LLM interactions or LLM-based agents, GUI agents introduce additional complexities due to their multimodal nature. Unlike textbased agents, GUI agents interact with users through both textual commands and visual UI elements, exposing them to a wider range of privacy threats. Beyond text-based privacy attacks such as adversarial jailbreaking, GUI agents can also be manipulated through dark patterns, including misleading UI elements, subtle nudging mechanisms, or obfuscated privacy settings designed to influence agent behavior without raising user and agent awareness. This multimodal nature presents new challenges and calls for novel evaluation approaches.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6">Human-Centered Evaluation for GUI Agents</head><p>Most current evaluations (see Section 5) automate the assessment and auditing process to achieve large-scale and more efficient evaluation, with some leveraging the power of LLMs to do so <ref type="bibr">[16,</ref><ref type="bibr">25,</ref><ref type="bibr">30,</ref><ref type="bibr">36]</ref>. However, several studies <ref type="bibr">[37,</ref><ref type="bibr">41]</ref> have shown that LLMs are inherently vulnerable when making ethical or moral judgments, particularly due to their lack of awareness of social and privacy norms. To mitigate these risks, human-centered evaluation which involves human inputs and governance, is needed to ensure that LLM agents operate ethically and in alignment with human values.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.1">Human Oversight and Auditing</head><p>Current human-centered evaluations for LLM agents primarily fall under human oversight <ref type="bibr">[14,</ref><ref type="bibr">17,</ref><ref type="bibr">24]</ref> and user-engaged algorithm auditing <ref type="bibr">[23,</ref><ref type="bibr">38]</ref>.</p><p>Human oversight has been recognized as a critical mechanism in AI governance to enhance system accuracy and safety and to uphold human values in technology <ref type="bibr">[14]</ref>. Regulations such as the EU AI Act emphasize that high-risk AI systems should be designed to allow "natural persons can oversee their functioning, ensure that they are used as intended and that in their impacts are addressed over the system's lifecycle" <ref type="bibr">[14]</ref>. For example, Operator, an OpenAIdeveloped GUI agent for computer use, integrates human oversight as a key approach to ensuring safety and privacy <ref type="bibr">[34]</ref>. It includes "Watch Mode" allowing users to monitor the agent's operations in real-time and directly catch potential mistakes, "User Confirmations" requiring users to approve any significant actions, and "Detection Pipeline" supporting human post-auditing to identify threats in the agent's behavior <ref type="bibr">[34]</ref>.</p><p>User-engaged algorithm auditing is a more specific process that assesses, mitigates, and ensures an algorithm's safety, legality, and ethical compliance with the involvement of end-users <ref type="bibr">[11,</ref><ref type="bibr">22]</ref>. For example, real-time auditing allows users to review an algorithm's outputs in daily tasks <ref type="bibr">[38]</ref>. In contrast, post-hoc auditing enables users to verify past or simulated examples at scale <ref type="bibr">[23]</ref>.</p><p>However, the multimodal nature, higher system complexity, increased agency, and seamless data transmission of GUI agents present novel challenges for human-centered evaluation. These challenges arise from factors such as knowledge barriers, flawed mental models, overtrust, limited privacy awareness, cognitive burden, and the need to rethink evaluation goals.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.2">Knowledge Barriers and Mental Model Challenges for Human Evaluators</head><p>One of the main criticisms of human oversight in AI governance is the capability of individuals responsible for overseeing AI systems <ref type="bibr">[17,</ref><ref type="bibr">40,</ref><ref type="bibr">44]</ref>. A lack of technical expertise or domain-specific knowledge can lead to ineffective oversight, increasing the risk of errors or biases. To address this concern, many studies emphasize the need for training professionals with expertise in both AI technology and its application domains <ref type="bibr">[40,</ref><ref type="bibr">43]</ref>. For example, Sterz et al. <ref type="bibr">[40]</ref> developed a framework to define the requirements for oversight professionals, emphasizing that individuals who focus on oversight should have a comprehensive understanding of how AI systems function and their associated risks in different situations. However, in GUI agents, the increasing complexity of systems and the invisible nature of backend data transmissions place even higher demands on human evaluators' knowledge and mental models. For example, there are different types of GUI agent perception interfaces, and each is often associated with distinct privacy risks. Nguyen et al. <ref type="bibr">[32]</ref> suggests that screen-visual-based interfaces could visually expose sensitive information, as the agent continuously captures screenshots. While HTML-based interfaces could also include sensitive information through interactions, depending on the structure of the web environment the agent operates in <ref type="bibr">[32]</ref>. Moreover, compared to screen-visual-based interfaces, where both the agent and the user perceive the same content, HTML/DOM-based and accessibility-based interfaces are more vulnerable to environmental injection attacks <ref type="bibr">[28]</ref>. These attacks manipulate the agent's perception by injecting misleading or malicious content into the environment. Even worse, such attacks can be difficult for humans to detect, particularly when designed to be invisible <ref type="bibr">[28]</ref>.</p><p>In addition, the high level of agency and seamless backend data transmission in GUI agents make it challenging for human evaluators to develop and maintain accurate mental models of these systems. Prior studies have shown that people often hold flawed or incomplete mental models of LLM-based conversational agents <ref type="bibr">[26,</ref><ref type="bibr">47,</ref><ref type="bibr">58]</ref>. GUI agents, however, introduce even greater complexity, as they seamlessly integrate with users' databases, applications, and services to ensure agency <ref type="bibr">[32,</ref><ref type="bibr">46]</ref>. This deeper level of integration and automation increases the difficulty for users to fully understand how data flows within the system, making it harder to anticipate potential privacy risks.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Challenge 1</head><p>The increasing complexity of systems and the invisible nature of backend data transmissions in GUI agents place higher demands on human evaluators' knowledge and mental models.</p><p>An even more pressing concern is the growing role of endusers in AI oversight, which further exacerbates these challenges.</p><p>Consumer-facing GUI agents, such as Operator <ref type="bibr">[34]</ref> and Claude's computer-use agent <ref type="bibr">[2]</ref>, are increasingly being adopted for both personal and professional tasks. Since privacy preferences vary between individuals, it is important to incorporate end-user perspectives in agent evaluation and assess whether GUI agents effectively protect user privacy based on users' perceptions and expectations. However, unlike professional evaluators who undergo training, endusers often struggle to fully understand how GUI agents function. They face significant challenges in developing accurate mental models that allow them to foresee risks, effectively oversee AI actions, and conduct proper auditing.</p><p>Challenge 2 End-users are playing an increasingly critical role in GUI agent oversight but face greater challenges than expert evaluators in acquiring the necessary knowledge and developing accurate mental models for effective oversight.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.3">Overtrust, Lack of Privacy Awareness, and Increased Cognition Burden in Evaluation</head><p>Many prior studies have found that humans tend to overtrust AI systems and often rely on AI-generated decisions without sufficient scrutiny <ref type="bibr">[19,</ref><ref type="bibr">21]</ref>. A recent study about text-based LM agents for interpersonal communication revealed that users exhibited overtrust in AI, overlooked privacy leakage in the agents' actions and made decisions that ultimately led to even greater privacy exposure <ref type="bibr">[57]</ref>. The phenomenon of "privacy paradox" was also observed in the use of LM agents, where users claim to care about privacy yet behave in ways that contradict their stated concerns, primarily due to a lack of privacy awareness. The findings suggest that both AI involvement and users' trust in AI capabilities collectively contribute to new challenges in privacy awareness, influencing how users manage and protect their own privacy. While GUI agents have greater agency, more advanced capabilities, and increased transparency in task execution (e.g., the"watch mode" in Operator <ref type="bibr">[34]</ref>), these features may inadvertently reinforce user reliance on the agent's decisions, assuming that the system is inherently safe and making oversight less effective <ref type="bibr">[3,</ref><ref type="bibr">57]</ref>.</p><p>Challenge 3 Overtrust in AI and limited privacy awareness may cause challenges in effectively overseeing GUI agents. GUI agents mimic human interaction patterns in operating systems, producing outputs not only in text but also as a sequence of visual actions, enhancing transparency in task execution. Studies suggest that increasing AI transparency and offering explanations can help humans better understand AI decision-making and reduce overreliance <ref type="bibr">[42]</ref>. However, this benefit hinges on cognitive forcing <ref type="bibr">[6]</ref>, which encourages slow and deliberative thinking. Without this cognitive engagement, more detailed explanations can sometimes make AI appear more rational and inadvertently increase human reliance on AI's decisions while overriding their own judgment <ref type="bibr">[3]</ref>. Similarly, Zhang et al. <ref type="bibr">[57]</ref> found that when users directly observed an agent's actions, most did not become aware of privacy leaks. Conversely, when provided with contextual privacy norms, users exerted greater cognitive effort and became more aware of the risks associated with disclosing certain information. Based on these findings, the authors advocate for a scaffolded evaluation process that guides human oversight of AI systems. However, overseeing GUI agents presents unique challenges due to their multimodal output. Unlike prompt-only interactions, GUI agents perform multiple actions across different information modalities, often requiring evaluators to process and assess multiple pieces of information simultaneously within a limited timeframe. As a result, human evaluators may experience cognitive overload, making it difficult to scrutinize each step carefully, provide consistent feedback, and maintain effective oversight.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Challenge 4</head><p>The multi-modal nature of GUI agent outputs increases cognitive burden, making it more difficult to oversee or audit multiple steps in complex tasks.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.4">Rethinking the Evaluation Goals</head><p>When GUI agents are designed to mimic human interaction patterns, should agent privacy behavior be evaluated based on the alignment with users' actual privacy practices? Gabriel <ref type="bibr">[15]</ref> raised a normative discussion on AI alignment goals, mentioning a concern that human behavior does not always reflect an individual's true preferences. Similarly, Zhang et al. <ref type="bibr">[57]</ref> found that aligning LLM agents solely with users' actual behavior can still result in privacy violations. Instead, recent studies suggest that informed preferences, where users are fully aware of privacy implications and make rational, deliberate choices, might serve as a more appropriate alignment target <ref type="bibr">[8,</ref><ref type="bibr">15,</ref><ref type="bibr">57]</ref>. However, eliciting informed preferences is inherently challenging because they are implicit and require users to be fully informed on privacy risks before making deliberate decisions. This process can place additional cognitive burdens on users, potentially reducing engagement or usability. Furthermore, privacy is not solely an individual concern, especially when individual privacy preferences conflict with those of others or broader societal expectations. Aligning GUI agents purely with individual preferences can still lead to harm, such as breaches of confidentiality, interpersonal privacy violations, or broader social risks. Therefore, we argue that evaluation goals should not be limited to either general privacy norms or individual privacy preferences. Instead, they should encompass a holistic assessment of privacy implications across all affected parties. Challenge 5 Evaluating GUI agents based solely on users' actual privacy behavior may reinforce privacy violations, requiring a more comprehensive assessment approach.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="7">Call for Actions</head><p>To ensure trustworthy deployment of GUI agents, we call for the following actions:</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="7.1">Human-Centered Evaluation for Privacy Risk Assessment</head><p>Unlike traditional GUI automation, GUI agents require in-context evaluations involving user oversight. The increasing complexity of systems and invisible backend data transmissions (Challenge 1) necessitate systematic privacy risk assessments across UI perception, intent generation, and action execution. Since end-users may lack the expertise to develop accurate mental models (Challenge 2), evaluation frameworks should enhance their ability to recognize and manage privacy risks. The multi-modal nature of GUI agent outputs also increases cognitive burden (Challenge 4), complicating oversight of automated workflows. Therefore, evaluations should assess unintended data exposure, ensuring transparency and minimizing oversight challenges. To prevent privacy violations from being reinforced by user behavior (Challenge 5), evaluations must proactively measure trust and satisfaction while systematically mitigating risks.</p><p>7.2 Enhancing Users' Privacy Awareness with In-Context Consent GUI agents should enhance privacy awareness through explicit warnings and in-context consent mechanisms. Since users may struggle to understand privacy risks (Challenge 2) and tend to overtrust AI (Challenge 3), agents must retrieve and process online privacy policies, providing contextualized explanations and actionable guidance. To prevent reinforcing privacy violations (Challenge 5), structured consent requests should precede privacy-sensitive actions-such as sending emails or conducting transactions-ensuring user control. Configurable privacy settings should allow users to balance automation convenience with data protection based on their needs.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="7.3">Integrating Privacy Measures into Agent Creation</head><p>Privacy safeguards must be embedded in both prompt-based and training-based GUI agent development. In prompt-based methods, data protection should be enforced through explicit instructions, limited data retention, and required user consent before accessing sensitive information. To counter overtrust in AI (Challenge 3), constraints such as restricting memory retention should mitigate unwarranted reliance. In training-based methods, privacy protections should be integrated throughout development: pre-training with privacy-focused datasets, fine-tuning to prevent breaches, and reinforcement learning to reward protective behaviors while penalizing unauthorized data exposure. These measures ensure privacy is a core design principle, fostering informed oversight rather than blind trust.</p></div><note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="1" xml:id="foot_0"><p>https://www.selenium.dev/documentation/</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="2" xml:id="foot_1"><p>https://www.autoitscript.com/site/autoit/documentation-localization/</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="3" xml:id="foot_2"><p>https://robotframework.org/robotframework/</p></note>
		</body>
		</text>
</TEI>
