<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>SemanticOn: Specifying Content-Based Semantic Conditions for Web Automation Programs</title></titleStmt>
			<publicationStmt>
				<publisher></publisher>
				<date>10/28/2022</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10386614</idno>
					<idno type="doi">10.1145/3526113.3545691</idno>
					<title level='j'>Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology</title>
<idno></idno>
<biblScope unit="volume"></biblScope>
<biblScope unit="issue"></biblScope>					

					<author>Kevin Pu</author><author>Rainey Fu</author><author>Rui Dong</author><author>Xinyu Wang</author><author>Yan Chen</author><author>Tovi Grossman</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[Figure 1: The workow of SemanticOn. There are three steps to creating a web automation program with semantic conditions using SemanticOn. (Step 1) To specify semantic conditions, users can either describe their intent in text (User Enters, c ) or indicate the section of interest by brushing through an image a or highlighting parts of a text b (System Suggests). SemanticOn then encodes these specications with computer vision and natural language processing techniques into web program conditions. (Step 2) To create the intended web automation program, users demonstrate the actions on the website using WebRobot, including image downloading d and text scraping e . (Step 3) Once the program is executed, users can also easily coordinate with SemanticOn to rene the semantic conditions ( f , h ) or take back control to add or remove data manually g .]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><p>Web automation oers a solution that leverages bots to mimic human interactions on web applications. It assists users with tedious and recurring tasks and has proven to be faster and more accurate for various task types compared to manual eort <ref type="bibr">[42]</ref>.</p><p>Past research has developed techniques to help users of all expertise levels to quickly and accurately create their intended web automation programs <ref type="bibr">[25-27, 44, 68]</ref>. However, these techniques are limited to creating programs with requirements at the website syntax or structural level (e.g., scraping the rst two items in each row of a table). Tools capable of creating logic based on the meaning of the content (semantics) remain unexplored. For instance, commercial tools such as iMacros <ref type="bibr">[2]</ref> and UiPath <ref type="bibr">[8]</ref> enable users to perform record-and-replay interactions for web automation and testing. Research tools such as Helena <ref type="bibr">[15]</ref> further this technique by lowering the learning curve, allowing users with little programming experience to create complex programs that can handle hierarchical data (e.g., tree-structured data) and distributed data spread across multiple websites.</p><p>We identied a need for web automation with semantic conditions through prior user studies <ref type="bibr">[24]</ref> and analysis of real user requests in online forms <ref type="bibr">[3,</ref><ref type="bibr">7]</ref>. This includes vision-related semantic conditions, such as scraping images that meet specic criteria (e.g., a photography student wants to study group interaction portraits on a gallery website with thousands of photos) or text-related semantic conditions, such as scraping text only when it expresses particular sentiments (e.g., a lm critic wants to evaluate positive reviews of a movie star's acting from dozens of news articles in a journal). With current techniques, users cannot specify these semantic intents in web automation programs. As noted above, semantic information often varies by content type, which makes it hard to design a universal interaction that is both easy to use and suciently expressive. Additionally, unlike other AI systems that provide results immediately after the provision of user inputs (e.g., chatbots), once executed, a web automation program will continuously output results as it iterates over web contents. This makes monitoring and error handling dicult, as the program may encounter unforeseen and problematic cases.</p><p>This paper explores interactive techniques to enable contentbased semantic condition specication for web automation programs. We introduce SemanticOn, 1 a system that allows users to specify, rene, and incorporate visual and textual semantic information as conditions in web automation programs via two methods: natural language description via prompts or detailed information highlighting with system support. We dene them as User Enters and System Suggests, respectively. SemanticOn combines the relative strengths of neural models (Transformer) for unstructured information and program synthesis techniques for web automation. By doing so, we introduce a new interaction paradigm for users to continuously add/rene semantic conditions in a programming-bydemonstration system. Specically, SemanticOn builds upon We-bRobot <ref type="bibr">[24]</ref>, a program synthesis system that enables users to create web automation programs by demonstrating actions on the target websites. WebRobot employs a no-code development approach that requires only web interactions in place of programming knowledge from users, which is consistent with our design goal. 1 SemanticOn is an acronym for semantic condition SemanticOn uses similarity-based computer vision and natural language processing techniques to encode these specications into web program conditions. <ref type="bibr">(</ref>Step 2) To create the intended web automation program, users will demonstrate actions on the website using WebRobot, including image downloading (Fig. <ref type="figure">1</ref>.d) and text scraping (Fig. <ref type="figure">1</ref>.e). <ref type="bibr">(</ref>Step 3) Once the program is executed, users can also easily coordinate with SemanticOn to rene the semantic conditions based on the automatically detected information (Fig. <ref type="figure">1</ref>.f, Fig. <ref type="figure">1</ref>.h) or reclaim control to manually add or remove data if the program has misjudged (Fig. <ref type="figure">1</ref>.g). To our knowledge, SemanticOn is the rst system to explore content-based semantic specication interactions for web automation programs.</p><p>We conducted a user study with 10 participants to evaluate SemanticOn's overall usability and eciency and to compare the semantic condition specication of each method (User Enters and System Suggests). We found that participants using SemanticOn successfully extracted 80.8% of the required data with an average time of 06:10 minute:second per task. The participants found that SemanticOn helped them eectively express their semantic intent by prompting them to consider their visual and textual perceptions of the tasks. We found a sense of control vs. eort trade-o, where participants enjoyed composing their conditions in User Enters but had to spend more time and mental eort devising a description to encapsulate the semantic condition. On the other hand, while participants could specify and rene conditions more easily via highlighting content details and selecting generated conditions in System Suggests, they had less freedom to express their intent when system suggestions were inaccurate.</p><p>In the nal section of this work, we analyze the human-AI collaboration workow in SemanticOn, discuss the implications of adding similarity-based models in a symbolic PBD system, and explore future work that can adapt our approach to other types of interactive AI systems that require semantic conditions. This work is an essential step towards the vision of natural, intent-unambiguous end-user programming with a focus on web automation creation. This paper makes the following contributions:</p><p>&#8226; The User Enters, System Suggests interaction designs, implementations, and evaluations that allow users to specify and demonstrate their intent during web automation creation, &#8226; The renement and error-handling techniques to clarify and improve semantic lters in a continuous human-AI collaboration process, &#8226; SemanticOn, along with a user study showing its usability and eectiveness in helping users specify semantic conditions for web automation programs.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2">RELATED WORK</head><p>SemanticOn builds on decades of web automation systems and innovations. In this section, we draw our design goals and guidance from three areas of work: web automation, programming-bydemonstration, and user intent specication and renement.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.1">Web Automation</head><p>Web automation is a software technique that leverages bots to perform tedious and recurring web tasks by mimicking human interactions, such as data entry and data extraction. Data scientists, UI testers, and clerks all use web automation to help complete their domain-specic tasks <ref type="bibr">[40,</ref><ref type="bibr">43,</ref><ref type="bibr">49,</ref><ref type="bibr">69]</ref>. Social scientists, for example, might want to develop web data scraping programs to collect necessary web datasets. UI testers might want to create an automated browser testing program to help developers nd frontend defects. Data workers might envision a data entry program for routine tasks like entering large amounts of data into a digital system (e.g., booking ights for all employees). Creating web automation programs is a non-trivial and complex task. Many web automation tools require users to have domain knowledge (e.g., understand the Document Object Model (DOM) structure) and programming experience. Commonly used tools like Puppeteer <ref type="bibr">[4]</ref>, Selenium <ref type="bibr">[6]</ref>, Scrapy <ref type="bibr">[5]</ref>, and Beautiful Soup [1] require users to learn code syntax, understand the task content architecture (e.g., DOM tree hierarchy), and have software testing experience. Prior work has shown that even for professional developers, creating automation programs is time-consuming. Krosnick and Oney studied the challenges of writing web macros using common web automation frameworks for experienced programmers <ref type="bibr">[35]</ref>. They found that a primary challenge for participants was the labor of checking syntactical element selectors to create their programs, which was inecient and prone to mistakes. In addition, the program might not generalize to cross-webpage selections where the elements don't have syntactic similarity. Our work enables users to specify the semantic meaning of their target content, bypassing the issues caused by implementation.</p><p>Researchers have developed many helpful tools to reduce the eort of program creation. For desktop application automation, systems like Sikuli <ref type="bibr">[76]</ref> allow users to identify a GUI element (e.g., an icon or a toolbar button) by taking its screenshot. Using computer vision techniques, it analyzes patterns in the screenshots to locate the appropriate elements when automating GUI interactions. Although this approach is promising, it requires programming knowledge and cannot disambiguate similar elements or text information. For UI testing, researchers have proposed and studied crowdsourcing and automated testing strategies to help increase the testing coverage and reduce the eort of creating programs <ref type="bibr">[21,</ref><ref type="bibr">23]</ref>. While helpful, outputs produced with these tools are hard to generalize to new UIs or contexts.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.2">Programming-by-Demonstration</head><p>To further reduce the expertise required, many tools have used a programming-by-demonstration (PBD) approach where users only have to interact with the target applications rather than writing code <ref type="bibr">[12,</ref><ref type="bibr">36,</ref><ref type="bibr">38]</ref>. These span a variety of application domains including text manipulation <ref type="bibr">[13,</ref><ref type="bibr">39,</ref><ref type="bibr">54,</ref><ref type="bibr">60,</ref><ref type="bibr">75]</ref>, image or video editing <ref type="bibr">[37,</ref><ref type="bibr">47,</ref><ref type="bibr">51]</ref>, and GUI synthesis <ref type="bibr">[55,</ref><ref type="bibr">57,</ref><ref type="bibr">61,</ref><ref type="bibr">71]</ref>. In the context of web applications, PBD delivers on this rst design requirement, oering web automation without requiring users to understand browser internals or manually reverse-engineer target pages. The PBD approach has produced great successes in the web automation domain, most notably CoScripter <ref type="bibr">[42]</ref>, Vegemite <ref type="bibr">[48]</ref>, Rousillon <ref type="bibr">[15]</ref>, and iMacros <ref type="bibr">[2]</ref>. Some of these systems require users to edit their traces to add parametrization. For instance, CoScripter and iMacros oer recordand-replay functionality; users record themselves interacting with the browser-clicking, entering text, and navigating between pagesand the tool writes a loop-free script that replays the recorded interaction. Because they lack support for control constructs and function composition, these systems require users to have logic skills. Other systems support iteration using program synthesis, automatically discovering loops given a demonstration of one or a few iterations. While less domain knowledge is needed, the synthesizer can make mistakes in which the user must provide more demonstrations or edit the DSL to correct it (e.g., Helena), which can be frustrating. SemanticOn instead allows users to eectively coordinate with PBD systems by smoothly switching agency and editing constraints at any time during program execution.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.3">User Intent Specication and Renement</head><p>User intent specication is an important and challenging component of human-AI collaboration. Ideally, users should be able to easily and naturally specify their intent to a system while understanding its states. However, given the limited capabilities of AI understanding techniques, high-level user intent can be dicult for systems to comprehend. Many systems have proposed bridging the gap between user intent and system understanding. For instance, PLOW <ref type="bibr">[9]</ref> and PUMICE <ref type="bibr">[46]</ref> allow users to express concepts (e.g., hot weather) in natural language and then learn the concepts to generalize the automation. Systems like Scout <ref type="bibr">[70]</ref>, Designscape <ref type="bibr">[64]</ref>, and Iconate <ref type="bibr">[80]</ref> allow users to iteratively rene their intent by directly manipulating the AI-generated artifacts. Other studies have shown that this renement interaction can even be delegated to crowd workers <ref type="bibr">[18]</ref>. Another work, APPINITE <ref type="bibr">[45]</ref>, also encapsulates user's intent in natural language instructions and claries the intention in a back-and-forth conversation with the AI. While these approaches are promising, user intents can involve visual and cognitive details such as identifying visual relationships in images or parsing texts to match a high-level idea. The user's semantic level intents are often not fully or accurately expressed through natural language or limited examples only, leading to information loss during communication and rendering the communication ineective <ref type="bibr">[19]</ref>.</p><p>Similar to PBD systems, programming-by-example (PBE) is another approach to facilitate program creation for various tasks such as data wrangling <ref type="bibr">[29,</ref><ref type="bibr">30,</ref><ref type="bibr">34]</ref> and data visualization <ref type="bibr">[52,</ref><ref type="bibr">72]</ref>. Many PBE and PBD systems require users to provide additional examples to disambiguate user intent. Falx allows users to specify visualization examples using a small amount of data and then infers and transforms the data to match the design <ref type="bibr">[73]</ref>. Sporq allows users to more accurately and quickly search code patterns in large codebases by prompting them to rene their intent by annotating a batch of negative examples and adding specic constraints <ref type="bibr">[58]</ref>. Other works enable users to directly annotate their input examples (augmented examples) to disambiguate user intent <ref type="bibr">[66,</ref><ref type="bibr">78]</ref>. Or they employ data visualization techniques to showcase the generated programs, allowing users to tweak the path of program generation in a tree view <ref type="bibr">[77]</ref>. While promising, providing additional examples increases users' cognitive demand. In this work, we focus on addressing the ambiguous semantic conditions and designing human-AI collaboration interaction solutions to help rene the constraints based on the content.</p><p>Using machine learning (ML) models to rene intent has been a recent focus in the eld of interactive ML. One common interactive ML approach allows users to oer feedback during the model training process for more eective ML model creation <ref type="bibr">[16]</ref>. Work by Cai et al. allows users to adjust the search algorithm iteratively with dierent types of similarities at dierent moments <ref type="bibr">[14]</ref>. Projects by Austin et al. and Jiang et al. allow users to interact with large language models to help rene their intent when writing code snippets <ref type="bibr">[11,</ref><ref type="bibr">33]</ref>. Work by Amershi et al. allows users to identify new friend groups on social media by analyzing the examples presented <ref type="bibr">[10]</ref>. Software developed by Fogarty et al. helps users to create their own rules to improve the search results <ref type="bibr">[28]</ref>. This research inspires our work, but instead, we focus on helping users rene their intent while interacting with continuous AI systemsweb automation programs that require constant monitoring and that eectively coordinate the turn-taking.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3">BACKGROUND AND DESIGN GOALS</head><p>Our work is built upon an existing web automation system, We-bRobot <ref type="bibr">[24]</ref>, that only uses web interactions and requires no programming knowledge from its users. This is consistent with our design goal. In addition to prior work, we derive our design goals from WebRobot's user study. In this section, we provide necessary background information on the WebRobot system and then discuss the design goals for our system SemanticOn. </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.1">The WebRobot System Workow</head><p>WebRobot is designed to facilitate web automation program creation. Figure <ref type="figure">2</ref> shows the WebRobot user interface. To create a web automation program for a data entry or scraping task, a user rst starts recording their actions (Fig. <ref type="figure">2</ref>.a). Then they can either upload a JSON le (Fig. <ref type="figure">2</ref>.b) if the task involves data entry, or they can choose an appropriate action (e.g., Scrape Text) in the action panel (Fig. <ref type="figure">2</ref>.c) and perform the required actions (e.g., clicking the desired text data on the website). After each scraping action, they will see the data appended to the output panel (Fig. <ref type="figure">2.d</ref>). Behind the scenes, WebRobot records every user action on the website and its associated action type (e.g., Scrape Text). After a few demonstrations, WebRobot synthesizes a program P from the trace A of demonstrated actions. In particular, WebRobot guarantees that P not only reproduces the demonstrated actions from A but also generalizes beyond A. This typically implies P would contain loops that can be used to automate the user-intended task. Finally, WebRobot executes P to automate the rest of the actions in the task.  </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2">WebRobot's Synthesis Algorithm</head><p>In a nutshell, WebRobot's synthesis algorithm (Algorithm 1) generalizes an input action trace A into a program P (with loops) by iteratively rewriting A to loops in P from the inside out. Initially, it creates a program P 0 with exactly those actions in A (line 1): while P 0 reproduces A, it does not generalize A (i.e., it does not produce new actions after A). Therefore, the algorithm performs iterative rewriting to gradually "compress" P 0 into more compact and general programs using a worklist algorithm (lines 2-8). The worklist W is initialized to have only P 0 , and we use P e to keep track of all programs that generalize A (line 2). Whenever W is not empty (line 3), the algorithm would remove a program P from W (line 4). It then checks to see whether P generalizes A; if so, P is added to P e (line 5). After this, in line 6, the algorithm tries to rewrite P into more general programs, which are stored in W 0 . The key idea underlying our R procedure is to perform semantic rewriting using a methodology called speculate-and-rewrite. Intuitively, it inspects P, identies repetitive patterns in P, hypothesizes potential loops that correspond to P, and nally synthesizes programs with one more level of loop. How WebRobot's speculative writing process works is beyond the scope of this work; we refer interested readers to the original WebRobot paper <ref type="bibr">[24]</ref> for details. Once P is rewritten to a new set of programs W 0 (line 6), the algorithm simply merges W 0 into W (line 7). The worklist loop terminates when no programs can be rewritten and it nally returns the smallest program in P e using a ranking function (line 8).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.3">User Feedback</head><p>In WebRobot's user study, participants reported that while We-bRobot can help lower barriers of entry for the creation of web automation programs and handling a more comprehensive range of tasks, they wished that they could express conditions to lter the content. For instance, one participant said, "maybe some conditional scraping [can be included], not based on whether the element exists in the webpage, but based on some other conditions." Consistently, we found posts on forums such as iMacros [3] and Stack Overow [7] that request the creation of web automation programs with content-based conditions. Participants also wished to rene their intent when interacting with the WebRobot system. For instance, participants reported that they wanted to "undo my wrong manipulations" or "edit my history. " However, as noted in the related work, WebRobot and other systems do not eectively support actions such as undo or history manipulation.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.4">Design Goals</head><p>Based on this prior work, we devised the following three design goals to help users easily create web automation programs with semantic conditions.  </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4">SEMANTICON</head><p>With the three design goals above, we created SemanticOn to help users specify, rene, and incorporate semantic conditions in automated web data scraping. Figure <ref type="figure">3</ref> shows SemanticOn's system architecture and main user interactions at a high level. Instead of writing web macros from scratch for each website and ltering the scraped content in post-processing, users can interact with Seman-ticOn to compose semantic conditions on the content they want to scrape (Step 1 Fig. <ref type="figure">3</ref>), then demonstrate actions on the target website to synthesize automation programs for dierent websites without writing a single line of code (Step 2 Fig. <ref type="figure">3</ref>). Throughout this process, users can communicate with SemanticOn, which is capable of parsing text and image content through machine learning models. The users and SemanticOn work together to rene the condition set and repair errors in result selection, continuously improving both the system's and user's understanding of the lter criteria (Step 3 in Fig. <ref type="figure">3</ref>). In this section, we rst illustrate SemanticOn's user experience with a sample scenario that embodies common semantic conditions. We then detail the design and implementation of SemanticOn.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.1">The SemanticOn User Experience</head><p>Mia, an outdoor enthusiast, wants to extract online information about travel destinations where outdoor activities are available. To help her make an informed decision, Mia wants to scrape the text description and the image for each location from an article to build potential itineraries. One option is to read through every paragraph, look at each picture, and manually copy and paste the relevant information, but that process would be tedious and repetitive. On the other hand, Mia could write a web scraping script using Python. She has some coding experience, but writing a script and ltering the results based on her preference would also be time-consuming and laborious. Instead, Mia uses SemanticOn to eciently demonstrate her conditions and web actions and synthesize a web automation program that completes the task for her.</p><p>To begin, Mia sets the semantic conditions for the intended content (Step 1 Fig. <ref type="figure">3</ref>). She rst clicks "Text Condition". She then selects User Enters (Fig. <ref type="figure">4</ref>.d) to specify the semantic condition in her own words. Mia represents her high-level requirement in the system prompt by typing, "This is a great location for outdoor activities" (Fig. <ref type="figure">6</ref>.c). She believes this sentence is likely semantically similar to the relevant content in this article. After clicking "Add", the condition is appended to the text condition table (Fig. <ref type="figure">4</ref>.h) in the condition panel. Furthermore, Mia decides to add a condition to the corresponding destination image. She wants to travel to a place where hiking and water activities are accessible. To that end, she uses System Suggests, clicks on an ideal image, and highlights the mountain and lake in the picture (Fig. <ref type="figure">6</ref>.a). The system detects several objects and summarizes the image content into a sentence. Mia also adds the relevant objects and caption to the corresponding tables (Fig. <ref type="figure">4</ref>.f,g) in the condition panel.</p><p>After specifying two initial conditions, Mia decides to start the demonstration process (Step 2, Fig. <ref type="figure">3</ref>). She clicks the "Start Recording" button (Fig. <ref type="figure">4</ref>.a) to start the web macro recording for program synthesis. Then, Mia species the task name as "Travel Destination Search" and sets the column number to 2, one for text descriptions and one for the associated images.</p><p>Next, Mia selects "Download Image" and hovers the mouse to highlight the desired image element (Fig. <ref type="figure">4</ref>.j). As she clicks on the element, the image is downloaded and put into the rst column A unique table displays image object conditions f , image description conditions g , and text description conditions h . The user may pause i at any time to make changes that will not aect automation. To interact with the web page and demonstrate actions for program synthesis, the user will select their action type by choosing an option in the action panel j . The system will predict the next action, shown in the temporary row k . If the semantic conditions are met, the temporary row will be appended to the output panel l . To learn more about how the content match with the conditions, the user may click on the information icon m . of the Temporary Row (Fig. <ref type="figure">4</ref>.k). SemanticOn evaluates conditions associated with each column before the row is added to the output panel or skipped. However, at the demonstration stage, the condition will not be triggered, as we assume users will only demonstrate macro actions on the content they want to include. Mia then selects the "Scrape Text" action and repeats the steps for the destination description. As both columns in the temporary row are lled, the entire row is added to the output panel (Fig. <ref type="figure">4</ref>.l). Mia repeats her demonstration for a second destination that ts her requirement. After two rows of user demonstration, WebRobot can synthesize the web automation program and predict the user's next step, so the system now enters a guided semi-automation mode. SemanticOn scrolls into the next row of the destination and highlights the next element to be extracted. For the image, a prompt states that the generated caption matched with the initial image condition "a lake beneath the mountain range". Mia conrms this selection and lets SemanticOn continue. However, Mia nds that her initial text condition is too general, as the system informs her the text condition does not match the current text and presents key phrases from this element to be potentially used as renement (Fig. <ref type="figure">5</ref>). She realizes that "scenic hiking trails" would be a good textual condition to include. She also discovers "pet travel" as a key phrase. Mia would love to take her dog Sushi on the trip, so she also selects that key phrase. After she clicks "Finish, " those two phrases are added to the text condition table, and the highlighted text is considered accepted.</p><p>After inspecting several elements in semi-automation mode and rening the condition set with system prompts, Mia feels satised with the system's interpretation of her requirements. She clicks on the "Hide Prompt" button (Fig. <ref type="figure">4</ref>.b) to allow the system to automatically predict and lter the rest of the web page content without stopping for user conrmation. Mia sits back and waits for the program to nish. However, she notices that for one destination, "Grand Canyon, USA," the system lters the image as it does not match "a lake beneath the mountain range". However, she actually wants to include this destination in her list, as she can still engage in outdoor activities such as hiking and kayaking. To repair this error,  Then, a prompt a containing detected objects within that region, and a general description of the image will be shown. Similarly, the user may highlight sections of text, and the system will prompt b important key phrases within the highlighted portion. Users may manually enter conditions by clicking on an image or section of text and entering their conditions c .</p><p>Mia pauses the automation process (Fig. <ref type="figure">4</ref>.i, 7.b). She then manually scrapes the text description and downloads the image using the action panel (Fig. <ref type="figure">4</ref>.j). It's worth noting that during the pause, Mia's web macros are not used for program synthesis and therefore do not aect the current automation program. Mia also adds another image condition using User Enters, specifying "a canyon valley with river" as another instance of her ideal destination. While the program is paused, Mia scrolls through, evaluates the output panel, and can delete output rows that were added by system mistake or modify/delete conditions during the pause (Fig. <ref type="figure">7</ref>.d). After she is satised with the rened condition set and the current results, Mia clicks "Resume" to continue the automation. The program iterates through the rest of the web page and collects ltered data based on Mia's semantic conditions. After the program ends, Mia stops the web macro recording, and the collected results are exported as a JSON le. Mia uses SemanticOn to create a tailored web automation program without writing any code. She also obtains a high-accuracy result set of her ideal travel destinations by continuously working with SemanticOn to clarify and rene conditions and repair errors.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.2">Design and Implementation</head><p>We implemented SemanticOn as a Chrome browser extension, incorporating the core program synthesis engine from an existing system, WebRobot. Primarily, it uses plain JavaScript for the front-end interactions. For semantic condition comprehension, we adopted two o-the-shelf Transformer models.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>4.2.1</head><p>Step 1: Specify Semantic Conditions. To enable content-based semantic conditions specication (DG1), SemanticOn allows users to add condition criteria to specic content on the target website. Currently, the system supports conditions on text and image content (Fig. <ref type="figure">4</ref>.c). The user can choose one of the two methods of specication: 1) User Enters, where the user composes the condition using analyze and provide suggested conditions (Fig. <ref type="figure">4</ref>.d). Upon selecting the type of condition and the specication method, the user chooses an element on the web page as the basis of the semantic condition.</p><p>Inspired by recent prompt-based interactions <ref type="bibr">[33,</ref><ref type="bibr">50,</ref><ref type="bibr">79]</ref>, a prompt displays next to the selected text or image element for User Enters, encouraging the user to enter a semantic description of their search criteria (Fig. <ref type="figure">6</ref>.c). This description, along with the context of the text or image element, is added to the conditional panel on SemanticOn (Fig. <ref type="figure">4</ref>.f,g,h). In comparison, for System Suggests, the user needs to highlight the crucial part of the content to set the condition on.</p><p>For images, the user brushes over an area of interest with their mouse (Fig. <ref type="figure">6</ref>.a) to illustrate. The image is fed through an o-theshelf pre-trained multi-layer Transformer model that learns to align image-level tags with their corresponding image region features <ref type="bibr">[32]</ref>. The model detects objects for the illustrated section and generates a caption for the entire image. For texts, the user highlights relevant phrases or sentences with their mouse (Fig. <ref type="figure">6</ref>.b). Similarly, the text is processed by an o-the-shelf unsupervised language model where the noun phrases in the input text are rst detected and then ranked based on frequency and co-occurrence. The model generates key phrases in the passage highlighted by the user. The user can then pick any object, caption, or phrase tags to add to the condition table (Fig. <ref type="figure">4</ref>.f,g,h). We used these models through the Microsoft Azure Cognitive Services and Cloud platform. <ref type="foot">2</ref> Guided by the design of other systems <ref type="bibr">[17,</ref><ref type="bibr">31]</ref>, we also implemented edit, delete, and logical operations for multiple condition specications, allowing the user to modify, remove, or set the AND/OR logic switch on the conditions (Fig. <ref type="figure">4</ref>.e). They can also apply the condition to one or all of the output columns.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>4.2.2</head><p>Step 2: Demonstrate Actions and Automate. Once the user sets initial conditions, they can start the demonstration process by clicking "Start Recording" (Fig. <ref type="figure">4</ref>.a). This initializes the WebRobot natural language, or 2) System Suggests, a novel specication technique where the user highlights relevant content for SemanticOn to  After the user demonstrates their intended pattern of actions twice, WebRobot generates an automation program and predicts the next action. SemanticOn also enters a guided semi-automation mode. That is, for every data extraction action predicted, the system parses the content through machine learning models. A new image is considered matching with the current conditions if 1) an object specied in the image condition is detected (e.g. user specied a dog, and the new image consists of a human petting a dog), or 2) the new caption yields a similarity score above a certain threshold when compared with captions specied in the image condition. The image caption and text content similarity thresholds were set to 0.5 and 0.4, respectively, based on our benchmark testing. A new text is considered matching if it yields a similarity score above a threshold when compared with specied text conditions. We used a Sentence-Transformer to calculate the semantic similarity between captions and texts <ref type="bibr">[67]</ref>. The model utilizes Sentence-BERT (SBERT), a pretrained network that derives semantically meaningful sentence embeddings that can be compared using cosine-similarity. To avoid the low-score comparison between an entire paragraph and a key phrase, we chunk the text content and conditions into sentences and see if any pair results in a high similarity score.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>4.2.3</head><p>Step 3: Refine, Repair, and Coordinate. To help users easily rene their intent (DG2) and repair mistakes (DG3), we adopted the human-in-the-loop approach <ref type="bibr">[24,</ref><ref type="bibr">59,</ref><ref type="bibr">63]</ref> and introduced a semiautomation mode to ease the transition between manual demonstration and full system automation. First, to repair unexpected results or conditions, SemanticOn oers users a program pause function, which is important and unique in continuous AI systems. In the semi-automation mode, if new text or image content matches with the condition set, the user is notied by a prompt informing them which part of the content was matched. In contrast, if the new content is ambiguous or does not match with the condition set, the user is prompted with a set of suggested conditions (objects and captions for images, key phrases for text) generated by ML model processing (Fig. <ref type="figure">5</ref>). The user can clarify their intent by selecting suggested conditions to append to the condition set. The new content will be accepted and added to the output results. Alternatively, the user can reject the system's prediction by clicking "Finish", in which case the condition set remains unchanged, and the new content is discarded. The semi-automation mode guides the user through condition specication and renement. The user might begin with a high-level idea of their search query but can rene and adjust it upon seeing SemanticOn's interpretation of the result. The user can add new conditions to cover unforeseen cases or to modify/delete vague or over-specied conditions that lter results in an unintended way.</p><p>After going through predicted content, administering decisions, and rening conditions, the user might feel satised with the set of conditions. In that case, the user can choose to hide system prompts (Fig. <ref type="figure">4</ref>.b) for rejected content and enter full automation. In this mode, SemanticOn continuously executes the predicted web macros and assesses each new piece of content, outputting the information accordingly. This is done by adding a condition evaluation step after executing every macro before extracting the data. However, when the user detects a ltering error, they can use the "Pause Automation" button to reclaim manual control (Fig. <ref type="figure">7</ref>.b). When the system is paused, the user is free to modify the output result table. For each assessed piece of output, an information icon is displayed at the end of the table cell (Fig. <ref type="figure">4</ref>.m). Once clicked, it will expand and display the ML processing results for that content. The user can repair system-made errors by deleting rows of ambiguous results accepted by the system but misaligned with the user's mental model (Fig. <ref type="figure">7</ref>.d). They can also manually use the action panel to add content rejected by the system but matches the user's needs. In addition, the user can learn from the ltering results to add new conditions they omitted or under-specied. Likewise, condition editing and deletion are also available during the pause (Fig. <ref type="figure">7</ref>.c). This way, the user can feel condent about the automatic selection of results, as they always have the power to reclaim manual control, repair system mistakes, and clarify their intents. Note that during pause, none of these steps are recorded for program synthesis and do not aect the automation program.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5">SYSTEM EVALUATION</head><p>We conducted an in-person user study to evaluate SemanticOn's usability and compare the two methods for condition specications. This evaluation was guided by the usage evaluation in the HCI toolkit evaluation strategy classication <ref type="bibr">[41]</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.1">Participants</head><p>We recruited 10 people (5F5M, mean age 24.3, mean coding experience 4.6 years) at a large public university. Participants are denoted as P1-P10 in subsequent sections. Seven of the participants were graduate students, and three were undergraduates. Five participants have written web automation programs or used commercial tools to some extent. The participants were contacted by email to participate in a study where they would interact with computer software to create web automation programs.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.2">Study Design</head><p>After signing our consent form, each participant rst watched a tutorial video of SemanticOn's interface and features. Then participants performed ve tasks using SemanticOn. For each task, they were given a task sheet with a semantic conditional intent. Similar to the posted tasks in online forums, the descriptions were intentionally vague, so the user could not copy the instructions verbatim and would need to formalize their own idea of how to specify and rene the conditions. The participants could request the experimenter's assistance at any time during the session. After the participants completed the tasks, we conducted a short interview with them regarding their experience. Additionally, they lled out a short survey with Likert scale and short-answer questions when they exited. The participants were compensated $25 for their time. Each session took 60-75 minutes and was conducted in person on our machine. All sessions were screen-and audio-recorded. Our study is approved by the IRB at our institution.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.3">Tasks</head><p>Through an analysis of online web scraping request posts from iMacros Forum [3], and Stack Overow [7], we designed ve tasks that represent the challenge of creating web automation programs with semantic conditions. The Appendix (Fig 8 ) shows the details of the user study tasks. To evaluate the usability and utility of the system and compare the eectiveness of two intent specication methods, we designed the tasks with the following goals in mind: 1) the web content to be scraped should be realistic and easily understood by participants, 2) the tasks should extract information based on text and image content, 3) the condition criteria and the corresponding results should contain some ambiguity to allow room for renement, and 4) the tasks should help SemanticOn to demonstrate the system's full capabilities. To achieve this goal, we adopted and piloted ve web scraping requests by real users on online forums, two based on image-heavy websites (pet wallpapers<ref type="foot">foot_1</ref> , street photography<ref type="foot">foot_2</ref> ), two with text-heavy content (movie star biographies <ref type="foot">5</ref> , list of novels <ref type="foot">6</ref> ), and one task with a variety of both image and text content (beautiful places in the world<ref type="foot">foot_5</ref> ). Participants needed to extract only images or only text content for the rst four tasks. In addition, they were randomly assigned to use User Enters and System Suggests as their specication method for the rst four tasks. We counterbalanced the ordering of image versus text tasks and the assigned specication method to eliminate sequence and learning eects. To standardize task diculty, participants needed to extract the top 15 text passages and/or images from the website based on a semantic condition. For the nal task, participants needed to scrape both images and text in two separate columns and were free to use both condition specication methods. Task 5 served as an exploration task for participants to evaluate and compare the two specication methods and was not constrained by a result set or time. Participants engaged in this task until the end of the study. Therefore, Task 5 was not included in the quantitative analysis for accuracy and duration in the later sections.</p><p>We designed the conditions based on the content itself and established a ground truth result set that was compared with the participant's result to measure accuracy. The conditions were designed so that in the ground truth set, the number of results passing and failing the condition across all tasks was approximately the same (31 passes, 29 fails).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.4">Results</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.4.1">Time and accuracy.</head><p>The user study recorded 40 scenarios (10 participants x 4 tasks); one instance was discarded from analysis due to a recording issue, resulting in 39 total task completions. Table <ref type="table">1</ref> lists the average time (in minute:second) each participant spent and the accuracy (percentage of correctly included and excluded results) on each task. The overall average duration is 06:10 (image tasks mean=05:55, text tasks mean=06:25), and the overall average task accuracy is 80.7% (image tasks mean = 83.0%, text tasks mean = 78.5%). We could not identify statistical signicance in the dierence in accuracy and duration across image and text tasks.</p><p>We also analyzed the data comparing usages of User Enters versus System Suggests. The average duration was 06:22 for User Enters tasks and 05:57 for System Suggests tasks. In terms of mean accuracy, participants achieved 83.0% for User Enters tasks and 78.5% for System Suggests tasks. Again, the dierences are unable to be identied as statistically signicant.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.4.2">Overall eectiveness in intent specification.</head><p>In the exit survey, participants rated the ease of use of SemanticOn overall, the ease of use of each specication method, their mental eort, and their trust in the system. Table <ref type="table">2</ref> displays the average score for Likert scale questions on SemanticOn's usability. On a scale of 1 (strongly disagree, very negative) to 7 (strongly agree, very positive), participants believed their specied conditions were displayed in an easily understandable way (mean = 6.0, SD = 1.1) and that the system-generated prompts in semi-automation mode helped them rene their intent (mean = 5.6, SD = 0.97). However, participants had polarized opinions on the statement "it was easy to coordinate (claim controls, pause/resume, show/hide prompts) with SemanticOn to repair under-or over-specications, " with 5 participants agreeing and 3 participants disagreeing (mean = 4.7, SD = 1.7). Overall, the users widely accepted the semi-automation workow towards automation and the opportunity to adjust conditions later on by pausing. 9 out of 10 participants added renement conditions for Tasks 1-4, and all participants utilized the system prompts in semiautomation mode to rene their conditions in Task 5.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.4.3">Error-handling Analysis.</head><p>As per DG3, SemanticOn oers error-handling techniques for users during full automation. We measured participants' usage of three error repair methods: condition deletion, result deletion, and manual result addition. Most users opted to use the pause and repair functionalities, as 7 participants deleted their specied conditions due to inaccuracies, 6 participants removed undesired output to repair system mistakes, and 2 participants manually added desired contents missed by the lter. The utilization of the error-handling features varied based on the task type. For condition deletion, we found 16 instances across image tasks and 0 instances for text tasks. Similarly, there were 11 instances of result deletion for image tasks and 2 for text tasks. As for manual result addition, we found 3 instances for image tasks and none for text tasks. This dierence can be related to the content processing speeds for visual versus textual materials. The condition specication method also plays a role in the usage of error-handling features. We observed 4 instances of condition deletions for User Enters and 12 for System Suggests. In addition, we found 4 result deletion instances for User Enters and 9 for System Suggests. This can be attributed to the dierence in the required eort for each workow. We expand on these varieties in our Discussion.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.4.4">Comparison between User Enters and System</head><p>Suggests. Regarding the dierence between the two specication methods, participants found both experiences comparable but thought System Suggest was easier to use. P10 said, "[My preference] really depends on the task, but I really liked the System Suggests mode, especially for words it was really easy to use and accurate." When assessing the overall experience to specify conditions, participants rated User Enters slightly higher (mean = 5.3, SD = 1.1) than System Suggests (mean = 5.1, SD = 1.2). But when evaluating the ease of use for each interaction, users rated entering their own natural language descriptions lower (mean = 5.1, SD = 1.1) than brushing, highlighting, or choosing system-suggested conditions (mean = 5.7, SD = 0.82). We were unable to identify statistical signicance in these dierences.</p><p>However, when measuring the average number of initial conditions set before the demonstration, participants specied an average of 2.6 conditions for User Enters and 4.1 conditions for System Suggests (p &lt; 0.05). This points to a dierence in ease-of-use and conditiongeneration capability between the two methods, which is further explored in Discussion. </p><p>Table <ref type="table">2</ref>: Survey Responses. For section one (top), 1 is very negative, and 7 is very positive. For section two (bottom), 1 is very high mental demand, eort, insecurity and stress, and feelings of being hurried and rushed.</p><p>5.4.5 Mental eort and AI system trust. Participants also reported on their mental eort in completing the tasks and their trust in the AI system in the survey. Participants generally reported medium to high mental eort using SemanticOn, especially at the start of the study when they had to familiarize themselves with the interface and remember the workow (P2, P5, P6, P7). On a scale of 1 (very demanding, very hard, not successful) to 7 (not demanding, not hard at all, very successful), participants experienced medium mental demand (mean = 4.6, SD = 1.8) and medium eort (mean = 4.9, SD = 1.4). They believed they were relatively successful in accomplishing the tasks (mean = 5.0, SD = 0.94).</p><p>We also found that the participants exhibited a medium level of trust (mean = 4.5, SD = 1.6) towards the AI system's prediction in full automation. Six users were comfortable entering full automation mode for at least one task. However, P3 and P7 expressed very low trust in the automated prediction (both rated 2 out of 7 in survey). During the interview, some participants (P5, P9) also commented that their trust in the system depended on the content type and the number of renements required for each task. For example, P5 mentioned they "trust this picture task a bit more...[because] text has more group[ing]s and potential variations" when they used 11 text renements and only 1 image renement. We analyze the eect of content type on users' perception of the task and our system in the Discussion.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6">DISCUSSION</head><p>Based on the evaluation, we present analysis of the human-AI collaboration techniques and the role of each agent in SemanticOn. We also discuss the implications of adding similarity-based machine learning models in symbolic PBD systems. In addition, we report ndings on users' trade-os in using User Enters and System Suggests, as well as the intent specication eectiveness and system limitations to provide insights on future designs.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.1">Human-AI Collaboration</head><p>The user workow of SemanticOn consists of both a human and an articial intelligence agent; they exchange control in dierent parts of the interaction and collaborate to build a complete and accurate web automation program. In SemanticOn, we utilize machine learning models' classication eciency to rapidly process texts and images, summarize the content, and provide suggested semantic lters to the user. This saves the user's mental eort signicantly, as it is no longer the human's role to parse content and identify an inclusion or exclusion decision. However, it also introduces a new source of error to web automation, as similarity-based models can mislabel inputs and result in false positive and false negative data. This is why human is always involved in the continuous human-AI collaboration in SemanticOn. After a couple of demonstrations, the user is prompted with potential semantic conditions for each processed content in the semi-automation mode. Additionally, even when the user feels satised with the set of conditions, the program can still be paused at any time if a mistake is spotted or the user wants to edit the conditions.</p><p>From the evaluation of this interaction paradigm, most participants (7/10) reported that the workow's path from demonstration to guided semi-automation to full automation was intuitive and useful. The semi-automation phase allowed users to calibrate conditions with SemanticOn and better understand the AI system's interpretation of the conditions. This is important as users might not have had a well-dened semantic condition at the program's start. P4 believed that the "human brain still needs to think about [how to describe the condition]...and link to all the keywords. These words might not specically come to mind to the human. " In this case, the initial conditions may not include all of the user's needs, resulting in low lter accuracy. The semi-automation mode provides aid for this. For example, P3 mentioned that even if the results were not accurate at the start, "I can interactively try to make [automated prediction] better as it goes. " P4 also mentioned that "the transitional period to trust the process is good, saves a lot of fuss. "</p><p>However, the steps to rene conditions and results through system prompts and automation pauses could require too much user eort and become time-consuming. P5 commented that the entire user experience was "a little slow...I think the ne-tuning part is slow. Once you start [full automation], then it's ne." P2 reported that "it took me multiple times to gure out the sequence of the buttons. " When they spotted system mistakes, P1 and P4 decided not to pause and amend the errors because it took too much eort. They believed the misclassied content was not detrimental to the automation task, as there was no requirement for accuracy. While supplying users with multiple renement tools enhances robustness, the system must be cautious not to overload users with interaction techniques and interruptions.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.2">Similarity-based Model in PBD Systems</head><p>A main contribution of SemanticOn is introducing a new interaction paradigm for users to continuously add/rene semantic conditions in programming-by-demonstration (PBD) systems. In particular, it enables users to express intent via similarity-based machine learning models. This specication is at a higher abstraction level than pure symbolic systems based on DOM structure. Here, we discuss the advantages and disadvantages of expressing intent using similarity-based statistical models versus pure symbolic systems, the implications of adding statistical models in PBD systems, and the generalizability and viability of SemanticOn against errors.</p><p>Traditional symbolic automation systems like WebRobot are robust at understanding specic user actions and generalizing them into repeatable steps in a synthesized program. This is achieved by iterating over the elements in the DOM structure of the websites and generalizing the pattern. Meanwhile, statistical machine learning models are useful at parsing high-level user intent and matching unstructured, but semantically similar concepts, for example extracting key entities from a natural language input. However, both systems alone have their limitations. Similarity-based models lack reasoning capability. The machine learning models we employ classify the inclusion and exclusion of content based on semantic similarity, but they cannot derive further actions based on specic criteria. For example, the models alone could not construct a program that repeatedly takes in an image URL, searches it, and downloads the image based on semantic conditions. It is ecient in the last ltering step but ineective when structural elements need to be generalized. In contrast, symbolic models that reason about a set of user instructions over a certain website structure might fail to generalize if the query is unstructured or outside the scope of the task domain, while similarity models can cover a wider range of input content that are not constrained by the structure. SemanticOn combines these two techniques in an eective way that seals the gap between users' semantic intent and web automation.</p><p>From a practical standpoint, training an intelligent statistical model for the tasks we cover in SemanticOn's user evaluation would be eortful. Systems like Calendar.help <ref type="bibr">[22]</ref> require many expert heuristics and even human workers to automate a very specic task in event scheduling across dierent parties. In SemanticOn, users compromise some eort by doing two rounds of demonstrations of their desired web actions. Still, the automation tasks can be generalized to a diverse set of data on many dierent websites. Introducing similarity-based models in symbolic systems does present a new source of error as the models can lter content incorrectly, in addition to program synthesis errors in pure-symbolic systems. To amend this, SemanticOn provides continuous condition renement and output edit/delete options so that users can repair classication errors easily and yield more accurate results.</p><p>Our work does not rely on a specic program synthesizer to generate a web automation program or a specic machine learning model to lter content semantically. The novelty of SemanticOn is the set of condition specication, renement, and error-correction interactions, which can be generalized to other PBD systems.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.3">User Enters and System Suggests</head><p>One emphasis of this project is to compare the usability of the two semantic condition specication methods. To summarize the ndings, participants shifted their preference between System Suggests and User Enters based on two factors: perceived eort and the type of content presented. When participants expressed their preferences and the rationale behind using each specication method, we also discovered a trade-o between their sense of control and the perceived eort.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>6.</head><p>3.1 Perceived eort based on content types. Perceived eort plays an important role in user preference. When it was easy to summarize the target content on which users needed to set conditions, participants preferred to use User Enters to express their intent instead of selecting System Suggests, which could be too broad or too specic. P3 remarked that "[User Enters makes] you feel like [you have] more control because you are looking for a specic thing. "</p><p>On the contrary, when the target content was information-heavy and hard to encapsulate in a short natural language description, participants preferred to use System Suggests to highlight the details and let SemanticOn present potential conditions. P6 commented that "I do like the possibility of being able to say this is what I see in the image [based on System Suggests]... and [highlight] a part of the image that I think is important. "</p><p>When participants were free to use both User Enters and System Suggests (i.e. Task5 in Appendix Fig <ref type="figure">8</ref>), we found that text content inherently took more eort to process, and 7 out of 10 participants opted to use System Suggests to avoid reading a large block of text. In contrast, despite containing a variety of objects and entity relationships, the corresponding image content could be encoded and summarized more quickly, and 9 out of 10 participants chose to specify their intents with User Enters.</p><p>This preference based on content type persisted during condition renement in semi-automation and full automation modes. Participants expressed diculty in parsing textual information rapidly, as they could not distinguish whether the system made the correct selection. P3 said that "text task in general is much more dicult," and P5 commented, "when it comes to text task, I'm a little lazy to read through all this... so system can just get me relevant keywords. " Our observations support this as users were far more likely to repair errors for image tasks than text tasks by deleting inaccurate conditions and incorrect results. Participants can quickly identify whether an image ts the condition set, but they can not parse through a block of text as the automation quickly processes each row of content. Based on this nding, future designs should account for the processing eort for text content and provide users more time and aid in understanding and summarizing the information.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.3.2">Sense of Control and mental eort trade-o.</head><p>Another key factor for specication preference was the sense of control. Three participants (P3, P7, P10) listed a better sense of control as to why they preferred User Enters, where they could tailor the condition using their terms. In addition, in cases where System Suggests suggestions were inaccurate (e.g., failing to detect relevant text entities or generate image captions at the appropriate granularity), some participants (P2, P4, P6, P10) switched to User Enters.</p><p>However, there was a trade-o between the sense of control and the required eort. Participants rated a lower ease-of-use score on average for User Enters (5.1 versus 5.7 for System Suggests), despite it having fewer interaction steps. Three participants (P4, P5, P10) pointed out that they preferred System Suggests due to the convenience and ease of mental eort. System Suggests is also advantageous in reporting details that escaped the user's attention. For example, P4 pointed out that they preferred System Suggests mode because it sometimes captured things that the user failed to recognize or think of. Additionally, P7 mentioned that they "quite like System Suggests more, [because it is] more systematic and highlights specic things." Participants could avoid cognitive overload by handing the processing labor to SemanticOn through System Suggests, but they were limited to the generated set of conditions for each item. This trade-o also aects participants' usage of error handling features during automation. With the increased cognitive load in User Enters, participants are much less likely to pause and repair their mistakes than when using System Suggests. However, this could be explained by participants adopting less accurate prompts from System Suggests, which leads to more errors overall for users to x. Future work could potentially create a combined approach where users can edit system-suggested conditions, yielding a sense of control and avoiding heavy user eorts.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.4">System Eectiveness</head><p>Most participants (8/10) agreed that SemanticOn is an eective tool for web automation and that the condition specication aspect is useful. Compared with manual eort and traditional data scraping methods such as writing automation scripts, participants found Se-manticOn's no-code solution novel and easier to use. P5 mentioned that in manual scraping scenarios, "Control-F would only get you so far" and that the system is "much easier than writing your own script...[and] more functionalit[ies]" In addition, P7 thought this tool "would be super helpful for someone not comfortable with code. [It provides] low eort but maximum reward. "</p><p>More than half of the participants (6/10) believed that Seman-ticOn could be applied to realistic tasks. However, some focused on the conditional selection aspect, and others emphasized on the complexity of the task. For example, P2 commented that the system is "very good for selective tasks [when you] only want a few images from a lot of them. " Similarly, P5 suggested that SemanticOn could be used for websites without robust ltering, search engine, or categories because "it could easily make any website sort-able. " On the other hand, P1 enjoyed using this tool on Task 5 with both image and text conditions combined and proposed that "this type of tool would benet... power users [for tasks] like creating a dataset... <ref type="bibr">[and]</ref> dealing with complex things." These reports provide evidence for the need of no-code solutions in conditional web automation tasks.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.5">System Limitations</head><p>There are several limitations to our system. First is the eciency of coordination. As mentioned, some participants found the renement process complicated and slow and were reluctant to utilize the functionality to increase system accuracy. In future designs, the researchers could simplify the user interface and the interaction steps and condense the system prompt information to reduce cognitive load.</p><p>Another limitation is the relatively low accuracy of the machine learning models it relies on to classify text and image content. Partially, this relates to a design decision we made when building SemanticOn. We rst used two state-of-the-art models-OFA <ref type="bibr">[74]</ref>, which uses a sequence-to-sequence learning framework, and Im-ageAI, which uses a convolutional neural network <ref type="bibr">[56]</ref>, for image captioning and object detection, and we were able to achieve high accuracy. However, the image processing was computation-heavy and took more than 30 seconds per image on an o-the-shelf laptop. To reduce the gulf of evaluation <ref type="bibr">[62]</ref>, we switched to the Microsoft Azure computer vision model <ref type="bibr">[32]</ref> for near-instant cloud processing but with lower accuracy and detail level for some content domains. Similarly, for key phrase extraction, we employed the Microsoft Azure language model, sending text contents via REST API calls for near-instant processing. But the extracted key phrases often have a low level of abstraction (i.e. extracting entities and nouns in a sentence without indication of interaction) and could not provide high-level descriptions like those used for image captions.</p><p>Additionally, SemanticOn builds upon an existing symbolic program synthesis system, WebRobot, which has its own constraints and requirements. And the introduction of similarity-based machine learning models also presents ambiguity to the source of error between incorrect program synthesis steps and result misclassications. The user might be confused when encountering unexpected behavior, as it could be a product of a user demonstration error or the system's misunderstanding of user intents, which requires a dierent workow to repair. However, in our user study, we emphasize repairing errors from machine learning model mislabelling as that is the novel part of SemanticOn.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.6">Future Work</head><p>We summarized three points of feedback that are relevant for future works. First, we found that participants spent a longer time encoding and processing texts than images. This is coherent with previous studies on human capability in consuming dierent types of information <ref type="bibr">[53]</ref>. But, this nding is signicant in full automation, when the system rapidly loops through web page content, giving users little time to identify potential selection errors. Future works could enable multi-modal interaction (e.g., voice) to reduce the eort of information processing <ref type="bibr">[20]</ref>. Additionally, one can provide more indicators for users to quickly recognize the information represented in text elements (e.g. highlighting matching key phrases) and decide whether the content should be included based on the semantic condition.</p><p>Second, participants separately favored one of the two condition specication methods, User Enters and System Suggests, depending on the content type and the amount of information they needed to process. Similar to the neurosymbolic program synthesis approach <ref type="bibr">[16,</ref><ref type="bibr">65]</ref>, future works could create a unied technique where users could benet from the instant and comprehensive results from machine learning models while preserving users' power to specify conditions on their own terms.</p><p>Lastly, future designs could further improve the system usability by reducing the mental eort. The dual process of collaborating with an AI on both demonstrating web macros to synthesize an automation program and specifying semantic conditions to lter web content requires a decent amount of mental eort, especially for users who are less familiar with programming or web scraping. Therefore, future designs should alleviate users' cognitive load with a more minimalist UI and more user guidance.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="7">CONCLUSION</head><p>In this work, we designed and developed SemanticOn, a collaborative system that allows users to specify and rene visual and textual conditions through user-entered descriptions and system-suggested prompts in a web automation program. In a system evaluation, we found that participants can eectively use SemanticOn to create conditional lters and rene them via continuous human-AI collaboration, collecting selective web content with high accuracy. Participants' feedback also suggested that a guided semi-automation mode, where users authorize system predictions, helped clarify user intents. We also found a trade-o between User Enters and System Suggests regarding user eort and the sense of control. Our work can point directions to future system and interaction designs for user-intent specication and renement in a continuous human-AI collaboration setting.</p></div><note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="2" xml:id="foot_0"><p>Microsoft Azure, https://tinyurl.com/55puwcf2</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="3" xml:id="foot_1"><p>Pet Wallpapers, https://tinyurl.com/47kr3mz6</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="4" xml:id="foot_2"><p>Street Photographers, https://tinyurl.com/bdppfa48</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="5" xml:id="foot_3"><p>Movie Stars, https://tinyurl.com/zsnkne5r</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="6" xml:id="foot_4"><p>Best Romance Novels, https://tinyurl.com/mryjs47h</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="7" xml:id="foot_5"><p>Vacation Destinations, https://tinyurl.com/4peucx3p</p></note>
		</body>
		</text>
</TEI>
