More Like this

1. Screen2Vec: Semantic Embedding of GUI Screens and GUI Components

   https://doi.org/10.1145/3411764.3445049  

   Li, Toby Jia-Jun; Popowski, Lindsay; Mitchell, Tom; Myers, Brad A (May 2021, Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2021))

   null (Ed.)

   Representing the semantics of GUI screens and components is crucial to data-driven computational methods for modeling user-GUI interactions and mining GUI designs. Existing GUI semantic representations are limited to encoding either the textual content, the visual design and layout patterns, or the app contexts. Many representation techniques also require significant manual data annotation efforts. This paper presents Screen2Vec, a new self-supervised technique for generating representations in embedding vectors of GUI screens and components that encode all of the above GUI features without requiring manual annotation using the context of user interaction traces. Screen2Vec is inspired by the word embedding method Word2Vec, but uses a new two-layer pipeline informed by the structure of GUIs and interaction traces and incorporates screen- and app-specific metadata. Through several sample downstream tasks, we demonstrate Screen2Vec’s key useful properties: representing between-screen similarity through nearest neighbors, composability, and capability to represent user tasks. 

   more » « less
2. An Open-Source Deep Learning-Based GUI Toolbox For Automated Auditory Brainstem Response Analyses (ABRA)

   https://doi.org/10.1101/2024.06.20.599815  

   Erra, Abhijeeth; Chen, Jeffrey; Chrysostomou, Elena; Barret, Shannon; Miller, Cayla; Kassim, Yasmin M; Friedman, Rick A; Ceriani, Federico; Marcotti, Walter; Carroll, Cody; et al (June 2024, bioRxiv)

   Abstract In this paper, we introduce a new, open-source software developed in Python for analyzing Auditory Brainstem Response (ABR) waveforms. ABRs are a far-field recording of synchronous neural activity generated by the auditory fibers in the ear in response to sound, and used to study acoustic neural information traveling along the ascending auditory pathway. Common ABR data analysis practices are subject to human interpretation and are labor-intensive, requiring manual annotations and visual estimation of hearing thresholds. The proposed new Auditory Brainstem Response Analyzer (ABRA) software is designed to facilitate the analysis of ABRs by supporting batch data import/export, waveform visualization, and statistical analysis. Techniques implemented in this software include algorithmic peak finding, threshold estimation, latency estimation, time warping for curve alignment, and 3D plotting of ABR waveforms over stimulus frequencies and decibels. The excellent performance on a large dataset of ABR collected from three labs in the field of hearing research that use different experimental recording settings illustrates the efficacy, flexibility, and wide utility of ABRA. 

   more » « less
3. Detecting Outdated Screenshot from GUI Document

   https://doi.org/10.1145/3750041  

   Tang, Ye; Yan, Aoyang; Liu, Hui; Meng, Na; Zhong, Hao (July 2025, ACM Transactions on Software Engineering and Methodology)

   In software development, many documents (e.g., tutorials for tools and mobile application websites) contain screenshots of graphical user interfaces (GUIs) to illustrate functionalities. Although screenshots are critical in such documents, screenshots can become outdated, especially if document developers forget to update them. Outdated screenshots can mislead users and diminish the credibility of documentation. Identifying screenshots manually is tedious and error-prone, especially when documents are numerous. However, no existing tools are proposed to detect outdated screenshots in GUI documents. To mitigate manual efforts, we propose DOSUD, a novel approach for detecting outdated screenshots. It is challenging to identify outdated screenshots since the differences are subtle and only specific areas are useful to identify such screenshots. To address the challenges, DOSUD automatically extracts and labels screenshots and trains a classification model to identify outdated screenshots. As the first exploration, we focus on Android applications and the most popular IDE, VS Code. We evaluated DOSUD on a benchmark comprising 10 popular applications, achieving high F1-scores. When applied in the wild, DOSUD identified 20 outdated screenshots across 50 Android application websites and 17 outdated screenshots in VS Code documentation. VS Code developers have confirmed and fixed all our bug reports. 

   more » « less
4. FUNSC: A GUI Software for Stochastic Computing

   https://doi.org/10.1109/HORA61326.2024.10550497  

   Huse, Jackson P; Salehi, Sayed A (May 2024, IEEE)
5. Checking conformance of applications against GUI policies

   https://doi.org/10.1145/3468264.3468561  

   Zhang, Zhen; Feng, Yu; Ernst, Michael D.; Porst, Sebastian; Dillig, Isil (January 2021, FSE)