More Like this

1. Memory-Aware Active Learning in Mobile Sensing Systems

   https://doi.org/10.1109/TMC.2020.3003936  

   Esna Ashari, Zhila; Chaytor, Naomi S; Cook, Diane J; Ghasemzadeh, Hassan (June 2020, IEEE Transactions on Mobile Computing)

   null (Ed.)

   We propose a novel active learning framework for activity recognition, which takes limitations of the oracle into account when selecting wearable sensor data for annotation. It is inspired by human-beings' limited capacity to respond to prompts on their mobile device, introducing the notion of mindful active learning and proposing a computational framework, called EMMA, to maximize the active learning performance taking informativeness of sensor data, query budget, and human memory into account. We formulate this optimization problem, propose an approach to model memory retention, discuss the complexity of the problem, and propose a greedy heuristic to solve it. Additionally, we design an approach to perform mindful active learning in batch where multiple sensor observations are selected simultaneously for querying and design two instantiations of EMMA to perform active learning in batch mode. We demonstrate the effectiveness of our approach using three datasets and by simulating oracles with various memory strengths. Our results indicate that EMMA achieves an accuracy of, on average, 13.5% higher than the case when only informativeness of samples is considered. We observe that mindful active learning is most beneficial when the query budget is small and/or the oracle's memory is weak. 

   more » « less
2. Cost-Effective Multitask Active Learning in Wearable Sensor Systems

   https://doi.org/10.3390/s25051522  

   Arefeen, Asiful; Ghasemzadeh, Hassan (March 2025, Sensors)

   Multitask learning models provide benefits by reducing model complexity and improving accuracy by concurrently learning multiple tasks with shared representations. Leveraging inductive knowledge transfer, these models mitigate the risk of overfitting on any specific task, leading to enhanced overall performance. However, supervised multitask learning models, like many neural networks, require substantial amounts of labeled data. Given the cost associated with data labeling, there is a need for an efficient label acquisition mechanism, known as multitask active learning (MTAL). In wearable sensor systems, success of MTAL largely hinges on its query strategies because active learning in such settings involves interaction with end-users (e.g., patients) for annotation. However, these strategies have not been studied in mobile health settings and wearable systems to date. While strategies like one-sided sampling, alternating sampling, and rank-combination-based sampling have been proposed in the past, their applicability in mobile sensor settings—a domain constrained by label deficit—remains largely unexplored. This study investigates the MTAL querying approaches and addresses crucial questions related to the choice of sampling methods and the effectiveness of multitask learning in mobile health applications. Utilizing two datasets on activity recognition and emotion classification, our findings reveal that rank-based sampling outperforms other techniques, particularly in tasks with high correlation. However, sole reliance on informativeness for sample selection may introduce biases into models. To address this issue, we also propose a Clustered Stratified Sampling (CSS) method in tandem with the multitask active learning query process. CSS identifies clustered mini-batches of samples, optimizing budget utilization and maximizing performance. When employed alongside rank-based query selection, our proposed CSS algorithm demonstrates up to 9% improvement in accuracy over traditional querying approaches for a 2000-query budget. 

   more » « less
3. Deep Active Learning via Open-Set Recognition

   https://doi.org/10.3389/frai.2022.737363  

   Mandivarapu, Jaya Krishna; Camp, Blake; Estrada, Rolando (February 2022, Frontiers in Artificial Intelligence)

   In many applications, data is easy to acquire but expensive and time-consuming to label, prominent examples include medical imaging and NLP. This disparity has only grown in recent years as our ability to collect data improves. Under these constraints, it makes sense to select only the most informative instances from the unlabeled pool and request an oracle (e.g., a human expert) to provide labels for those samples. The goal of active learning is to infer the informativeness of unlabeled samples so as to minimize the number of requests to the oracle. Here, we formulate active learning as an open-set recognition problem. In this paradigm, only some of the inputs belong to known classes; the classifier must identify the rest as unknown . More specifically, we leverage variational neural networks (VNNs), which produce high-confidence (i.e., low-entropy) predictions only for inputs that closely resemble the training data. We use the inverse of this confidence measure to select the samples that the oracle should label. Intuitively, unlabeled samples that the VNN is uncertain about contain features that the network has not been exposed to; thus they are more informative for future training. We carried out an extensive evaluation of our novel, probabilistic formulation of active learning, achieving state-of-the-art results on MNIST, CIFAR-10, CIFAR-100, and FashionMNIST. Additionally, unlike current active learning methods, our algorithm can learn even in the presence of out-of-distribution outliers. As our experiments show, when the unlabeled pool consists of a mixture of samples from multiple datasets, our approach can automatically distinguish between samples from seen vs. unseen datasets. Overall, our results show that high-quality uncertainty measures are key for pool-based active learning. 

   more » « less
4. Towards Joint Segmentation and Active Learning for Block-Structured Data Streams

   Conrad Holtsclaw, Meet P. (August 2019, The ACM SIGKDD Conference on Knowledge Discovery and Data Mining Workshop on Data Collection, Curation, and Labeling for Mining and Learning)

   Stream-based active learning methods assume that data instances arrive in sequence and the decision must be made to query an instance or not as it arrives. In mobile health and human activity recognition, the data stream is often block-structured where instances in the same block have the same label, but the boundaries between blocks are unobserved. In this paper, we propose an approach to active learning in this setting where we simultaneously learn to segment the stream while learning an instance-level discriminative classifier. We show that by propagating collected labels into inferred segments, we can learn improved predictive models with significantly fewer queries. 

   more » « less
5. Composite Active Learning: Towards Multi-Domain Active Learning with Theoretical Guarantees

   https://doi.org/10.1609/AAAI.V38I11.29119  

   Hao, Guang-Yuan; Huang, Hengguan; Wang, Haotian; Gao, Jie; Wang, Hao (March 2024, Proceedings of the AAAI Conference on Artificial Intelligence)

   Active learning (AL) aims to improve model performance within a fixed labeling budget by choosing the most informative data points to label. Existing AL focuses on the single-domain setting, where all data come from the same domain (e.g., the same dataset). However, many real-world tasks often involve multiple domains. For example, in visual recognition, it is often desirable to train an image classifier that works across different environments (e.g., different backgrounds), where images from each environment constitute one domain. Such a multi-domain AL setting is challenging for prior methods because they (1) ignore the similarity among different domains when assigning labeling budget and (2) fail to handle distribution shift of data across different domains. In this paper, we propose the first general method, dubbed composite active learning (CAL), for multi-domain AL. Our approach explicitly considers the domain-level and instance-level information in the problem; CAL first assigns domain-level budgets according to domain-level importance, which is estimated by optimizing an upper error bound that we develop; with the domain-level budgets, CAL then leverages a certain instance-level query strategy to select samples to label from each domain. Our theoretical analysis shows that our method achieves a better error bound compared to current AL methods. Our empirical results demonstrate that our approach significantly outperforms the state-of-the-art AL methods on both synthetic and real-world multi-domain datasets. Code is available at https://github.com/Wang-ML-Lab/multi-domain-active-learning. 

   more » « less