More Like this

1. Using Topic Modeling for Code Discovery in Large Scale Text Data

   https://doi.org/https://doi.org/10.1007/978-3-030-67788-6_2  

   Cai, Zhiqiang; Siebert-Evenstone, Amanda; Eagan, Brendan R.; Shaffer, David W. (January 2021, Advances in Quantitative Ethnography: Second International Conference, ICQE 2020, Malibu, CA, USA, February 1-3, 2021, Proceedings)

   Ruis, Andrew; Lee, Seung B. (Ed.)

   When text datasets are very large, manually coding line by line becomes impractical. As a result, researchers sometimes try to use machine learning algorithms to automatically code text data. One of the most popular algorithms is topic modeling. For a given text dataset, a topic model provides probability distributions of words for a set of “topics” in the data, which researchers then use to interpret meaning of the topics. A topic model also gives each document in the dataset a score for each topic, which can be used as a non-binary coding for what proportion of a topic is in the document. Unfortunately, it is often difficult to interpret what the topics mean in a defensible way, or to validate document topic proportion scores as meaningful codes. In this study, we examine how keywords from codes developed by human experts were distributed in topics generated from topic modeling. The results show that (1) top keywords of a single topic often contain words from multiple human-generated codes; and conversely, (2) words from human-generated codes appear as high-probability keywords in multiple topic. These results explain why directly using topics from topic models as codes is problematic. However, they also imply that topic modeling makes it possible for researchers to discover codes from short word lists. 

   more » « less
2. Natural Language Generation From Ontologies

   Van Nguyen, Tran Cao (April 2019, Lecture notes in computer science)

   This paper addresses the problem of automatic generation of natural language descriptions for ontology-described artifacts. The original motivation for the work is the challenge of providing textual narratives of automatically generated scientific workflows (e.g., paragraphs that scientists can include in their publications). The paper presents two systems which generate descriptions of sets of atoms derived from a collection of ontologies. The first system, called nlgPhylogeny, demonstrates the feasibility of the task in the Phylotastic project, providing evolutionary biologists with narrative for automatically generated analysis workflows. nlgPhylogeny utilizes the fact that the Grammatical Framework (GF) is suitable for the natural language generation (NLG) task; the paper shows how elements of the ontologies in Phylotastic, such as web services and information artifacts, can be encoded in GF for the NLG task. The second system, called 𝚗𝚕𝚐𝙾𝚗𝚝𝚘𝚕𝚘𝚐𝚢𝐴, is a generalization of nlgPhylogeny. It eliminates the requirement that a GF needs to be defined and proposes the use of annotated ontologies for NLG. Given a set of annotated ontologies, 𝚗𝚕𝚐𝙾𝚗𝚝𝚘𝚕𝚘𝚐𝚢𝐴 generates a GF suitable for the creation of natural language descriptions of sets of atoms derived from these ontologies. The paper describes the algorithms used in the development of nlgPhylogeny and 𝚗𝚕𝚐𝙾𝚗𝚝𝚘𝚕𝚘𝚐𝚢𝐴 and discusses potential applications of these systems. 

   more » « less
3. On Leave-One-Out Conditional Mutual Information For Generalization

   Rammal, R; Achille, A; Golatkar, A; Diggavi, S N; Soatto, S (January 2022, Advances in Neural Information Processing Systems (NeurIPS))

   We derive information theoretic generalization bounds for supervised learning algorithms based on a new measure of leave-one-out conditional mutual information (loo-CMI). Contrary to other CMI bounds, which are black-box bounds that do not exploit the structure of the problem and may be hard to evaluate in practice, our loo-CMI bounds can be computed easily and can be interpreted in connection to other notions such as classical leave-one-out cross-validation, stability of the optimization algorithm, and the geometry of the loss-landscape. It applies both to the output of training algorithms as well as their predictions. We empirically validate the quality of the bound by evaluating its predicted generalization gap in scenarios for deep learning. In particular, our bounds are non-vacuous on large-scale image-classification tasks. 

   more » « less
4. GLOBEM: Cross-Dataset Generalization of Longitudinal Human Behavior Modeling

   https://doi.org/10.1145/3569485  

   Xu, Xuhai; Liu, Xin; Zhang, Han; Wang, Weichen; Nepal, Subigya; Sefidgar, Yasaman; Seo, Woosuk; Kuehn, Kevin S.; Huckins, Jeremy F.; Morris, Margaret E.; et al (December 2022, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies)

   There is a growing body of research revealing that longitudinal passive sensing data from smartphones and wearable devices can capture daily behavior signals for human behavior modeling, such as depression detection. Most prior studies build and evaluate machine learning models using data collected from a single population. However, to ensure that a behavior model can work for a larger group of users, its generalizability needs to be verified on multiple datasets from different populations. We present the first work evaluating cross-dataset generalizability of longitudinal behavior models, using depression detection as an application. We collect multiple longitudinal passive mobile sensing datasets with over 500 users from two institutes over a two-year span, leading to four institute-year datasets. Using the datasets, we closely re-implement and evaluated nine prior depression detection algorithms. Our experiment reveals the lack of model generalizability of these methods. We also implement eight recently popular domain generalization algorithms from the machine learning community. Our results indicate that these methods also do not generalize well on our datasets, with barely any advantage over the naive baseline of guessing the majority. We then present two new algorithms with better generalizability. Our new algorithm, Reorder, significantly and consistently outperforms existing methods on most cross-dataset generalization setups. However, the overall advantage is incremental and still has great room for improvement. Our analysis reveals that the individual differences (both within and between populations) may play the most important role in the cross-dataset generalization challenge. Finally, we provide an open-source benchmark platform GLOBEM- short for Generalization of Longitudinal BEhavior Modeling - to consolidate all 19 algorithms. GLOBEM can support researchers in using, developing, and evaluating different longitudinal behavior modeling methods. We call for researchers' attention to model generalizability evaluation for future longitudinal human behavior modeling studies. 

   more » « less
5. Recovering Structured Probability Matrices.

   https://doi.org/10.4230/LIPIcs.ITCS.2018.46  

   Huang, Qingqing; Kakade, Sham M.; Kong, Weihao; Valiant, Gregory (May 2018, 9th Innovations in Theoretical Computer Science Conference (ITCS 2018))

   We consider the problem of accurately recovering a matrix B of size M by M, which represents a probability distribution over M^2 outcomes, given access to an observed matrix of "counts" generated by taking independent samples from the distribution B. How can structural properties of the underlying matrix B be leveraged to yield computationally efficient and information theoretically optimal reconstruction algorithms? When can accurate reconstruction be accomplished in the sparse data regime? This basic problem lies at the core of a number of questions that are currently being considered by different communities, including building recommendation systems and collaborative filtering in the sparse data regime, community detection in sparse random graphs, learning structured models such as topic models or hidden Markov models, and the efforts from the natural language processing community to compute "word embeddings". Many aspects of this problem---both in terms of learning and property testing/estimation and on both the algorithmic and information theoretic sides---remain open. Our results apply to the setting where B has a low rank structure. For this setting, we propose an efficient (and practically viable) algorithm that accurately recovers the underlying M by M matrix using O(M) samples} (where we assume the rank is a constant). This linear sample complexity is optimal, up to constant factors, in an extremely strong sense: even testing basic properties of the underlying matrix (such as whether it has rank 1 or 2) requires Omega(M) samples. Additionally, we provide an even stronger lower bound showing that distinguishing whether a sequence of observations were drawn from the uniform distribution over M observations versus being generated by a well-conditioned Hidden Markov Model with two hidden states requires Omega(M) observations, while our positive results for recovering B immediately imply that Omega(M) observations suffice to learn such an HMM. This lower bound precludes sublinear-sample hypothesis tests for basic properties, such as identity or uniformity, as well as sublinear sample estimators for quantities such as the entropy rate of HMMs. 

   more » « less