An official website of the United States government
Visual relationship reasoning is a crucial yet challenging task for understanding rich interactions across visual concepts. For example, a relationship 'man, open, door' involves a complex relation 'open' between concrete entities 'man, door'. While much of the existing work has studied this problem in the context of still images, understanding visual relationships in videos has received limited attention. Due to their temporal nature, videos enable us to model and reason about a more comprehensive set of visual relationships, such as those requiring multiple (temporal) observations (e.g., 'man, lift up, box' vs. 'man, put down, box'), as well as relationships that are often correlated through time (e.g., 'woman, pay, money' followed by 'woman, buy, coffee'). In this paper, we construct a Conditional Random Field on a fully-connected spatio-temporal graph that exploits the statistical dependency between relational entities spatially and temporally. We introduce a novel gated energy function parametrization that learns adaptive relations conditioned on visual observations. Our model optimization is computationally efficient, and its space computation complexity is significantly amortized through our proposed parameterization. Experimental results on benchmark video datasets (ImageNet Video and Charades) demonstrate state-of-the-art performance across three standard relationship reasoning tasks: Detection, Tagging, and Recognition.
more »
« less
Analysis of conditional colocalization relationships and hierarchies in three-color microscopy images
Colocalization analysis of multicolor microscopy images is a cornerstone approach in cell biology. It provides information on the localization of molecules within subcellular compartments and allows the interrogation of known molecular interactions in their cellular context. However, almost all colocalization analyses are designed for two-color images, limiting the type of information that they reveal. Here, we describe an approach, termed “conditional colocalization analysis,” for analyzing the colocalization relationships between three molecular entities in three-color microscopy images. Going beyond the question of whether colocalization is present or not, it addresses the question of whether the colocalization between two entities is influenced, positively or negatively, by their colocalization with a third entity. We benchmark the approach and showcase its application to investigate receptor-downstream adaptor colocalization relationships in the context of functionally relevant plasma membrane locations. The software for conditional colocalization analysis is available at https://github.com/kjaqaman/conditionalColoc.
more »
« less
- Award ID(s):
- 2114417
- PAR ID:
- 10367056
- Publisher / Repository:
- DOI PREFIX: 10.1083
- Date Published:
- Journal Name:
- Journal of Cell Biology
- Volume:
- 221
- Issue:
- 7
- ISSN:
- 0021-9525
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Background: Relationships between bio-entities (genes, proteins, diseases, etc.) constitute a significant part of our knowledge. Most of this information is documented as unstructured text in different forms, such as books, articles and on-line pages. Automatic extraction of such information and storing it in structured form could help researchers more easily access such information and also make it possible to incorporate it in advanced integrative analysis. In this study, we developed a novel approach to extract bio-entity relationships information using Nature Language Processing (NLP) and a graph-theoretic algorithm. Methods: Our method, called GRGT (Grammatical Relationship Graph for Triplets), not only extracts the pairs of terms that have certain relationships, but also extracts the type of relationship (the word describing the relationships). In addition, the directionality of the relationship can also be extracted. Our method is based on the assumption that a triplet exists for a pair of interactions. A triplet is defined as two terms (entities) and an interaction word describing the relationship of the two terms in a sentence. We first use a sentence parsing tool to obtain the sentence structure represented as a dependency graph where words are nodes and edges are typed dependencies. The shortest paths among the pairs of words in the triplet are then extracted, which form the basis for our information extraction method. Flexible pattern matching scheme was then used to match a triplet graph with unknown relationship to those triplet graphs with labels (True or False) in the database. Results: We applied the method on three benchmark datasets to extract the protein-protein-interactions (PPIs), and obtained better precision than the top performing methods in literature. Conclusions: We have developed a method to extract the protein-protein interactions from biomedical literature. PPIs extracted by our method have higher precision among other methods, suggesting that our method can be used to effectively extract PPIs and deposit them into databases. Beyond extracting PPIs, our method could be easily extended to extracting relationship information between other bio-entities.more » « less
-
Abstract Single-molecule fluorescence experiments have transformed our understanding of complex materials and biological systems. Whether single molecules are used to report on their nano-environment or provide for localization, understanding their blinking dynamics (i.e., stochastic fluctuations in emission intensity under continuous illumination) is paramount. We recently demonstrated another use for blinking dynamics called blink-based multiplexing (BBM), where individual emitters are classified using a single excitation laser based on blinking dynamics, rather than color. This study elucidates the structure-activity relationships governing BBM performance in a series of model rhodamine, BODIPY, and anthraquinone fluorophores that undergo different photo-physical and-chemical processes during blinking. Change point detection and multinomial logistic regression analyses show that BBM can leverage spectral fluctuations, electron and proton transfer kinetics, as well as photostability for molecular classification—even within the context of a shared blinking mechanism. In doing so, we demonstrate two- and three-color BBM with ≥ 93% accuracy using spectrally-overlapped fluorophores.more » « less
-
Semantic relationships, such as hyponym–hypernym, cause–effect, meronym–holonym etc., between a pair of entities in a sentence are usually reflected through syntactic patterns. Automatic extraction of such patterns benefits several downstream tasks, including, entity extraction, ontology building, and question answering. Unfortunately, automatic extraction of such patterns has not yet received much attention from NLP and information retrieval researchers. In this work, we propose an attention-based supervised deep learning model, ASPER, which extracts syntactic patterns between entities exhibiting a given semantic relation in the sentential context. We validate the performance of ASPER on three distinct semantic relations—hyponym–hypernym, cause–effect, and meronym–holonym on six datasets. Experimental results show that for all these semantic relations, ASPER can automatically identify a collection of syntactic patterns reflecting the existence of such a relation between a pair of entities in a sentence. In comparison to the existing methodologies of syntactic pattern extraction, ASPER’s performance is substantially superior.more » « less
-
Abstract Inspired by biology’s most sophisticated computer, the brain, neural networks constitute a profound reformulation of computational principles1–3. Analogous high-dimensional, highly interconnected computational architectures also arise within information-processing molecular systems inside living cells, such as signal transduction cascades and genetic regulatory networks4–7. Might collective modes analogous to neural computation be found more broadly in other physical and chemical processes, even those that ostensibly play non-information-processing roles? Here we examine nucleation during self-assembly of multicomponent structures, showing that high-dimensional patterns of concentrations can be discriminated and classified in a manner similar to neural network computation. Specifically, we design a set of 917 DNA tiles that can self-assemble in three alternative ways such that competitive nucleation depends sensitively on the extent of colocalization of high-concentration tiles within the three structures. The system was trained in silico to classify a set of 18 grayscale 30 × 30 pixel images into three categories. Experimentally, fluorescence and atomic force microscopy measurements during and after a 150 hour anneal established that all trained images were correctly classified, whereas a test set of image variations probed the robustness of the results. Although slow compared to previous biochemical neural networks, our approach is compact, robust and scalable. Our findings suggest that ubiquitous physical phenomena, such as nucleation, may hold powerful information-processing capabilities when they occur within high-dimensional multicomponent systems.more » « less
