In accelerated MRI reconstruction, the anatomy of a patient is recovered from a set
of under-sampled and noisy measurements. Deep learning approaches have been
proven to be successful in solving this ill-posed inverse problem and are capable of
producing very high quality reconstructions. However, current architectures heavily
rely on convolutions, that are content-independent and have difficulties modeling
long-range dependencies in images. Recently, Transformers, the workhorse of
contemporary natural language processing, have emerged as powerful building
blocks for a multitude of vision tasks. These models split input images into nonoverlapping
patches, embed the patches into lower-dimensional tokens and utilize
a self-attention mechanism that does not suffer from the aforementioned weaknesses
of convolutional architectures. However, Transformers incur extremely
high compute and memory cost when 1) the input image resolution is high and 2)
when the image needs to be split into a large number of patches to preserve fine
detail information, both of which are typical in low-level vision problems such
as MRI reconstruction, having a compounding effect. To tackle these challenges,
we propose HUMUS-Net, a hybrid architecture that combines the beneficial implicit
bias and efficiency of convolutions with the power of Transformer blocks
in an unrolled and multi-scale network. HUMUS-Net extracts high-resolution
features via convolutional blocks and refines low-resolution features via a novel
Transformer-based multi-scale feature extractor. Features from both levels are then
synthesized into a high-resolution output reconstruction. Our network establishes
new state of the art on the largest publicly available MRI dataset, the fastMRI
dataset. We further demonstrate the performance of HUMUS-Net on two other
popular MRI datasets and perform fine-grained ablation studies to validate our
design.
more »
« less
AttCAT: Explaining Transformers via Attentive Class Activation Tokens
Transformers have improved the state-of-the-art in various natural language processing and computer vision tasks. However, the success of the Transformer model
has not yet been duly explained. Current explanation techniques, which dissect
either the self-attention mechanism or gradient-based attribution, do not necessarily
provide a faithful explanation of the inner workings of Transformers due to the following reasons: first, attention weights alone without considering the magnitudes
of feature values are not adequate to reveal the self-attention mechanism; second,
whereas most Transformer explanation techniques utilize self-attention module,
the skip-connection module, contributing a significant portion of information flows
in Transformers, has not yet been sufficiently exploited in explanation; third, the
gradient-based attribution of individual feature does not incorporate interaction
among features in explaining the model’s output. In order to tackle the above
problems, we propose a novel Transformer explanation technique via attentive
class activation tokens, aka, AttCAT, leveraging encoded features, their gradients,
and their attention weights to generate a faithful and confident explanation for
Transformer’s output. Extensive experiments are conducted to demonstrate the
superior performance of AttCAT, which generalizes well to different Transformer
architectures, evaluation metrics, datasets, and tasks, to the baseline methods. Our
code is available at: https://github.com/qiangyao1988/AttCAT.
more »
« less
- Award ID(s):
- 2211897
- NSF-PAR ID:
- 10448646
- Date Published:
- Journal Name:
- Advances in neural information processing systems
- ISSN:
- 1049-5258
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Despite the advances in the field of Face Recognition (FR), the precision of these methods is not yet sufficient. To improve the FR performance, this paper proposes a technique to aggregate the outputs of two state-of-the-art (SOTA) deep FR models, namely ArcFace and AdaFace. In our approach, we leverage the transformer attention mechanism to exploit the relationship between different parts of two feature maps. By doing so, we aim to enhance the overall discriminative power of the FR system. One of the challenges in feature aggregation is the effective modeling of both local and global dependencies. Conventional transformers are known for their ability to capture long-range dependencies, but they often struggle with modeling local dependencies accurately. To address this limitation, we augment the self-attention mechanism to capture both local and global dependencies effectively. This allows our model to take advantage of the overlapping receptive fields present in corresponding locations of the feature maps. However, fusing two feature maps from different FR models might introduce redundancies to the face embedding. Since these models often share identical backbone architectures, the resulting feature maps may contain overlapping information, which can mislead the training process. To overcome this problem, we leverage the principle of Information Bottleneck to obtain a maximally informative facial representation. This ensures that the aggregated features retain the most relevant and discriminative information while minimizing redundant or misleading details. To evaluate the effectiveness of our proposed method, we conducted experiments on popular benchmarks and compared our results with state-of-the- art algorithms. The consistent improvement we observed in these benchmarks demonstrate the efficacy of our approach in enhancing FR performance. Moreover, our model aggregation framework offers a novel perspective on model fusion and establishes a powerful paradigm for feature aggregation using transformer-based attention mechanisms.more » « less
-
Transformers have shown great promise in medical image segmentation due to their ability to capture long-range dependencies through self-attention. However, they lack the ability to learn the local (contextual) relations among pixels. Previous works try to overcome this problem by embedding convolutional layers either in the encoder or decoder modules of transformers thus ending up sometimes with inconsistent features. To address this issue, we propose a novel attention-based decoder, namely CASCaded Attention DEcoder (CASCADE), which leverages the multiscale features of hierarchical vision transformers. CASCADE consists of i) an attention gate which fuses features with skip connections and ii) a convolutional attention module that enhances the long-range and local context by suppressing background information. We use a multi-stage feature and loss aggregation framework due to their faster convergence and better performance. Our experiments demonstrate that transformers with CASCADE significantly outperform state-of-the-art CNN- and transformer-based approaches, obtaining up to 5.07% and 6.16% improvements in DICE and mIoU scores, respectively. CASCADE opens new ways of designing better attention-based decoders.more » « less
-
Transformers have shown great success in medical image segmentation. However, transformers may exhibit a limited generalization ability due to the underlying single-scale selfattention (SA) mechanism. In this paper, we address this issue by introducing a Multiscale hiERarchical vIsion Transformer (MERIT) backbone network, which improves the generalizability of the model by computing SA at multiple scales. We also incorporate an attention-based decoder, namely Cascaded Attention Decoding (CASCADE), for further refinement of the multi-stage features generated by MERIT. Finally, we introduce an effective multi-stage feature mixing loss aggregation (MUTATION) method for better model training via implicit ensembling. Our experiments on two widely used medical image segmentation benchmarks (i.e., Synapse Multi-organ and ACDC) demonstrate the superior performance of MERIT over state-of-the-art methods. Our MERIT architecture and MUTATION loss aggregation can be used with other downstream medical image and semantic segmentation tasks.more » « less
-
Deep learning models have been studied to forecast human events using vast volumes of data, yet they still cannot be trusted in certain applications such as healthcare and disaster assistance due to the lack of interpretability. Providing explanations for event predictions not only helps practitioners understand the underlying mechanism of prediction behavior but also enhances the robustness of event analysis. Improving the transparency of event prediction models is challenging given the following factors: (i) multilevel features exist in event data which creates a challenge to cross-utilize different levels of data; (ii) features across different levels and time steps are heterogeneous and dependent; and (iii) static model-level interpretations cannot be easily adapted to event forecasting given the dynamic and temporal characteristics of the data. Recent interpretation methods have proven their capabilities in tasks that deal with graph-structured or relational data. In this paper, we present a Contextualized Multilevel Feature learning framework, CMF, for interpretable temporal event prediction. It consists of a predictor for forecasting events of interest and an explanation module for interpreting model predictions. We design a new context-based feature fusion method to integrate multiple levels of heterogeneous features. We also introduce a temporal explanation module to determine sequences of text and subgraphs that have crucial roles in a prediction. We conduct extensive experiments on several real-world datasets of political and epidemic events. We demonstrate that the proposed method is competitive compared with the state-of-the-art models while possessing favorable interpretation capabilities.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
