Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups

Introduction

Machine learning (ML) models are powerful at complex tasks, but also notoriously opaque. In high-stakes domains such as science (Li et al., 2021;Zednik & Boelsen, 2022) and medicine (Reyes et al., 2020;Tjoa & Guan, 2021), experts need explanations to trust the models' decisions. For instance, physicists use interpretable coefficients to validate an ML model's rediscovery of Kepler's first law (Li et al., 2021), while physicians require explanations to trust MLdriven diagnostic decisions (Klauschen et al., 2024). . Sum-of-Parts (SOP) linearly aggregates outputs from multiple feature groups. This maintains performance while ensuring interpretability. SOP first generates groups using a group generator Γ, predicts with a pre-trained backbone h, and aggregates the group predictions with a group selector θ.

Self-explaining neural networks (SENNs) were proposed as a way to create neural networks with guaranteed linear interpretations (Alvarez Melis & Jaakkola, 2018). Specifically, SENNs express predictions as linear combinations of interpretable atoms scaled by learnable coefficients, a natural generalization of the statistical interpretation of the classic linear model. These interpretable atoms represent semantic notions such as object segments in images or example image prototypes (Alvarez Melis & Jaakkola, 2018).

A common class of self-explaining neural networks, which we refer to as Self-Attributing Neural Networks (SANNs), use embedded feature subsets as interpretable atoms (Brendel & Bethge, 2019;Jain et al., 2020;Agarwal et al., 2021). SANNs have predictions faithfully decomposable as linear combinations of feature subset contributions. On the other hand, post-hoc feature attributions fail to pass the sanity checks for faithfulness (Sundararajan et al., 2017;Adebayo et al., 2018).

However, SANNs often exhibit performance trade-offs and rely on specific components such as per-feature modules in NAM (Agarwal et al., 2021), convolutional layers in Bag-Net (Brendel & Bethge, 2019), or attention mechanisms in FRESH (Jain et al., 2020). This architecture require-ment hinders SANN from leveraging pre-trained models that achieve high performance on target tasks.

To understand these limitations, we first analyze a theoretical barrier that limits SANN's performance. Specifically, we prove a lower bound on the error of a class of self-attributing neural networks that we refer to as per-feature SANNs. This result shows that it is impossible for per-feature SANNs to achieve high performance when features are highly correlated, a critical limitation in high-dimensional vision and language data. In contrast, we further prove that group-based SANNs can achieve high performance in these settings, but require a careful selection of feature groups.

To overcome these provable limitations for per-feature SANNs and inflexible group-based SANNs, we propose Sum-of-Parts (SOP), a flexible framework that transforms any differentiable model into a group-based SANN (Figure 1). Specifically, given a backbone model and an input, the framework (1) identifies feature groups with a learned attention module, (2) encodes each group using a modelagnostic backbone, and (3) aggregates predictions with a second sparse attention module. This framework can then be learned end-to-end with only the final prediction labels, notably without the direct supervision of feature groups. Learned feature groups can capture the dynamic correlations in high-dimensional data, enabling SOP to overcome the theoretical limitations of per-feature and fixed-groupbased SANNs.

Our main contributions are as follows:

1. We propose Sum-of-Parts (SOP), a model-agnostic framework which transforms any model into a groupbased SANN. The groups in SOP are end-to-end learnable without the need for group label supervision.

2. We formally prove that groups are essential for SANNs to achieve low errors for highly correlated features.

In contrast, we prove a lower bound on per-feature SANNs' errors, which grows as the number of features increases.

3. We show that SOP achieves state-of-the-art performance among SANNs on vision and language tasks as informed by the theory, with learned interpretable groups validated on a range of quantitative and semantic metrics.

4. We validate the utility of SOP in debugging if correct/incorrect model predictions rely more on the background/objects, as well as a scientific discovery setting within cosmology by using the groups and scores to uncover new insights about galaxy formation.

2 Overcoming Self-Attributing Neural Networks' Limitations with Groups

In this section, we first review self-attributing neural networks (Section 2.1), and prove that the poor performance of previously explored per-feature SANNs is theoretically limited due to correlated features in high dimensional data (Section 2.2). In contrast, we further prove that group-based SANNs can overcome these fundamental limitations, motivating our proposed framework for learnable group-based SANNs (Section 2.3).

Self-Attributing Neural Networks

Self explaining neural networks model predictions as a linear combination f (x) = i θ(x) i h(x) i , where θ(x) are linear coefficients and h(x) are referred to as interpretable atoms (Alvarez Melis & Jaakkola, 2018). A common strategy for creating self-explaining neural networks is to use feature subsets as interpretable atoms. For example, Bag-Net (Brendel & Bethge, 2019) or Neural Additive Models (Agarwal et al., 2021) decompose predictions into a linear combination of terms, where each term is directly computed from and attributed to a subset of input features. We denote such models, which combine the interpretability of linearity with guaranteed attributions to input features, as Self-Attributing Neural Networks (SANNs).

Definition 2.1. A self-attributing neural network f : R d → R given input x ∈ R d decomposes predictions as follows:

where θ(x) ∈ R m are linear coefficients, and h(x Gi ) ∈ R m are embeddings of the feature subset x Gi corresponding to the subset G i ⊆ [d]. Note that m can be different from the number of raw features d.

The resulting linear combination constitutes a faithfulby-construction explanation for the decision process of the model (Lyu et al., 2024). SANNs are only as interpretable as the underlying feature subsets (Alvarez Melis & Jaakkola, 2018;Zytek et al., 2022), and different SANNs have explored various feature subsets x Gi . For example, NAM (Agarwal et al., 2021) uses individual features, Bag-Net (Brendel & Bethge, 2019) relies on large patches, and FRESH (Jain et al., 2020) selects a single subset using attention scores. However, across these subsets, SANNs have consistently exhibit significant trade-offs in performance in exchange for interpretability. In this section, we theoretically analyze the underlying cause for this trade-off and how SANNs can overcome these barriers.

Lower Bounds on the Error of Per-feature SANNs

One class of SANNs uses individual features as interpretable atoms (Agarwal et al., 2021), where each feature subset G i = {i} and the corresponding interpretable atom h(x Gi ) = h(x i ) is the encoding of exactly one feature x i . We begin by analyzing the error of per-feature SANNs.

In order for such models to be accurate, the contribution of a feature to the true label should match the change in prediction when the feature is removed or added. Similarly, the contribution of a subset, Gi⊆S α i , should capture the change in prediction when the subset S is excluded or included. We formalize this difference between a SANN prediction and the ground truth when inserting or deleting subsets of features as insertion and deletion errors, respectively. Definition 2.2. (Insertion Error) Let α i = θ(x) i h(x Gi ) be the total contribution of the ith feature group to the prediction of a SANN. Then, the insertion error of a self attributing neural network f

The total insertion error over all possible insertions is

The insertion error captures the difference between the ground truth effect of inserting a subset of features S and the corresponding change in the SANN. If these two quantities are close, then the error is low. The insertion error of a perfeature SANN is a special case where G = {{1}, . . . , {d}}.

For brevity of presentation, in this section we focus on insertion, and present analogous definitions and theorems for deletion to Appendix A. We note that the insertion and deletion procedures are analogous to insertion and deletion tests for post-hoc explanations (Petsiuk et al., 2018;Samek et al., 2017), but used here to capture the error of a SANN.

Error Lower Bounds for Data with Correlated Features.

We now prove that it is impossible for per-feature SANNs to perform well when the data contains correlated features. Specifically, in Theorem 2.3, we show that when estimating polynomials function with correlated features, per-feature SANNs have a non-trivial lower bound on the total insertion error.

Theorem 2.3 (Lower Bound on Insertion Error for Binomials). Let p : {0, 1} d → {0, 1, 2} be a multilinear binomial polynomial function. Furthermore suppose that the features can be partitioned into (S 1 , S 2 , S 3 ) of equal sizes where p(x) = i∈S1∪S2 x i + j∈S2∪S3 x j . Then,

, where D ins ( λ) = ( λ1 -λ2 ) ⊤ c is the lower bound, λ is a dual feasible point, and c is a constant as defined in (16).

To derive this lower bound, we formulated the minimum total error of any SANN as a linear program, and used a dual feasible point to compute a lower bound. The proof and an analogous theorem for deletion error on monomials (Theorem A.2) are presented in Appendix A.

Lower Bounds Grow Rapidly with Dimension. In Figure 2, we calculate the lower bound using ECOS (Domahidi et al., 2013) and plot the lower bound for total insertion and deletion errors as the feature dimension grows (Figure 2), and observe that these errors increase exponentially with d. Altogether, these theoretical lower bounds and empirical trends suggest that per-feature SANNs are fundamentally in-

Algorithm 1 The Sum-of-Parts Framework Require: Backbone Pred h, Backbone Encoder h h , Embedding Model h e , Class Weights C h

{Group Generating (trained)}

capable of modeling high-dimensional data, as they cannot model even simple polynomials.

Group-based Self-Attributing Neural Networks Can Overcome the Performance Barrier

The fundamental limitation of per-feature SANNs comes from its choice of interpretable atom: individual features are unable to capture correlations between multiple features. If we allow SANNs to use more expressive interpretable atoms composed of feature groups, can we get past this limitation?

To answer this question, we carry out an analogous analysis for more general, group-based SANNs.

In this section, we summarize our main theoretical result in Theorem 2.4: we prove that there exist group-based SANNs that can not only perfectly capture the earlier settings in Theorem 2.3 (and Theorem A.2 in the appendix), but also far more complex, general polynomials with zero error. In other words, the right groups can enable SANNs to capture correlated signals and overcome the performance barrier. Theorem 2.4 (Informal: Zero Group Insertion and Deletion Error). For any general m-nomial polynomial p, having at most m groups is sufficient for a group-based selfattributing neural network to achieve zero insertion and deletion error. See Theorem A.5 for full theorem and proof.

Intuitively, a SANN can achieve low error if its groups align with the correlations in the data. Specifically, consider data generated from a polynomial with multiple terms p

, where each q i (x G ′ i ) is a multiplicative term that depends on the group of features in G ′ i . Then, a group-based SANN can achieve low error if each correlated feature group G ′ i aligns with a SANN group G i . On the other hand, misaligned or insufficiently many groups lead to nonzero errors, highlighting how groupalignment is key to SANN performance.

While this theorem demonstrates that SANNs can model highly complex polynomials, it also provides insight into why existing SANNs have suffered major performance trade-offs. For example, SANNs that rely on rigid patterns (Brendel & Bethge, 2019), use groups that are too small (Agarwal et al., 2021) or use too few groups (Jain et al., 2020) are unlikely to align with the ground truth correlations. In contrast, a high-performing SANN requires the ability to use many groups of flexible patterns to capture the diverse signals in high-dimensional data. These criteria motivate a new type of SANN that can overcome this theoretical performance barrier: the Sum-of-Parts framework.

The Sum-of-Parts Framework

In this section, we introduce our main technical contribution Sum-of-Parts (SOP), a framework that transforms any differentiable model into a group-based self-attributing model.

Suppose we have an input x ∈ R d and a backbone model h : R d → R that makes a prediction with the input, and we hope to convert h to a SANN. A SANN requires components to generate and encode feature subsets and another component to assign them coefficients. SOP therefore naturally consists of three parts: a group generator Γ : R d → {0, 1} m×d that generates groups g 1 , . . . , g m ∈ {0, 1} d , a backbone predictor h : R d → R that makes a prediction with each group of features, and a group selector θ : {0, 1} m×d × R d → [0, 1] m that assigns scores to the groups:

where g i ∈ Γ(x).

group generator (trained)

Here we consider a single predicted logit, while the process can be repeated in batch for multiple logits or classes. Our algorithm is illustrated in Figure 1 and Algorithm 1.

Group Generator Γ : R d → {0, 1} m×d takes in an input x ∈ R d and outputs m binary group masks g 1 , . . . , g m ∈ {0, 1} d , where g ij = 1 if and only if the feature j is included in group g i . 2 It uses a multi-headed self-attention module (Vaswani et al., 2017) to assign scores to features, and threshold each attention distribution to include top τ = 20% features to each group.

where the encoder h e : R d → R d×e , which we typically take up to the penultimate layer of the backbone model, embeds each feature x i into a vector with embedding dimension e. The learnable group generator dynamically creates feature groups for each input, enabling better correlation compared to fixed groups (e.g., patches). Moreover, it has no specific architectural constraints on the backbone such as attention mechanisms.

Backbone Predictor h : R d → R then makes a prediction with the input x masked by each binary group mask g i :

where y i ∈ R is the output logit and ⊙ is Hadamard product. The backbone predictor can be arbitrary high-performing pre-trained model.

Group Selector θ : {0, 1} m×d × R d → [0, 1] m then takes in the encoding of each group and uses a sparse crossattention module to assign each group a score.

where the query C h ∈ R k is initialized using the target class's weights with k hidden dimensions and the key z = (h h (g 1 ⊙x), . . . , h h (g m ⊙x)) ∈ R m×k are last hidden states of all groups. As using a sparse number of groups avoids overloading human users, we replace the softmax in the cross attention with a sparse variant, the sparsemax operator (Martins & Astudillo, 2016). Dynamically assigning scores allows the model to choose the most helpful groups for prediction, while the sparse number of groups ensures easy human interpretability.

The final prediction is made by aggregating predictions from each group g i with its assigned score c i .

To address the gradient flow issue caused by binarized groups, we incorporate a scaling factor based on the attention score in the final loss, as detailed in Appendix B.3. Additional details on self-attention, cross-attention, and embedding models are provided in Appendix B.

In summary, the learnable group generator dynamically creates correlated groups needed for high-performing SANNs, the model-agnostic framework supports arbitrary backbone predictors, and the sparse group selector assigns contributions for a small number of groups. Together, these components are essential for SOP to be a high-performing SANN, as informed by theory in Section 2.

Empirical Evaluations of Sum-of-Parts

We conduct experiments using our framework on image and text tasks to see if our theory-informed framework actually uses the learned groups for (RQ1) high performance. Next, we quantitatively measure the interpretability of SOP with (RQ2) the performance at different inference-time sparsity levels, (RQ3) various faithfulness metrics, and (RQ4) whether the group masks are leaking predictive signals.

Then, we measure the interpretability of SOP semantically using (RQ5) semantic coherence of the groups. Finally, we validate SOP's utility on (RQ6) model debugging for unwanted behaviors in correct and incorrect predictions and (RQ7) scientific discovery in cosmology.

Experiment Setups. We evaluate on two vision and one language datasets: ImageNet-S (Gao et al., 2022) image classification using Vision Transformer (Dosovitskiy et al., 2021) backbone, CosmoGrid (Kacprzak et al., 2023) cosmology image regression using CNN (Matilla et al., 2020), and MultiRC (Khashabi et al., 2018) text classification using BERT (Devlin et al., 2019). We use patches as image features and tokens as text features. For baselines, we compare with other SANNs including XDNN (Hesse et al., 2021), BagNet (Brendel & Bethge, 2019), and FRESH (Jain et al., 2020). Additionally, we convert post-hoc feature attribution methods to self-attributing models by using top 20% scored features as single groups and denote with "-F" (e.g. "LIME-F"), including LIME (Ribeiro et al., 2016), SHAP (Lundberg & Lee, 2017), IG (Sundararajan et al., 2017), GC (Selvaraju et al., 2016), FG (Srinivas & Fleuret, 2019), RISE (Petsiuk et al., 2018), Archipelago (Tsang et al., 2020), MFABA (Zhu et al., 2024), AGI (Pan et al., 2021), AMPE (Zhu et al., 2023) and BCos (Böhle et al., 2022). We use a maximum of 20 groups for SOP and the same 20 forward passes for all controllable baselines (Table 2). Details for datasets, models and baselines are in Appendix C.

Quantitative Measures for Interpretability

(RQ2) Can SOP Perform Well at Different Sparsity Levels? Sparser explanations are easier to understand for humans (Lombrozo, 2007;Poursabzi-Sangdeh et al., 2021), but the best sparsity level is often unknown at training time. SOP's group generator learns to generate groups at a specific sparsity, and we test whether it performs well across other sparsity levels without retraining. Figure 3 shows that as sparsity increases (≥ 80%, keeping ≤ 20% features per group), SOP's error grows more slowly than other SANNs, maintaining much lower errors at extreme sparsity. Similar trends are observed in CosmoGrid and MultiRC (Appendix C.4.2). For MultiRC, FRESH slightly outperforms SOP for untrained sparsity levels, as it is optimized for language tasks. Overall, SOP trained on one sparsity also performs well on other sparsity levels at inference time.

3 Requires specialized architectures and thus only included for ImageNet-S where pre-trained models exist.

0.0 0.2 0.4 0.6 0.8 Sparsity 0.2 0.4 0.6 0.8 1.0 Error ImageNet VisionTransformer Beyond group size, we examine how the total number of groups affects performance. Ablation studies on ImageNet-S using 1, 2, 5, 10, 20 groups show that performance improves with more groups but saturates at 5 groups. Using only 2 groups increases the error rate by 6 percentage points compared to 5 groups, suggesting SOP can achieve approximately 4x greater computational efficiency while maintaining comparable performance.

(RQ3) How Faithful Is SOP with Respect to Classic Metrics? While self-attributing neural networks ensure faithfulness by construction through linear aggregation, we check how well SOP performs on classic faithfulness metrics: fidelity (Yu & Varshney, 2017;Chen et al., 2019), insertion, and deletion (Petsiuk et al., 2018;Samek et al., 2017). Fidelity measures how well summed explanations match the model's prediction using KL-divergence, while insertion and deletion evaluate the impact of high-scoring features via perturbations and area-under-curve (AUC) computation.

Definitions and results are in Appendix C.4.3, C.4.4.

All SANNs, including SOP, achieve a perfect fidelity score of 0, while no post-hoc method does (Table A5), indicating only SANNs faithfully match predictions. SOP outperforms all baselines on insertion across all tasks, and on deletion for ImageNet, while methods like Archipelago, LIME, and FRESH perform better on deletion in some cases (Tables 3 and A6). In fact, as deletion score measures how fast the predicted probability drops when the most scored features are deleted, it biases towards models that use a small number of features, regardless of how faithful the model is. SOP which relies on signals from multiple groups then naturally performs better on insertion than deletion. Ablations using smaller step sizes and occlusion values (Table A7) show consistent and thus robust results. In summary, SOP performs strong on most classic faithfulness metrics in addition to its built-in faithfulness.

(RQ4) Do SOP's Attributions Contain Predictive Signals? The SANN's claim of linear interpretability relies on the assumption that feature groups (which form the interpretable atoms) do not inherently encode label information. If these groups already contain predictive signals, the backbone model-not the coefficients-would drive predictions, rendering the coefficients uninformative. To validate this, we train probing models to predict labels solely from group masks: high accuracy indicates that label information is pre-encoded in the groups. Figure 4 shows that group masks from SOP achieves random accuracy (0.10%) on ImageNet-S using a CNN probing model. In comparison, probing models for groups from other SANNs achieve significantly higher accuracies, such as FG-F (13.40%) and AGI-F (10.66%). Thus SOP's generated groups do not leak information about the label compared to other SANNs, and the model's interpretablity is not weakened.

Semantic Measures for Interpretability

(RQ5) How Semantically Coherent are SOP Groups? Explanations need to be semantically coherent such as relating to object segments or human-understandable con-

R E S H S O P 0.00 0.05 0.10 0.15 Group Pred Acc ImageNet VisionTransformer Figure 4. (ImageNet Group Probing Accuracy) The powerful group generator in SOP is not doing all the work and not compromising SOP's interpretability. A CNN model trained on group masks from SOP is unable to obtain accuracies more than random (0.1% accuracy), while MFABA-F, AMPE-F, IG-F etc. do. RISE and Archipelago is omitted for the significant computational cost. Results for linear and ViT probing models are in Appendix C.4.5.

cepts. We thus ask how semantically coherent are the group explanations generated by SOP compared to other models? To evaluate this, we compute intersection-over-union (IOU) for ImageNet-S and MultiRC where ground-truth annotations exist, and threshold-based purity for Cosmo-Grid using expert-informed metrics for cosmological structures (Matilla et al., 2020). Table 1 shows that SOP has the best semantic coherency on two vision tasks and a close second on the language task. Figure 5 illustrates an example of an image displaying a beagle, where SOP learns more semantically coherent groups than other methods without direct supervision. The exact formulations for IOU and purity, threshold ablations for CosmoGrid, and additional examples can be found in Appendix C.4.1. Additional experiments for human evaluation can be found in Appendix C.4.6.

Utility of SOP Explanations

(RQ6) Can We Use SOP Explanations to Debug Models?

As SOP explanations directly compose the final prediction, we want to see if we can use the group explanations to identify undesirable model behaviors. One such behavior is overly relying on background features for correct predictions, indicating potential spurious correlations. We analyze SOP's explanations to check which feature groups the model use to make correct and incorrect predictions. We find that SOP uses more objects in correct (64.9%) than incorrect (57.3%) examples. Figure 6 shows similar behavior for other SANN baselines. We conjecture that using objects, instead of relying on spurious background correlations, helps the model make correct predictions. Thus, explanations from SOP and other SANNs can help illuminate the reasoning behind model behaviors.

(RQ7) Can SOP Explanations Assist Scientific Discovery? The ultimate goal of interpretability methods is for domain experts to use these tools and explanations in real settings. To validate the usability of our approach, we collaborate with cosmologists and use SOP to discover new knowledge about the expansion of the universe and the growth of the cosmic structure.

Problem Formulation. Cosmologists hope to understand the relations of cosmological structures with two cosmological parameters related to the initial state of the universe: Ω m and σ 8 . The parameter Ω m captures the average energy density of all matter in the universe, while σ 8 describes the fluctuation of matter distribution (Abbott et al., 2022). However, these parameters are not directly measurable. What we can obtain are weak lensing mass maps, which are spatial distribution of matter density in the universe calculated using precise measurements of the shapes of ∼ 100 million

E S H S O P 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 % Object Pixels ImageNet-S -Vision Transformer Correct Incorrect Figure 6. (ImageNet-S Percent of Groups that are Objects) We analyze the proportion of groups that are objects on ImageNet-S, and find that groups consistently consist more of objects when predicting correctly. This trend is consistently across methods.

galaxies (Gatti et al., 2021). While the direct relation from weak lensing maps to Ω m and σ 8 is unknown, cosmologists create simulated weak lensing maps from different Ω m and σ 8 values and train CNNs (Ribli et al., 2019;Matilla et al., 2020;Fluri et al., 2022) to reversely predict Ω m and σ 8 from the mass maps. An open question in cosmology remains:

What structures from weak lensing maps drive the inference of the cosmological parameters Ω m and σ 8 ?

As validated in Section 4.3 (RQ5), the groups from SOP correspond to more cosmological structures-voids and clustersthan other self-attributing models. Voids are large regions under-dense relative to the mean density (pixel intensity < 0σ) and appear as dark regions in the mass maps, whereas clusters are areas of concentrated high density (pixel intensity > 3σ) and appear as bright dots, as shown in Figure 7.

Cosmological Findings. We then use the groups automatically generated by SOP and find the following new findings related to voids/clusters and Ω m /σ 8 :

1. Figure 8a shows that for both voids and clusters, when they are used in the prediction, they weigh higher when predicting Ω m than σ 8 .

2. Figure 8b shows that a lot of voids contribute 100% to the prediction, while most clusters contribute partially. This finding is consistent with previous work (Matilla et al., 2020) that found that voids are the most important feature in prediction for low noise maps, as when the model does use voids, it relies on them more.

Therefore, we are able to obtain explanations meaningful to expert cosmologists from SOP, which is a promising step towards meaningful scientific discovery. We include additional cosmology background in Appendix D.

Related Works

Self-attributing neural networks uses feature-based interpretable atoms and include per-feature models such as NAM (Agarwal et al., 2021), and group-based models such as BagNet (Brendel & Bethge, 2019) and FRESH (Jain et al., 2020). While previous works did use per-feature SANNs mainly in tabular data domains (Agarwal et al., 2021) and group-based SANNs in image and text domains (Brendel & Bethge, 2019;Jain et al., 2020), they did not formalize the problem of growing error in per-feature SANNs. Also, previous models rely on specific architectures while our framework allows the use of arbitrary pre-trained models (Li, 2023;Jo & Kim, 2023). Other self-explaining neural networks include prototype-based (Ma et al., 2024;Wen et al., 2024) and concept-based (Koh et al., 2020;Yang et al., 2023;Lai et al., 2024;Yang et al., 2024) models, which explain with prototypical examples or concepts instead of input features. Self-attributing models that we use have attributions to input features, it is not directly comparable to prototypes and concepts. Program-execution models (Lyu et al., 2023) provide faithful reasoning but do not attribute to features.

For evaluating SANNs, Nauta et al. (2023) advocates prioritizing performance over faithfulness as the primary metric; we adopt this approach in our evaluation. Coherence (Nauta et al., 2023) is also proposed to evaluate how well the explanation aligns with domain knowledge ground truth, using metrics such as Intersection over Union (Bau et al., 2017;Wang & Vasconcelos, 2020), outside-inside relevanace ratio (Nam et al., 2020), and pointing game accuracy (Du et al., 2018;Huang & Li, 2020). We thus use IOU when there is ground truth (Gao et al., 2022;DeYoung et al., 2020) and an expert-informed threshold-based purity when the ground truth is not available (Matilla et al., 2020). Human evalua- Ω m σ 8 Cosmological Parameter 0.0 0.5 1.0 Attribution Void Cluster (a) Ωm vs. σ8 Attributions 0.00 0.25 0.50 0.75 1.00 Contribution Density Void Cluster (b) Voids vs. Clusters Weights Figure 8. (a) Ωm relies more on cosmological structures than σ8, as shown by box plots of group weights from SOP for predicting Ωm and σ8. Voids on average contribute 50.7% to Ωm and 41.5% to σ8, and clusters contribute 34.0% to Ωm and 28.5% to σ8. (b) Many voids contribute 100% to the predictions, while clusters are weighted less, as shown by the density plot.

tions are used to validate utilities (Kim et al., 2022;Akula et al., 2020;Hase & Bansal, 2020). We are using a standard human distinction task from the HIVE protocol (Kim et al., 2022). Matilla et al. (2020) attempted to use post-hoc explanations in scientific discovery for cosmology. Our work uses a self-attributing model to validate the same results.

A Theory Details and Proofs

In addition to the insertion error, we can define the deletion error for a SANN and a target true function when excluding a subset. Definition A.1. (Deletion Error) Let α i = θ(x) i h(x Gi ) be the total contribution of the ith feature group to the prediction of a SANN. Then the deletion error of a self-attributing neural network f

Then the total deletion error over all possible deletions is S⊆ [d] DelErr(G, α, S).

We can solve the exact total deletion error for multilinear monomials with linear programs. These lowerbounds are then the minimum error in performance of SANNs attempting to learn the simple monomials.

where (M S , c S ) are defined as (M S ) i = 1 if i ∈ S 0 otherwise, and c S contains the remaining constant terms.

This captures the faithfulness notion that α i is faithful if it reflects a contribution of x i to the prediction of the target function.

Then, the self-attributing neural network f (x) = d i=1 α * i with α * that achieves the lowest possible faithfulness error over all possible subsets is α * = arg min

This can be more compactly written as

The minimum total deletion error can then be solved by the following linear program

To obtain the lower bound for total deletion error, we can solve the dual of this linear program. Given the above primal linear program, we can find the Lagrangian

For the dual function to be bounded below, the coefficients of α and β must be zero:

Since we are minimizing over α, β, the dual objective is to maximize

In summary, the dual problem is

Let λ be feasible, then D ins ( λ) = (λ 1λ 2 ) ⊤ c ≤ S∈P InsErr(α, S). We can then maximize the lower bound to the primal program (5) in linear programming solvers such as CVXPY which maximizes the dual program (9). x i . Then, the lower bound of total deletion error for p follows an exponential trend as dimension d grows, where the lower bound is approximately γ 0 + e γ1+γ2d , where (γ 0 , γ 1 , γ 2 ) = (-1.030, -1.171, 0.665).

We solve for α * in (5) using ECOS in the cvxpy library for d ∈ {2, . . . , 20}. To fit the exponential function, we fit a linear model to the log transform of the output which has high degree of fit (with a relative absolute error of 0.012), with the resulting exponential function shown in Figure 2b.

In other words, Theorem A.2 states that we can find the exact lower bound of the total deletion error of a monomial, and Conjecture A.1 posits that lower bound of the total deletion error of any feature attribution of a monomial grows exponentially with respect to the dimension, as visualized in Figure 2b For high-dimensional problems, this suggests that there does not exist a feature attribution that satisfies all possible deletion tests. On the other hand, monomials can easily achieve low insertion error, as formalized in Theorem A.3. Theorem A.3 (Insertion Error for Monomials). Let p : {0, 1} d → {0, 1} be a multilinear monomial function of d variables, p(x) = d i=1 x i . Then, for all x, there exists a self-attributing neural network f (x) = d i=1 α i for p at x that incurs at most 1 total insertion error.

Proof. Consider α = 0 d . If x ̸ = 1 d then this achieves 0 insertion error. Otherwise, suppose x = 1 d . Then, for all subsets S ̸ = [d], p(x S ) = 0 = i∈S α i so α incurs no insertion error for all but one subset. For the last subset S = [d], the insertion error is 1. Therefore, the total insertion error is at most 1 for α = 0 d . However, once we slightly increase the function complexity to binomials, we find that the total insertion error of any feature attribution likely follows an exponential trend with respect to d, as shown in Figure 2a. Theorem 2.3 (Lower Bound on Insertion Error for Binomials). Let p : {0, 1} d → {0, 1, 2} be a multilinear binomial polynomial function. Furthermore suppose that the features can be partitioned into (S 1 , S 2 , S 3 ) of equal sizes where p(x) = i∈S1∪S2 x i + j∈S2∪S3 x j . Then, S⊆[d] InsErr(G, α, S) ≥ D ins ( λ), where D ins ( λ) = ( λ1 -λ2 ) ⊤ c is the lower bound, λ is a dual feasible point, and c is a constant as defined in (16).

Proof. Consider x = 1 d . The addition error for a binomial function can be written as

where (M S , c S ) are defined as (M S ) i = 1 if i ∈ S 0 otherwise, and c S contains the remaining constant terms. Then, the least possible insertion error that any attribution can achieve is

The minimum total insertion error can then be solved by the following linear program

To obtain the lower bound for total insertion error, we can solve the dual of this linear program. Given the above primal linear program, we can find the Lagrangian

For the dual function to be bounded below, the coefficients of α and β must be zero:

Since we are minimizing over α, β, the dual objective is to maximize

In summary, the dual problem is

Let λ be feasible, then D ins ( λ) = (λ 1λ 2 ) ⊤ c ≤ S∈P InsErr(α, S). We can then maximize the lower bound to the primal program (12) in linear programming solvers such as CVXPY which maximizes the dual program (16).

Conjecture A.2 (Insertion Error for Binomials Grows Exponentially with Dimension). Let p be a multilinear binomial function of d variables as defined in Theorem 2.3. Then, the lower bound of total insertion error for p follows an exponential trend as dimension d grows, where the lower bound is approximately λ 0 + e λ1+λ2d , where (λ 0 , λ 1 , λ 2 ) = (4. 778, 1.332, 0.198).

We maximize the lower bound by solving the dual linear program in (16) using ECOS in the cvxpy library for d ∈ {2, . . . , 20}. To get the exponential function, we fit a linear model to the log transform of the output, doing a grid search over the auxiliary bias term. The resulting function has a high degree of fit (with a relative absolute error of 0.188), with the resulting exponential function shown in Figure 2a.

A.1 Groups

Theorem A.4 (Insertion and Deletion Error for Groups). Consider p 1 and p 2 , the polynomials from Conjecture A.1 and Conjecture A.2. Then, there exists a group-based self-attributing neural network with zero deletion and insertion error for both polynomials.

Proof. Let [d] denote {1, . . . , d}. First let p 1 (x) = i x i and consider a self-attributing neural network with one group,

Otherwise, no matter what subset S is being tested, S ⊆ [d] is always true, thus:

Therefore the total grouped deletion error for p 1 is 0. Next let p 2 (x) = i∈S1∪S2 x i + j∈S2∪S3 x j and consider a self-attributing neural network with two groups,

then

If S empty, then the insertion error is trivially 0. Otherwise suppose S is missing an element from one of S 1 or S 3 . WLOG suppose it is from S 1 but not S 2 or S 3 . Then,

Otherwise, suppose we are missing elements from both S 1 and S 3 . Then,

Lastly, suppose we are missing elements from S 2 . Then,

Thus by exhaustively checking all cases, p 2 has zero grouped insertion error. Therefore the total grouped insertion error for p 2 is 0. , where

is the set of indices for terms associated with group G i , and b ijk ∈ Z + is the exponent for feature j in subset i in the term indexed by k ∈ K i . Then, a self-attributing neural network needs at most m groups to achieve zero deletion and insertion error for polynomial p.

Proof. Let [d] denote {1, . . . , d}. Let q i (x) = k∈Ki a ik j∈Gi x b ijk j be the ith polynomial with K i terms, and then we can rewrite p(x) = q 1 (x) + • • • + q m (x). We prove by induction that we can have a self-attributing neural network of m groups G = {G 1 , . . . , G m } with contributions α = {q 1 (x), . . . , q m (x)} to achieve zero deletion and insertion error for polynomial p. For groups and contribution scores for up to m groups, we denote with G (m) and α (m) , and omit the superscripts when the context is clear.

Insertion. Base Case: Suppose m = 1, then p 1 (x) = k∈K1 a 1k j∈G1 x b 1jk j = q 1 (x) and consider a self-attributing neural network with one group, G (1) = {G 1 } with contribtuions α (1) = {q 1 (x)}. As there are no other input features,

We proved that we can have a self-attributing neural network with groups G (1) = {G 1 } and group contributions α (1) = {q 1 (x)} for polynomial p 1 (x) = q 1 (x), which only has one group. Therefore, we can need at most one group to achieve zero grouped insertion error for monomial p 1 .

Inductive

Step: Assume that it holds for (m -1)-nomial polynomial p m-1 (x) = q 1 (x) + • • • + q m-1 (x) that we need at most (m -1) groups to achieve zero insertion error, where the groups are G (m-1) = {G 1 , . . . , G m-1 } with group contributions α (m-1) = {q 1 (x), . . . , q m-1 (x)} which means that

This holds for all S ⊆ [d].

Now, we prove it for m-nomial polynomial p m (x) = q 1 (x) + • • • + q m (x). There are two cases. First, if G m ⊈ S, meaning that not all G m features are in S, but there are some parts of G m in ¬S. Then f * (x S ) does not contain the polynomial term q m (x) that uses G m . Thus,

= InsErr(G (m-1) , α (m-1) , S) = 0

Otherwise, if G m ⊆ S, meaning that all G m features are in S, and no features that are in G m are contained in S, then f * (x S ) contains the polynomial term q m (x) that uses G m . Thus,

The last steps of the above derivations use the induction from (17). Thus by exhaustively checking all cases, p m has zero group insertion error with self-attributing neural networks with groups G (m) and group contributions α (m)

Deletion. Base Case: Suppose m = 1, then p 1 (x) = k∈K1 a 1k j∈S1 x b 1jk j = q 1 (x) and consider a self-attributing neural network with one group G (1) = {G 1 } and contribution α (1) = {q 1 (x)}. As there are no other input features,

We proved that we can have a self-attributing neural network f (x) = θ(x) i h(x [d] ) = q 1 (x) with groups G (1) = {G 1 } and group contributions α (1) = {q 1 (x)} for polynomial p 1 (x) = q 1 (x), which only has one group. Therefore, we can need at most one group to achieve zero grouped deletion error for monomial p 1 .

A.2 Alternative Mean Square Error Insertion/Deletion Errors

In the main paper, we use mean absolute error for Insertion Error and Deletion Error. Here we also compute the exact Deletion Errors for monomials and Insertion Errors for binomials up to d = 20. We conjecture that the two errors will also follow an exponential growth. Definition A.6. (Mean Square Deletion Error) Let α i θ(x) i h(x Gi ) be the total contribution of the ith feature group to the prediction of a SANN. Then the mean square deletion error of a self-attributing neural network f (x) = m i=1 θ(x) i h(x Gi ) = i α i for a target function f * : R d → R when removing a subset of features S from an input x is

d}. Then the total mean square deletion error over all possible deletions is S⊆[d] DelErr(G, α, S). Theorem A.7 (Lower Bound on Mean Square Deletion Error for Monomials). Let p : {0, 1} d → {0, 1} be a multilinear monomial function of d variables, p(x) = d i=1 x i . Then, S⊆[d] DelErr MSE (G, α, S) ≥ α * , where α * del

is the lower bound, and M and c are constants as defined in (28).

Proof. Let x = 1 d , and let α ∈ R d be any feature attribution. Consider the set of all possible perturbations to the input, or the power set of all features P, We can write the error of the attribution under a given perturbation S ∈ P as

where (M S , c S ) are defined as (M S ) i = 1 if i ∈ S 0 otherwise, and c S contains the remaining constant terms.

This captures the faithfulness notion that α i is faithful if it reflects a contribution of α i to the prediction. Then, the feature attribution α * that achieves the lowest possible faithfulness error over all possible subsets is

where

for an enumeration of all elements S i ∈ P.

This is a quadratic function without constraint, and thus we can analytically solve the exact minimum by finding where the gradient is zero.

To solve min

We first expand the squared norm as

Substituting into the objective function:

Since 1 ⊤ is a summation operator over all elements, this simplifies to:

Then, we compute the gradient. We define

Taking the derivative with respect to α:

Setting the gradient to zero:

Finally, we can solve for α. If M ⊤ M is invertible, we obtain the optimal solution:

As M is the enumeration of all elements S i ∈ P, the columns in M are linearly independent to each other, and thus M ∈ {0, 1} 2 d ×d is invertible with column rank d.

By the fundamental rank theorem,

Since M ⊤ M ∈ Z d×d , it is then invertible.

Thus we prove that we can solve the optimal solution with (28). x i . Then, the lower bound of the total mean square deletion error for p follows an exponential trend as dimension d grows, where the lower bound of total mean square deletion error is approximately γ 0 + e γ1+γ2d , where (γ 0 , γ 1 , γ 2 ) = (-1.104, -1.720, 0.626).

We solve for α * in ( 20) using ( 28) for d ∈ {2, . . . , 20}. To fit the exponential function, we fit a linear model to the log transform of the output which has high degree of fit (with a relative square error of -0.293), with the resulting exponential function shown in Figure A9b.

Definition A.8. (Mean Square Insertion Error) Let α i = θ(x) i h(x Gi ) be the total contribution of the ith feature group to the prediction of a SANN. Then, the mean square insertion error of a self attributing neural network

The total mean square insertion error over all possible insertions is S⊆ [d] InsErr(G, α, S).

Theorem A.9 (Lower Bound on Mean Square Insertion Error for Binomials). Let p : {0, 1} d → {0, 1, 2} be a multilinear binomial polynomial function of d variables. Furthermore suppose that the features can be partitioned into (S 1 , S 2 , S 3 ) of equal sizes where p(x) = i∈S1∪S2 x i + j∈S2∪S3 x j . Then, S⊆[d] InsErr MSE (G, α, S) ≥ α * , where α * del = (M ⊤ M ) -1 M ⊤ c is the lower bound, and M and c are constants as defined in (40).

Proof. Consider x = 1 d . The addition error for a binomial function can be written as

where (M S , c S ) are defined as (M S ) i = 1 if i ∈ S 0 otherwise, and c S contains the remaining constant terms. Then, the least possible insertion error that any attribution can achieve is

where (M, c) are constructed by stacking (M S , c S ) for some enumeration of S ∈ P.

This is a quadratic function without constraint, and thus we can analytically solve the exact minimum by finding where the gradient is zero.

A.3 Discussion on Theoretical Proofs

While our theorems and that of Bilodeau et al. (2024) both present impossibility results for feature attributions, the assumptions and resulting theorem are different. Bilodeau et al. (2024) put forth a result that says (put simply) that linear models cannot accurately capture complex models, where complexity is measured by having a large number of piece-wise linear components. Indeed, if we had shown that a linear model is not a good approximation of a highly non-linear model, then this would not be a novel contribution. This is also an unsurprising result (it is not surprising that a linear model cannot approximate a highly non-linear model).

However, our result paints a significantly bleaker picture: we show that a linear feature attribution is unable to model the extremely simple functions in our theorems. Our examples distill the problem to the fundamental issue in its purest form: correlated features. Specifically, we show feature attribution is impossible with only one group of correlated features. This is the polar opposite assumption than that of Bilodeau et al. (2024), and we argue that it is more surprising for feature attribution to be impossible for simpler functions than for complex functions.

Second, we provide not only a negative impossibility result for standard feature attributions, but also a positive result for grouped feature attributions that provides a path forward and motivates the approach in our submission. This is in contrast to Bilodeau et al. (2024), which only presents negative impossibility results in standard feature attributions without clear suggestions on where to go.

In summary, our theoretical results differ in Assumption (we assume simple functions with a single correlation whereas Bilodeau et al. ( 2024) assume complex functions with many piece-wise linearities) Theoretical results (we show both positive and negative results, whereas Bilodeau et al. (2024) only show negative results).

B Method Details B.1 Group Generator Details

The group generator Γ : R d → {0, 1} m×d takes in an input x ∈ R d and outputs m groups (g 1 , . . . , g m ), Each g i ∈ {0, 1} d is a binary mask such that if the feature j is included in group g i , then g ij = 1, otherwise 0.

To generate these groups, we first project the input x into embedding dimension e with embedding function h e : R d → R d×k .

In our experiments, we typically use up to the second to last layer in the backbone model as the embedding function to obtain the contextualized embedding of each feature, and we finetune this copy of projection layer while keeping the original backbone model unchanged.

Soft Multiheaded Self-Attention. Then, we use a self-attention mechanism a : R d×e → [0, 1] d×d (Vaswani et al., 2017) to parameterize a probability distributions over features.

where W Q , W K ∈ R k are learned parameters, and α is a temperature scaling hyperparameter. In practice, one might not need that many groups and can use m < d groups, resulting in a(x) ∈ [0, 1] m×d . However, the outputs of self-attention are continuous and dense. To make groups sparse, we binarize the attention by taking top τ = 0.2 features for each group. Each group g i is then the binarized a(x) where g ij = 1 if jth feature is within top τ according to a(x), and otherwise g ij = 0.

Input Embedding Encoder. In practice, the embedding function h e is initialized with parts of the backbone predictor h.

The embedding function h e can be finetuned as part of the group generator while not changing the backbone predictor.

Binary Groups and the Risk of Out-of-Distribution Data. The groups (i.e. the image patch subsets or language token subsets) are binary. However, the scores for each group are real valued. Since we are using discrete groups of features for making each prediction, the group masks need to be binarized. Nonbinary groups (i.e. close to zero but non-zero) will have information leakage from the nonzero input features.

A potential risk of blacking out pixels is that it will create Out-of-Distribution (OOD) data. However, modern Transformers are not significantly affected by this bias due to the significant data augmentations used during pretraining. For example, Jain et al. (2022) show that ResNet suffers more from masking out tokens while ViT mostly is able to maintain its original prediction even when some parts are blacked out. In our ImageNet and MultiRC experiments, we use these transformer based models, and all baselines use the same transformer in fairness, except XDNN, BCos and BagNet which do not have Transformer counterparts. For CosmoGrid, though we are using a CNN-based model, the data form is very different from natural images, and zeroing parts of the map is not as OOD as it would be for natural images.

B.2 Group Selector Details

Sparse Multiheaded Cross-Attention. The group selector θ : {0, 1} m×d × R d → [0, 1] m then assigns a weight to each group such that a sparse subset of the groups get nonzero weights. The scores (c 1 , . . . , c m ) are produced by a cross attention using the target class's weights C h ∈ R k as the query and all groups' hidden states z = (h h (g 1 ⊙ x), . . . , h h (g m ⊙ x)) ∈ R m×k as the key, where h h : R d → R k is usually the same backbone as h but outputs the hidden states with hidden dimension k:

where

parameters, and β is a temperature scaling factor.

Having scores on a sparse subset of groups helps avoid overloading human users. As the outputs of softmax are continuous and dense, we use a sparse variant, the sparsemax operator (Martins & Astudillo, 2016), to assign scores to the groups. The sparsemax operator uses a simplex projection to make the attention weights sparse and only assigns nonzero scores to a few groups.

In practice, we can initialize the query C h to a row of the weight matrix in the linear classifier of the pretrained model h, and W Q ′ and W K ′ to the identity matrix, since the pretrained model already learns a relation between the hidden states z and class weights C h .

Hidden State Encoder. In practice, the embedding function h e , and encoder for hidden states h h are both initialized as parts of the backbone predictor h. The embedding function h e can be finetuned as part of the group generator, while the hidden state encoder h h is always kept frozen and computed at the same time of making the prediction with h. Therefore, h h does not incur additional forward passes.

B.3 Scaling the Loss

Since the gradient cannot pass through the thresholded binary mask, we use a scale λ ∈ [-1, 1] that is the difference between sums of attention weights of selected and unselected features and multiply that to the logits for optimization.

The gradient-passing scales (λ 1 , . . . , λ m ), the group scores (c 1 , . . . , c m ), and predictions (y 1 , . . . , y m ) are then combined with a weighted average to make the final prediction

Classification. For classification, we multiply the scaler λ on the logits. If the prediction from group i is prefered, then λ i will also increase and thus up-weigh the selected features in the group. Conversely, if the prediction from group i is not prefered, then λ i will decrease and thus down-weigh the selected features and up-weigh other features in the group.

Scaled CrossEntropyLoss(⃗ y, y * , ⃗ c, ⃗ λ) = CrossEntropyLoss(

where y i is the predicted logit from each group and y * is the ground truth label.

Regression. For regression, we multiply the negated scaler on the loss instead of the logits because we need to scale the loss instead of the absolute value.

Scaled MSELoss(⃗ y, y * , ⃗ c, ⃗ λ) = min(

Method Abbr Model-Agnostic Self-Attributing Learnable Groups Self-Attributing Ver.

LIME (Ribeiro et al., 2016 (Petsiuk et al., 2018) RISE ✓ ✗ ✗ RISE-F Archipelago (Tsang et al., 2020) Archi. ✓ ✗ ✗ Archi.-F MFABA (Zhu et al., 2024) MFABA ✓ ✗ ✗ MFABA-F AGI (Pan et al., 2021) AGI ✓ ✗ ✗ AGI-F AMPE (Zhu et al., 2023) AMPE ✓ ✗ ✗ AMPE-F BCos (Böhle et al., 2022) BCos ✗ ✗ ✗ BCos-F XDNN (Hesse et al., 2021) XDNN

Table A4. Properties of all post-hoc feature attributions and self-attributing neural networks we use. SOP is the only attribution method that is both model agnostic and self-attributing and has learnable groups.

• Input Features: We use tokens as individual features.

•

C.3 Baselines

As we are building a new type of self-attributing neural networks, we compare with self-attributing neural networks that attribute to input features for all main experiments. The baselines we compare with are either already self-attributing neural networks, or converted from post-hoc attributions. The models that already fall under the class of self-attributing neural (a) Image: Tench (b) LIME-F (Ribeiro et al., 2016) (c) SHAP-F (Lundberg & Lee, 2017) (d) IG-F (Sundararajan et al., 2017) (e) GC-F (Selvaraju et al., 2016) (f) FG-F (Srinivas & Fleuret, 2019) (g) RISE-F (Petsiuk et al., 2018) (h) Archi.-F (Tsang et al., 2020) (i) MFABA-F (Zhu et al., 2024) (j) AGI-F (Pan et al., 2021) (k) AMPE-F (Zhu et al., 2023) (l) BCos-F (Böhle et al., 2022) (m) XDNN (Hesse et al., 2021) (n) BagNet (Brendel & Bethge, 2019) (o) FRESH (Jain et al., 2020) (p) SOP (Ours) Figure A11. Example groups from different feature attribution methods for an image of Tench. "-F" indicates self-attributing models converted from post-hoc methods. The highlights show the groups selected by each method (top 20% attributed patches), with unused patches hatched-out.

Fidelity can be measured by the KL-Divergence between the model predicted probabilities p(x) ∈ [0, 1] k for all k classes and the probability using summed attributions p(x) ∈ [0, 1] k for all k classes (Yu & Varshney, 2017;Chen et al., 2019;Anders et al., 2020).

where p = softmax(f (x)) is the predicted probability of the model, and p = softmax( i α i ) is the probability of summed attribution scores for data x in distribution D. As self-attributing neural networks all follow the design f (x) = m i=1 θ(x) i h(x) i and the attribution scores α i = θ(x) i h(x) i , self-attributing neural networks by design achieve 0 fidelity. We report fidelity of post-hoc models in Table A5 and show that self-attributing neural networks are the only ones that achieve perfect fidelity.

For SOP, the sum of attribution scores is the sum of group predictions weighted by the group selector scores i α i = i c i y i . For FRESH (Jain et al., 2020), they only have one group, so the score for each class is the same as the prediction from that one group. All the self-attributing neural networks achieves fidelity of 0 by construction.

For most post-hoc baselines, the sum of attribution scores is simply the sum of all the scores for each feature. We report the fidelity scores for post-hoc methods in Table A5. We can see that no post-hoc method achieves perfect fidelity of 0. (a) Image: Cassette tape (b) LIME-F (Ribeiro et al., 2016) (c) SHAP-F (o) FRESH (Jain et al., 2020) (p) SOP (Ours) Figure A13. Example groups from different feature attribution methods for an image of Cassette tape. "-F" indicates self-attributing models converted from post-hoc methods. The highlights show the groups selected by each method (top 20% attributed patches), with unused patches hatched-out.

C.4.4 INSERTION AND DELETION

Original Insertion and Deletion. Intuitively, if we add features from the most important to the least important one-by-one starting with a blank image, the probability for the predicted class should go up quickly. Conversely, if we delete them from the most to least important, the probability should ideally drop instantly. The mainstream insertion and deletion metrics (Samek et al., 2017;Petsiuk et al., 2018) intend to evaluate if the features are actually important by inserting or deleting them one-by-one and computing the area-under-the-curve (AUC).

As insertion and deletion criteria are designed for pixel-level instead of groups of features, we consider a modified version of insertion and deletion for groups, which grow or shrink the groups instead of adding pixels to the model. We report the percentage probability AUC instead of the absolute, to accommodate the difference in accuracy when the models are different. We use step size of 10% following previous work (Wu et al., 2024). Table 3 shows that SOP performs the best for both insertion and deletion on ImageNet. Among other methods, the perturbation-based methods such as SHAP and LIME perform better than gradient-based methods. Table A6 shows that SOP performs the best for insertion on Cosmogrid and MultiRC. (a) Image: Castle (b) LIME-F (Ribeiro et al., 2016) (c) SHAP-F (o) FRESH (Jain et al., 2020) (p) SOP (Ours) Figure A15. Example groups from different feature attribution methods for an image of Castle. "-F" indicates self-attributing models converted from post-hoc methods. The highlights show the groups selected by each method (top 20% attributed patches), with unused patches hatched-out.

The mainstream insertion and deletion metrics (Samek et al., 2017;Petsiuk et al., 2018) are approximations intended to evaluate the faithfulness of post-hoc methods (Nauta et al., 2023). Nevertheless, we include results for insertion and deletion for completeness.

Adapting Insertion and Deletion for Groups. The mainstream insertion and deletion criteria has a problem when directly used to evaluate group attribution. When inserting a single group of features, it is unclear what order one should use to insert individual features. In images, inserting pixels by row versus randomly can result in very different results. As a result, it is not immediately obvious how to directly apply the standard insertion and deletion criteria to the group setting. Nevertheless, we can consider a modified version of insertion criteria for groups, where instead of adding a single pixel at a time, we instead gradually grow the size of all groups simultaneously. This procedure is equivalent to the classic insertion criteria when there is a single group. Specifically, we grow the size of the groups gradually from with an interval of 10% of all features (approximately around 5017 pixels), and compute the AUCs of the highest predicted probability of the faithful models.

For deletion, we perform a similar procedure by shrinking the groups gradually with the same interval of 10% pixels and computing the AUCs on the highest predicted probability of the faithful models. Because each faithful model's prediction start with a different value, it is hard to compare the deletion score. Therefore, we make a small modification to the original deletion metric and penalize the cases where the prediction is wrong by setting the score of that point to 1.

Method ImageNet CosmoGrid MultiRC Fidelity↓ Fidelity↓ Fidelity↓ LIME 3.866 ± 0.244 0.100 ± 0.000 1.550 ± 0.271 SHAP 0.015 ± 0.006 0.100 ± 0.000 1.628 ± 0.283 IG 7.161 ± 0.212 0.100 ± 0.000 1.921 ± 0.276 GC 10.406 ± 1.098 1.697 ± 0.277 1.697 ± 0.277 FG 13.567 ± 0.158 1.921 ± 0.277 1.921 ± 0.277 RISE 0.884 ± 0.533 0.100 ± 0.000 2.771 ± 0.231 Archi. 10.850 ± 0.354 1.684 ± 0.277 1.684 ± 0.277 MFABA 6.674 ± 0.166 1.690 ± 0.267 1.690 ± 0.267 AGI 5.416 ± 0.549 1.665 ± 0.254 1.665 ± 0.254 AMPE 13.671 ± 0.326 2.532 ± 0.493 2.532 ± 0.493 BCos 13.372 ± 0.373 --Table A5. Fidelity scores for post-hoc methods on ImageNet, CosmoGrid, and MultiRC. Lower values indicate better fidelity. We omit numbers for self-explaining models as they by design achieve fidelity of 0. Post-hoc attribution methods are not able to achieve fidelity of 0, which empirically shows that they are not faithful to the model prediction. For CosmoGrid, to compute KL-divergence for fidelity, an additional softmax operation is applied on top of the CNN predicted logits as KL-divergence only works on probability distributions. Category Method MultiRC CosmoGrid Ins.↑ Del.↓ Ins.↓ Del.↑ Post-Hoc-Converted LIME 0.869 ± 0.020 0.778 ± 0.005 0.028 ± 0.001 0.028 ± 0.001 SHAP 0.840 ± 0.016 0.839 ± 0.017 0.023 ± 0.001 0.023 ± 0.001 IG 0.878 ± 0.009 0.852 ± 0.019 0.027 ± 0.001 0.027 ± 0.001 GC 0.882 ± 0.011 0.876 ± 0.014 0.026 ± 0.001 0.026 ± 0.001 FG 0.928 ± 0.009 0.834 ± 0.014 0.025 ± 0.001 0.025 ± 0.001 RISE 0.961 ± 0.017 0.832 ± 0.016 0.027 ± 0.001 0.027 ± 0.001 Archi. 0.669 ± 0.021 0.920 ± 0.015 0.036 ± 0.002 0.036 ± 0.002 MFABA 0.863 ± 0.011 0.873 ± 0.007 0.023 ± 0.004 0.023 ± 0.004 AGI 0.929 ± 0.013 0.901 ± 0.007 0.024 ± 0.004 0.024 ± 0.004 AMPE 0.868 ± 0.007 0.892 ± 0.018 0.027 ± 0.004 0.027 ± 0.004 Self-Attributing FRESH 0.937 ± 0.013 0.710 ± 0.028 --SOP 1.018 ± 0.022 0.949 ± 0.007 0.020 ± 0.001 0.027 ± 0.002 Table A6. Insertion/Deletion metrics for MultiRC and CosmoGrid. For MultiRC (left columns), higher insertion (Ins.↑) and lower deletion (Del.↓) are better. Best result is bolded, second-best is italicized. As FRESH is designed for text, it is unsurprising that it achieves better results on some metrics in text. We compute percent insertion/deletion scores as discussed in Appendix C.4.4, which is why it is possible for SOP to obtain an insertion score larger than 1. For CosmoGrid (right columns), lower insertion (Ins.↓) and higher deletion (Del.↑) are better due to MSE loss. Best result is bolded, second-best is underlined. All tables report percent insertion and deletion scores with an interval of 10%. FRESH is not applicable to CosmoGrid.

Also, as we are using different self-attributing neural networks, the base prediction can be different, making it unfair when comparing the insertion and deletion scores. We thus use the percent probability comparing with using all the input instead of the raw probability.

For self-attributing neural networks that select groups by thresholding attention, such as SOP and FRESH, we grow the size of the group according to the attention scores. For models that have fixed group sizes, such as XDNN which uses groups of single pixels, and BagNet which uses groups of fixed patches, we perform insertion/deletion by adding/removing patches from the aggregation, following the same percentage for step sizes. Similar things are performed when evaluating accuracy for different sparsity levels.

Different

Step Sizes. Here in the Appendix, we also report more finegrained insertion and deletion scores using step sizes of 16 × 16 patches (which equals 256 pixels) in

Table A7. The result is very similar and SOP still achieves the best insertion and deletion scores. Different Occlusion Strategies. There can be different occlusion values to use when masking out the groups. To test if the choice of feature deletion values affects the results, we ran additional insertion/deletion experiments using an alternative deletion value as used in Bluecher et al. (2024) (specifically, randomly sampled color from the image). The results in Category Method ImageNet -ViT Step size 256 pix. Occlusion w. Colors Ins.↑ Del.↓ Ins.↑ Del.↓ Post-Hoc-Converted LIME-F 0.815 ± 0.005 0.428 ± 0.004 0.797 ± 0.007 0.555 ± 0.023 SHAP-F 0.831 ± 0.006 0.373 ± 0.008 0.909 ± 0.018 0.524 ± 0.023 IG-F 0.611 ± 0.006 0.617 ± 0.008 0.679 ± 0.030 0.786 ± 0.059 GC-F 0.772 ± 0.007 0.366 ± 0.007 0.789 ± 0.021 0.540 ± 0.031 FG-F 0.759 ± 0.005 0.383 ± 0.004 0.830 ± 0.029 0.447 ± 0.026 RISE-F 0.590 ± 0.008 0.661 ± 0.004 0.652 ± 0.043 0.776 ± 0.019 Archi.-F 0.676 ± 0.003 0.501 ± 0.004 0.824 ± 0.058 0.608 ± 0.025 MFABA-F 0.674 ± 0.006 0.499 ± 0.010 0.855 ± 0.048 0.610 ± 0.085 AGI-F 0.735 ± 0.006 0.462 ± 0.008 0.875 ± 0.048 0.585 ± 0.050 AMPE-F 0.675 ± 0.007 0.534 ± 0.005 0.738 ± 0.078 0.645 ± 0.041 BCos-F 0.257 ± 0.005 0.288 ± 0.008 0.574 ± 0.175 0.380 ± 0.032 Self-Explaining XDNN 0.199 ± 0.007 0.156 ± 0.003 0.245 ± 0.038 0.254 ± 0.044 BagNet 0.560 ± 0.006 0.417 ± 0.007 0.878 ± 0.023 0.228 ± 0.022 FRESH 0.713 ± 0.002 0.369 ± 0.004 0.746 ± 0.033 0.512 ± 0.046 SOP 0.890 ± 0.004 0.014 ± 0.000 0.910 ± 0.010 0.106 ± 0.000 Table A7. ImageNet Insertion (Ins.↑) and Deletion (Del.↓) metrics. Higher Ins. and lower Del. are better. Best results are bolded, second-best are italicized. Step size results use 16 × 16 patches (256 pixels, 5% of image). Occlusion uses a random color. All tables report percent scores with an interval of 10%.

Table A7 aligns with our previous results of replacing the deletion value with 0 and SOP still performs the best for both insertion and deletion on ImageNet. This shows that the evaluation is consistent with other feature deletion values.

Discussion: Biases in Insertion and Deletion Tests. SOP performs well on the insertion test for all tasks, while being only the best on deletion task for one task. We notice a bias in the deletion test. As the deletion test favors SANNs such that deleting the most important feature results in a sharp performance drop, it biases towards models that rely primarily on a small number of features. On the other hand, if a model distributes its dependence to multiple different groups of features, removing the new most important features will not lead to a large performance drop. While deletion test is initially designed to evaluate post-hoc feature attributions that attempt to explain the same backbone model, it is not well-suited to evaluate self-attributing models, as it will score models that depend on a few features more, regardless of the underlying faithfulness of the explanation.

For example, if there are multiple flowers in an image, and the model only looks at the upper left corner, while ignoring all other parts of the image, then removing the flower in the upper left corner will reduce the model predicted probability for the flower to 0. However, if the model averages the prediction from different parts of the image, its predicted probability for the flower will only drop a little bit if its most used flower is removed. This doesn't mean that the second model's explanation is any less faithful than the first one. On the other hand, the model's score for the one flower could be a smaller number that faithfully reflects the small amount of confidence reduced.

Figure A16 shows average deletion curves for SOP. We see that even as we delete features from groups, SOP is able to maintain relatively high performance, resulting in a worse deletion score. Such behavior is natural because the training objective in SOP encourages the group selector to select highly predictive groups, and multiple groups can compensate for the information missing in another group.

Insertion/Deletion Results for CosmoGrid and MultiRC. Table A6 shows insertion and deletion results for MultiRC and CosmoGrid. We see that SOP is consistently good on the insertion metric while LIME is better at deletion. We conjecture that this is due to different groups in SOP compensating for each other.

C.4.5 INFORMATION LEAK FROM THE GROUP GENERATOR

We show all the results for probing if the groups contain the labels in Table A8. We can see that models trained on the SOP groups are not able to predict the labels well. While results from linear and ViT models are not differentiating different methods, CNN probing models show that the groups from other methods like FG-F and AGI-F are leaking a lot of information. Archipelago is omitted because of the significant computational cost. to generate explanations for training 0.0 0.3 0.6 0.9

Fraction of Patches Removed

0.0 0.5 1.0 Percent Probability Deletion Curve ImageNet Method LIME-F SHAP-F IG-F FG-F GC-F RISE-F Archi.-F MFABA-F AGI-F AMPE-F BCos-F XDNN BagNet FRESH SOP 0.0 0.3 0.6 0.9 Fraction of Patches Removed 0.00 0.05 0.10 Loss Deletion Curve CosmoGrid Method LIME-F SHAP-F IG-F FG-F GC-F RISE-F Archi.-F MFABA-F AGI-F AMPE-F SOP 0.0 0.3 0.6 0.9 Fraction of Patches Removed 0.7 0.8 0.9 1.0 Percent Probability Deletion Curve MultiRC Method LIME-F SHAP-F IG-F RISE-F Archi.-F MFABA-F AGI-F AMPE-F FRESH SOP Figure A16. Deletion curves across datasets. SOP's multi-group approach allows it to maintain low loss when only a few input features are preserved. Other methods show unstable behavior, especially in CosmoGrid. LIME-F SHAP-F IG-F GC-F FG-F MFABA-F AGI-F AMPE-F BCos-F XDNN BagNet FRESH SOP Model CNN 0.10 0.17 9.38 2.96 13.40 8.27 10.66 6.67 0.12 0.14 0.06 2.55 0.10 Linear 2.58 0.15 3.41 3.46 2.95 2.95 3.06 3.28 2.32 3.14 0.15 3.61 2.62 ViT 0.13 0.11 0.10 0.07 0.16 0.11 0.08 0.12 0.13 0.09 0.28 0.08 0.07 Table A8. (ImageNet Group Probing Model Accuracy) A model trained on group masks from SOP is unable to obtain accuracies much more than random. This indicates that the powerful group generator in SOP is not doing all the work and not compromising SOP's interpretability. RISE and Archipelago are omitted because of the significant computational cost to generate explanations for training examples.

examples, while the experiments with all other baselines already demonstrate the interpretability of linear combination in SOP.

C.4.6 HUMAN DISTICTION TEST SOP learns to use predictions from feature groups for the final prediction. However, if the group explanations are the same for different classes, they are not semantically relevant to the classes. We follow the HIVE protocol (Kim et al., 2022) and conduct a human distinction task on ImageNet-S to assess if people can predict which class the model predicts based solely on the group explanations for the classes. In the study, human evaluators are shown an image with group explanations for four classes and asked to guess the model prediction. Figure A18 shows a bar plot of human distinction task accuracies, where explanations from SOP is among the ones having most distinctive explanation for the predicted class. The large error bars do not declare a single winner, while being consistent with original experiments in HIVE (Kim et al., 2022). 7Nevertheless, SOP groups for different classes are distinctive enough to provide meaningful explanations.

We follow HIVE (Kim et al., 2022) for constructing our distinction task for a human simulation test. Figure A18 shows the accuracy of the human distinction task and its standard deviation bootstrapped 1000 times. This human evaluation is conducted on Amazon Mechanical Turk (MTurk) on 100 examples each evaluated by 3 mturk users.

The large standard deviation is consistent with the literature and thus there is no one absolute winner method for human simulation (Kim et al., 2022). Figure A17 show the interface we display to the human evaluators.

superclusters and voids, develop throughout cosmic history. However, Ω m and σ 8 are not directly measurable, and the inverse relation from cosmological structures in the weak lensing map to Ω m and σ 8 is unknown.

One approach to inferring Ω m and σ 8 from weak lensing maps, as demonstrated for example by Ribli et al. (2019); Matilla et al. (2020);Fluri et al. (2022), is to apply deep learning models that can compare measurements to simulated weak lensing maps. Even though these models have high performance, we do not fully understand how they predict Ω m and σ 8 . As a result, the following remains an open question in cosmology:

What structures from weak lensing maps drive the inference of the cosmological parameters Ω m and σ 8 ?

In collaboration with expert cosmologists, we use convolutional networks trained to predict Ω m and σ 8 as the backbone of an SOP model to get accurate predictions with faithful group attributions. Crucially, the guarantee of faithfulness in SOP provides confidence that the attributions reflect how the model makes its prediction, as opposed to possibly being a red herring. We then interpret and analyze these attributions and understand how structures in weak lensing maps of CosmoGridV1 (Kacprzak et al., 2023) influence Ω m and σ 8 .

It will be interesting to explore how these results change as we mimic realistic data by adding noise and measurement artifacts. Other aspects worth exploring are the role of "super-clusters" that contain multiple clusters, and how to account for the fact that voids occupy much larger areas on the sky than clusters (i.e., should we be surprised that they perform better?).