元記事の説明文
<!-- twitter -->
<p style="text-align: center;">
<!-- <img src="https://bair.berkeley.edu/static/blog/spex/image1.png" alt="different_tests" width="600"><br> -->
<img alt="different_tests" src="https://bair.berkeley.edu/static/blog/spex/image1.png" width="600" /><br />
<i style="font-size: 0.9em;">
</i>
</p>
<p>Understanding the behavior of complex machine learning systems, particularly Large Language Models (LLMs), is a critical challenge in modern artificial intelligence. Interpretability research aims to make the decision-making process more transparent to model builders and impacted humans, a step toward safer and more trustworthy AI. To gain a comprehensive understanding, we can analyze these systems through different lenses: <strong>feature attribution</strong>, which isolates the specific input features driving a prediction (<a href="https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html">Lundberg & Lee, 2017</a>; <a href="https://dl.acm.org/doi/abs/10.1145/2939672.2939778">Ribeiro et al., 2022</a>); <strong>data attribution</strong>, which links model behaviors to influential training examples (<a href="https://proceedings.mlr.press/v70/koh17a/koh17a.pdf">Koh & Liang, 2017</a>; <a href="https://proceedings.mlr.press/v162/ilyas22a/ilyas22a.pdf">Ilyas et al., 2022</a>); and <strong>mechanistic interpretability</strong>, which dissects the functions of internal components (<a href="https://papers.nips.cc/paper_files/paper/2023/hash/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Abstract-Conference.html">Conmy et al., 2023</a>; <a href="https://openreview.net/forum?id=91H76m9Z94">Sharkey et al., 2025</a>).</p>
<!--more-->
<p>Across these perspectives, the same fundamental hurdle persists: <strong><em>complexity at scale</em></strong>. Model behavior is rarely the result of isolated components; rather, it emerges from complex dependencies and patterns. To achieve state-of-the-art performance, models synthesize complex feature relationships, find shared patterns from diverse training examples, and process information through highly interconnected internal components.</p>
<p>Therefore, grounded or reality-checked interpretability methods must also be able to capture these <strong>influential interactions</strong>. As the number of features, training data points, and model components grow, the number of potential interactions grows exponentially, making exhaustive analysis computationally infeasible. In this blog post, we describe the fundamental ideas behind <a href="https://openreview.net/forum?id=pRlKbAwczl">SPEX</a> and <a href="https://openreview.net/forum?id=KI8qan2EA7">ProxySPEX</a>, algorithms capable of identifying these critical interactions at scale.</p>
<h3 id="attribution-through-ablation">Attribution through Ablation</h3>
<p>Central to our approach is the concept of <strong>ablation</strong>, measuring influence by observing what changes when a component is removed.</p>
<ul>
<li><strong>Feature Attribution:</strong> We mask or remove specific segments of the input prompt and measure the resulting shift in the predictions.</li>
<li><strong>Data Attribution:</strong> We train models on different subsets of the training set, assessing how the model’s output on a test point shifts in the absence of specific training data.</li>
<li><strong>Model Component Attribution (Mechanistic Interpretability):</strong> We intervene on the model’s forward pass by removing the influence of specific internal components, determining which internal structures are responsible for the model’s prediction.</li>
</ul>
<p>In each case, the goal is the same: to isolate the drivers of a decision by systematically perturbing the system, in hopes of discovering influential interactions. Since each ablation incurs a significant cost, whether through expensive inference calls or retrainings, we aim to compute attributions with the <strong><em>fewest possible ablations</em></strong>.</p>
<p style="text-align: center;">
<!-- <img src="https://bair.berkeley.edu/static/blog/spex/image1.png" alt="different_tests" width="600"><br> -->
<img alt="different_tests" src="https://bair.berkeley.edu/static/blog/spex/image1.png" width="600" /><br />
<i style="font-size: 0.9em;">
Masking different parts of the input, we measure the difference between the original and ablated outputs.
</i>
</p>
<h3 id="spex-and-proxyspex-framework">SPEX and ProxySPEX Framework</h3>
<p>To discover influential interactions with a tractable number of ablations, we have developed <a href="https://openreview.net/forum?id=pRlKbAwczl">SPEX</a> (Spectral Explainer). This framework draws on signal processing and coding theory to advance interaction discovery to scales orders of magnitude greater than prior methods. SPEX circumvents this by exploiting a key structural observation: while the number of total interactions is prohibitively large, the number of <strong><em>influential</em></strong> interactions is actually quite small.</p>
<p>We formalize this through two observations: <strong>sparsity</strong> (relatively few interactions truly drive the output) and <strong>low-degreeness</strong> (influential interactions typically involve only a small subset of features). These properties allow us to reframe the difficult search problem into a solvable <strong>sparse recovery</strong> problem. Drawing on powerful tools from signal processing and coding theory, SPEX uses strategically selected ablations to combine many candidate interactions together. Then, using efficient decoding algorithms, we disentangle these combined signals to isolate the specific interactions responsible for the model’s behavior.</p>
<p style="text-align: center;">
<!-- <img src="https://bair.berkeley.edu/static/blog/spex/image2.png" alt="image2" width="600"><br> -->
<img alt="image2" src="https://bair.berkeley.edu/static/blog/spex/image2.png" width="600" />
</p>
<p>In a subsequent algorithm, <a href="https://openreview.net/forum?id=KI8qan2EA7">ProxySPEX</a>, we identified another structural property common in complex machine learning models: <strong>hierarchy</strong>. This means that where a higher-order interaction is important, its lower-order subsets are likely to be important as well. This additional structural observation yields a dramatic improvement in computational cost: it matches the performance of SPEX with around <strong><em>10x fewer ablations</em></strong>. Collectively, these frameworks enable efficient interaction discovery, unlocking new applications in feature, data, and model component attribution.</p>
<h3 id="feature-attribution">Feature Attribution</h3>
<p>Feature attribution techniques assign importance scores to input features based on their influence on the model’s output. For example, if an LLM were used to make a medical diagnosis, this approach could identify exactly which symptoms led the model to its conclusion. While attributing importance to individual features can be valuable, the true power of sophisticated models lies in their ability to capture complex relationships between features. The figure below illustrates examples of these influential interactions: from a double negative changing sentiment (left) to the necessary synthesis of multiple documents in a RAG task (right).</p>
<p style="text-align: center;">
<!-- <img src="https://bair.berkeley.edu/static/blog/spex/image3.png" alt="image3" width="600"><br> -->
<img alt="image3" src="https://bair.berkeley.edu/static/blog/spex/image3.png" width="600" />
</p>
<p>The figure below illustrates the feature attribution performance of SPEX on a sentiment analysis task. We evaluate performance using <em>faithfulness</em>: a measure of how accurately the recovered attributions can predict the model’s output on unseen test ablations. We find that SPEX matches the high faithfulness of existing interaction techniques (Faith-Shap, Faith-Banzhaf) on short inputs, but uniquely retains this performance as the context scales to thousands of features. In contrast, while marginal approaches (LIME, Banzhaf) can also operate at this scale, they exhibit significantly lower faithfulness because they fail to capture the complex interactions driving the model’s output.</p>
<p style="text-align: center;">
<!-- <img src="https://bair.berkeley.edu/static/blog/spex/image4.png" alt="image4" width="600"><br> -->
<img alt="image4" src="https://bair.berkeley.edu/static/blog/spex/image4.png" width="600" />
</p>
<p>SPEX was also applied to a modified version of the trolley problem, where the moral ambiguity of the problem is removed, making “True” the clear correct answer. Given the modification below, GPT-4o mini answered correctly only 8% of the time. When we applied standard feature attribution (SHAP), it identified individual instances of the word <em>trolley</em> as the primary factors driving the incorrect response. However, replacing <em>trolley</em> with synonyms such as <em>tram</em> or <em>streetcar</em> had little impact on the prediction of the model. SPEX revealed a much richer story, identifying a dominant high-order synergy between the two instances of <em>trolley</em>, as well as the words <em>pulling</em> and <em>lever,</em> a finding that aligns with human intuition about the core components of the dilemma. When these four words were replaced with synonyms, the model’s failure rate dropped to near zero.</p>
<p style="text-align: center;">
<!-- <img src="https://bair.berkeley.edu/static/blog/spex/image5.png" alt="image5" width="600"><br> -->
<img alt="image5" src="https://bair.berkeley.edu/static/blog/spex/image5.png" width="600" />
</p>
<h3 id="data-attribution">Data Attribution</h3>
<p>Data attribution identifies which training data points are most responsible for a model’s prediction on a new test point. Identifying influential interactions between these data points is key to explaining unexpected model behaviors. Redundant interactions, such as semantic duplicates, often reinforce specific (and possibly incorrect) concepts, while synergistic interactions are essential for defining decision boundaries that no single sample could form alone. To demonstrate this, we applied ProxySPEX to a ResNet model trained on CIFAR-10, identifying the most significant examples of both interaction types for a variety of difficult test points, as shown in the figure below.</p>
<p style="text-align: center;">
<!-- <img src="https://bair.berkeley.edu/static/blog/spex/image6.png" alt="image6" width="600"><br> -->
<img alt="image6" src="https://bair.berkeley.edu/static/blog/spex/image6.png" width="600" />
</p>
<p>As illustrated, <strong>synergistic interactions</strong> (left) often involve semantically distinct classes working together to define a decision boundary. For example, grounding the synergy in human perception, the <em>automobile</em> (bottom left) shares visual traits with the provided training images, including the low-profile chassis of the sports car, the boxy shape of the yellow truck, and the horizontal stripe of the red delivery vehicle. On the other hand, <strong>redundant interactions</strong> (right) tend to capture visual duplicates that reinforce a specific concept. For instance, the <em>horse</em> prediction (middle right) is heavily influenced by a cluster of dog images with similar silhouettes. This fine-grained analysis allows for the development of new data selection techniques that preserve necessary synergies while safely removing redundancies.</p>
<h3 id="attention-head-attribution-mechanistic-interpretability">Attention Head Attribution (Mechanistic Interpretability)</h3>
<p>The goal of <strong>model component attribution</strong> is to identify which internal parts of the model, such as specific layers or attention heads, are most responsible for a particular behavior. Here too, ProxySPEX uncovers the responsible interactions between different parts of the architecture. Understanding these structural dependencies is vital for architectural interventions, such as task-specific attention head pruning. On an MMLU dataset (highschool‐us‐history), we demonstrate that a ProxySPEX-informed pruning strategy not only outperforms competing methods, but can actually <em>improve model performance on the target task</em>.</p>
<p style="text-align: center;">
<!-- <img src="https://bair.berkeley.edu/static/blog/spex/image7.png" alt="image7" width="600"><br> -->
<img alt="image7" src="https://bair.berkeley.edu/static/blog/spex/image7.png" width="600" />
</p>
<p>On this task, we also analyzed the interaction structure across the model’s depth. We observe that early layers function in a predominantly linear regime, where heads contribute largely independently to the target task. In later layers, the role of interactions between attention heads becomes more pronounced, with most of the contribution coming from interactions among heads in the same layer.</p>
<p style="text-align: center;">
<!-- <img src="https://bair.berkeley.edu/static/blog/spex/image8.png" alt="image8" width="600"><br> -->
<img alt="image8" src="https://bair.berkeley.edu/static/blog/spex/image8.png" width="600" />
</p>
<h3 id="whats-next">What’s Next?</h3>
<p>The SPEX framework represents a significant step forward for interpretability, extending interaction discovery from <strong><em>dozens to thousands of components</em></strong>. We have demonstrated the versatility of the framework across the entire model lifecycle: exploring feature attribution on long-context inputs, identifying synergies and redundancies among training data points, and discovering interactions between internal model components. Moving forwards, many interesting research questions remain around <em>unifying</em> these different perspectives, providing a more holistic understanding of a machine learning system. It is also of great interest to systematically evaluate interaction discovery methods against existing scientific knowledge in fields such as genomics and materials science, serving to both ground model findings and generate new, testable hypotheses.</p>
<p>We invite the research community to join us in this effort: the code for both SPEX and ProxySPEX is fully integrated and available within the popular SHAP-IQ repository.</p>
<ul>
<li><a href="https://github.com/mmschlk/shapiq">https://github.com/mmschlk/shapiq</a> (SHAP-IQ Github)</li>
<li><a href="https://openreview.net/forum?id=KI8qan2EA7">https://openreview.net/forum?id=KI8qan2EA7</a> (ProxySPEX NeurIPS 2025)</li>
<li><a href="https://openreview.net/forum?id=pRlKbAwczl">https://openreview.net/forum?id=pRlKbAwczl</a> (SPEX ICML 2025)</li>
<li><a href="https://openreview.net/forum?id=glGeXu1zG4">https://openreview.net/forum?id=glGeXu1zG4</a> (Learning to Understand NeurIPS 2024)</li>
</ul>