Behavioral Compositionality of Sparse Autoencoder Representations

If the effects of individual SAE features are known, can their joint effect be predicted by summing them?

Haotian Ma, Ziye Mei, Frederic Sala University of Wisconsin–Madison

Individual feature effects need not add up

Prompt

→

Layer 21

→

16 SAE features

→

Remove some

→

“she” or “he”

After Emily went to the store, she / he

Click a cell to remove that pair of features.

Joint effect larger than the sum Joint effect smaller than the sum Effects add up

How much of the behavior do individual feature effects miss?

Behavior

g = logit (she) - logit (he)

—

Predicted by adding single effects

—

Model output

The intervention

At layer 21 and the last token, the activation $h_{x}$ is split into the 16 most active SAE contributions $δ_{i} = z_{i} d_{i}$ and all other activation content $h_{x}^{0}$ , which includes the reconstruction residual. Keeping a subset $S$ gives

{\tilde{h}}_{x} (S) = h_{x}^{0} + \sum_{i \in S} δ_{i}

The upper layers run unchanged and give $g (S)$ . For a pair of features the sum of single effects predicts $g (N) - [g (N) - g (N ∖ i)] - [g (N) - g (N ∖ j)]$ . The colour of a cell is the model output minus this prediction.

The selected pair is SAE features 21097 and 29794 of the Llama Scope SAE. For this prompt = 0.1677 and = 0.771.

Paper Section 3.1, Figure 9 and Table 22, prompt gendered_pronoun/00310.

~55 s

Talk track

A prompt goes into Llama-3.1 8B. At layer 21 we take the 16 most active SAE features. We remove some of them while the rest of the activation stays fixed, and read whether the model prefers she or he as the next word.
This is a real prompt from the paper. Each cell of the grid is one pair of the 16 features, and clicking it removes that pair. Red means removing both changes the behavior more than the two single effects added together. Teal means less. Most cells are pale, so most pairs do add up.
This number is the prediction from single effects. We start from the full preference and subtract what each feature costs on its own.
This is what the model does. For the selected pair the sum predicts 1.63 and the model gives 0.56. The bars show each step.
So we need a measure of how much individual feature effects miss.

SAR measures what individual feature effects leave unexplained

{SAR}_{r} = \frac{Var (g - Π_{r} g)}{Var (g)}

?

Explained by first order Added by pairs Unexplained

SAE Additivity Residuals

Subsets are drawn by keeping each feature independently with probability 1/2. Let $Π_{r} g$ be the best approximation of $g$ with terms of at most $r$ features and $V$ the variance of $g$ . Then

{SAR}_{r} = \frac{{‖ g - Π_{r} g ‖}_{μ}^{2}}{V}, Δ_{2} = {SAR}_{1} - {SAR}_{2}, ρ_{2} = \frac{Δ_{2}}{{SAR}_{1}}

Under this sampling ${SAR}_{r}$ is exactly the normalized energy of interactions above order $r$ , and it bounds the coalition ordering error of the best order $r$ approximation.

With 16 real features there are 65,536 subsets. The paper samples 384, fits on 256 and reports ${\hat{SAR}}_{r} = {\hat{L}}_{r} / (\hat{V} + ϵ)$ on the other 128.

The toy uses all 16 subsets of four features, so its values are exact. Its behavior is $g = A + 0.5 B + C + 0.4 D - A C + A B D$ .

Paper Sections 3.2 to 3.4.

~70 s

Talk track

We use a toy with four features, A to D, and all sixteen subsets. A dot means the feature is kept, and the number is the behavior with only those features kept.
One pair of features does not add up. Click two column headers to guess it, then press Reveal.
After the reveal two columns appear. The first is the best prediction with one coefficient per feature. The second is its residual. The large residuals are in rows where A and C are both kept, because A and C carry the same information.
Now switch to pairwise. The model gets one more coefficient for each pair of features, and the residuals almost vanish.
SAR one is the share of the behavior's variance that the first order model leaves unexplained, 0.22 in this toy. SAR two is the same share for the pairwise model. Rho two is the part of SAR one that pairs recover, 88 percent here. With 16 real features we sample 384 subsets, fit on 256, and measure the residual on the other 128.

SAE coordinates have lower first order residuals than random rotations

A rotation mixes the 16 selected contributions into 16 new directions. The subspace, the total contribution and the upper layers stay the same.

Each dot is one prompt. Each diamond is the median of one model and behavior.

18 / 18

model and behavior settings where the SAE median is lower

57 / 57

settings where is above zero. A linear readout gives 1.55 × 10⁻¹³.

Highlight a model

Rotated coordinates

With the contributions as the columns of $D_{x}$ , a rotation uses $D_{x} Q$ where $Q$ is orthogonal and $Q 𝟏 = 𝟏$ . Each prompt gets eight random rotations and keeps its subsets and split.

From early to late layers falls in eight of nine comparisons and rises in all nine. Rotations show the same decrease, so part of this trend comes from the shorter downstream computation. SAE coordinates stay lower in all 33 model, behavior and layer comparisons.

Median, SAE/rotation	Gemma	Mistral	Llama

Paper Sections 4.2 and 4.5, Figure 3 and Table 10.

~55 s

Talk track

Now real data. To see whether this depends on the SAE basis, we rotate the 16 contributions into random new directions. The subspace, the total contribution and the upper layers stay the same.
Each dot is one of 1,800 prompts from three models and six behaviors. Up is the first order residual in SAE coordinates and right is the residual after rotation. Dots below the dashed line favor the SAE coordinates. Diamonds are medians.
Pick a model to highlight it. Each one stays below the line.
The SAE median is lower in all 18 settings. For Mistral subject verb agreement it is 0.022 against 0.120.
The residual is never zero. It is positive in all 57 settings we measured, while a linear readout gives numerical zero. SAE coordinates are more compositional than rotations, but not fully additive.

Pairwise terms recover most of the missing structure

{\hat{a}}_{1} (S) = θ_{0} + \sum_{i \in S} θ_{i}

{\hat{a}}_{2} (S) = {\hat{a}}_{1} (S) + \sum_{i < j \in S} θ_{i j}

Both are fit on 256 random subsets and evaluated on 128 held out subsets.

Rank fidelity

F_{rank}^{(r)} = {Spearman}_{S \in 𝒰} ({\hat{a}}_{r} (S), g (S))

1 means every held out subset is ordered correctly.

0.9782 → 0.9959

Average rank fidelity over 18 model and behavior settings. Pairwise terms improve it in every setting and beat the same number of random third order terms.

The same prompt, 128 held out subsets

Pairs and a matched control

The pairwise model has 137 coefficients. The control keeps the 17 first order terms and replaces the 120 pair terms with 120 terms of three features drawn at random from 560. Each prompt gets 50 such controls, and this slide shows the one with the median rank fidelity. Over the 50 controls rank fidelity ranges from .

Over the 18 settings pairwise terms also lower sign error in every setting and cut selection regret from 0.0272 to 0.0051. Both gains hold in all 20 repeated splits.

From the lowest to the highest decile, first order rank fidelity falls from 0.9857 to 0.9539 in Gemma, from 0.9939 to 0.9413 in Mistral and from 0.9790 to 0.9289 in Llama.

Paper Sections 4.3 and 4.4, Figures 4 and 5, Tables 15 to 19 and 22.

~60 s

Talk track

We compare two models of a subset's behavior. The first order model has one coefficient per feature. The pairwise model adds one coefficient for each pair. Both are fit on 256 random subsets and evaluated on 128 subsets they did not see.
We judge them by rank fidelity, the Spearman correlation between predicted and actual behavior over the held out subsets.
Back to the Emily prompt. Each dot is one held out subset. Right is the prediction and up is the model output. The red lines show the residuals.
Click through the three models. With pairs the residuals shrink, and pairs recover 77 percent of what single features miss. The control adds 120 random third order terms instead, the same number of coefficients, and does worse.
Over 18 settings, rank fidelity rises from 0.9782 to 0.9959 with pairs, and the gain appears in every setting.

Pairwise modeling better predicts combined steering

We add 16 published SAE steering directions to Llama-3.1 8B in 1,024 combinations and measure accuracy on 1,400 MMLU-Pro questions.

Each dot is one of 512 held out combinations.

Predict each combination with

Rank fidelity, higher is better

First order

Pairwise

Ordering error, lower is better

ℰ = \frac{𝔼 [(Δ g)^{2} 𝟏 {Δ g Δ a \leq 0}]}{2 V}

First order

Pairwise

Residual estimate , lower is better

First order

Pairwise

The steering experiment

The 16 MMLU-Pro features come from CorrSteer for the Llama Scope SAEs at layers 12 to 30, with their published coefficients. A combination adds each chosen direction at its layer. We sample 1,024 of the 65,536 combinations and split them 512 and 512. Each of the 14 MMLU-Pro domains contributes 100 test questions.

In ordering error, $Δ g$ and $Δ a$ are the differences in actual and predicted accuracy between two combinations. Domain results are mixed. Pairs raise rank fidelity in 10 of the 14 domains.

Domain	Residual, first order	Rank, first order	Rank, pairwise

Paper Section 4.6 and Tables 1, 7, 20 and 21.

Takeaways

SAE coordinates are comparatively compositional, but faithful joint explanations still need interactions

Individual effects leave a residual

is above zero in all 57 settings.

SAE coordinates add up better

They beat random rotations of the same subspace in all 18 settings.

Pairs recover most of the rest

Rank fidelity rises from 0.9782 to 0.9959, and from 0.5874 to 0.7790 for steering.

What these results do not show

A property of a whole SAE dictionary. The results depend on the selected features, the behavior and the intervention.
That pairs are always enough. Some higher order structure remains, and pairs did not help in every steering domain.
What the interacting features mean.