Subspace dictionaries across depth: comparing SASA and sparse autoencoders in Gemma 3 1B
- 26
- layers swept, 3 seeds each
- 0.51 → 0.83
- dictionary utilization, layer 0 → 24
- 3.4×
- R² over random controls at layer 17
- 1.00 → 0.45
- cause AUROC once dataset is held fixed
A language model can tell you how uncertain it is. Semantic entropy, linear probes, and a handful of causal neurons all compress that signal into one number. This paper asks whether the model’s internal state also encodes why it is uncertain, and whether that structure is a subspace rather than a direction.
Setup
Two dictionary-learning methods are compared on residual-stream activations of Gemma 3 1B. Vanilla sparse autoencoders (Gemma Scope 2) assign one direction per feature. SASA, a group-sparse method, assigns each feature a rank-6 subspace and gates ten groups per token, matching the SAE’s sparsity of 60. SASA dictionaries were trained at all 26 layers with three seeds each, on 50M tokens of FineWeb-Edu per layer, and set against a random-subspace null of matched dimensionality.
Utilization rises with depth
How much of its rank-6 capacity a SASA block actually uses climbs from 0.507 at layer 0 to 0.826 at layer 24. Two dips, at layers 8 and 11, reproduce across all three seeds and do not align with the model’s global-attention layers. The trend tracks the residual stream’s own intrinsic dimensionality (r = 0.82), so the reading is that SASA follows the stream’s structure rather than that deep layers are intrinsically richer.
Blocks persist across layers
Because consecutive layers share one activation space, a block’s decoder subspace at layer L can be compared directly with every block at L+1 by principal angles. At all 25 transitions the typical block’s best partner scores far above the random baseline (mean 0.14–0.37 against a random 95th percentile of 0.095). Persistence is broad in early layers, churns mid-stack, and in late layers is carried by a minority of strongly aligned blocks. A vanilla SAE can only ask this question as a cosine between two vectors.
The structure carries uncertainty signal
Projected onto six selected blocks, activations at layer 17 predict within-dataset-centred semantic entropy with R² of 0.119 (SASA) and 0.107 (Gemma Scope 2), against 0.035 and 0.033 for matched random controls. The two trained dictionaries track each other within 0.02 across the whole sweep. A caveat the paper states plainly: the signal comes mostly from how strongly a block fires, not from the direction within it.
The negative result
Does the structure separate four hypothesized causes of uncertainty (ambiguity, knowledge gap, missing information, computational difficulty)? On a pool where each cause comes from a different dataset, surface features alone reach a worst-pair AUROC of 1.000. Under a passage-matched control that holds dataset identity fixed, it falls to 0.453. Cause and dataset cannot be separated on that pool. The paper reports this as a confound, and constructs the passage-matched pool that cause-resolved work will require.
Reported as a limitation, not a positive finding. The Institute’s research standards require negative results to be written up alongside positive ones.