Outliers by Design: Activation Structure and Ablation Sensitivity in Vision Transformers

• Daniel Mihovch, Computer Science, University of Delaware

• Nazim Karaca, Computer and Information Sciences, University of Delaware

Massive activation outliers in Vision Transformers are a deliberate architectural feature, not anomalous noise, and they are vital for accuracy. We profile a pretrained ViT-B/16 across six measurement sites: residual stream, post-LayerNorm, pre-softmax attention logits, post-softmax attention weights and pre-GELU activations, in all twelve encoder blocks. Pre-GELU activations display heavy-tailed per-channel distributions with a 12x spread in standard deviation from the lowest standard deviation of ~2 and the highest of ~25 and per-channel mean shifts spanning 97 units within a single block. Zeroing pre-GELU elements exceeding k·σ from the per-channel mean drops top-1 accuracy from 84.8% to 42.4% at k=3, while random zeroing of matched sparsity levels leaves accuracy relatively unchanged. Pre-softmax attention logits are substantially less sensitive (74.5% at k=3), while residual stream ablation is catastrophic (5.5% at k=3). Per-channel thresholding of pre-GELU activations preserves 46.7% at k=3, and decomposing this benefit reveals that it derives entirely from correcting for per-channel mean shifts: using per-channel means with a global standard deviation recovers 63.5% accuracy, while using per-channel standard deviations with a global mean collapses the model to just 6.6% accuracy. The effect is concentrated in later blocks, with blocks 8-11 accounting for 38% of the total accuracy drop. The per-channel variance pattern correlates strongly with the effective gain of the fully-connected layer one weights scaled by the LayerNorm gain vector (Pearson r = 0.65 to 0.77 in late blocks), confirming that these outliers are encoded in trained weights.