Abstract
<title>Abstract</title> <p>Exclusive Self-Attention (XSA) modifies standard self-attention by removing the component of the attention output aligned with a token's own value representation, with the goal of reducing self-referential behavior and encouraging stronger contextual interactions. While prior work has demonstrated XSA under conventional AdamW-based training, its behavior under alternative optimization regimes remains less explored. In this work, we study the stability and generalization behavior of XSA under Muon-based optimization within the nanochat language-model training framework. We implement a parameterized XSA mechanism supporting configurable projection strength and layer placement, and validate its correctness through numerical equivalence tests and controlled training runs. Experiments at 12-layer and 24-layer model depths show that XSA preserves optimization stability and closely tracks baseline training-loss trajectories. At d12, validation bits-per-byte improves from 0.8484 to 0.8457, although the corresponding Base CORE metric decreases, indicating a mixed downstream effect. At d24, XSA improves validation bits-per-byte from 0.7193 to 0.7171, Base CORE from 0.2593 to 0.2606, and ChatCORE from 0.3638 to 0.3682. These results suggest that XSA remains compatible with Muon optimization and can provide modest generalization improvements without destabilizing training. More broadly, the findings indicate that reducing self-alignment in attention can act as a representation-level inductive bias whose effectiveness depends on model scale and evaluation setting.</p>