Article
Understanding Attention Masks in Transformers
Why masks use very negative values before softmax, and how padding and causal masks solve different problems.
The core idea
Attention is a weighted average over tokens. A mask changes which tokens are allowed to receive weight.
Before softmax, masked positions are set to a very negative value. After softmax, those positions become effectively zero.
Padding masks
Padding masks prevent the model from attending to fake tokens that only exist to make sequences the same length in a batch.
Causal masks
Causal masks prevent a decoder from looking at future tokens. This keeps generation
autoregressive: token n can only depend on tokens before it.
Takeaway
Masks do not remove tokens from the tensor. They change the probability distribution so unwanted positions stop contributing to the contextual representation.