Self-Attention Mechanism
Mathematical operation computing context weights by comparing Query, Key, and Value vector dot-products.
Mechanism & Definition
The core mathematical module in Transformer models. For input tokens transformed into Query ($Q$), Key ($K$), and Value ($V$) matrices, attention weights are computed via $\text{Softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$, allowing tokens to dynamically focus on distant contextually relevant tokens regardless of position.