Token
Sub-word atomic chunk created by tokenizers (e.g. BPE) mapping text strings to numerical vocabulary IDs.
Mechanism & Definition
The fundamental unit of text processed by Large Language Models. Sub-word tokenizers like Byte-Pair Encoding (BPE) or WordPiece slice raw text into sub-word pieces or characters, which map to unique integer IDs within a fixed vocabulary index.