Token

A token is a unit of text processed by a language model. It may be a whole word, part of a word, punctuation, or whitespace, depending on the model’s tokenizer.

Tiny example

“Vector databases” is two words, but a tokenizer may split it into more than two tokens. Token counts therefore cannot be derived reliably from word counts alone.

Why it matters

Input and output limits, model pricing, and much of request latency are expressed in tokens.

FDE note

Count with the tokenizer for the exact model when hard limits matter. Reserve output space before filling the context window with history or retrieved passages.