By the Variask team · Updated
Definition
Tokenization is how an AI model splits text into small pieces called tokens, often parts of words, before it reads or writes. Limits, costs and context windows are all measured in tokens.
Also called Tokens
A token is roughly a short word or part of a longer word. Common words are often one token; rare words, names and some languages take several. Models read and write token by token, and providers price and limit usage by tokens.
Questions
In English, roughly 700 to 800 words, though it varies by text and model.
Related terms
How AI search works
Context window
A context window is the amount of text an AI model can take in at once, including the question, the conversation so far and any documents or search results added to it, measured in tokens.
ReadHow AI search works
Large language model (LLM)
A large language model (LLM) is an AI model trained on very large amounts of text to understand and write language. LLMs power assistants such as ChatGPT, Gemini, Claude, Grok and DeepSeek.
ReadHow AI search works
Embeddings
An embedding is a list of numbers that captures the meaning of a piece of text, so that texts with similar meaning get similar numbers. AI systems use embeddings to find, group and compare content.
Read