2:25 What Is Token?
Token in AI and NLP
In AI and NLP, a token is a chunk of text that a language model processes as a single unit. Tokens can be words, parts of words, or individual characters. GPT-4 uses roughly 1 token per 4 English characters. Understanding tokens is essential for managing context windows and API costs.
Learn ChatGPT & LLMs from these channels
How Token Works
The sentence 'Hello, world!' is 4 tokens in most LLM tokenizers: 'Hello', ',', ' world', '!'. Common words are single tokens; rare words get split into subword tokens. 'Tokenization' might become 'Token' + 'ization'. This subword approach balances vocabulary size with coverage.
Key Concepts
- Tokenizer — The algorithm that splits text into tokens — each LLM has its own tokenizer (GPT uses tiktoken, Claude uses its own)
- Context Window — The maximum number of tokens a model can process in one call — ranges from 4K to 200K+
- Token Cost — API pricing is per token — both input (your prompt) and output (the model's response) tokens are billed
Learn Token — Top Videos
Frequently Asked Questions
How do I count tokens?
Use tiktoken (Python) for OpenAI models, or the Anthropic tokenizer for Claude. As a rough estimate, 1 token ≈ 4 characters in English, or about 75% of a word.
Why do tokens matter?
Tokens determine API cost (you pay per token), context limits (how much text the model can consider), and response length. Managing token usage is essential for production AI applications.