1:56:20 Let's build GPT: from scratch, in code, spelled out. Andrej Karpathy
6.8M views · 149.3K likes · 3 years ago
What Is Token?
Token in AI and NLP
In AI and NLP, a token is a chunk of text that a language model processes as a single unit. Tokens can be words, parts of words, or individual characters. GPT-4 uses roughly 1 token per 4 English characters. Understanding tokens is essential for managing context windows and API costs.
How Token Works
The sentence 'Hello, world!' is 4 tokens in most LLM tokenizers: 'Hello', ',', ' world', '!'. Common words are single tokens; rare words get split into subword tokens. 'Tokenization' might become 'Token' + 'ization'. This subword approach balances vocabulary size with coverage.
Key Concepts
- Tokenizer — The algorithm that splits text into tokens — each LLM has its own tokenizer (GPT uses tiktoken, Claude uses its own)
- Context Window — The maximum number of tokens a model can process in one call — ranges from 4K to 200K+
- Token Cost — API pricing is per token — both input (your prompt) and output (the model's response) tokens are billed
Learn Token — Top Videos
1:56:20 
34:05 ChatGPT Tutorial - A Crash Course on Chat GPT for Beginners Adrian Twarog
6M views · 86.2K likes · 3 years ago
3:31:24 Token Educators
Leila Gharani
@leilagharani
AI Coding
Excel. Power Query. Copilot. ChatGPT. Power BI. PowerPoint. You use them every day to automate Excel and your work - s...
3M Subs
625 Videos
117.7K Avg Views
3.06% Engagement
OpenAI
@openai
AI Coding
OpenAI’s mission is to ensure that artificial general intelligence benefits all of humanity.
1.9M Subs
456 Videos
36.2K Avg Views
2.18% Engagement
Academind
@academind
AI Coding
There's always something to learn! We create courses and tutorials on tech-related topics since 2016! We teach develop...
929K Subs
752 Videos
17K Avg Views
2.39% Engagement
Frequently Asked Questions
How do I count tokens?
Use tiktoken (Python) for OpenAI models, or the Anthropic tokenizer for Claude. As a rough estimate, 1 token ≈ 4 characters in English, or about 75% of a word.
Why do tokens matter?
Tokens determine API cost (you pay per token), context limits (how much text the model can consider), and response length. Managing token usage is essential for production AI applications.
Want a structured learning path?
Plan a Token Lesson →