DevelopmentGlossary
Token
Everything an LLM sees and produces is tokens. Your prompt, its response, the context window limit, the per-request cost: all are measured in tokens. Different models use different tokenizers (BPE, SentencePiece, tiktoken variants), so "1000 tokens" on GPT is not identical to 1000 on Claude.
Rules of thumb for English prose: 1 token ≈ 0.75 words, ≈ 4 characters. Code tokenizes denser (more tokens per char) because of punctuation and identifiers. Whitespace counts. A 1000-word article ≈ 1300 tokens.
Example
# Approximating cost before you call
word_count = len(text.split())
estimated_tokens = int(word_count * 1.3) # English prose
estimated_cost_usd = estimated_tokens * (INPUT_PRICE_PER_M / 1_000_000)
# Exact count via tokenizer:
import tiktoken
enc = tiktoken.get_encoding("o200k_base") # one OpenAI tokenizer; other providers count differently
exact_tokens = len(enc.encode(text))When to use it
- Estimating cost before calling
- Deciding whether content fits the context window
- Budgeting prompts and retrievals in a RAG pipeline
When not to use it
- You don't have a tokenizer handy; a rough char-count approximation is good enough for first pass
Articles that use this
- AI Debugging Prompt: Make Your Agent Start From a Failing TestDevelopment
- Writing gh skill Definitions That Work Across Claude Code, Copilot, and CursorDevelopment
- Prompting Parallel Coding Agents: Task Decomposition That Actually ShipsDevelopment
- The System Prompt Architecture for Coding Agents: What Claude Code and Cursor Get RightDevelopment
- When to Use a Reasoning Model, and When Thinking Just Costs MoreTechniques

