02 · How LLM Tokens Actually Work
The Token Illusion
You probably think of an LLM as a machine that reads text. It doesn't.
When you type "hello world", the LLM sees something like [31373, 995]. Two tidy concepts.
But type " hello" (with a leading space) and suddenly it might see [220, 23748].
Wait, why did adding a single space completely change the underlying numbers? Why does the word "hello" translate to entirely different IDs based on its position, punctuation, or capitalization?
Welcome to the strange, hidden world of tokens.
Tokens are the true atomic unit of every LLM. They dictate everything from the cost of your API calls to the limits of your context window. Understanding how tokenization works isn't just academic—it's the mental model every developer needs to optimize prompts, debug bizarre LLM behaviors, and build robust AI applications.
Let's dive into how text is actually sliced, diced, and digested by models like GPT-4 and Claude.
Solution
Tokens are the atomic unit of every LLM API call. Learn how they drive cost, shape context limits, and parse text.
selected — 0 / 8
sequence — none
status — idle
- Click squares in any order (center is inert).
- Clicking again deselects without touching the sequence.
- Fill all 8 → auto-deselect in reverse, 200ms apart.
Code Walkthrough
Six steps from empty component to interruptible reverse animation. Expand a step to read the reasoning and copy the snippet.
A Set gives O(1) membership checks for the coloured squares; an array preserves the click order.
selected tracks active squares, sequence tracks the order of selections only, isReverting blocks input while the animation runs.
React1const [selected, setSelected] = useState(new Set());2const [sequence, setSequence] = useState([]);3const [isReverting, setIsReverting] = useState(false);
Everything together
The finished component, same code as the Solution tab.
selected — 0 / 8
sequence — none
status — idle
- Click squares in any order (center is inert).
- Clicking again deselects without touching the sequence.
- Fill all 8 → auto-deselect in reverse, 200ms apart.