The video explains an MIT result on why making language models wider eventually stops helping. Because models pack far more concepts (~50,000+) than they have internal dimensions (~12,000 for GPT-3), concepts overlap in a phenomenon called superposition, and that overlap causes "interference" errors. Doubling a model's width halves that interference, but each doubling costs roughly six times more, the improvement rate is pinned in place by the statistics of human language (Zipf's law), and once width matches the number of genuinely independent concepts there is no interference left to remove — so extra width buys nothing.
- LLMs learn via self-supervised training on huge text corpora and build a geometric map of how words relate, without being taught grammar rules explicitly.
- A model must represent well over 50,000 concepts inside an internal space only a few thousand dimensions wide (a bit over 12,000 for GPT-3), so concepts get squeezed into overlapping directions — superposition, a concept Anthropic described in a 2022 paper.
- When overlapping concepts get partially activated together, the resulting 'interference' is one reason models output words that are slightly off or completely wrong.
- The MIT paper's core formula: doubling model width halves interference and therefore halves the errors — this is a big part of why bigger models have been better.
- Cost kills the strategy first: each doubling of width costs about six times more, so a $300 million frontier model becomes $1.8B, $11B, $65B, and then $390B after four doublings.
- Even with unlimited money, the improvement exponent is locked by Zipf's law — the fact that a few words appear constantly while most appear very rarely — so errors can't be made to fall faster.
- Error never extrapolates to zero because language is genuinely ambiguous: after 'I really like,' cats, dogs, or pizza are all valid, so there's no single correct answer to learn.
- The hard ceiling arrives when width finally matches the number of independent concepts in language: with no superposition left, there is no interference to reduce, and extra width gains nothing.
- The video notes some researchers have already pivoted toward AI that learns from video and sensory input about the physical world rather than human text.
This is the clearest plain-language explanation of the mechanism the whole argument rests on.
When space runs out, some of the things it learns, like banana, elephant, balloon, and the semicolon, end up pointing in almost the same direction... That bleed is called interference.
The concrete cost arithmetic shows how fast brute-force scaling becomes economically absurd.
A frontier model today costs somewhere around $300 million to train. So doubling from here gets you to around 1.8 billion, doubling again puts you at 11 billion, once more and you are looking at 65 billion. One more time after that and we are looking at 390 billion dollars for a single model.
The lesson everyone took from this was bigger equals better, and much bigger equals much better.
Once the model's width catches up with the actual number of genuinely independent things in language, there's no superposition left anywhere in it.
Language is not a math equation where 2 + 2 always equals 4.