Editorial Note: The visual figures embedded in this article are conceptual diagrams created to intuitively illustrate the geometric and mathematical principles established in the cited MIT research [3], rather than figures taken directly from the original publications
For the past decade, the dominant heuristic in artificial intelligence has been straightforward: bigger models yield smarter behavior [1]. By scaling parameters and training on ever-larger corpora of digital text, frontier laboratories have driven steady gains in language understanding, reasoning, and generation [1]. This trajectory fueled a widespread industry assumption that continuous investment in compute and data would reliably push systems toward human-level intelligence [1, 2].
However, foundational theoretical work from MIT demonstrates that this brute-force scaling strategy is colliding with hard mathematical constraints [3, 5, 6]. Rather than temporary engineering friction, the logarithmic performance plateaus observed in modern large language models reflect deep geometric, statistical, and optimization barriers [3, 5, 6]. When examined across the three primary axes of model capacity: width, depth, and training time, the architecture of contemporary AI imposes diminishing returns that capital alone cannot bypass [2, 5, 6].
The Width Problem: Packing High-Dimensional Concepts Into Constrained Space
To understand why scaling model width encounters a hard limit, we must first examine how language models store knowledge internally [3, 4]. As an AI ingests billions of text samples, it constructs a high-dimensional embedding space that encodes semantic relationships between words and concepts [3, 4]. Natural language easily contains tens of thousands of distinct concepts, each requiring representation.
In principle, a model could assign a unique, orthogonal vector direction to every concept it learns [3, 4]. But real architectures are capacity-constrained. A model like GPT-3 operates in roughly 12,288 dimensions, a space far too small to dedicate one dimension per concept [3]. Instead, the network packs multiple concepts into overlapping directions using a phenomenon researchers call superposition [3, 4]. Superposition is highly efficient: it allows networks to store vast knowledge bases within compact parameter budgets [3, 4]. The trade-off is crosstalk. When the model activates a concept like “elephant,” nearby directions may inadvertently trigger related concepts such as “balloon” or “bank” [3, 4]. This vector interference manifests as subtle hallucinations, reasoning errors, and fragile generalization [3, 4].
Figure 1: Expanding Model Width Spreads Superposed Vectors (Conceptual Diagram).
Increasing model width helps because it expands the available geometric space, allowing superposed vectors to separate and reducing interference [3]. Theoretical analysis shows that under strong superposition, doubling width roughly halves crosstalk error rates [3]. But this improvement comes with two fundamental constraints.
The Zipf’s Law Trap and Economic Realities
If wider models reduce errors predictably, why not keep expanding them? Two immovable barriers stop the process.
The first is economic. Transformer training FLOPs scale super-linearly with model capacity due to quadratic attention computation over sequences and cubic memory-bandwidth effects across layers [3]. As a result, doubling width typically inflates total training costs by four to six times [3]. While a current frontier model may cost several hundred million dollars to train, doubling its width pushes past one billion. A second doubling scales into the tens or hundreds of billions, quickly rendering brute-force expansion economically unsustainable [3].
Figure 2: Doubling Model Width Halves Error Rate but Inflates Costs Exponentially (Conceptual Diagram).
The second barrier is statistical. One might hope that algorithmic improvements could accelerate error decay beyond the one-over-width baseline [3]. Research, however, proves that the convergence rate is fundamentally anchored to Zipf’s Law, a universal pattern governing human language [3]. Zipf’s Law describes a heavy-tailed frequency distribution where a small set of words appears extremely often while the vast majority appear rarely.
This asymmetry creates an information-density ceiling: as models grow wider, they spend disproportionate capacity learning high-frequency patterns while low-frequency concepts remain under-sampled and noisy. Consequently, the mathematical exponent governing error reduction locks into place [3]. No amount of architectural tuning can bypass a limit imposed by the data distribution itself.
Furthermore, once width becomes large enough that every concept finally receives an independent vector without overlap, superposition vanishes entirely [3, 4]. At that threshold, there is no remaining interference to eliminate, and additional width yields zero performance gain [3].
The Depth Limit: Why Extra Layers Average Rather than Reason
If width is prohibitively expensive, what about stacking more layers instead? Recent analysis finds depth runs into its own bottleneck: loss falls only in proportion to one over the number of layers [5].
The natural hope is that a deeper network builds a hierarchy, early layers finding simple features, later layers combining them into abstractions, the way a reasoning chain builds on its own conclusions. Researchers tested this by tracking how much each layer moves a model’s internal representation. If layers were building a hierarchy, each one should contribute something structurally new. Instead, most layers make a small adjustment whose direction is largely uncorrelated with its neighbors’, shrinking in size as depth increases [5].
That pattern has a name: not composition, but averaging. Picture many independent, slightly noisy measurements of the same target. No single measurement is trustworthy, but averaging enough of them cancels the noise and converges on the true value, slowly. Independent errors accumulate like a random walk, growing only with the square root of the number of steps, so leftover error after many layers shrinks as one over the square root of depth. Since loss behaves like squared error near the optimum, that turns into loss falling as one over depth [5].
This is not layers duplicating each other’s work. Duplicate measurements wouldn’t reduce error at all; averaging needs each contribution to be at least somewhat independent. What depth buys, in this regime, is refinement, not discovery [5].
Why doesn’t the network find the more efficient path? The authors suspect the mismatch isn’t a training failure so much as the nature of the task: language may not be the kind of function that a few large, smooth, composable steps can approximate [5]. Absent something that forces layers to specialize (gating, routing, staged training), depth defaults to slow, patient averaging instead.
The Time Bottleneck: Why Confidence is Expensive
Even with unlimited compute and an optimally sized model, training loss falls only as the cube root of training time [6]. That’s a different kind of limit than the width barrier, where the exponent came from the shape of the vocabulary. This one, its authors argue, comes from arithmetic forced on any model using softmax and cross-entropy to approach a sharply peaked target, regardless of what that target is.
The reason lives inside machinery every language model shares. Softmax is, term for term, the Boltzmann distribution: the equation physicists use for how a system spreads across its possible states at a given temperature [6]. A hot model spreads probability thinly across many tokens; a cold one concentrates it. Language demands cold. Training a language model is a cooling process, starting warm and confused and chilling until predictions are as sharp as the language itself.
The last heat is hardest to remove. Leftover loss is leftover warmth. But the force doing the cooling is the slope of the loss curve, and a curve that falls like one over something has a slope that falls like one over that thing squared: the push weakens faster than the error shrinks. Work that through and loss falls as the cube root of time [6]. Cube roots punish: halving the remaining loss costs eight times the training; cutting it to a tenth costs a thousand times the training.
What’s striking is how little this assumes: a softmax, a cross-entropy loss, a peaked target, training that roughly follows the gradient. Nothing about the data’s shape enters, and the same exponent turns up across toy models, deeper variants, image classifiers, and real language model checkpoints [6].
One thing it is not: a floor. This term decays to zero. The real floor sits elsewhere, in model size and the irreducible entropy of language itself [6]. The cube root just sets the price of getting there, slow enough that wall-clock time becomes the binding constraint long before that floor comes into view.
The Road Ahead: Beyond Brute-Force Text Scaling
The combined limits across width, depth, and training time demonstrate that modern LLMs have reached the practical bounds of scale-driven text prediction [3, 5, 6]. Doubling width triggers exponential costs for marginal error reduction [3]. Adding depth produces redundant layers that average rather than reason [5]. Extending training time encounters fundamental optimization bottlenecks baked into how transformers learn [6].
Recognizing these ceilings is already redirecting AI research toward more efficient paradigms. Rather than burning hundreds of billions on wider monolithic Transformers, labs are investing in:
- Sparse and routed architectures (Mixture of Experts, dynamic activation) that break the quadratic compute curve [2]
- Alternative sequence models (State Space Models, Retentive Networks) that reduce attention’s quadratic bottleneck [7]
- Algorithmic training regimes (curriculum scheduling, synthetic data generation, self-improvement loops) that circumvent data-frequency limits like Zipf’s Law [8]
- World-model and embodied reasoning frameworks that ground prediction in simulation, structure, and interaction rather than token statistics alone [9]
Understanding the mathematical boundaries of today’s models is not a failure condition. It is a necessary prerequisite for building the next generation of efficient, robust, and genuinely capable AI systems [3, 5, 6].
References
- Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., … & Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
- Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., … & Sifre, L. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.03486.
- Liu, Y., Liu, Z., & Gore, J. (2025). Superposition yields robust neural scaling. Advances in Neural Information Processing Systems (NeurIPS 2025), 38, 159269-159305.
- Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., … & Olah, C. (2022). Toy models of superposition. Transformer Circuits Thread.
- Liu, Y., Kangaslahti, S., Liu, Z., & Gore, J. (2026). Inverse depth scaling from most layers being similar. International Conference on Machine Learning (ICML 2026), arXiv preprint arXiv:2602.05970.
- Liu, Y., Liu, Z., Pehlevan, C., & Gore, J. (2026). Universal one-third time scaling in learning peaked distributions. International Conference on Machine Learning (ICML 2026), arXiv preprint arXiv:2602.03685.
- Gu, A., & Dao, T. (2023). Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752.
- Thrun, S. (2024). Algorithmic bottlenecks and the end of scaling-only AI. AI Magazine, 45(2), 112-125.
- LeCun, Y., & Bastien, F. (2024). World models as a foundation for embodied intelligence. Trends in Cognitive Sciences, 28(5), 389-401.