Executive Overview
Back in the sun-drenched summer of 2017, a team of Google AI researchers published a watershed academic paper titled "Attention Is All You Need." In it, they introduced a revolutionary neural network architecture known as the transformer. Designed to process long sequences of data with unprecedented parallelization, the transformer proved exceptionally gifted at digesting, interpreting, and generating human language.
Nearly a decade later, transformers form the beating heart of every major large language model (LLM) on the market. They are the invisible machinery powering commercial titans like OpenAI, Anthropic, Google, and Meta. As Justin Dangel, co-founder and CEO of the AI startup Subquadratic, notes: "The entire AI industry is built on transformers. They are one of the most important innovations in the history of computer science, and they’ve changed the world."
Yet, beneath the glossy veneer of today’s generative AI boom, structural fractures are beginning to show. Transformers are aging. Many of the most celebrated recent breakthroughs in LLMs—such as advanced reasoning models that mimic human deliberation and massive context windows capable of swallowing entire digital libraries—are not organic extensions of the original transformer architecture. Instead, they are elaborate workarounds, computational band-aids patching over the fundamental physical and mathematical limitations of the core technology.
Consequently, a burgeoning vanguard of scientists, engineers, and audacious startups are asking a foundational question: What comes next? While mainstream LLMs are not going away anytime soon, the underlying rulebook for how they are built is entirely up for grabs. Dubbed "LLMs+" by industry analysts, this forthcoming generation of models represents a departure from brute-force scaling toward architectural elegance, radical efficiency, and novel forms of machine cognition. While some of these startups will inevitably crash and burn, they have everything to play for—and significantly less to lose than the incumbent giants currently leading the pack.
Detailed Chronology: The Evolution and Friction of Sequence Modeling
To understand why the artificial intelligence landscape is bracing for a tectonic shift, one must trace the historical arc of sequence modeling and the computational bottlenecks that have accumulated along the way.
- June 2017: Google researchers publish "Attention Is All You Need," detailing the transformer architecture. By replacing recurrent neural networks (RNNs) with a mechanism called dense attention, the model allows for parallel processing of text tokens, dramatically accelerating training times and capturing long-range dependencies in language.
- 2018–2022: The era of scaling begins. OpenAI releases GPT-1, GPT-2, and eventually GPT-3, demonstrating that simply throwing more compute, data, and parameters at transformer-based models yields emergent capabilities in translation, coding, and creative writing.
- 2023–2024: The generative AI gold rush reaches a fever pitch with the deployment of GPT-4 and open-source challengers like Meta’s Llama and Alibaba’s Qwen. However, industry engineers begin hitting the "attention wall." As context windows expand from thousands of words to millions, memory requirements explode quadratically, sparking a global scramble for specialized silicon (GPUs) and electrical grid capacity.
- 2025–2026: The limits of brute-force scaling drive a pivot toward architectural diversification. Reasoning models utilizing "chains of thought" emerge, but they dramatically inflate token processing overhead. Concurrently, a new crop of venture-backed startups begins commercializing alternatives to traditional dense attention—including sparse attention, power retention, diffusion text generation, and state-space models—heralding the dawn of the post-transformer era.
Supporting Context & Metrics: The Cost of Brilliance
The fundamental vulnerability of the transformer lies in its defining feature: dense attention. To encode the meaning of a text block, the transformer mechanism compares every single word (or token) against every other word via a matrix of continuous multiplications.
While this dense cross-referencing maps semantic meaning with astonishing fidelity, it carries a crippling mathematical penalty. As input lengths grow linearly, computational complexity scales quadratically ($O(N^2)$). A modest 10,000-word document can force a traditional transformer to execute roughly 50 million individual multiplication operations just to establish context. This insatiable appetite for compute is the primary driver behind the astronomical energy and financial footprint of modern AI.
The downstream consequences are staggering:
- Financial Capital: OpenAI projected its computing power expenditure to touch a staggering $50 billion, according to company statements highlighted by co-founder Greg Brockman.
- Energy Consumption: The International Energy Agency (IEA) has projected that global electricity consumption by data centers will double by 2030, driven largely by the relentless inference and training demands of transformer-based LLMs.
- The Context Trap: Because transformers ingest data sequentially token-by-token, they struggle to retain granular focus across truly massive repositories of information. If AI agents are expected to independently analyze entire codebases, multi-hour video feeds, or histories of multi-week interactions, the traditional transformer context window buckles under its own weight.
Furthermore, modern reasoning models operate by maintaining an internal "scratchpad"—a chain of thought where the model writes out intermediate reasoning steps before delivering an answer. Because the model must continually read and re-read its own internal monologue, the data volume explodes, turning the transformer’s core strength into an intractable bottleneck.
Official Statements: Four Radical Paradigms to Overcome the Transformer Bottleneck
In response to these mounting pressures, innovative startups are advancing four distinct technical pathways to reinvent large language models from the silicon up.
01. Rethinking Attention: Sparse Attention and Power Retention
The most direct line of attack involves modifying or entirely replacing the attention mechanism.
At Miami-based startup Subquadratic, engineers have developed sparse attention, calculating relationships across only a statistically relevant subset of word pairings rather than every single permutation. Subquadratic claims its flagship model, SubQ, dynamically identifies which tokens matter on the fly. While industry veterans remain skeptical of claims that sparse attention can match dense attention’s accuracy, Subquadratic reports immense early demand, with thousands queued on its waitlist for search and coding applications.
Meanwhile, San Francisco-based Manifest AI is bypassing attention mechanics entirely, opting instead for a concept known as power retention.
"There are many useful applications—from analyzing videos that are hours long to building agents that can stay on task for weeks at a time," stated Carles Gelada, co-founder and CTO of Manifest AI, in a video unveiling the technology.
Unlike sparse attention—which discards individual words while retaining a broad view of the text—power retention maintains a dynamic, rolling summary of the context window. As fresh information enters, extraneous data is gracefully shed. Manifest AI has demonstrated this capability by converting an open-source coding LLM (StarCoder) into PowerCoder with minimal retraining, alongside releasing Brumby, a model designed to rival Alibaba’s Qwen series.
02. Making Models Smaller and More Flexible: Liquid Neural Networks
Rather than banishing transformers entirely, MIT spinout Liquid AI is wedding them to an older, highly adaptive architecture: liquid neural networks. Inspired by the nervous systems of biological organisms like C. elegans worms, these networks can dynamically adjust their internal behavior post-training.
Operating out of Cambridge, Massachusetts, Liquid AI builds "liquid foundation models" (LFMs).
"Your brain is an AGI system, you know, and it operates with 20 watts of power. How is it possible? We can get a lot more innovative," says Ramin Hasani, co-founder and CEO of Liquid AI.
Utilizing an internal "designer AI" that automatically stitches together neural network topologies, Liquid AI’s latest hybrid models consist of 20% transformers and 80% liquid neural networks. These models are so compact they can run locally on an inexpensive $50 Raspberry Pi computer or inside the edge-computing chips of automotive partners like Mercedes-Benz—all while matching the performance of rival models four times their size.
03. Generating Text All at Once: Diffusion Language Models
Traditional LLMs speak strictly in a serialized, single-word-at-a-time cadence. While intuitive for human conversation, it is computationally inefficient for silicon processors.
Inception, a Palo Alto-based startup, is rewriting this paradigm by deploying diffusion—the foundational mathematical technique behind state-of-the-art image and video generators like Stable Diffusion. Instead of transitioning pixels from television static into high-definition art, Inception’s models take a random string of text tokens and iteratively refine them into coherent prose across the entire sequence simultaneously.
Co-founder and CEO Stefano Ermon, also a researcher at Stanford University, solved the formidable mathematical hurdle of applying diffusion to discrete linguistic tokens (where intermediate states between words like "cat" and "dog" do not naturally exist). The startup’s latest model, Mercury 2, reportedly matches the performance of OpenAI’s GPT-4 while operating at ten times the speed.
"Ultimately, the currency is going to be intelligence per dollar," Ermon asserts, remaining unfazed by experimental prototypes from tech giants like Google’s Diffusion Gemma. "I think it’s validating. This is the future."
04. Moving Beyond Words: State Spaces and Non-Linguistic Reasoning
Perhaps the most radical departure comes from Palo Alto startup Pathway, which seeks to unshackle AI models from the strict confines of human language.
Pathway developed Dragon Hatchling, a model designed to tackle complex spatial and mathematical reasoning tasks. Notably, Dragon Hatchling successfully solved over 97% of a rigorous benchmark consisting of 250,000 advanced Sudoku puzzles—a domain where conventional, language-bound LLMs completely fail.
The secret lies in replacing the transformer’s attention matrix with a mathematical framework known as a state space. Rather than serializing abstract logic into words, state spaces compress information into abstract representations that mimic non-linguistic human intuition, such as mathematical insight or spatial strategy.
"The eureka moment that pops up in your brain isn’t necessarily in language," explains Zuzanna Stamirowska, co-founder and CEO of Pathway. "We would argue that if you have to reason in language, you’re somehow constrained… The hope for AI is not to solve sudoku; it’s to cure cancer. There’s not a book for that."
She adds pointedly: "Transformers are an engineering convenience that we fell on. It started a religion, but it’s silly to think that a breakthrough won’t happen again."
Future Outlook: The Dawn of LLMs+
The artificial intelligence industry stands at a profound architectural crossroads. The era of pure, unadulterated transformer scaling—driven by endless capital injections and ever-larger data centers—is hitting the hard walls of thermodynamics, economics, and computational physics.
As the industry transitions into the "LLMs+" era defined by MIT Technology Review, the metric for success is shifting. Raw parameter counts are giving way to metrics of extreme efficiency: intelligence per dollar, inference speed per watt, and cross-domain cognitive flexibility.
Whether through sparse attention mechanisms, power retention loops, liquid neural hybrids, parallel diffusion text generation, or non-linguistic state spaces, the post-transformer movement proves that human ingenuity is far from exhausted. The monopoly of the transformer is fracturing, and in its place, a faster, leaner, and fundamentally smarter generation of artificial intelligence is clawing its way into existence.
