Executive Overview
Nine years ago, a team of Google researchers published a landmark paper titled "Attention Is All You Need." The work introduced the transformer, a novel neural network architecture that proved remarkably adept at processing long sequences of data, particularly human language. Today, that single architectural breakthrough serves as the beating heart of every major large language model (LLM) on the market. From conversational agents to complex multimodal systems, the entire modern AI industry is built upon the transformer.
Yet, as the industry matures, these foundational models are showing their age. Far from being infinitely scalable, transformers are hitting structural bottlenecks. Many recent advancements in LLMs—such as reasoning models that generate "chains of thought" or systems capable of digesting massive inputs—are not natural extensions of the core transformer architecture. Instead, they are intricate workarounds patched over fundamental flaws.
The primary culprit is dense attention, the mechanism by which transformers process information. By requiring every token in a text to interact with every other token via heavy matrix multiplication, transformers demand staggering computational resources as input lengths grow. OpenAI is projected to spend upwards of $50 billion on computing power this year alone, while the International Energy Agency (IEA) predicts that global data center electricity consumption will double by 2030.
Recognizing that the transformer’s core strength has become its greatest limitation, a new guard of scientists, engineers, and venture-backed startups is asking a vital question: What comes next? While LLMs are here to stay, the methods used to build them are up for grabs. Dubbed "LLMs+" by industry analysts, this upcoming generation of artificial intelligence seeks to shatter the current computational ceilings. This report examines four radical technological shifts that could redefine how AI is built, making it faster, vastly more efficient, and fundamentally smarter.
Detailed Chronology: From Google’s 2017 Breakthrough to the Post-Transformer Era
To understand the urgency of today’s architectural pivot, it is helpful to trace the timeline of modern generative AI:
- June 2017: Google researchers publish "Attention Is All You Need," outlining the transformer neural network. It replaces recurrent neural networks (RNNs) by allowing parallel processing of sequential data through self-attention mechanisms.
- 2018–2020: The rise of foundational models. OpenAI introduces Generative Pre-trained Transformer (GPT) models, followed by Google’s BERT and later GPT-3. The transformer proves to be an unprecedented vehicle for scaling text generation.
- 2022–2024: The generative AI boom goes mainstream with the launch of ChatGPT. LLM sizes balloon into hundreds of billions of parameters. However, power consumption skyrockets, data center queues stretch thin, and the physical limits of GPUs begin to clash with economic realities.
- 2024–2025: Researchers tackle reasoning and long-context windows. Models begin using internal "scratch pads" (chains of thought) and ingesting massive codebases, pushing dense attention architectures past their breaking point. Energy consumption metrics raise severe environmental and financial flags.
- 2026 and Beyond (The LLM+ Era): Startups challenge the status quo. Alternative mechanisms—including sparse attention, power retention, liquid neural networks, diffusion text generation, and state-space models—emerge from academic labs into commercial testing, signaling the beginning of the post-transformer transition.
Supporting Context & Metrics: The Cost of Scale
The push to replace or augment transformers is driven by hard economics and thermodynamic realities. The fundamental mathematics of dense attention create a punishing scaling curve. For instance, a document 10,000 words long requires a standard transformer to perform roughly 50 million individual multiplications just to compute the relationships between words.
This computational overhead translates directly into astronomical financial and environmental costs:
- Compute Expenditures: Industry leaders are pouring unprecedented capital into infrastructure. OpenAI’s projected $50 billion computing spend for the year illustrates a financial model that is deeply reliant on raw brute force.
- The Energy Crisis: According to the International Energy Agency (IEA), global data center electricity use surged dramatically through the mid-2020s and remains on track to double by 2030, driven largely by the insatiable demands of training and running dense attention models.
- The Context Window Wall: While users demand that LLMs read entire corporate codebases, legal libraries, or multi-hour video transcripts at once, transformers struggle to maintain coherence as context windows expand due to memory bandwidth limits.
Four Radical Innovations Solving the Transformer Problem
01: Rethinking Attention (Sparse Attention vs. Power Retention)
The most direct way to fix the transformer problem is to reform attention itself. Swapping dense attention for sparse attention—which calculates relationships only between select word pairings rather than every single one—drastically reduces computational load. Historically, sparse attention mechanisms struggled to match the semantic precision of dense attention. However, Miami-based startup Subquadratic claims to have solved this hurdle. Their model, SubQ, dynamically determines on the fly which words matter for any given piece of text, achieving performance that rivals mainstream LLMs in search and coding tasks.
Taking a different path, San Francisco-based Manifest AI has bypassed attention entirely, replacing it with a mechanism known as power retention.
- How it works: Traditional attention forces an LLM to retain everything in its context window. Sparse attention discards individual words but keeps a macro-level footprint. Power retention, by contrast, maintains a rolling summary of the context window. As new data flows in, less relevant information is dropped.
- Practical Impact: Manifest AI has adapted existing open-source models, such as the coding LLM StarCoder, into PowerCoder, and released Brumby to challenge Alibaba’s Qwen. Co-founder and CTO Carles Gelada notes that this technology enables applications like analyzing hours of continuous video or powering autonomous agents that run uninterrupted for weeks.
02: Making Models Smaller and More Flexible (Liquid AI)
Rather than abandoning transformers entirely, MIT spinout Liquid AI has pioneered a hybrid approach. Based in Cambridge, Massachusetts, the firm pairs transformers with liquid neural networks—an evolution inspired by biological worm brains and convolutional networks—to create Liquid Foundation Models (LFMs).
- Adaptive Behavior: Standard transformers are structurally static; once trained, their behavior is fixed. Liquid neural networks feature mechanisms that allow models to adapt their behavior to incoming information in real-time, learning continuously as they operate.
- Extreme Efficiency: LFMs are remarkably lightweight. While traditional models require massive clusters of enterprise GPUs, Liquid AI’s models can run locally on low-powered automotive chips (such as those used by Mercedes-Benz) or even on a $50 Raspberry Pi computer.
- Automated Architecture Design: Using an internal meta-AI system, Liquid AI tests millions of combinations of transformers, convolutional networks, and liquid architectures. Their latest flagship models are hybrid structures consisting of 20% transformers and 80% liquid neural networks, matching the performance of competitor models four times their size.
03: Generating Text All at Once (Diffusion LLMs)
Virtually all contemporary LLMs generate text sequentially, word by word. While natural for human speech, this serial generation is highly inefficient for silicon processors. Palo Alto-based Inception is addressing this bottleneck by applying diffusion—the technology behind state-of-the-art image and video generation models—to text.
- Parallel Generation: Diffusion models take a randomized field of data (akin to television static) and iteratively refine it into a coherent output across all points simultaneously. Inception trains its models to take random strings of tokens and resolve them into coherent sentences in a single pass.
- Overcoming the Language Hurdle: Applying diffusion to text required entirely new mathematics. While an image can smoothly transition from blue to red through intermediate pixel values, discrete text ("cat" vs. "dog") has no middle ground. In 2024, Inception co-founder and Stanford researcher Stefano Ermon successfully resolved these mathematical constraints, building a model that matched OpenAI’s older GPT-2 while operating 10 times faster.
- The Pursuit of Efficiency: Inception’s latest model, Mercury 2, reportedly matches the performance of OpenAI’s GPT-4 while maintaining a 10x speed advantage. As Ermon succinctly states, the ultimate currency of the future will be intelligence per dollar.
04: Moving Beyond Words (State-Space Models & Pathway)
Perhaps the most philosophically radical departure comes from Palo Alto-based Pathway, which seeks to free artificial intelligence from the constraints of human language altogether.
- The Limitations of Language: Pathway’s flagship model, Dragon Hatchling (named after the reality-materializing dragons in Terry Pratchett’s The Colour of Magic), recently solved over 97% of a rigorous benchmark consisting of 250,000 highly complex Sudoku puzzles—a task where multiple leading mainstream LLMs scored zero. According to CEO Zuzanna Stamirowska, this proves that language is an inefficient medium for deep logical reasoning. "It’s very difficult to represent a Sudoku board word by word," she notes.
- State-Space Mathematics: Pathway replaces the transformer’s attention mechanism with state spaces, a mathematical structure that compresses information into abstract, multi-dimensional representations rather than sequential words.
- Toward Novel Discovery: By bypassing the need to "think in sentences," Pathway’s models can execute types of non-verbal reasoning akin to human mathematical intuition or chess calculations. As Stamirowska observes, human eureka moments do not manifest as long internal monologues: "The hope for AI is not to solve Sudoku; it’s to cure cancer. There’s not a book for that."
Official Statements and Industry Perspectives
The consensus across the research community is that while transformers sparked the current generative AI revolution, they should be viewed as a stepping stone rather than a permanent destination.
- Justin Dangel (Cofounder and CEO, Subquadratic): "The entire AI industry is built on transformers. They are one of the most important innovations in the history of computer science, and they’ve changed the world. But they are starting to show their age."
- Ramin Hasani (Cofounder and CEO, Liquid AI): "Your brain is an AGI system, you know, and it operates with 20 watts of power. How is it possible? We can get a lot more innovative." Pointing to hybrid architectures, he views pure transformers as just the opening chapter of machine intelligence.
- Stefano Ermon (Cofounder and CEO, Inception): Commenting on tech industry validation as giants like Google experiment with diffusion models (e.g., Diffusion Gemma), Ermon maintains an optimistic outlook: "I think it’s validating. This is the future."
- Zuzanna Stamirowska (Cofounder and CEO, Pathway): Challenging the dogma surrounding text-based models, Stamirowska remarks: "Transformers are an engineering convenience that we fell on. It started a religion, but it’s silly to think that a breakthrough won’t happen again."
Future Outlook: The Dawn of LLMs+
The transition away from unyielding transformer dominance marks a maturing phase for the artificial intelligence sector. While incumbent tech giants continue to optimize massive, brute-force transformer clusters, agile startups are demonstrating that alternative paths to machine intelligence are not only viable, but necessary for sustainable growth.
Whether through sparse attention that prunes redundant calculations, power retention that gracefully manages infinite contexts, liquid neural networks that bring edge-computing adaptability, diffusion models that write paragraphs in a single computational beat, or state-space mathematics that transcend the limits of language, the blueprint of AI is undergoing a fundamental rewrite.
As the industry enters the LLM+ era, the defining competitive advantage will no longer simply be who can throw the most capital at a data center. Instead, the future belongs to those who can build systems that deliver maximum intelligence at minimal energy cost—ushering in an era of faster, cheaper, and profoundly more capable machine minds.
