Executive Overview
In the hyper-competitive landscape of business-to-business (B2B) software, the rules of marketing are undergoing a seismic shift. For decades, tech vendors have relied on polished slide decks, static feature checklists, self-serving analyst quadrants, and gated PDF reports to claim market dominance. In this traditional playbook, every company claims to be the fastest, the most reliable, and the most advanced.
However, the dawn of generative artificial intelligence and autonomous software agents has rendered these legacy marketing tactics obsolete. Unlike traditional software—where a CRM button performs the exact same programmatic function on a Tuesday afternoon as it does on a Friday morning—AI agents are dynamic, probabilistic systems. They can deliver stellar, human-like reasoning in one customer interaction and hallucinate or fail catastrophically in the next. Worse yet, silent model updates deployed by upstream providers can degrade an agent’s performance overnight without any explicit notification to the end-user.
Recognizing this volatility, enterprise buyers have grown increasingly skeptical of traditional vendor claims. They no longer trust static product comparisons or glossy marketing materials. They demand empirical proof.
Enter the era of open-source, live competitive evaluations ("evals"). Leading the charge is Gorgias, a $100 million ARR (Annual Recurring Revenue) leader in ecommerce customer experience (CX) automation. Backed by the SaaStrFund, Gorgias recently shattered industry norms by publishing a comprehensive, highly rigorous, open-source benchmark comparing its proprietary AI support agent against 18 competing vendors across 8,356 live customer conversations. Crucially, the report does not paint Gorgias as the undisputed winner in every category; it transparently highlights competitors—such as Yuma and Envive—surpassing Gorgias in specific performance metrics.
This watershed moment signals a broader industry truth: in the age of AI, radical transparency is no longer just a moral high ground—it is a vital competitive moat. By open-sourcing their test harnesses, publishing their rubrics, and openly acknowledging where they lose, forward-thinking tech companies are reshaping buyer diligence, neutralizing skepticism, and preparing for a future where autonomous AI agents curate vendor shortlists.
Detailed Chronology: The Evolution of Software Benchmarking
To understand the magnitude of Gorgias’s recent move, it is necessary to examine how software evaluation has evolved over the past thirty years.
Phase 1: The Era of Feature Checklists (Late 1990s–2010s)
During the rise of traditional Software-as-a-Service (SaaS), vendor differentiation was straightforward. Software was deterministic. A vendor either supported single sign-on (SSO), localized currency conversions, or custom API webhooks, or they did not. Buyers relied on binary feature checklists compiled by procurement teams or IT consultants. Industry analysts like Gartner and Forrester aggregated these capabilities into rigid, quadrant-based maps. While these reports were occasionally biased by vendor relationships or marketing budgets, they served as an acceptable baseline for enterprise procurement because software capabilities changed slowly, usually across annual release cycles.
Phase 2: The G2 and Peer-Review Revolution (2010s–2020s)
As the SaaS market saturated, buyers lost faith in top-down analyst firms. Platforms like G2, Capterra, and TrustRadius rose to prominence by crowdsourcing reviews from actual users. While this introduced a democratic element to software evaluation, it remained fundamentally subjective. Reviews could be gamed, incentivized, or skewed by emotional reactions to customer support interactions rather than systematic product capabilities. Furthermore, peer reviews struggled to keep pace with rapid feature deployments.
Phase 3: The Black Box of Generative AI (2023–2025)
The explosion of Large Language Models (LLMs) and conversational AI agents in 2023 caught the B2B market unprepared. Suddenly, enterprise buyers were expected to purchase complex, autonomous agents that could reason, execute multi-step workflows, and interact directly with end-users.
However, vendors reverted to old habits. They published benchmark metrics derived from closed-door tests, cherry-picked enterprise use cases, or generic academic benchmarks (such as MMLU or GSM8K) that bore little resemblance to actual commercial workflows. Buyers were forced to rely on limited proof-of-concept (PoC) trials or take sales reps at their word. The industry was plagued by a "trust deficit."
Phase 4: The Open-Source Evals Movement (Late 2026 and Beyond)
The release of the Gorgias AI Agent Benchmark in September 2026 marks the definitive beginning of Phase 4. By open-sourcing its entire test harness—complete with a versioned GitHub repository, 8,356 live conversations, and explicit scoring rubrics—Gorgias established a new template for software validation. This chronology highlights a clear trajectory: from static vendor promises to crowdsourced opinions, and finally, to verifiable, reproducible code and empirical data.
Supporting Context & Metrics: Inside the Gorgias AI Benchmark
The scale and depth of the Gorgias benchmark set it apart from any traditional software comparison. Operating at approximately $100 million in ARR—with roughly 80% of that revenue tied directly to AI-driven customer support solutions for ecommerce brands—Gorgias had a massive commercial footprint to defend. Yet, rather than hiding behind marketing spin, leadership embraced radical exposure.
The Anatomy of the Benchmark
The Gorgias AI Agent Benchmark is built on several foundational pillars designed to eliminate ambiguity:
- Real-World Volume: The evaluation process tested agents across 8,356 live customer conversations, simulating the chaotic, unpredictable nature of actual ecommerce support tickets.
- Comprehensive Vendor Scope: The study evaluated 18 distinct market vendors, providing a holistic view of the ecommerce CX landscape rather than a bilateral comparison against a single rival.
- Methodological Transparency: The entire testing harness was open-sourced on GitHub, allowing any developer, data scientist, or enterprise buyer to inspect the underlying code, audit the prompts, and replicate the results.
Embracing Vulnerability: Where Gorgias Lost
The true genius—and credibility—of the benchmark lies in its refusal to claim perfection. While Gorgias secured the #1 position in overall support efficacy, the report explicitly highlighted areas where competitors outperformed them.
For instance, the data revealed clear automation and resolution rate advantages held by competitors like Yuma, while other niche players demonstrated superior response latency (such as Envive’s impressive 7.9-second response metric). By putting these vulnerabilities front and center on the very first page of the report, Gorgias neutralized the natural skepticism inherent in B2B buying cycles.
Official Statements & Industry Perspectives
The philosophy driving this transparency movement has found a vocal champion in Jason Lemkin, founder of SaaStr and the SaaStrFund (which led Gorgias’s seed round). Lemkin’s advocacy for open competitive evals has galvanized the SaaS and AI communities.
"Everyone should publish the deepest, most direct competitive evals they can," Lemkin asserted in a widely shared industry commentary. "Will there always be some bias? Yes. You wrote the rubric, picked the weights, and chose what to measure. Say that on the first page. Then make everything checkable."
Lemkin noted that Gorgias executed this strategy to near-perfection within the ecommerce CX ecosystem. According to SaaStr leadership, investors actively pushed Gorgias to build the most rigorous, unvarnished evaluation possible.
“We know the company well—SaaStrFund led the seed round, and in fact, we pushed them to do the most honest, detailed evals in the CX+ space,” Lemkin explained. “And they did. The results don’t all go Gorgias’s way, and they published them anyway.”
Industry analysts have echoed these sentiments, pointing out that B2B buyers are exhausted by the friction of traditional enterprise sales cycles. By doing the heavy lifting of comparative diligence upfront, vendors can drastically shorten sales cycles and build immediate, lasting trust.
Why Publishing Your Evals Wins: Strategic Advantages for AI Vendors
For software executives hesitant to expose their product flaws to the public eye, the Gorgias case study offers four undeniable strategic imperatives.
1. Buyers Already Discount Traditional Marketing
Modern procurement leaders and customer support VPs have been burned by vendor exaggerations too many times. When a company presents a self-generated chart showing a 100% win rate across every conceivable metric, buyers instinctively tune it out. Conversely, when a vendor willingly highlights areas where competitors excel, it lends profound credibility to the categories where that vendor does win. Gorgias claiming the #1 spot in support carries immense weight precisely because Yuma’s automation lead is published alongside it on the exact same page.
2. Outsourcing Vendor Diligence
No mid-market or enterprise ecommerce brand has the engineering resources to independently test 18 different AI vendors across thousands of live conversations. Typically, a buyer relies on three vendor demos and a superficial pilot. By publishing a exhaustive, open-source evaluation, a vendor effectively conducts the market’s diligence for them. The published report becomes the definitive source of truth that buyers rely on, positioning the publishing vendor as an authoritative market leader.
3. Adapting to AI-Driven Shortlisting
The nature of software discovery is shifting rapidly. Increasingly, enterprise technology evaluations do not begin with a human browsing Google or attending trade shows; they begin with a query posed to advanced reasoning agents like Claude or ChatGPT.
An AI research agent can read, verify, and cite a versioned GitHub repository, open-source scoring weights, and public rubrics. It cannot, however, extract meaningful insights from a gated, corporate PDF marketing whitepaper. Vendors that structure their competitive evals as open, machine-readable datasets will be systematically favored and cited by AI procurement agents. (Note: While Gorgias’s live report currently restricts automated readers via robots.txt, the open-source GitHub repository provides the necessary transparency).
4. Internal Accountability and Continuous Improvement
Publishing live, weekly-updated evaluations transforms competitive intelligence from a secret slide deck buried in the product marketing folder into an operational heartbeat. Inside Gorgias, competitor metrics—such as Envive’s latency or Yuma’s resolution rates—are no longer abstract concepts; they are public, permanent fixtures sitting directly next to internal performance figures. Because the rubric is versioned in a public repository, internal engineering teams cannot quietly alter scoring parameters to mask performance gaps. It creates a culture of relentless, measurable product iteration.
How to Execute an Open Evals Strategy: A Template for Founders
For B2B founders and product leaders looking to replicate the Gorgias playbook, the path forward requires courage, technical rigor, and absolute methodological honesty. Industry leaders recommend following a strict five-step template:
- Open-Source the Harness: Do not merely publish the final score; publish the underlying code, prompts, and test scripts in a public repository (such as GitHub) so that anyone can audit or rerun the evaluation.
- Test on Real Workflows: Move beyond synthetic benchmarks and academic quizzes. Evaluate agents using actual, anonymized customer conversations that reflect the messy reality of your target market.
- Include Named Competitors: Name names. Vague references to "Competitor A" or "Industry Average" breed suspicion. Test against the actual market leaders and direct rivals your buyers are considering.
- Admit Your Flaws: Explicitly feature the metrics, use cases, or categories where your product loses. A benchmark where you win 100% of the time is treated as marketing fiction; a benchmark where you win 70% and lose 30% is treated as empirical fact.
- Version Your Rubric: Acknowledge that you designed the scoring weights, but make those weights transparent, static, and version-controlled. Ensure that scoring criteria cannot be covertly manipulated over time to hide product regressions.
Future Outlook: The Death of the Gated PDF
As artificial intelligence matures from an experimental novelty into the core operating system of enterprise software, the tolerance for opaque marketing will approach zero.
The strategy pioneered by Gorgias and championed by SaaStr is not a temporary marketing stunt; it is the early blueprint for a permanent structural shift in enterprise technology sales. In the near future, B2B buyers—augmented by their own procurement AI agents—will simply refuse to engage with vendors that refuse to open-source their performance data.
In this transparent new world, the companies that win will not be those with the loudest marketing budgets or the most persuasive sales teams. They will be the ones confident enough in their engineering to run the tests, publish the losses, and let the code speak for itself.
