OpenClaw has become one of the most widely adopted agentic frameworks, but it has yet to prove itself at enterprise scale. Agents need real credentials — API keys, OAuth tokens, service accounts — to work effectively, and Brex found that traditional guardrails couldn’t contain what those agents were doing with them.
Brex set out to overcome these limitations by building an internal platform it calls CrabTrap. The open-source HTTP/HTTPS proxy intercepts all network traffic, examines policy rules, and uses a LLM-as-a-judge to decide whether agent requests should be approved or denied.
“What we noticed was that the network layer was an untapped enforcement point,” Brex co-founder and CEO Pedro Franceschi told VentureBeat. “Every request an agent makes is an opportunity to intercept, reason about, and make a policy decision.”
The takeaway Franceschi wants IT leaders to draw: agent governance should shift from SDK-level permissions and model guardrails toward a centralized network control plane that enforces and learns from real in-the-wild agent behavior.
The “obvious fix” (at least initially) to the agent security gap was guardrails, and much of the early work has centered on scoped tools, per-action permissions, and human-in-the-loop approvals. But as agents evolve, each new capability means there’s another API to tune or surface to audit, Franceschi noted.
“Any agentic system with multiple tools and access to the open internet creates an immediate tension for builders: The more capable you make an agent, the more dangerous it becomes, and the safer you make it, the less useful it is,” he said.
Existing solutions to this tradeoff were “weak”: Fine-grained API tokens help at the margins but can still be misused and constrain functionality. Semantic guardrails (such as context, skills, or prompt steering) are easily bypassed by prompt injection, especially for agents connected to the internet.
Agents can be “defanged” when given read-only access or limited toolsets, but then they can’t do meaningful work, Franceschi said. On the other hand, granting broad write access and a large tool surface can result in hallucinations and real production consequences.
Model context protocol (MCP) gateways enforce policy at the protocol layer — but only for traffic using MCP. Meanwhile, guardrails from LLM providers are tied to a single model and can be “opaque” to customize with enterprise-specific policies. And powerful tools like Nvidia OpenShell offer more of a “per-sandbox egress control.”
“When we started, we hadn’t found a solution to deploying harnesses like OpenClaw safely,” Franceschi said. “Instead of waiting for the industry to catch up, we decided to own the problem and invent the necessary tools.”
Notably, they needed a platform that sat between every agent and every network request, and could make “nuanced decisions about what to allow,” he said.
This made the transport layer a core architectural component and natural starting point, he said.
By operating at this layer, CrabTrap is framework-agnostic, language-agnostic, and API-agnostic. It doesn’t require SDK wrappers or per-tool integration. Users set HTTP_PROXY and HTTPS_PROXY in the agent’s environment, and every outbound request routes through the proxy before it reaches a destination.
However, Franceschi emphasized, Brex didn’t start at the transport layer because it thought it was the only answer; rather, they believe in “security by layers.”
“The transport layer was simply an underinvested one, and we saw an opportunity to add meaningful enforcement there alongside everything else,” he said.
CrabTrap combines deterministic static rules with an LLM-as-a-judge for requests that fall outside known patterns, Franceschi explained. The judge only “fires on the long tail of unfamiliar endpoints or unusual request shapes,” which for a mature agent is typically fewer than 3% of requests.
The more pressing problem was how to know that a policy is the right one? With static rules, it’s “relatively straightforward” to reason about accuracy. But with an LLM judge, the system is nondeterministic, and users need confidence that the policy approves the right requests and blocks the rest.
“Our key insight was to bootstrap policy from observed behavior rather than write it from scratch,” Franceschi said. Beginning with real behavior and editing down based on real-world learnings turned out to be “dramatically more effective than starting from a blank page.”
Brex’s team built a policy builder (itself an agentic loop) that runs underlying agents in shadow mode, analyzes historic network traffic, samples representative calls, and drafts a natural-language policy that matches what the agent actually does.
From there, they built an eval system that tests policy changes before they go live. CrabTrap compares historical audit entries against a draft policy and reports the exact changes to be made. Users can slice results by method, URL, original decision, and agreement status.
All of this runs with concurrent judge calls, so replaying thousands of requests “takes minutes, not hours,” Franceschi said. Brex also developed a live feedback loop: Full audit trails are stored in PostgreSQL and queryable through the admin API and dashboard. In cases where a resource is continuously denied, the system can notify a human or an agent to propose a policy update for review.
“That closes the loop between observed denials and policy refinement,” Franceschi said.
Of course, the build wasn’t without its challenges. A big one was latency: “Putting an LLM between an agent and every outbound API request sounds like it would grind things to a halt,” he said.
However, it didn’t turn out to be as big a problem as expected. This was for two reasons: The LLM judge only activates on a small fraction of requests (the aforementioned 3%). Agents quickly settle into predictable traffic patterns; once observed, high-volume patterns become static rules. Second, by using small, fast models like Claude Haiku meant that, even when the judge did fire, added latency was “negligible.” This can be further reduced with local models and prompt caching, Franceschi said.
The harder and less obvious challenge was prompt injection, he said. The judge receives the full HTTP request and all content is user-controlled, so potentially, a crafted URL, header, or request body could manipulate the judge’s decision.
Brex addressed this by structuring the request as a JSON object before sending it to the model, so all user-controlled content is “escaped rather than interpolated as raw text,” Franceschi said.
Brex tracks a few factors to measure CrabTrap’s internal impact: Engagement with agents, network traffic patterns, and net promoter scores (NPS). The most meaningful result of CrabTrap has been “organizational confidence,” Franceschi said.
Previously, the team had “real hesitation” when it came to deploying autonomous agents broadly across business operations, because the existing guardrail options didn’t provide enough assurance.
“CrabTrap changed that calculus,” Franceschi said. They now have an enforcement layer they trust, increasing confidence around expanding agent deployment into more parts of the business and delegating more agent configuration and management to users.
Franceschi described the policies derived from traffic as “surprisingly strong.” The team expected the policy builder to produce a “rough starting point” requiring heavy manual editing. In practice, though, pointing the platform at a few days of real traffic produced policies that matched human judgment on the “vast majority of held-out requests.”
Additionally, CrabTrap revealed how much noise agents generate. “The audit trail made this visible for the first time,” Franceschi said. They used denial logs and traffic analysis not only to tune policies, but to tighten agents themselves, remove tools, and cut out entire categories of requests that were wasting both time and tokens.
“The proxy became a discovery tool, not just an enforcement one,” he said.
Brex anticipates CrabTrap to continue to evolve, particularly as they have released it as open-source. “We hope the community helps shape it,” Franceschi said.
Areas of improvement include deeper authentication functionality such as single-sign on (SSO), fine-grained role-based access control (RBAC); escalation workflows that allow agents to request additional permissions; and policy recommendations based on denial patterns.
Programmatic configuration, or developing API endpoints for “creating, forking, and applying” policies to agents, could allow the whole policy lifecycle to be automated rather than managed manually, Franceschi said.
As for escalation, if an agent is continuously denied a given resource or endpoint, it should be able to route requests to humans or other AI agents for review and back that up with a rationale for why it needs access.
“That turns CrabTrap from a hard enforcement boundary into something more like a managed permission system,” Franceschi said.
Additionally, the policy was built to bootstrap from network traffic, but there is opportunity to incorporate additional signals around agent traces and resource-calling, as well as broader context on what agents are ultimately trying to accomplish. This can help produce more accurate and nuanced policies.
Finally, there’s an “open philosophical question” about the right posture for CrabTrap: Should it be a fully transparent layer that the agent itself is unaware of, or should it operate more like a “well-intentioned manager”? (that is, the agent knows about the layer and can interact with it).
The open-source community can help shape these developments, and CrabTrap will only get better with more users, Franceschi said. Brex’s agents speak to a specific set of APIs; teams using CrabTrap with different agents, services, and policy requirements will surface “edge cases and patterns we can’t hit alone.”
“We have ambitious plans for where it could go, and we’d rather build in the open,” Franceschi said.
The response has been stronger than expected. CrabTrap has more than 700 stars on GitHub. Franceschi said Brex has also heard from OpenAI, Y Combinator CEO Garry Tan, and programmer Pete Steinberger, all expressing interest in deploying similar internal infrastructure.
The broader lesson: “Don’t let infrastructure gaps become excuses to wait,” Franceschi advised. There are “real blockers” for every enterprise looking to seriously deploy AI agents, including security concerns, lack of tooling, or unclear guardrails.
“It’s tempting to sit on your hands until the industry catches up,” he said. “The lesson from CrabTrap is that you can own those problems directly.”
The enterprise AI industry has a math problem. Cisco data shows 85% of enterprises are piloting AI agents, but only 5% have shipped them to production. At VB Transform 2026 on Tuesday, Bryan Silverthorn, Director of AGI Autonomy at Amazon, explained why that gap persists — and why the answer isn’t better benchmarks.
Silverthorn, who joined Amazon through its acquisition of Adept AI and now leads multimodal agent training inside the company’s AGI lab, argued that reliability must be broken into four distinct dimensions: consistency, robustness, predictability, and safety — a framework he credits to research from Princeton.
“It unpacks different factors that I see tangled together in almost every eval I’ve ever seen,” he said.
The framework matters because agents routinely ace internal evaluations and then collapse in the wild. Silverthorn described a customer that deployed an agent for software QA involving serial number extraction from screens. It worked flawlessly for two months — then began intermittently reading wrong numbers. The culprit: the underlying vision encoder behaved differently depending on where the serial number appeared on screen, and a software change imperceptible to humans triggered the failure.
The lesson, Silverthorn said, is about measurement, not just models. “The models have to be better. Obviously, we’re working hard on making the models better,” he said. But the deeper takeaway, he added, is that teams need to identify their dimensions of variability and match measurement rigor to the stakes of the application. VentureBeat’s own proprietary research, presented before the session, reinforces the point: half of surveyed companies shipped agents that passed internal evals but failed real customers, and enterprises overwhelmingly track uptime while ignoring accuracy — checking the pulse without checking the diagnosis. A related finding underscored how few guardrails exist: most enterprises default to the model makers’ own evaluations and little else, leaving their testing strategy, as I described it on stage, a coin flip between trusting the vendor and trusting nothing.
Silverthorn’s most memorable prescription was cultural, not technical. Inside Amazon’s AGI lab, researchers literally call their agents “interns” — as in, “I’ll have my intern talk to your intern.” The joke carries a serious operational philosophy. Agents, like interns, are powerful but occasionally clueless, capable of amazing work and spectacular derailment.
Managing them, he argued, requires management skills rather than software skills: asking what could go wrong, adding backups and undo capabilities, and consciously deciding what risk you can accept. “You can ask the intern, ‘Hey, what might you do wrong here? How might you mitigate your negative outcomes?'” he said. Amazon’s lab has embraced that trade-off, accepting agents occasionally running the wrong experiment in exchange for research velocity — including one agent running experiments around the clock on its own high-level research plan.
Silverthorn was candid about the limits of today’s technology. Self-improving AI remains “a loaded term,” he said — Amazon uses AI to improve its models constantly, but fully autonomous self-improvement is distant. Computer use remains a core focus of his lab, with a commercial trucking customer already using browser automation to stitch together warranty claims across fragmented systems**, though he stressed that no future agent will rely on computer use alone — it will work alongside MCP, APIs, and other tools to complete end-to-end workflows**. And LLM-as-judge techniques, while promising, are just one of several strategies for aligning agent capability with acceptable risk.
For enterprises stuck in pilot purgatory, the path forward starts with a mindset shift: stop asking whether your agent can do something impressive once, and start asking whether it can do it correctly a thousand times in a row.
In other words, the enterprises that escape the 85% ceiling won’t be the ones with the smartest agents. They’ll be the ones with the best managers.
Model routing is becoming a key component of the enterprise AI stack, dynamically sending prompts to the right AI model to optimize speed and costs. However, current frameworks mostly treat routing as a static classification problem, which severely limits their potential.
A new open-source framework called Agent-as-a-Router tackles this bottleneck, treating the router as a dynamic, memory-building agent. It uses a Context-Action-Feedback (C-A-F) loop to track model successes and failures and update the behavior of the router.
The researchers also released ACRouter, a concrete implementation of this paradigm. In their tests, ACRouter significantly outperformed static routers and the expensive strategy of defaulting to premium models, all without requiring teams to train massive models or write endless heuristics.
For real-world applications, this framework provides the option to replace hard-coded AI infrastructure with self-optimizing systems that can adapt to changes in user behavior and foundation models used in the enterprise AI stack.
Single-model setups are useful for experiments but detrimental when scaling AI applications. AI engineers use model routing to map tasks to cheaper and faster open models when possible, while reserving expensive frontier models for complex reasoning.
Currently, developers rely on two main mechanisms for this task. The first is heuristics-based routing, which relies on hard-coded manual rules. For example, a developer might write a rule dictating that if a prompt contains certain keywords, it is routed to GPT-5.5. Otherwise, it goes to a self-hosted open source model like Kimi K2.7.
The second mechanism is static trained policies. These are machine learning classifiers trained on historical datasets that look at the prompt’s embeddings and predict the best model based on past training data.
Both approaches are static. When the researchers tested these existing mechanisms on real-world coding and agentic workflows, they found a hard ceiling on accuracy. The key finding shows that static routers suffer from a severe information deficit. Because they only evaluate the input text and never see if the model actually succeeded in executing the task, they guess blindly when faced with complex edge cases.
This results in three distinct points of failure. First, static routers suffer from a frozen information state, meaning they cannot accumulate new execution feedback during deployment. Second, they fail in out-of-distribution (OOD) generalization. They break down during day-two operations when enterprise data or user behavior shifts because their training data no longer matches reality. Finally, they are highly vulnerable to model churn. A static classifier trained on today’s models may become obsolete when a better model drops the following week.
The core thesis of the Agent-as-a-Router is that a truly effective router must acquire and accumulate execution-grounded information during deployment, essentially learning on the job.
The researchers achieved this through the C-A-F loop. When a new prompt arrives, the router examines the prompt and task metadata, such as the programming language or difficulty. It then searches its historical memory for similar tasks to see which models succeeded or failed in the past. The router uses this context to select the target model and execute the task. Finally, the system observes the real-world outcome, extracts a success or failure signal, and writes this feedback back into its memory to inform future routing decisions.
Consider an automated enterprise data analytics pipeline. The router receives a SQL generation task and sends it to an open-source model like Kimi. The model hallucinates a column name and fails to compile the SQL. The C-A-F loop observes the compiler error, registers it as feedback, and logs it. The next time a similar obscure SQL query arrives, the router checks its context and routes the task to a more advanced model like Claude Opus 4.8.
The researchers developed ACRouter as the concrete instantiation of this framework. It is composed of three core components: the Orchestrator, the Verifier, and Memory. This architecture is supported by a tool layer to physically execute the C-A-F loop.
The Memory module powers the context phase. Built on a vector store, it retrieves relevant past interactions and updates the historical database with new outcomes. The Orchestrator handles the action phase. It processes the user prompt alongside the retrieved memory to select the most capable target model from the available pool. The Verifier manages the feedback phase by evaluating the chosen model’s output to generate a clear success or failure signal.
The tool layer hooks the Verifier into real-world execution environments, like a Python code interpreter, an agentic sandbox, or a database engine. The tool layer allows the system to execute the generated code or query and observe the exact outcome, providing the verifiable signal the router needs to learn.
The Orchestrator itself is lightweight. Instead of a massive, computationally heavy large language model, the researchers trained a sub-billion parameter adapter based on Qwen 3.5 (0.8B parameters), which means it can be self-hosted on a device of your choice.
To stress-test the framework, the researchers introduced CodeRouterBench, an evaluation environment comprising roughly 10,000 tasks with verified scores across eight frontier models, including Claude Opus 4.6, GPT-5.4, Qwen3-Max, and GLM-5. The evaluation was split between in-distribution (ID) tests (covering nine single-turn coding dimensions like algorithm design and test generation) and an out-of-distribution (OOD) agentic programming testbed. The OOD tasks were qualitatively different, requiring multi-step planning, file navigation, and iterative debugging to see if the router could adapt to fundamentally new domains.
The baseline results revealed why a single-model strategy is flawed: no single model dominates every category. For example, while Claude Opus 4.6 achieved the highest average performance, it was outperformed in algorithm design by GLM-5 (an 86% relative improvement) and in test generation by Qwen3-Max (a 111% improvement), despite Opus costing roughly 12 times as much as smaller models like Kimi-K2.5.
In the benchmarks, static routers continuously failed by sending a specific niche coding task to a model ill-equipped for that exact syntax. The static router had no way to know the code was failing to execute. In contrast, ACRouter adjusted its strategy after receiving negative feedback signal from the execution environment.
According to the researchers’ benchmarking, ACRouter sits firmly at the Pareto frontier of cost and performance. On both the ID task streams and the complex OOD agentic tests, ACRouter achieved the lowest cumulative regret, a metric measuring sub-optimal routing decisions over time. On the in-distribution test set, ACRouter cost $13.21 across the full task run, compared to $34.02 for always defaulting to Opus — a 2.6x savings.
It dynamically matched tasks to the most capable model for that specific niche, suggesting that enterprises can achieve or exceed frontier-level accuracy across diverse workloads without paying a premium price for every query.
While the Agent-as-a-Router paradigm solves the information deficit, it is not a blanket solution for all AI workflows.
The framework shines in verifiable tasks where the Verifier gets a clear success or failure signal from the environment, such as coding or data retrieval. It is effective for applications with distribution shifts and domains where different models excel in completely distinct niches.
Conversely, the setup is overkill for trivial tasks where any model will suffice, or for low-volume applications that do not justify the engineering overhead. It is also unsuitable for subjective domains, such as creative writing, where a correct answer cannot be easily verified and feedback signals are impossible to standardize.
The researchers open-sourced the code on GitHub and released the orchestrator model weights on Hugging Face under the Apache 2.0 license. The router is compatible with Claude Code, Codex, and OpenCode.
DeepSeek’s recent decision to drastically cut pricing on its V4-Pro model by 75% should have been unequivocally good news for enterprise AI vendors and developers. Instead, many are discovering that cheaper models don’t automatically translate into healthier margins.
The reason is simple: While inference costs plummet, agent systems are voraciously consuming tokens faster than prices are declining. For the last 2 decades, software economics was dictated by the same rule. Infra became cheaper every year whereas applications became more capable. AI was initially hypothesized to follow the same pattern. As frontier models improved and token prices dropped, many assumed inference would become a negligible operating expense.That assumption has begun crumbling exponentially.
A chatbot usually turns one user question into one model call. An agent turns it into a chain of planning, retrieval, tool use, verification, summarization, and follow-up decisions. The user sees one answer. The vendor pays for the loop. That is the 100x problem: The same user-visible request can cost a lot more to serve as an agentic workflow than as a chatbot or retrieval-augmented generation (RAG) response. In longer-running workflows, the multiplier is higher. Falling model prices help, but they do not fix a product architecture that turns one prompt into dozens of billable operations.
The scale of what is now at stake is clear in how model providers themselves are pricing developer relationships. OpenAI’s proposed program to give every Y Combinator startup $2 million in API credits — a number that would have funded an entire seed round in any prior tech cycle, and when the same cohort got by on a few thousand dollars of AWS credits — is less a recruiting perk than an admission of what it now costs to run an AI-native company through its first year of product. For established enterprises retrofitting agents into existing product lines, the absolute numbers are larger still.
In a single-turn chatbot, one user message produces roughly one model call. Input-to-billed ratio is about 1:5.
In a multi-step agent rolled out across customer support, sales operations, finance, legal review, and engineering, that ratio routinely lands at 1:700 or higher. Every loop iteration carries forward the cumulative conversation, tool outputs, and reasoning traces. Each step appends; nothing is dropped.
A “simple” agent query like “What did our top customer ask about last week?” typically touches seven priced operations before returning an answer:
User prompt (~50 tokens)
System prompt and tool definitions (~3,000 tokens, repeated on every call)
Retrieval (~5,000 tokens of context)
Model call #1 — tool selection (8,000 in / 200 out)
Tool execution (~4,000 tokens returned)
Model call #2 — summarization (12,000 in / 400 out)
Model call #3 — follow-up decision (12,400 in / 100 out)
One sentence in, roughly 35,000 input tokens billed. Somewhere between $0.10 and $0.40 per query on a frontier model. Multiply that by a million queries a month — the table-stakes volume for any enterprise B2B feature — and the line item is six figures.
The dominant pricing story for enterprise AI has been seat-based SaaS: Pay per-user per-month, deliver agent capability, capture margin. That model assumes a reasonably bounded cost-per-user.
Token amplification breaks the assumption. A power user running 50 agent invocations a day on a $40/seat plan can cost more in inference than the plan charges. Token amplification shatters the traditional SaaS pricing model. When a power user’s daily agent activity costs more in inference than their monthly subscription fee, vendor gross margins turn negative, a paradox that compounds as customers deepen their agent adoption, the very usage curve vendors are selling to their boards. Several vendors are now privately reporting negative gross margins on heavy users, mirroring recent cloud expenditure reports from the Bessemer ‘Supernova’ cohort, where the correlation between AI-agent adoption and gross margin contraction has moved from a theoretical risk to a primary P&L headwind.
The visible symptoms have started leaking into public coverage. Bloomberg this week documented a widening gap between Salesforce’s Agentforce marketing demos and the capabilities actually shipping to customers. This is the kind of gap that opens predictably when promised functionality is technically possible but uneconomical to serve at the price the seat plan implies. Salesforce is the most-watched case, not a unique one.
“For my team, the cost of compute is far beyond the costs of the employees.” — Bryan Catanzaro, VP of Applied Deep Learning, Nvidia
The strategic implication is not “AI is expensive.” It is that the dominant business model assumed by most AI-native company plans does not survive contact with agentic workloads.
Consider an enterprise software vendor charging $40 per-user per-month for an AI-enabled support assistant. A traditional chatbot might cost only a few cents per user per day in inference, leaving healthy gross margins.
Now replace that chatbot with a fully agentic workflow capable of investigating tickets, querying internal systems, drafting responses, validating outputs, and escalating exceptions. If a heavy user executes 50 to 100 agent requests per day, inference consumption can increase by an order of magnitude. What was once a negligible infrastructure cost becomes a material operating expense.
This creates an unusual dynamic: The customers receiving the most value from the product are often the customers generating the highest inference costs. In extreme cases, vendors can find themselves with their most engaged users contributing the least profit. The result is a growing realization across enterprise software that agent adoption and margin expansion are no longer automatically aligned.
The technical responses are known and converging. They are not novel, but they are critical for survival
Cost-aware routing: This technique involves a small classifier model that decides which tier (Haiku, Sonnet, Opus equivalents) handles each query. Well-tuned routers cut inference bills by around 60% without any degradation in quality
Prompt caching: Anthropic, OpenAI, and Google now offer 75 to 90% discounts on cached prefixes.
Context discipline: You can truncate tool outputs, prune reasoning traces, and cap tool depth to prevent your agent from going down a rabbit hole
Speculative decoding: for self-hosted deployments, this technique guarantees 2 to 3X effective throughput on the same GPUs.
“Organizations using orchestration-led governance report stronger productivity gains — a holistic orchestration layer is associated with six times greater productivity impact than compliance‑only approaches” — IBM
The companies building this layer well are starting to look less like microservice operators and more like financial trading systems: Every routing decision priced, every path with its own P&L, every tenant on a metered budget.
Four moves separate the companies that will still have margin in 24 months from the ones that won’t:
Make inference cost a first-class metric. Track it per-feature, per-tenant, per-query class the same way cloud cost was tracked starting in the mid-2010s.
Budget like a media buyer. Set cost-per-thousand-queries ceilings per feature. Cap them. Alert on overruns. Engineering will not enforce this on its own.
Treat the router as core infrastructure, not an optimization. It is the new load balancer.
Audit prompts quarterly. A 4,000-token system prompt that grew organically over six months is a six-figure bill in slow motion. Most teams have never read their own production prompts end to end.
Negotiate volume commits early. Frontier-model vendors now offer reserved-instance-style prepaid commits at substantial discounts. List price is the worst price any enterprise will ever pay.
The structural shift underneath agentic AI is not that it is expensive. As DeepSeek’s price cut today underscores, frontier inference unit costs are dropping roughly 3X per year, and the curve is not slowing.
The shift is that amplification is outrunning the price cuts. Cutting per-token costs 75% does not help a company whose agents are doing 700X more tokens per user query than its pricing model assumed. For the first time since the cloud era began, architecture decisions are again financial decisions in real time. A prompt redesign is a margin event. A poorly bound agent loop is an outage with a credit card attached.
The companies that survive the next 24 months of AI infrastructure pricing will not be the ones running the cheapest model. They will be the ones whose agents are smart and know what they cost to think.
That is the 100X problem. And it is arriving faster than the price cuts can hide it.
Maitreyi Chatterjee is a senior software engineer at a big tech company.
Devansh Agarwal works as an ML engineer at a leading tech company.
OpenAI on Thursday launched ChatGPT Work, a new AI agent embedded inside its flagship chatbot that aims to transform ChatGPT from a question-and-answer tool into an autonomous work platform capable of executing complex, multi-step tasks across users’ email, calendars, code repositories, and messaging apps.
The product is powered by OpenAI’s latest flagship model, GPT-5.6, and is designed to go far beyond generating text. ChatGPT Work can gather context from connected apps, files, and workflows to produce finished documents, spreadsheets, presentations, reports, and websites. The agent takes a stated outcome, breaks it into smaller steps, and stays with complex projects for hours, completing them independently.
The launch marks OpenAI’s clearest attempt yet to reposition ChatGPT as a workplace platform rather than a chatbot — and it arrives at a moment of extraordinary financial significance for the company. Last month, OpenAI confidentially submitted a draft S-1 registration statement to the SEC, initiating what could become one of the largest technology IPOs in history, with reported valuations clustering between $730 billion and $852 billion and annualized revenue that has blown past $25 billion.
In a short demonstration and conversation with VentureBeat on Friday, Ty Geri, a product manager at OpenAI who helped build ChatGPT Work, said the product’s mission is to democratize the kind of agentic AI capabilities that OpenAI’s internal engineering tool, Codex, has already demonstrated. “What’s really exciting is we’ve seen how much Codex has been able to push the frontier of what we can get done with these AI tools, as opposed to just getting information or answers or guidance,” Geri said. “Our internal adoption of Codex is literally an exponential curve across every single product function and every single use case.”
The core architectural bet behind ChatGPT Work is a persistent cloud-based virtual machine that runs on OpenAI’s servers, always available to the user regardless of which device they happen to be on. That marks a deliberate departure from competitors whose agents require a local machine to remain powered on and connected.
“What’s really exciting about ChatGPT Work is that it’s a virtual machine in the cloud that’s always on for you, and this is available across all of our paid tiers,” Geri said. “All Plus users are getting this. I think that’s a very unique aspect of this.”
The mobile-first aspect of the launch is something Geri described as “missing from the market.” He pointed to the ability to create a website on a phone and share it with collaborators as a particularly novel capability. “Sites are new in general to Codex. They launched in Codex about a week and a half ago, but now we’re launching also in web and mobile. You can create a site on your phone at the beach and share it with your friends,” he said.
ChatGPT Work will roll out beginning with Pro, Enterprise, and Edu users, and will expand to Plus and Business users over the next few days. In the interview, Geri emphasized that the availability of the product to Plus subscribers — not just premium tiers — is central to OpenAI’s strategy. “It’s accessible to all paid plans, including Plus users, which in my opinion is a really big feat, and really part of that OpenAI mission, which is about bringing all this power to as many people,” he said.
The product relies on MCP-based plugins to connect to external services like Gmail, Google Calendar, Slack, and GitHub. When asked whether the plugin architecture is based on the Model Context Protocol standard, Geri confirmed: “These are all based on MCP.” He added that connecting multiple Gmail accounts — a frequent user request — “is definitely on the roadmap.”
The experience is designed to be action-oriented from the first interaction. ChatGPT Work offers a personalized onboarding flow that surfaces different suggested use cases depending on the user’s role. Geri demonstrated how the system, detecting his role as a product manager, immediately suggested tasks like evaluating AI systems, building research artifacts, and managing his calendar. “You can start with a simple task like catch me up on Slack or Teams or read today’s calendar,” Geri said. He described a scenario where the system reviewed his calendar, identified scheduling conflicts, flagged meetings requiring preparation, and then — on his instruction — declined, accepted, or rescheduled events directly.
Users can also customize the agent by teaching it their writing style, organizing outputs into projects, and — in a lighter touch — choosing a virtual pet that accompanies them in the interface. The interface also introduces a hosted website feature that allows users to build and share interactive sites directly through ChatGPT Work, turning what would typically be a static slide deck into a dynamic, collaborative artifact. “Now we suddenly have a collaborative interface that’s actually more exciting and more accessible than a slide deck, which has all these formatting restrictions,” Geri said.
Geri’s own usage of ChatGPT Work illustrates the breadth of tasks the system can handle. In the run-up to the product’s launch, he needed to organize pre-release testing sessions — known internally as “bug bashes” — across dozens of features and team members.
“I just come to ChatGPT Work and say, ‘Set up a bug bash for all the distinct features in ChatGPT Work. Add all the people that worked on that feature,’ and it can check Slack, it can check GitHub, it can check Docs, and find a time that works for the four highest contributors to that feature,” Geri said. “It went and scheduled 10 bug bashes, all coordinated across all those different people. That would have taken me 30 minutes at least.”
But Geri pushed back against the characterization that ChatGPT Work is limited to rote administrative work. He described using it for analytically complex tasks like identifying the biggest causes of user churn for specific product features and generating product solutions — work he said would previously have taken months. “Things that we would have spent three months doing, we can now spend a week doing — and do much more, and make a much better product,” Geri said. “Bugs that we would have found three or four weeks from now, we can now find within two days and fix for our users.”
He also described handing off the tedium of product testing itself. “It used to be that even though like the most interesting part of my job is like what to test, I would actually end up having to spend most of my job doing the testing, which is like me taking a mouse and like clicking on the same thing over and over again, like five times,” Geri said. “Instead, now I can define what do we want to test, and ChatGPT Work or Codex can actually go test it for me, deliver me that bug report, and then we can work on fixing that bug.”
When pressed on data privacy concerns — given that ChatGPT Work pulls sensitive information from workplace tools like Slack, Google Drive, and email — Geri said privacy “is incredibly important, and the most important part of this is it’s always in the user’s control.”
He pointed to OpenAI’s existing enterprise security infrastructure, noting that “enterprise accounts have ZDR, and users can always opt out of letting their conversations help improve future models, which many users do.” The comment aligns with assurances OpenAI made when it first launched ChatGPT Enterprise in August 2023, when the company wrote in a blog post that it does “not train on your business data or conversations.”
The privacy question carries additional weight now because of the sheer volume of sensitive workplace data ChatGPT Work is designed to access. Unlike a chatbot session where a user voluntarily pastes text into a prompt, ChatGPT Work actively reaches into connected systems — reading Slack messages, scanning calendar invitations, pulling GitHub commit histories — to assemble context for its tasks. That represents a fundamentally different data surface area than anything OpenAI has offered before, and one that enterprise security teams will scrutinize carefully before granting access.
ChatGPT Work lands squarely in the middle of what has become the defining competitive battlefield in enterprise AI: the race to build autonomous workplace agents that can go beyond generating text and actually execute tasks.
The product arrives months after Anthropic took Claude Cowork out of preview and into general availability in April, bringing its AI agent to web and mobile platforms aimed at helping enterprise users monitor and manage long-running AI-driven tasks from anywhere. Meanwhile, Microsoft made Copilot Cowork generally available worldwide on June 16, built in partnership with Anthropic to move beyond chat and into execution. The three products — ChatGPT Work, Claude Cowork, and Microsoft Copilot Cowork — now compete directly for the attention of enterprise IT departments and individual knowledge workers alike.
The convergence is striking. All three products share a remarkably similar vision: a persistent AI agent running in the cloud that can break complex tasks into steps, connect to workplace tools via plugins, and produce finished outputs rather than just conversational replies. All three work across desktop, web, and mobile.
What distinguishes OpenAI’s approach is its raw consumer distribution advantage. ChatGPT has reached 900 million weekly active users, and OpenAI now has 50 million paying subscribers. More than 9 million paying business users rely on ChatGPT for work, and 92% of Fortune 500 companies now use ChatGPT. By making ChatGPT Work available to Plus subscribers at $20 a month — not just Enterprise or Pro customers — OpenAI is betting that broad accessibility will drive adoption faster than any competitor can match.
When asked about the potential impact on the labor market, Geri was careful with his framing. He declined to speak broadly about workforce disruption but offered his personal experience as a product manager whose day-to-day work has been substantially reshaped by the tool.
“My job is not to schedule bug bashes and find out who contributed to a specific feature. That’s a task I do in my job, but that’s not my job,” Geri said. “My job is to make an amazing product.” He described ChatGPT Work as “a partner” and “an extension of me, certainly not a replacement,” adding: “Everybody feels far more productive than before, but is also almost working harder than before, because you get to work on all the things you want to work on as opposed to the drudgery around it.”
But Geri was also careful not to minimize the sophistication of the work the agent can handle. “I also don’t want to say that it’s only doing mundane tasks because, like something like hill climbing retention curves on a given feature is not mundane. It’s actually really hard to do,” he said. The distinction matters. If ChatGPT Work were merely automating calendar invitations and expense reports, it would be a convenience tool. The fact that Geri describes it compressing three months of analytical product work into a single week suggests something with far greater implications for how teams are structured and staffed.
The timing of ChatGPT Work’s launch is impossible to separate from OpenAI’s IPO trajectory. The company needs to demonstrate that it can convert its massive consumer user base into durable enterprise revenue — a narrative that becomes significantly more compelling with a product explicitly designed around professional workflows.
OpenAI said it is generating $2 billion in revenue per month, growing four times faster than Alphabet and Meta did at comparable stages, with enterprise now making up more than 40% of revenue and on track to reach parity with consumer by the end of 2026. But OpenAI remains heavily loss-making, and the company does not expect to reach profitability until around 2030, with internal projections suggesting losses of $14 billion in 2026 alone.
The competitive dynamics are unprecedented. Anthropic filed for its own IPO on June 1 at a $965 billion valuation, setting up simultaneous public listings from the two most prominent AI startups in history. Whether both can sustain their lofty valuations under the scrutiny of public market investors will depend in large part on whether products like ChatGPT Work and Claude Cowork deliver measurable productivity gains to paying enterprise customers.
The launch also caps a product trajectory that began with ChatGPT Enterprise in August 2023, accelerated through the release of OpenAI’s Operator agent in January 2025, and continued through Operator’s deprecation and shutdown on August 31, 2025, when its capabilities were folded into the ChatGPT agent framework. ChatGPT Work is the consolidation of those efforts into a single, unified product — one that pairs GPT-5.6’s three model variants (Sol for power, Luna for speed, and Terra for balanced everyday use) with a persistent cloud environment and an expanding library of MCP plugins.
When asked whether ChatGPT Work signals a shift toward a new kind of operating system — one where users interact with their computers primarily through an AI agent rather than through traditional mouse-and-keyboard interfaces — Geri stopped short of making sweeping predictions. But he hinted at the direction OpenAI sees ahead.
“Anybody who has worked with Codex or now ChatGPT Work will realize how exciting it is to interact with your environment and your computer via the agent,” he said. “Especially in the desktop app, where the model has access to your entire machine and can interact with websites on your behalf — it’s really able to be an extension of you and a real partner, and that certainly feels like the future.”
At the end of the interview, Geri circled back to something personal. “I’ve never enjoyed work as much as I have in the last month using ChatGPT Work and Codex,” he said — a striking admission from a product manager who, until recently, spent a meaningful share of his days clicking through the same interface five times in a row just to see if it would break. OpenAI is now asking 900 million users to believe that feeling scales. For a company weeks away from one of the largest public offerings in history, the answer to that question is worth roughly $850 billion.
Enterprise companies are running AI agents ahead of the controls needed to manage them — and they deployed that way knowingly. That is the central finding from VentureBeat Research’s June survey of 573 technical leaders at companies with 100 or more employees, fielded across five parallel surveys of the agentic stack.
Enterprises are now retrofitting to catch up with their own standards, and they are budgeting for it: Roughly six in 10 enterprises plan to switch or add vendors in each of five control layers within the next 12 months, and roughly a third — depending on the layer — plan to move within the quarter, the research finds.
There are five main layers where enterprises are building: identity for agents (which agent is allowed to do what, under whose credentials); evaluation of agent output (whether the work is any good); cost telemetry (what each agent costs to run); the context layer (the business data and definitions agents draw on to answer); and the orchestration control plane (the software that coordinates multi-step agent work).
Enterprises are already paying the price for deploying agents ahead of adequate control functions. Fifty-four percent of companies had an agent security incident or near-miss caught before harm in the past 12 months. Twenty-seven percent exercise only reactive control of agent spend — they learn what an agent costs when the invoice arrives, with no per-agent budget or ceiling in place.
Here are the five findings that anchor the set — one finding per layer of the tech stack — and what the data suggests doing first in each.
Eighty-six percent of enterprises that run their own GPUs report utilization of 50% or less. Wall Street has spent the quarter debating whether the AI buildout is overbuilt. This is buy-side measurement, from the enterprises doing the buying, and the research says the most expensive hardware in buildings of these enterprises runs at no more than half its capacity.
The measurement gap compounds it: A minority 44% rigorously track what their AI compute actually costs and returns. Everyone else is only estimating. And the enterprise shopping process continues regardless: 45% of these enterprises say the emerging compute option they are most likely to evaluate in the next 12 months is an AI-specialized cloud (CoreWeave, Lambda, Crusoe, Nebius). However, under 2% of these enterprises report using one of these neoclouds today.
Moreover, roughly one in three companies appears to be considering a hedge against Nvidia: Asked which emerging compute option they are most likely to evaluate in the next 12 months, 32% of enterprises named non-Nvidia accelerators (AWS Trainium, Google TPUs, AMD), while 28% named next-generation Nvidia GPUs. The data suggests that enterprises should measure the utilization and per-workload cost of the GPUs they already own before committing budget to new compute — whether that’s an AI-specialized cloud contract, new accelerators, or more GPUs.
Seventy-one percent of enterprises say a quarter or fewer of their deployed “agents” can complete multi-step work on their own; the rest are single-prompt chatbots. Only 10% say true agents are the majority of what they run. To be sure, the respondents reported that they are in a position to know these things: 81% said they recommend or decide AI purchases at their companies.
That finding — that most agents are actually just chatbots in trenchcoats — lands amid adoption claims across the industry running well ahead of what enterprises are actually running. Gartner predicted 40% of enterprise applications will be integrated with task-specific AI agents by the end of 2026, up from less than 5% in 2025. It also warned that the most common misconception is referring to these AI assistants as agents, a misunderstanding known as “agentwashing.”
Meanwhile, Zapier’s enterprise survey said 72% reported deploying or testing autonomous agents; and Writer’s 2026 survey has 97% of executives saying their company deployed AI agents in the past year.
Those surveys asked whether companies have deployed something called an AI agent, and companies said yes. Our survey asked the people running those deployments a harder question: Of the agents you have in production, how many can complete a multi-step task without a person driving each step? The gap matters for two practical reasons. First, the inflated adoption figures are the benchmark boards and vendors use to pressure technical leaders into moving faster — and this data says the real bar is far lower than the headlines suggest. Second, the label determines the bill: A single-prompt chatbot with a human reading every answer needs none of the identity, evaluation, and cost controls this report covers, while a true multi-step agent needs all of them.
Two-thirds of enterprises fall into one of two camps: 34% already allow an AI agent to push a code or system change to production based on automated evaluation results alone, with no human reviewing it, and another 33% are actively engineering their pipelines to allow that within the next 12 months. Only five percent fully trust the automated evaluations that would make that decision.
The distrust is earned. Half of enterprises shipped an agent that passed internal evaluations and then caused a customer-facing failure in the past year; a quarter watched it happen more than once. Asked to name the biggest weakness in their current evaluations, more enterprises chose “poor alignment with real-world outcomes” than any other answer — 29% of respondents.
And most of the checking happens before an agent ships, then stops. Once agents are live with real users, only 23% of enterprises run real-time quality checks on the answers those agents produce. Another 51% monitor system health only — uptime, request traces, and gateway logs — which tells them the agent is running, and nothing about whether its answers are right. The first move: Before removing human review from any workflow, test your evaluations against production outcomes rather than internal benchmarks, and instrument answer quality, not just uptime.
This finding is explored in more depth in VentureBeat’s related coverage of the evaluation gap, which found that larger enterprises are moving faster toward zero-human deployment while also failing more often — and outlines a regression-testing framework built on production outcomes rather than internal benchmarks.
Sixty-nine percent of companies allow agent credential sharing somewhere in their agent fleet during runtime – meaning multiple agents operating under one API key or service account. Those companies were far more likely to get hit: Organizations with credential sharing anywhere in the fleet experienced a security incident or near-miss at a 63.5% rate (47 of 74), against 40.9% (9 of 22) where every agent has its own scoped identity.
The takeaway for enterprises is this: Give every agent its own scoped identity, starting with the agents that touch production systems.
Fifty-seven percent of enterprises traced at least one confident, wrong agent answer in the past six months to missing or inconsistent business context: wrong metrics, stale definitions, absent documents. Most of them watched it happen more than once.
Most enterprise companies are fixing this, even though they’ve moved forward with agent deployment already: 25% already run a governed semantic layer, or one governed definition of the business that every AI reads from, in production. However, 34% are still building one, and 41% haven’t started. The takeaway: Govern the definitions your agents answer from, metrics and entities first, before scaling the agents that depend on them.
One more shift is worth reporting with its limits stated plainly. In our spring orchestration survey wave, the top concern about provider-controlled orchestration was security and permissioning limits (32%). By June, vendor lock-in led at roughly a third, with security limits at 28%.
Those are two snapshots one quarter apart, and here’s one possible explanation for why portability became a top issue for enterprises. Our June survey went into market after a June 12 U.S. Commerce Department export order took Anthropic’s Claude Fable 5 offline for enterprises for roughly three weeks. Meanwhile, Chinese company Z.ai released GLM-5.2’s open weights under an MIT license on June 16 at roughly one-sixth of GPT-5.5’s price; and Tencent’s Hy3 arrived July 6 under Apache 2.0; and OpenAI previewed GPT-5.6 on June 26 to a small group of government-vetted partners, opening it broadly on July 9 after the government’s review cleared. The open-weight releases in particular promise enterprises more control over their agents, and while we haven’t established a causal link here, the timing is worth noting.
The posture data matches the mood: 51% now expect their primary control plane for enterprise agents to be hybrid — provider-native plus external orchestration — by the end of 2026, up from 34% in the spring survey wave. Enterprises reporting that they rely purely on provider-managed agent services fell from 12% to 7%.
The synthesis across all five surveys reveals a huge “buying” window. In each of the five control layers, 57% to 64% of enterprises plan to switch or add vendors within 12 months — 64% in infrastructure and in evaluations, 59% in agent security, 57% in retrieval and context — and 26% to 38%, depending on the layer, plan to move within a quarter. No layer has an established incumbent: The most common evaluation tooling is the model provider’s built-in evals, tied with no dedicated tooling at all (17% each); 82% of respondents name provider-native or hyperscaler controls as their primary agent security layer; and provider-native retrieval leads the context technology layer (RAG, etc) as well.
Most enterprises are defaulting today to the built-in tools that ship with the big AI platforms they already use: Anthropic, OpenAI, Google, Microsoft, and AWS. That holds true across every one of these agentic technology layers: enterprises are looking to their primary cloud and model providers to supply the guardrails, evaluations, and retrieval solutions already bundled into those providers’ offerings.
Those defaults are winning on convenience, and they’re also what the coming spending decisions will test. The survey didn’t ask which direction that money moves — toward the platforms’ built-in tools or toward the specialists challenging them — which is exactly why every contract in these five layers is worth watching over the next four quarters.
The Q3 survey wave will measure whether the enterprises made good on these budget plans: whether their agents gained scoped identities, whether evaluations got tested against production outcomes, whether GPU utilization rose, and whether the semantic layers under construction shipped.
VentureBeat will release the full Q2 reports across all five VB Pulse trackers at VB Transform, July 14–15 at Hotel Nia in Menlo Park, where we convene enterprise technical leaders building autonomous agents in production.
Disclosure: VentureBeat produces both this research and VB Transform
Enterprise AI teams are giving agents more freedom at the same moment their confidence in automated testing is collapsing.
Half of enterprises have deployed an AI agent or LLM feature that passed internal evaluations and yet still caused a customer-facing failure — one in four more than once — according to the June 2026 VB Pulse survey of 157 qualified enterprise respondents at companies with 100 or more employees.
The sample is self-selected rather than a probability sample, so the findings should be read as directional, not precise.
But enterprises are not responding by slowing automation: 66% of respondents already permit some production deployment without human review or are building systems intended to do so within the next 12 months. Only 5% say they fully trust the automated evaluations that would make those release decisions.
That mismatch is the evaluation gap: the autonomy ceiling is rising faster than the assurance beneath it.
It also fits a broader thesis that will be explored at VB Transform 2026: enterprises ship agents first, while the control layers around identity, evaluation, cost, context and orchestration are arriving later. The next year will be a retrofit cycle, with buyers shifting budget toward the systems that make agentic deployments governable and dependable.
Traditional software testing usually asks whether a defined input produces an expected output. Agent testing is harder because the system may choose its own sequence of steps, call tools, retrieve data, alter state and respond differently from one run to the next.
An agent can make several individually plausible decisions and still reach the wrong result. It may retrieve the correct account but update the wrong field. It may draft a valid refund request but send it without approval. It may call five tools successfully before a sixth step leaks sensitive information or leaves a workflow incomplete.
The survey shows enterprises already recognize this limitation. The most common reason for distrusting automated evaluation is poor alignment with real-world outcomes, cited by 29% of respondents. Bias or inconsistency follows at 21%, lack of explainability at 18%, and data leakage or privacy concerns at 17%.
That hierarchy matters. Enterprises are saying the score often does not predict what happens when a customer, employee or business process encounters the agent in production — not that automated scoring is too slow or expensive.
NIST makes a similar point in its Generative AI Profile: measurements gathered in controlled environments may not transfer cleanly to deployment because behavior changes with prompts, users, context and operating conditions. Its guidance calls for field testing, post-deployment monitoring and clear processes for escalating failures.
A single successful run proves that an agent can complete a task. It does not prove that it will complete the task reliably.
Anthropic’s guidance on agent evaluation distinguishes between measuring whether a system succeeds at least once across repeated attempts and whether it succeeds every time. That distinction is essential for customer-facing or operational workflows. A model that occasionally produces an excellent answer may still be unacceptable if the same task fails unpredictably on the next attempt.
Enterprise teams should therefore treat repeatability as a first-class metric. That means running the same scenario multiple times, varying phrasing and context, testing tool failures, and measuring whether the final business outcome remains correct even when the route changes.
The evaluation set also has to evolve. Every production incident should become a permanent regression test. Customer escalations, failed tool calls, incorrect approvals and data-handling mistakes should feed back into the pre-deployment suite rather than remaining isolated support cases.
The survey does not imply that every agent action should require a person. Human review cannot scale across millions of low-consequence decisions.
But zero-human operation should be earned by demonstrated reliability and bounded by the consequences of failure.
Low-risk actions such as drafting internal summaries or categorizing documents can tolerate broader autonomy. Financial transactions, customer communications, code deployment, access-control changes and data deletion need stricter thresholds, repeated consistency tests, policy checks, rollback mechanisms and clear human escalation paths.
The risk isn’t evenly distributed by company size, either. Larger enterprises — those with 2,500 or more employees — are moving toward zero-human deployment fastest, at 70% versus 64% for smaller companies, and they’re also shipping more agents that go on to fail a customer, at 54% versus 48%.
That is the warning for enterprise leaders. Removing the human from the loop does not remove uncertainty. Without stronger assurance, it converts uncertainty into an automated production decision.
The market will keep pushing toward greater autonomy because the economic incentive is real. The organizations best positioned won’t be those that remove people fastest — they’ll be the ones that treat repeatability and regression testing as seriously as deployment speed.
A team routing queries across a coding specialist, a logic specialist, and a generalist model assumes each will cover the others’ blind spots. A new study evaluating 67 frontier models from 21 providers shows that assumption is mathematically flawed — and the flaw has a name: the co-failure ceiling.
The assumption works like this: as long as two models don’t usually fail on the exact same prompts, combining them is supposed to create a safety net against failures.
The real limit on orchestration is not how often models disagree, but the percentage of prompts where every model in the pool gives the wrong answer at once. By ignoring the co-failure ceiling, enterprises are building complex, expensive routing infrastructure to chase performance gains that do not exist. Fortunately, developers can use this same math to build a cost-free test that determines exactly when multi-model orchestration will actually pay off.
To orchestrate multiple language models, developers typically rely on three architectures. Model routers act as traffic cops, sending complex queries to expensive models and simple queries to cheaper ones. Cascades send every prompt to a cheap model first, only escalating to a premium model if the initial system signals low confidence. Finally, approaches like Mixture-of-Agents (MoA) fuse multiple models by asking them the same question and generating a synthesized answer from their combined outputs.
These architectures introduce a “shadow price” to inference costs. Every time a development team implements a router or a cascade, they pay a premium in added system latency, complex infrastructure maintenance, and increased governance risks across multiple API providers.
To justify these operational costs, engineers rely on “pairwise error correlation” to select their model pool. Imagine a developer has Model A, which writes excellent Python but fails at SQL, and Model B, which writes excellent SQL but fails at Python. Because they fail on different types of prompts, their pairwise error correlation is low. The developer assumes that by placing a routing layer in front of them, they have created a composite system that rarely fails at coding.
According to the study, throwing diverse models together based on low correlation can actually hurt performance if the models are not equally capable — when you vote across diverse but unequal models, the weaker ones often gang up and outvote the smartest one.
Josef Chen, author of the paper, told VentureBeat that in their experiments, “Naive majority voting across unequal models had negative mean gain (minus 10 points on our hard mix): diverse-but-weaker members outvote the strong one.” The actionable advice for developers is to “combine only models within a matched quality band.” If you cannot match quality, take the single-model baseline and spend your budget on the best model available.
The paper provides one bright spot for this approach regarding MoA architectures. When building ensembles, teams often use “Self-MoA,” where they query the same premium model multiple times to generate a synthesized answer. The researchers found that at matched quality, building a diverse ensemble of models with low pairwise correlation beats a high-correlation Self-MoA setup.
However, when teams use that same pairwise correlation metric to predict the absolute accuracy of their overall system, the math breaks down.
“So teams pay the orchestration overhead up front (latency, complexity, multi-provider operations) on the assumption that a diversity dividend arrives later,” Chen said. “Usually it doesn’t, because today’s best models agree, and, worse, they fail on the same queries … the prompt simply carries little signal about which model will be the one that’s right when the frontier disagrees.”
The core finding of the study centers on a metric called the “co-failure rate” — the formal name for the all-wrong scenario described above. No router, voting system, or cascade can ever achieve an accuracy higher than the ceiling it imposes.
The coding, logic, and generalist pool shows low pairwise correlation on routine prompts — they rarely fail together. But the co-failure ceiling represents the obscure, highly complex edge case that pushes past the limits of current AI architectures. If a prompt is so difficult that all three models hallucinate or fail, it does not matter how intelligently the router distributes the task. The entire pool wipes out at once.
The researchers tested their 67-model pool, which included GPT-5.5, Claude Opus 4.8, and Gemini 3.1 Pro, on the open-ended MATH-500 math benchmark. Based on standard pairwise correlation, statistical models predicted that the entire pool would wipe out simultaneously on only 2.3% of the questions. In reality, the co-failure rate was 5.2%.
Standard correlation metrics underestimated the failure rate by roughly 2.25 times. The culprit is not just independent difficulty, but a shared failure point.
“The driver is what we call a common-mode atom: a slice of queries on which the entire market fails together, which no pairwise statistic can see,” Chen said. “Adding a 20th model to your pool doesn’t buy tail coverage. The tail is shared.”
The researchers also found that task format directly triggers co-failure. When they took graduate-level science questions from the GPQA benchmark and changed them from multiple-choice to free-response formats, the all-wrong tail expanded to 12.7%.
Developers can engineer around the ceiling, though. “The engineering implication is uncomfortable: multi-model setups buy the least exactly where teams want them most, on open-ended generation,” Chen said. “Anywhere you can convert generation into verification or constrained selection (structured outputs, checkable answers, execution tests), you reopen the ceiling.”
Ultimately, the researchers found this ceiling limits AI applications in two distinct ways, depending on the domain:
Ceiling-bound environments (e.g., open-ended math): The co-failure rate is high. The task is too hard, and all models fail simultaneously. No amount of routing can bypass the lack of underlying capability.
Realizability-bound environments (e.g., graduate-level science): The co-failure rate is near zero, meaning at least one model in the pool usually knows the answer. However, the models disagree so subtly that a routing layer cannot reliably pick the correct answer without an omniscient oracle.
Before dedicating engineering hours to building a router, teams can calculate their absolute performance ceiling for free using a mathematical formula called a Clopper-Pearson bound.
The Clopper-Pearson bound operates as a worst-case scenario calculator. If you flip a coin ten times and get eight heads, you cannot guarantee the coin will land on heads 80% of the time forever. The bound takes a small sample of test questions and outputs a mathematically guaranteed ceiling.
Applied to language models, suppose a team tests a pool of five agents on 50 sample queries and finds they all fail together on just two questions. A developer might assume their multi-agent system will achieve 96% accuracy in production. The Clopper-Pearson formula corrects this optimism. It analyzes the small sample size and provides a mathematical guarantee that the true co-failure rate could actually be as high as 12%.
To use this in practice, enterprises must build a held-out dataset. A fintech company, for example, could take 200 complex customer support tickets from the previous quarter and have human agents write perfect resolutions to serve as a benchmark. While this sounds like a heavy manual project, mature engineering teams can automate the entire ceiling calculation.
“Integration is trivial: it’s a counting job over eval logs teams already produce,” Chen notes, “so it runs in the same CI stage as the eval suite and re-triggers whenever the model pool or the workload changes.”
The engineering team then runs its candidate models against these 200 tickets once and records the results. When they want to evaluate multi-model configurations, they can use the co-failure rate measure to predict the maximum accuracy they can get from the system without running extra queries.
One important conclusion the study draws is that on tasks where answers can be definitively checked, combining models rarely beats using the single best model on the market, unless the team possesses an exceptionally strong query-level routing signal.
In an enterprise environment, a definitively checked task has an objective, zero-tolerance answer. This includes generating a SQL query that must execute without error, extracting a specific invoice total from a 50-page PDF, or formatting a JSON payload that perfectly matches a strict schema. For these tasks, enterprises are usually better off paying a premium for the smartest frontier model rather than weaving together three cheaper models and hoping a router picks the correct output. The study didn’t test subjective, ungraded tasks like drafting marketing copy — the authors note that whether these findings hold outside their verifiable benchmarks remains an open question.
Because this mathematical check is free, enterprise teams can track their own co-failure rates as new models drop.
“The measurement costs nothing, so any team can track its own co-failure rate across model generations and watch whether the tail is closing,” says Chen. Ultimately, “the lever buyers hold is failure-mode heterogeneity and market churn, not model count.”
Presented by SAP
Generating code with AI is fast, but getting that code to run reliably inside a large enterprise, integrated with live systems, governed for compliance, and maintainable over years requires foundational work that most organizations underestimate.
While 81% of all organizations have a detailed strategy, only 12–16% reach AI‑driven execution, says SAP’s Michael Ameling, CPO of SAP Business Technology Platform, and the reasons rarely come down to the quality of the generated code.
“Across industries, enterprises that have invested heavily in AI tooling are hitting a wall when generated code meets the reality of their existing environments, because generating code and operationalizing it are not the same problem,” Ameling says.
There are specific requirements for deploying AI-generated logic at enterprise scale: what data and integration readiness actually look like, how governance works when AI agents move from producing recommendations to executing workflows, and how development teams are changing their role as AI takes over more of the coding work.
The productivity gains from AI code generation are real and well-documented, but the ease of prototyping has given many organizations a misleading sense of how far along they actually are.
“Generating code is one thing,” Ameling says. “Enterprise customers, including multinationals and large organizations, need to ensure there are no compromises in compliance or security. Code that runs reliably for ten or twenty years, as it does at many of SAP’s largest customers, also has to be maintained, patched, and understood by whoever inherits it. Life cycle management, in other words, does not generate itself.”
The issue is rarely the generation quality. Teams build something compelling, then discover they lack access to the data it depends on, or the integrations it assumes, or the permissions required to run it in a real environment. The problem is essentially that AI amplifies an organization’s existing data and process maturity, but it can’t substitute for it.
This dynamic intensifies as AI moves from producing code to executing actions. Latency, cost, and system load all increase when logic runs continuously against live data rather than rendering a one-time output. The performance requirements of an autonomous agent operating across a multinational’s transaction systems are categorically different from those of a developer copilot.
The architecture challenge that most enterprise AI projects underestimate is integration. Real enterprise environments are not clean slates: they combine cloud systems, legacy on-premise infrastructure, fragmented data stores, and dozens of business applications that were never designed to talk to each other. Getting AI-generated logic to operate reliably across all of them requires a layer that unifies data access, process context, and governance, and it has to be in place before any agent starts executing. And organizations that see AI as a reason to defer infrastructure modernization are making a mistake.
“The question is not whether to modernize or not. Of course you need to modernize,” Ameling says. “But the value you get on top of this is much higher with AI. Federated data access and harmonized process layers are not alternatives to upgrading a fragmented landscape, they’re what make the upgrade worthwhile.”
At the platform level, this translates into a set of practical requirements: structured data integration, end-to-end process visibility, and the ability to discover and connect to APIs across both modern and legacy systems. SAP’s approach with the Business AI Platform draws on tools including its Joule Studio, Integration Suite, Business Data Cloud, and SAP AI Agent Hub enterprise architecture layer to provide that context. The goal is to give AI-generated logic accurate, current knowledge of what a business is doing and how, rather than just access to raw data.
AI agents handle large challenges by dividing them into smaller, autonomous tasks, with each agent responsible for a specific domain, and all coordinated toward a shared outcome. A financial close, for example, involves dozens of discrete sub-processes. Agents handling each task in parallel, within defined constraints, can compress cycle times dramatically, but only if the underlying systems they interact with are coherent and accessible.
When AI moves from assistant to operational actor, the governance questions loom large, because agents that trigger workflows, update records, and interact with live business systems need the same accountability framework that applies to human employees, i.e., identities, defined privileges, and auditable behavior.
There are two distinct models:
Principal propagation, where an agent acts on a user’s behalf, inheriting that user’s permissions and scope.
System-triggered agents, where the agent operates under its own identity and role-defined privileges, functioning more like an automated HR role than a personal assistant.
Both models require the same underlying infrastructure: an agent hub where operators can see which agents exist, what APIs they can access, and what they are authorized to do. Observability also needs to be operationalized correctly for AI, combined with both technical and business evals.
“In production, openness is very important,” Ameling says. “We use OpenTelemetry as a framework, so we can integrate with other solutions, for end-to-end observability of the tool, third-party agents and the like.”
On top of that, standard technical evals, which test whether an agent produces consistent outputs, are necessary but not enough. Business evals assess whether an agent is actually moving the performance indicators it was deployed to improve, but it has to work end-to-end.
Where the testing happens is equally important. The traditional software development cycle across dev, test, and production environments breaks down when a model produces different outputs depending on whether it is running against test data or live data. Getting to trustworthy AI in production means accepting that validation looks fundamentally different from what engineering teams have practiced for decades, with live environment testing, even A/B/C testing to ensure outcomes are reliable.
The role of the developer is not disappearing in this environment, but its center of gravity is shifting. The productivity multiplier is significant when developers can run multiple coding agents in parallel across open terminals, each working on a separate problem and each taking several minutes to complete. But it introduces a new kind of cognitive demand, because humans have to stay in the loop. That means tracking context across concurrent workstreams, evaluating outputs that range across large codebases, and making architectural judgments that no agent can be trusted to make alone.
“The more specific and complete the prompt, the less intervention is required, and developers are learning that bringing more context upfront pays dividends in reduced back-and-forth,” Ameling says. “But the output still needs to be understood, not just accepted.”
The competitive edge will remain intellectual property, not tooling. The companies that pull ahead will be those that most effectively encode their domain knowledge into the systems they build.
“A manufacturer’s process expertise, a financial institution’s risk logic, a logistics firm’s routing intelligence, these are the assets that AI can accelerate, but only if the organizations that hold them do the work to make them accessible and usable,” Ameling says. “Protect that, and apply AI to accelerate your differentiation.”
Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.
Presented by Oracle NetSuite
Every major technology transition produces a set of assumptions about where the market is headed. The assumptions are often directionally correct, but they tend to underestimate the degree to which organizations adapt new technologies to their own circumstances. AI is following a similar trajectory.
Many current discussions about enterprise AI assume a future in which employees interact with business systems through a common interface. The details vary depending on the prediction, but the destination often looks similar: a conversational system that becomes the primary way people access information, complete tasks, and interact with software.
The history of enterprise technology suggests a more complicated outcome. Organizations rarely adopt new capabilities uniformly because different parts of the business operate under different constraints. A finance team responsible for reporting accuracy, controls, and approvals approaches technology differently than an analytics group exploring operational data. Both groups have different requirements than a customer service organization focused on response times and case resolution. Even when there is broad agreement that a technology is valuable, the path to adoption tends to vary across functions.
The shift to cloud software followed this pattern — some organizations moved aggressively while others spent years operating hybrid environments. Different departments often modernized on different timelines, reflecting the priorities of the work itself rather than any industry consensus about the correct pace of adoption.
AI has accelerated many aspects of technology development, but it has not changed this underlying dynamic. Organizations still evaluate new capabilities through the lens of existing processes, responsibilities, and operational requirements.
For some employees, the most useful AI capabilities may be the least visible ones. A finance manager closing the books is often less interested in a new interface than in shortening a reporting cycle. An operations leader dealing with inventory issues is usually focused on identifying problems earlier and resolving them more quickly. In these situations, the value of AI comes from reducing the amount of effort required to complete existing work.
At the same time, another group of users increasingly wants direct interaction with AI systems. Analysts, planners, and operational teams often benefit from the ability to explore information conversationally, compare scenarios, and investigate questions that do not fit neatly into predefined reports. For these users, the interface itself becomes valuable because it provides a more flexible way to work with business information.
A customer service representative handling a high volume of inquiries has different requirements than a financial analyst investigating a trend in operating expenses. One benefits from information appearing automatically within an existing process while the other may benefit from the freedom to ask follow-up questions, explore alternative explanations, and move through data more dynamically.
Many organizations are discovering that both patterns exist simultaneously, which reflects a broader reality about how businesses evolve. Operational complexity accumulates gradually, systems multiply, and processes become fragmented. Information becomes distributed across applications, reports, spreadsheets, and workflows and employees spend increasing amounts of time locating information before they can begin acting on it.
Much of the value created by enterprise software over the last several decades came from reducing that fragmentation. Bringing financials, operations, inventory, customer information, planning, and reporting into a common system created a more complete picture of how the business was operating.
AI is beginning to address a related problem. Once information exists within connected systems, employees still need to find it, interpret it, and apply it. Reporting cycles consume time. Routine questions require investigation. Managers often spend considerable effort assembling information before they can make decisions. As organizations grow, these activities become increasingly expensive because they consume attention from people whose expertise is often in short supply.
AI’s promise is to reduce the effort required to move from information to action.
At Dura Software, AI-connected workflows are helping automate portions of revenue reporting that previously required manual preparation during each reporting cycle. Sloan Session, CFO at Dura Software, described the arrangement in practical terms: “The agents handle the pull. The humans handle the judgment and the personal touch.”
That observation captures an important aspect of current AI adoption. Most organizations are not attempting to remove judgment from business processes. They are trying to reduce the amount of time spent gathering, organizing, and preparing information so that experienced employees can focus on the decisions that require expertise.
A similar pattern emerged at S&B Filters. Employees previously spent several minutes during customer interactions collecting backorder information from multiple systems. By connecting AI to operational data, the company reduced that process to seconds and eventually extended the capability directly to customers through self-service.
In both cases, the benefit comes from reducing the friction associated with finding and using information rather than introducing a new interface. The moment information becomes easier to access, questions about access itself become more important. Permissions, approval structures, and security policies exist because businesses need mechanisms for controlling access to information and managing risk. Those requirements do not disappear when employees begin interacting with data through AI systems. If anything, they become more important because AI can make information easier to access.
Berry Carter, CEO of S&B Filters, described the principle clearly. If a user cannot access specific information within NetSuite, that user should not gain access to the same information through an AI assistant. The statement sounds obvious. Implementing it consistently across systems, workflows, and models requires considerably more discipline than the statement itself suggests.
Lauren Polasek, former NetSuite administrator and board member of the Texas NetSuite User Group, recently made a related point. Connecting technology is often the easier part. Organizations still need to determine which tools should be used, who should have access to them, and how governance should evolve as adoption expands.
This is one reason predictions about a single AI interface are difficult to reconcile with how enterprises actually operate. The requirements of a finance organization closing the books are different from those of a customer service team handling thousands of interactions each day. Some AI capabilities will be embedded directly into business processes where employees may barely notice them. Others will provide more direct access to operational information through conversational systems. Many businesses will end up using both approaches because the underlying work is different.
That perspective has shaped how we think about AI at NetSuite. Some customers want AI embedded directly within operational workflows. Others want the ability to connect NetSuite data to external models and assistants so they can interact with business information through tools that are already part of their daily work. Increasingly, organizations are asking for both.
The NetSuite AI Connector Service and our support for Model Context Protocol (MCP) were designed with that reality in mind. The goal is to allow organizations to connect business information securely to the workflows and systems that make sense for them while continuing to benefit from AI capabilities built directly into NetSuite.
The history of enterprise software suggests that adoption rarely follows a straight line. As organizations adopt AI, business leaders should identify the business objective and the workflows involved so they can match the solution to the reality of the work.
Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.