How A/B tests compound agent intelligence

For teams shipping voice agents, the pattern is all too familiar: swap a model, redeploy, listen to a few calls, and call it an upgrade. Vibe-checking is a tempting shortcut, especially when changes feel minor. It’s also how hidden regressions make it into production.

The first wave of the agentic era was about getting agents to work. We're now in the second wave, the one where agents are running real economic processes, and the question isn't can they do the job but which version of them does it best. That question has a well-understood answer across the rest of the software industry: controlled experiments.

This post is about how we run experiments at HappyRobot, and some of the challenges that show up specifically when the thing you're experimenting on is an agent. We'll walk through a real experiment end-to-end: a Text-to-Speech (TTS) voice swap that produced a +20% relative lift on a primary business metric for a large telecom provider, and then get into the statistical challenges that show up as agents get harder to improve.

Why this is harder than it looks

A voice agent isn't a single model you're A/B testing. It's a stack: a transcriber turning audio into text, an LLM deciding what to say and which tools to call, a TTS engine turning the response back into audio, a set of prompts and policies wrapping all of it, and an orchestrator gluing the layers together with telephony. Changing any layer can shift outcomes through paths that aren't obvious from the change itself.

The outcomes that matter are also downstream of long, branching conversations. A typical metric, “did the customer agree to pay for the upgrade” is binary at the call level but conditional on dozens of upstream decisions: did authentication succeed, did the caller stay on the line, did the agent surface the right offer at the right time. Each of those introduces variance that has nothing to do with the change you're testing. The signal is real but it's buried in noise, and the more your metric aggregates over upstream sub-events, the more sample size you'll need to read it cleanly.

And every experiment burns real production traffic. A badly executed experiment isn't just statistically wrong, it costs you in the same currency you measure success in. Put together, these properties mean voice agent experimentation has to be careful in ways that aren't always obvious from the outside, and that careful work has to happen at the speed of the business.

How experiments work in the HappyRobot platform

The HappyRobot platform's experimentation feature lets you set up an A/B test on any workflow you've already built: define a control and a treatment variant, pick a traffic split, and ship. Randomization happens at the call level. When a call lands, the orchestrator assigns it to a variant, and that assignment is sticky for the duration of the call. The workflow runs the assigned variant end-to-end, and when the call ends, the outcome is written into the customer's Context, their data layer: whether the customer agreed to pay, whether a load was booked, or an appointment scheduled successfully. That outcome is the unit the experiment measures on.

The same machinery works regardless of channel. Voice calls, email threads, WhatsApp / SMS or chatbot conversations. Anything the agent can run, we can split traffic on. For this post we're focused on voice, but the experimentation primitives don't change.

You designate one primary metric in advance, and that's the metric you make the decision on. Everything else is diagnostic. We'll explain why that distinction matters later.

A real experiment: lifting plan upgrades by 20%

We A/B tested the Text-to-Speech (TTS) voice used by an agent operated for a large telecom provider. The agent calls existing customers who have been overspending on their current plan regularly incurring overage charges, for example on data, to walk them through a better-fitting plan and get them to accept the upgrade, by phone, by web link, or by any other channel the workflow supports.

• Control: Voice A (the production proprietary voice at the start of the experiment)
• Treatment: Voice B and Voice C (two proprietary voices we wanted to evaluate)
• Duration: Three weeks
• Primary metric: upgrade_accepted

Voice B

Choosing the right primary metric

The temptation when evaluating a TTS swap is to measure things that feel like they're about the voice: interruptions, latency, naturalness ratings, words-per-minute. These are all real metrics and we track them. But none of them is the metric you should make a decision on.

The right primary metric is as close as possible to the actual economic outcome you care about. For this customer, that outcome is duty paid. We can't directly observe payment at the moment the call ends - payments settle later, sometimes days later - so we use the closest in-call proxy: did the customer agree to pay during the call. We have sub-metrics that break this down (upgrade_accepted_by_phone, upgrade_accepted_by_web) but the decision is made on the aggregate. Channel mix is interesting; the headline outcome is what matters.

Secondary metrics (we tracked customer_engaged_with_agent and several others) don't drive the decision. They drive the next experiment. If the primary metric moves, secondaries help us understand why. If it doesn't, secondaries help us design what to try next.

Results

Two findings, not one. The obvious finding is that Voice B is meaningfully better than the production voice. A +2.67 pp lift on a baseline of 13.01% is a 20% relative improvement on the metric the business actually cares about, sustained over three weeks of real production traffic. At this customer's volume, that's a material number.

The less obvious finding is that Voice C would have cost the business roughly 7% of plan upgrades if we'd shipped it on intuition. The downside of vibe-checking isn't that you miss winners. It's that you ship losers and never find out.

Why Voice B won

The secondary metric we tracked alongside upgrade_accepted was customer_engaged_with_agent , whether the caller actually engaged with the agent rather than disengaging early. It tells a consistent story:

Engagement moves in the same direction as the primary metric, and Voice B picks up most of its lift early in the call, before any of the plan-specific content has been read out. That's a strong hint that the initial impression of the voice matters: how human it sounds in the first few seconds determines whether the caller stays on the line long enough to hear the rest.

Looking at a sample of call transcripts and audio alongside the secondary metrics, two factors stand out as the likely drivers:

• Naturalness without theatricality. Voice B sits in a sweet spot: conversational and human-sounding without the over-expressive prosody some modern TTS voices reach for. The engagement lift, concentrated in the early seconds of the call, lines up with this: if a voice sounds clearly like a bot or clearly like a theater performance, callers disengage before the agent gets to the substance.
• Entity pronunciation. Voice B turned out to be materially better at pronouncing complex strings: account numbers, dollar amounts, email addresses, reference codes. For a use case where the agent is reading back exact overage charges and the price of the new plan, this isn't cosmetic. Misread digits cause callers to ask for repetition, which costs time and increases the chance of an early hangup.

The second factor is the one you'd miss if you weren't measuring the right thing. It doesn't show up in a naturalness rating. It shows up in upgrade_accepted.

The statistical challenges of optimizing agents

Everything above is what a clean, high-signal experiment looks like. The interesting question is what happens when the easy wins are gone, when your agents are already optimized and the lifts you're chasing are 1-2% instead of 20%. That regime is where most of the technical work goes.

Estimating experiment length

Before starting an experiment, it's worth estimating how long it will need to run. The inputs are familiar: the baseline rate of the metric you're trying to move (p), the minimum effect size you'd care about (δ, the MDE), the desired statistical power, and the significance level. For a binary outcome compared across two equally-sized variants, a useful back-of-the-envelope is:

Plug in numbers and you get the required sample size per variant; divide by your daily call volume and you get the experiment duration.

Smaller MDEs need a lot more data. Halving the detectable effect quadruples the required sample size (note the δ² in the denominator). Detecting a 1 pp lift on the same 22% baseline would have required roughly 10x the sample size of detecting a 3 pp lift. This isn't a quirk of any particular setup — it's the geometry of statistical power. But it has a consequence: as agents improve, the cost of each subsequent experiment goes up, fast.

The MDE problem, and why variance reduction matters

Here's the loop we want every customer to be in: ship an agent, run experiments to improve it, ship the winners, run more experiments, repeat. The faster this loop turns, the steeper the curve.

The thing that slows the loop down is statistical power. A team running at high volume on big lifts can iterate weekly. A team running on small lifts — say, a 1% improvement on a metric that's already optimized — might need months per experiment at the same traffic level. That's not iteration. That's waiting.

You have three levers to pull. You can increase traffic (limited by how many real customers exist). You can run longer (limited by how many experiments fit in a quarter). Or you can reduce the variance of your estimator, so the same data carries more information about the true treatment effect.

The third lever is where the interesting techniques live. The intuition is straightforward: a chunk of the variance in your outcome metric isn't caused by your treatment, it's caused by stuff that was already true about the call before the treatment was assigned. The customer's prior behavior, the time of day, the language of the call, the LLM model. If you can explain away that pre-existing variance, what's left is closer to a clean read on the treatment effect, and your confidence intervals get tighter for free.

A few of the techniques we reach for, in increasing order of complexity:

Multivariate regression. The simplest move. Instead of comparing raw means between control and treatment, you regress the outcome on the treatment indicator and a set of covariates that you know predict the outcome — language, hour of day, customer segment, prior conversion rate. The coefficient on the treatment indicator is your effect estimate, and as long as the covariates aren't themselves affected by the treatment, the estimate is unbiased and lower-variance than the simple difference of means.

CUPED. Controlled-experiment Using Pre-Experiment Data. The covariate is the same unit's outcome measured in a pre-experiment period. For example, the historical booking rate of the specific freight carrier you're calling. The intuition is simple: part of the reason a call converts or doesn't is just that you're talking to this carrier, who already books at a certain rate regardless of which voice the agent uses. CUPED subtracts out that pre-existing variation, leaving a cleaner read on the treatment effect. It's asymptotically equivalent to regression with that one covariate, and it's the workhorse of variance reduction at most large experimentation programs. Easy to implement, hard to mess up, often buys you 20-50% variance reduction for free when historical behavior is predictive.

CUPAC. Control Using Predictors as Covariates. Same idea as CUPED, but the covariate is the prediction of an ML model trained on pre-experimental data rather than a single historical average. CUPED can be viewed as the special case where that ML model is a simple per-unit average; CUPAC generalizes it to any predictor you can fit on pre-experiment data. When the relationship between covariates and outcome is non-linear — which it often is for voice agents, where outcomes depend on interactions between caller traits, time, and prior history — a flexible predictor (gradient-boosted trees, say) can buy you another meaningful chunk of variance reduction on top of CUPED.

The common pitfall across all of these: the covariate must not be affected by the treatment. If it is, you'll inject bias into the estimate. Pre-experiment data is the safest source. Anything observed during the experiment is suspect unless you can argue it can't have been influenced by the assignment. For instance: call duration is tempting as a covariate — it's predictive of outcomes and easy to measure — but a better TTS voice that keeps callers engaged for longer will mechanically shift call duration in the treatment arm. Adjusting for it would partly cancel out the effect you're trying to measure.

For the canonical write-up of these methods, including a clean simulation comparing them in linear, non-linear, and adversarial scenarios, we recommend Bouzas and Masip's Glovo Engineering post.

Noisy metrics need more data than you think

Some metrics are well-behaved. Per-utterance latency or words-per-minute, for instance, are tightly distributed - you're sampling something close to a stable physical quantity, and the variance stays bounded.

The metrics you actually care about are usually not those. A primary metric like upgrade_accepted is binary per call, but the journey to that binary outcome aggregates many sub-events: did the caller pick up, did authentication succeed, did the agent surface the right offer, did the caller stay engaged long enough to hear it. Each of those is its own bernoulli with its own variance, and the further downstream your metric sits in the funnel, the more of that upstream variance it inherits. Aggregated business outcome metrics are noisier than the moment-to-moment metrics that feel "more scientific" - but they're also the only ones tied directly to the economics.

The right move isn't to abandon them. It's to plan for the data they need: longer experiments, more careful covariate adjustment, and discipline about not peeking at the cumulative numbers when the daily ones look encouraging.

Why this compounds

A human agent making collection calls gets better over time, but linearly. They learn the script, internalize the objections, plateau. An agentic system optimized through controlled experiments doesn't have that limit. Every experiment that ships permanently raises the floor for every subsequent call. The next experiment runs against Voice B as the new control, and improvements compound.

What determines how fast this curve climbs isn't the size of the model or the cleverness of any single intervention. It's the rate at which you can run reliable experiments. Compound intelligence is just the name for what happens when that loop runs cleanly, week after week: production traffic stops being a cost of doing business and starts being the input to a system that gets monotonically better at the thing the business is paying for.

References:

• Variance reduction in experiments using covariate adjustment techniques — Bouzas and Masip, Glovo Engineering.
• Autoresearch - Andrej Karpathy