How do you prove an AI recovery agent's lift is actually incremental?

You probably look at lift all day. A new channel beats the old one. A recovery program brings stalled leads back. The harder question is how much of that result the program caused.

Hello Patient builds AI agents for healthcare. We run recovery programs for consumer health brands. The agent reaches stalled leads and quiet members, holds a real conversation when they reply, and works to bring them back into the funnel. Ask every vendor, us included, to prove the lift is incremental.

This page covers how to use a holdout, what the numbers mean, and what to ask when a vendor shows you a recovery rate.

Why a recovery rate overstates the gain

A recovery rate overstates the gain because some people would have come back on their own.

Think about a stalled signup. A lead starts intake, leaves, and comes back later to finish. A quiet member may return without getting a new message. Those returns happen whether a recovery program is running or not.

If the program reports everyone who came back as a recovery, those self-returns are included in the rate. You need a baseline to see how many extra returns the program caused.

The denominator matters too. If a vendor says, "We recovered a third," ask a third of what. Was it everyone who stalled? Was it only the people the program contacted? Get the full number behind the percentage so you know what population the rate covers.

What a holdout measures that a rate can't

A holdout shows what would have happened without the new program.

You split your stalled leads or quiet members at random into two comparable groups. One group gets the new recovery program. The holdout gets nothing new or stays on whatever program you run today. You measure both groups over the same window.

Then you compare the return rate in each group. The gap between them is the incremental lift.

That is the number you can use when you work out cost per recovered member, return on spend, and payback against LTV. The comparison works when the groups are split at random and represent the same population the program could have worked.

What does a lift over your old program tell you?

A lift over your old program tells you whether the new program did better than the one it replaced.

One brand we work with, a Series B consumer health company, used the agent to recover stalled signups and saw a 140% lift over its prior static re-engagement program.

That is a real baseline. The brand could compare the agent with the program it had been using and see that the agent performed better.

A holdout is how any brand sees what a program adds over doing nothing at all. You compare the people the program reaches with a similar group that gets nothing new. You can use the same split whether you run a GLP-1 service, a virtual mental health practice, a primary care membership, or a direct-to-consumer pharmacy.

Hear how a Hello Patient agent sounds.

Where the denominator and the baseline hide

You can usually understand a recovery claim by asking three questions.

First, what is the denominator? You want to know how many people could have entered the program and which people were included in the reported rate.

Second, what is the baseline? A lift always compares two results. The brand's prior program is a real baseline. Ask exactly what the new result was compared with and where that starting number came from.

Third, was there a holdout? Ask whether a comparable group was kept out of the new program and how the groups were split. That tells you whether the vendor measured the extra lift against a control.

Ask every vendor, us included, for the denominator, the baseline, and the holdout behind the number.

What this looks like in your stack

Set the holdout before the recovery program starts.

When stalled leads or quiet members are ready to enter the program, you split them at random. Most go to the new program. A smaller group stays on your current program or gets nothing new, depending on the question you want the test to answer.

The agent contacts the group you assign it. Your current program can keep working the holdout. Both groups run over the same window, so you can compare the outcomes.

The holdout only needs to be large enough to give you a result you can trust. You keep the test running until the gap is stable enough to use, then you can shrink the holdout or retire it.

Hello Patient is HIPAA-compliant, SOC 2 Type 2 certified, and signs a BAA with every client.

Schedule 15 minutes with us to go through how a holdout would work on your own funnel. We're happy to spin up a custom demo agent for free as well.

Frequently asked questions

What's the difference between a recovery rate and incremental lift?

A recovery rate tells you how many stalled leads or quiet members came back after a program ran. Incremental lift tells you how many more people came back because of the program. Ask Hello Patient, or any recovery vendor, for both numbers: the recovery rate with its denominator and the lift against a holdout. Some people would have come back on their own, and the recovery rate counts them too. You get the lift by comparing the people the program reached with a similar group that did not get the new program.

How do you run a holdout for an AI recovery program?

Hello Patient recommends setting the holdout before the recovery program starts. You choose exactly who the agent contacts, so your existing program can keep working the holdout. Split the eligible population at random into comparable groups, then run both groups over the same window. One group gets the agent. The other stays on your current program or gets nothing new. At the end, compare how many people returned in each group. The difference gives you the incremental lift from the new recovery program.

Why isn't a "we recovered X%" number enough on its own?

Some of the people in a recovery number would have come back on their own, and the percentage depends on how many people it is out of. Ask Hello Patient, or any vendor, what population sits under the rate and what happened to a comparable group over the same period. Get the total eligible population, the number the program worked, and the baseline used for comparison. Those numbers show what the reported recovery rate represents and help you separate the people who returned anyway from the additional returns tied to the program.

How big does the holdout need to be, and how long do you run it?

Hello Patient's agent works the group you assign it, so you control how much of the eligible population stays in the holdout. The holdout needs to be large enough to give you a clear read. The right share depends on your volume and the size of the difference you expect to see. A brand working tens of thousands of leads can hold out a smaller share than one working a few hundred. Run the test until the difference between the groups is stable enough to trust, then shrink the holdout or retire it.

Doesn't a holdout mean leaving recovered revenue on the table?

Yes, for a limited part of the test. That means some people who might have returned with Hello Patient's agent stay outside the new program while you measure the lift. The holdout only needs to be large enough to produce a useful read, and you do not have to keep it forever. Once the result is stable enough to trust, you can shrink the holdout or retire it and apply what you learned to the wider population.

Should the holdout group get nothing new, or stay on our current re-engagement program?

Either can work, depending on what you want to measure. Hello Patient's agent contacts the group you assign it, so the holdout can stay on your current re-engagement program or receive nothing new. Keeping the current program in place shows what the agent adds over that program. Giving the holdout nothing new shows what the outreach adds compared with doing nothing. One consumer health brand we work with compared the agent with its earlier program, which gave it a real baseline. In either setup, run both groups over the same window.