流失客户能教给你什么：用幸存者偏差视角改进产品健康度评分

"Ride your winners, cut your losers" is a popular trope in business and investing, but it hasn't served the product org well.

It's fun to celebrate our most successful customers. But many of the ones we lost along the way could have also become winners, and ignoring the lessons about why they didn’t is just leaving future revenue on the table.

So this piece shares how I think about integrating lessons from churned customers into a product success lifecycle, with a few stories from IRL products to illustrate.

A solid instinct…with a blind spot

I recently talked to a VP of product who built a customer health dashboard. Their health scores mainly measured how closely any given customer resembled other customers who’d renewed many times over. Call it “affirmative pattern-matching.”

(Side note: This is something product leaders do when they are also taking on the role of general manager for their product, and I love to see it. Far too few take accountability for the business side of what they build. But I digress…)

This health score was a great instinct, but I’d make just one tweak to the analysis. By only studying the customers who renewed, the dashboard fell victim to survivorship bias.

But this bias is preventable once you understand how it works.

The planes that made it home

The best example of survivorship bias comes from WWII. Early on, the US military was losing way too many aircraft in battle. When technicians analyzed the damage on planes that survived, they noticed certain regions tended to be more shot up than others, and they began to reinforce the armor on the high-damage spots.

But a statistician named Abraham Wald convinced his colleagues to do the opposite, aka reinforce the areas with no damage. He argued that the undamaged spots must be the most critical, that the returning plans survived precisely because they weren’t hit there, and that if you could analyze the planes that were shot down, you’d find most of them hit in these critical regions.

Survivorship bias, aka learning only from the planes that made it home, nearly sent the armor to the wrong places. The status quo was well-intentioned, but ultimately it was an expensive way to lose just as many airmen as before.

Many products, and their customers, are stuck in a similar status quo today.

Look for important, boring metrics

Some metrics are great at measuring the success of “survivor” customers, for example:

• Net revenue expansion
• MAU growth
• NPS score
• Customer LTV
• Cumulative totals (for anything - users, workflows completed, etc.)

But the metrics that explain your losses often lie deeper in the product. For example, a legal ops software company I work with discovered that churned accounts often shared a few traits:

a) Small named-user counts

b) Low adoption of key features (e.g. <10% adoption of a bulk editing tool)

c) Just one admin user

These might not be headliners for the case study, but they were clear indicators of low customer value and likelihood to churn.

Track completion on key workflows

A hygiene services company I work with tracks a core workflow on its equipment monitoring platform. They discovered that just 11% of workflow users added the asset details that would help prevent a service outage, and only 17% logged financial impact details.

Steep drop-off for a core workflow is a strong indicator that customers aren’t getting value from your product, so check to see if this is a common bug for your churned customers.

Verify if your product is user-usable

For a non-discretionary workflow, completion isn’t a helpful indicator, but perhaps who completes it is. For example, take a compensation planning software company I work with (everyone has to run a comp cycle, no matter what). This company found that the professional services team starting the cycle for the customer, instead of the customer doing it for themselves, is a leading indicator of customer churn. That’s because the customer is leaning on a service to get value and may never learn to get it from the product.

Strike while the iron is hot

A privacy software company I work with mapped the setup steps a customer has to complete to be fully deployed. They found that a longer period from contract close to deployment completion meant a higher chance of cancellation.

So they switched to onboarding messages triggered by the steps each customer had and hadn't finished, and more than 80% of customers in their pilot were fully deployed within 30 days, versus 16% before.

While “deployment” still didn’t always equal “adoption,” it was a strong leading indicator for customer success, and many churned customers didn’t have it.

When in doubt, ask

In my World War II example, nobody could ask the pilots who were shot down where they got hit. Wald had to infer it from the planes that came back.

But in tech, you can go straight to the source. After checking which areas of the product your lost customers didn’t use, sometimes you just have to ask them: “why?”

Frankly, most teams skip this part of lost account analysis because it’s hard and doesn't fit on a clean dashboard (and a clean dashboard gives us all the warm fuzzies, right?). But the data only tells you part of the story.

Start with your last ten losses

Whenever someone asks me how to begin adding churned customer analysis to their health scoring, I suggest these starter steps:

1. List your last ten churned customers within a segment of interest (e.g. North America, 50-1,000 employees, <$5M revenue – you have to compare apples to apples). This is easiest if your org has Claude connected to your CSP or CRM.
2. Identify one commonality between at least eight of the ten (I try to guard against overzealous pattern-seeking). This might come from your product analytics, for which I use Pendo.
3. Verify that pattern, first with your own account owners, and then with the customers themselves (if they’re willing).
4. If it checks out, then incorporate it as a detractor variable in your health score (i.e. more of it = less likely to renew).
5. At the end of next quarter, list your newly churned customers and calculate if there was a positive, strong correlation between the new variable and churn (easy to do in Sheets/Excel). If yes, keep it.
6. Repeat steps 1-5 once every ~6 months (changing multiple variables too fast makes it hard to tell what had the most impact).

Reducing survivorship bias in your product success lifecycle is a muscle that you have to build. Our culture often tells us to manifest success by keeping “eyes on the prize” and tuning out everything else. So just remember that your real prize is a successful product, and spending some attention on your losses gets you closer to that, faster.

Even with the help of AI, there will always be limited armor to go around. But it’s the customers we lost who best tell us where to put it.