定义"好"的标准——All Things Product 播客：Teresa Torres 与 Petra Wille 谈 AI 评估

Listen to this episode on: Spotify | Apple Podcasts

AI evals feel like a brand-new skill — but the hardest part is a question product teams have always had to answer: what does "good" actually look like?

Petra Wille and Teresa Torres dig into Teresa's recent article on AI evals and the one idea in it that hasn't changed at all. Teresa argues that measuring the quality of AI outputs is really the same work as defining outcomes, writing a PRD, or instrumenting a product — and that teams keep repeating an old mistake: measuring technical performance (uptime, speed, "was it concise?") instead of whether they actually created value for the customer. Drawing on the evolution of engagement metrics, a DocuSign example, and why a student and a professor need opposite definitions of "concise," they make the case for defining "good" in your own context — and never outsourcing it to off-the-shelf evals.

A grounding listen for any team worried they're falling behind on AI: much of the work is the work you already know how to do.

Key Takeaways

• Defining "what good looks like" isn't new — AI evals are just the latest place that old work shows up.
• Don't stop at technical performance. An eval suite should measure technology performance, customer value, and business value.
• Don't outsource your definition of "good" to generic, off-the-shelf evals — define it in your own context.
• "Good" is context-dependent: a concise answer is right for an expert and wrong for a beginner.
• Metrics mature the same way every time — from universal (MAU) toward your product's real value-creation moment (e.g., DocuSign getting you to send a document).
• AI amplifies whatever shape you're already in — automating a poorly-defined product or process just produces more of what doesn't work.
• Underneath it all is one habit: be explicit about what you expect to happen, so you can test the assumption when you're wrong.

Notable Moments

• "Good is: the customer did something that created value for the customer." — Teresa
• "Don't outsource what good looks like. Really think about, in your context, what does good look like?" — Teresa
• "Helpful is my new NPS." — Petra
• "Even with conciseness, the definition of what good looks like changes with the context." — Teresa
• "The habit that underlies metrics and outcomes is really just: what do you expect to happen?" — Teresa

Resources & Links:

• Follow Teresa Torres: https://ProductTalk.org
• Follow Petra Wille: https://Petra-Wille.com

Mentioned in this episode:

• Teresa's article on AI evals: AI Evals: A Hands-On Guide for Product Teams — the piece that sparked this conversation, and the earlier All Things Product episode on it
• XING — the German professional network where Petra watched engagement metrics first emerge
• DocuSign — used to illustrate evolving from raw engagement to the real activation metric
• Product Leadership Coaching with Petra
• An observation Teresa attributes tentatively to Ethan Mollick — but it's actually Dave Kline who posted that we're rushing to automate work that shouldn't be done at all
• Cal Newport — his idea about how much of our work doesn't actually have impact

Join the Conversation:

Have thoughts on this episode? Leave a comment below.

Full Transcript