AI Evals and Analytics Podcast

What is Evaluation-driven development? When should you start building evals for your product? How to build it from scrach?Using a real-world example of a customer chatbot for a medical insurance company, we walk through the process of setting up evals from scratch: translating product requirements into quantifiable metrics, curating quality test datasets (hint: you need fewer examples than you think), and making go/no-go decisions based on eval scores.You'll learn why accuracy and safety require different approaches, how to avoid the trap of AI-generated test data, and why 94% vs 95% accuracy matters less than you'd expect—but safety guardrails are non-negotiable. This is the practical blueprint for anyone building AI products who wants to catch problems before users do.00:00 – Introduction: Why We Need to Talk About Evals Now00:39 – When to Start AI Evals?03:20 – Example Setup: Medical Insurance Customer Chatbot04:30 – Defining Evals in Product Requirements07:19 – What Is Evaluation-Driven Development?08:27 – Breaking Down "Accuracy": What Does It Really Mean?09:42 – Dataset Curation: Quality Over Quantity11:24 – How Big Should Your Test Set Be?12:25 – Safety Guardrails: Knowledge Boundary and PII Leakage15:29 – Making Release Decisions with Eval Metrics17:33 – Start with What's Critical to Your Use CaseStella Liu: https://www.linkedin.com/in/wenxingl/Amy Chen: https://www.linkedin.com/in/amy17519/More about AI Evals and Analytics -- https://ai-evals.org/We (Stella & Amy) created the AI Evaluation & Analytics Playbook, a practical framework that helps teams ship production-ready, trustworthy AI systems.Powered by Firstory Hosting

What are the skills required for AI evals? Why data scientists have a natural advantage in AI evals?Evaluating AI isn’t just about "vibe coding" with an AI assistant. It actually requires a solid foundation in statistics for picking sample sizes and coding to build your own testing frameworks. Data scientists have a huge head start here because they are already pros at designing metrics and communicating risks.In the augural episode, we also explain why Evals (pre-launch testing) and Analytics (post-launch user feedback) are two sides of the same coin: one makes sure the AI works, and the other makes sure people actually love using it.00:00 – Introduction to AI Evals & Analytics01:31 – Why Data Scientists Have a Natural Advantage01:59 – Technical Pillar: Statistics02:48 – Technical Pillar: Coding & Prompt Engineering05:03 – Technical Pillar: Dataset Generation08:35 – Soft Skills & Stakeholder Collaboration11:17 – Domain Expertise in Regulated Industries15:50 – New Skills for the GenAI Era19:25 – Why Evals and Analytics Must Come TogetherStella Liu: https://www.linkedin.com/in/wenxingl/Amy Chen: https://www.linkedin.com/in/amy17519/More about AI Evals and Analytics -- https://ai-evals.org/We (Stella & Amy) created the AI Evaluation & Analytics Playbook, a practical framework that helps teams ship production-ready, trustworthy AI systems.Powered by Firstory Hosting