Read the original at Lenny’s Newsletter ↗lennysnewsletter.com · subscriber post
By Lenny Rachitsky · lennysnewsletter.com · @lennysan on X · YouTube · LinkedIn
Caitlin Sullivan, a user-research practitioner, explains four recurring ways AI-generated customer analysis goes wrong: invented or blended quotes, generic insights, signal that doesn't guide decisions, and missed contradictions. She pairs each failure mode with a prompting fix such as quote rules, context loading, few-shot calibration, and a final verification pass. The core argument is that AI output looks confident, so verification must be built into the workflow.
Subscriber post — summary only01Key takeaways
- AI analysis of interviews and surveys often invents or blends quotes, so define quote rules and verify each one.
- Generic themes come from models defaulting to consensus; provide project, business, product, and participant context.
- Teach your scoring scale with concrete labeled examples and the reasoning behind each label, not just descriptions.
- Run a dedicated verification pass to catch fabricated evidence, contradictions, and weakly supported findings before presenting.
- Different LLMs have different strengths for analysis, so test across models and know each one's tendencies.
“These mistakes are invisible until a stakeholder asks a question you can’t answer, or a decision falls apart three months later.”Caitlin Sullivan · Lenny’s Newsletter
“Verification is the difference between AI output you’ll always second-guess and insights you can stand behind.”Caitlin Sullivan · Lenny’s Newsletter
02Frameworks mentioned
Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.