Generating Opportunity Solution Trees with AI: How Vistaly Rebuilt Its Product Around Interview Synthesis, Evals, and Repair Loops

This podcast episode follows Vistaly, an opportunity solution tree (OST) software company, as it rebuilt its product around AI-generated interview snapshots and an agentic workflow that drafts and updates trees from customer interviews. The founders describe why a chat-based synthesis experiment was too slow, how errors in lower analysis layers corrupt everything above them, and why a repair loop in the orchestration fixed an issue that prompt tuning could not. The episode also covers teaching the agent to log semantic change sets, data residency and Bedrock hosting, and why helping users comprehend changes is harder than output quality. It matters to product teams weighing how much synthesis to automate and how to evaluate AI outputs in discovery workflows.

01Key takeaways

• Judge AI synthesis against the alternative of shallow analysis that never happened, not against careful manual synthesis.
• Validate each layer of analysis, since an error in an early snapshot will corrupt the tree built on top of it.
• Use cheap code assertions as pre-filters before spending on LLM-as-judge evals.
• When two error modes pull against each other, consider changing the orchestration (such as a repair loop) rather than only tuning prompts.
• Have agents log semantic moves like merge, move, and reframe, since multiple valid diffs can exist between two trees.
• Design for users who want the AI's answer first and then the ability to correct it, not step-by-step collaboration.

02Key sections

Why rebuild from scratch

Vistaly chose to rewrite its V1 canvas for continuous discovery as a ground-up V2 instead of bolting AI onto it, pausing V1 signups to protect the rewrite. The team rebuilt most functionality in about two and a half months.

Chat synthesis experiment

An early chat interface walked users through insights one at a time. It was faster than manual work, but users still found it too slow, pushing the team toward bulk synthesis.

Layered analysis and the house of cards

V2 generates a snapshot per interview and then a draft tree, so a poor snapshot propagates errors upward. The team treats the architecture as layered analysis, where whether a step is a pipeline or an agent matters less than the quality of each layer.

Evals and the repair loop

A user complaint about having to clean up branches led to new evals and experiment variants that balanced missing subgroupings against badly framed parent opportunities. The fix was an agentic repair loop in orchestration, which then became a production guardrail.

Change sets, data, and deployment

The agent was taught the rules of the game so it logs semantic moves like merge, move, and reframe, rather than diffing trees afterward. Data residency requirements led the team to Bedrock, and the hardest remaining problem is helping users understand what changed.

03From the post

“Listen to this episode on: Spotify | Apple Podcasts What happens when you hand your opportunity solution tree to an AI? Vistaly rebuilt its entire product to find out—and the agents were the easy part. In this episode of Just Now Possible, Teresa Torres talks with Matt O'”

04Frameworks mentioned

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.