Behind the Scenes: Building AI-Generated Opportunity Solution Trees

Teresa Torres describes building AI-generated opportunity solution trees for Vistaly as a product manager stepping into engineering with AI assistance. The core difficulty was updating an existing tree with new interviews, because tree diffs are ambiguous compared to linear text diffs: splits and merges can disappear from the comparison, so the change steps shown to users were not always producing the final tree. Her first fixes via prompt design got error rates down but still failed in roughly half of runs. The breakthrough was letting the model call her own validation code as a tool in an agent loop, so it could repair its change sets until they passed. The piece matters as a case study in the gap between AI prototypes and production quality, and in how validation tooling can make model output reliable.

01Key takeaways

• Prototypes that look impressive can hide a large share of remaining production work, so plan for reliability early.
• Ambiguous structural changes like splits and merges can be invisible to naive diffs, so model the operations explicitly.
• Have models show their reasoning as predefined, user-meaningful actions rather than arbitrary edits.
• Reuse deterministic validators as tools inside an agent loop so the model can iteratively repair its own output.
• Don't blindly trust AI-written code; invest in planning, testing, and code-review infrastructure before relying on it.

02Key sections

Starting an AI engineering sprint

Torres introduces herself as a PM and former designer who learned to build with AI help, and frames the post as showing what is newly possible. She describes the alpha launch of AI-generated interview snapshots and opportunity solution trees with design partners.

Updating trees with new interviews

Adding interviews to an existing tree is much harder than generating one from scratch. Inspired by git diff, she decided users need a step-by-step walkthrough of what changed, not just the final output.

The prototype is not production

A quick working prototype came together in days, but cracks appeared at higher interview counts, with change sets not reproducing the emitted tree. Tree diffs prove ambiguous, since splits and merges leave no trace in a simple comparison.

Making the model show its work

She refactored prompts so the model outputs both recommendations and predefined moves, then categorized errors into invalid moves and mismatches between change set and result. Even after heavy iteration, errors persisted in about half of service runs.

Self-correction through a validation tool

After a conversation with her husband, she realized the model could call her validation code as a tool within an agent loop and repair its own mistakes. The final loop usually fixes errors within one or two turns, and the update shipped to Vistaly.

Reflections on the process

She reflects on shedding limiting beliefs about engineering, enjoying the puzzle of action design, and needing tighter oversight of Claude-written code. She also notes the intensity of the sprint and her growing sense of being an AI engineer.

03From the post

“I just finished an all-out engineering sprint. That sounds weird to me. I've been writing code on and off for several years. But I wouldn't call myself an engineer. I'm a product manager. And I used to be a designer. It's been a long time”

04Frameworks mentioned

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.