Every note in the archive that names LLM-as-a-Judge: summaries, key takeaways and links to the original essays, episodes and posts. Ask a question below to get a cited answer from just these notes’ authors.
Lenny Rachitsky shares a guest post by prompt engineer Mike Taylor covering eight prompting techniques, from role-playing and few-shot examples to chain-of-thought, retrieval, and using an LLM as a judge. Each tactic…
“"That is a lot of first layer opportunities there. I wish I could click on this card and be like, 'Clean this up.'" This was a Vistaly customer reviewing her first AI-generated opportunity solution tree. Up until this point, the feedback was extremely positive. But”
Teresa Torres introduces a practical test harness for running AI product evals. Manually running every input through a prompt, saving outputs, scoring them against evals, and repeating for each prompt variant is…
Teresa Torres argues that AI evals should be a core discovery habit for product teams, not just an engineering concern. Because LLM outputs are probabilistic and semantic, traditional pass/fail unit tests don't work…
Hamel Husain and Shreya Shankar outline a practical playbook for building AI evaluation systems that improve a product rather than just producing dashboards. The core argument is to start with error analysis by a domain…
"The only way to know if our AI products and workflows are any good is with evals." 💡 If you're using AI to write PRDs, analyze customer feedback, or build customer-facing AI products, you need to understand evals. This guide breaks down what evals actually are and why product teams should be…
💭 "I wish I could click on this card and be like, 'Clean this up.'" When a customer pointed out a flat, unstructured branch in her AI-generated opportunity solution tree, it sparked a three-week journey to fix the problem at its source. Here's what you'll learn from this article: How one customer…