Why experts writing AI evals is creating the fastest-growing companies in history

Brendan Foody is CEO and co-founder of Mercor, a labor marketplace that supplies domain experts to AI labs so they can write evals, rubrics, and post-training data. The conversation covers why evals are becoming the central bottleneck for improving models, how Mercor grew to a very large revenue run rate in record time, which jobs and skills remain valuable as AI advances, and his views on founder lessons and the realistic pace of AI progress.

01Key learnings

• Treat evals as the product requirement document for a model: if you cannot measure what success looks like for a task, you cannot effectively improve or deploy AI against it (Foody).
• Companies should build a systematic test of how AI automates their core value chain, since that measurement is the prerequisite to applying AI across the business.
• Evals also serve as sales collateral, showing customers and researchers concretely which real-world capabilities a model or product delivers.
• Expert-written rubrics enable scalable reinforcement learning from AI feedback, which is far more data-efficient than relying on human preference labeling alone.
• Focus career investments on areas with elastic demand, where higher productivity expands output (software, product) rather than areas with a fixed amount of work to do.
• Rather than fighting AI use, teach people to use the tools effectively; Foody's hiring assessments ask candidates to build products with available AI tools within an hour.
• Find the leading indicator of a fast-moving market, then pick the customers willing to pay most, and obsess over their experience before investing in marketing.
• Be patient with the first handful of hires to protect talent density, but speed up hiring once you know demand far exceeds your capacity.

Summary and takeaways written by PM Atlas; quotes are short excerpts. © the original author.