Evals before anyone sees a demo

Applied AI has a familiar shape right now: a sparkling demo, a week of applause, and a quiet month where nobody can say whether the thing is better than a search box. We do not start there. We start with the job someone already does, the corpus they already have, and a harness that can tell us we are wrong.

Agents, evals, and tool-use belong inside the product — and in the repo while we write it. Intelligence in the loop, not a slide at the kickoff. That is the whole practice. The rest is taste about where a model is allowed to act.

The corpus is the brief

Your tickets, your radio logs, your product catalog with the ugly SKUs, your clinic notes that were typed at speed — that is the brief. A clean sample set is a lie. We want the messages with the missing fields, the PDF that was scanned twice, the city name nobody spelled the same way twice.

An eval harness on that corpus is the first artifact. It is boring. It is the only thing that lets us change a prompt, a tool, or a model without arguing from vibes. If we cannot fail the current system on a real example, we are not ready to replace it.

Tool-use is the product

A chatbot that “knows your docs” is a search box with extra latency. A useful agent calls the same tools an operator already trusts: create the booking, hold the slot, open the incident, write the note, refuse the refund when the rule says no. MCP and function-calling are how we put those tools in reach without inventing a second admin that nobody will maintain.

Guardrails are not a policy PDF. They are review paths: what the model may do unattended, what a human has to see, what gets logged, what metric decides if this stays in the workflow next quarter. If we cannot name the metric, we will not put the agent on the hot path.

We build with the same class of tools

The studio is AI-native in the product and in the pipeline. We use models to move while a 2014 shop is still writing the specification. We do not use them to skip the judgment. A generated migration that nobody reviewed is not speed. It is a future incident with better autocomplete.

So the loop is: eval, thin slice in production, measure weekly use, keep the people who wrote it on the line. Demo day is a side effect. If the harness is green and the desk still uses the old spreadsheet, the product is not done. We stay until the spreadsheet is a backup, not the system of record.