Borrowed confidence is fragile in agentic systems

How do you know that your agentic system does what you mean for it to do? And how do you know that the tools you chose were the right ones to begin with?

I ran into both questions recently.

I was building a retrieval system. A user query comes in, I check if it’s semantically similar to something I’ve seen before, and if it passes a threshold I return the cached answer. I picked a popular embedding model, text-embedding-3-small, ran a few queries, and was quite pleased with myself.

Then I decided to stress-test it. With the help of an LLM, I generated production-like queries aimed to mimic what users would search for in that domain. Within seconds, my system broke.

“What is the paternity leave policy” was consistently passing my similarity threshold for “What is the maternity leave policy.” No matter what I tried, adjusting the threshold, adding in a cross-encoder, I couldn’t get the two queries to separate.

That’s when it dawned on me. I’d been forward-designing on hope, and I learnt that borrowed confidence is fragile.

When nothing worked, I eventually placed a lightweight LLM after the similarity checker to verify whether what was retrieved actually matched the intent of the query, and that worked. But honestly, that wasn’t the real lesson.

Defining the behaviour I wanted in production first, then testing against it, revealed the architecture I actually needed.

The evals told me what system to build.

I understood the premise of TDD in traditional software engineering. I just never put the same weight on finding relevant production evals first in AI engineering. Software engineering has been around for decades; we have design patterns, stress tests, and so on. Modern LLM application engineering is still young. The tools keep evolving and evolving around a non-deterministic engine, the LLM.

Eval-driven development is arguably more useful here than anywhere else, and we haven’t treated it that way yet.

For me, the key lesson is that confidence in your agentic system can’t and shouldn’t be borrowed. It is easy to pick the embedding model you see in a tutorial or an LLM based on benchmarks and hope things work out. Confidence has to be earned by designing against the behaviour you actually expect in production. Not just happy paths or random queries, but the full range of how real users would interact with your system.

And this doesn’t stop at launch. User behaviour in production will typically surprise you. Your evals need to evolve with it, and observability isn’t optional. This is how you keep your confidence earned rather than borrowed.

If you’re building agents without this, you might be the last person to know when it breaks, and that’s scary.

If you’re building agents, I’d love to know how you’re grounding their behaviour in truth.