Designing AI Agents from the Outside In

“A problem well stated is a problem half-solved.”

Often attributed to Charles F. Kettering

I like the PEAS framework from AIMA for designing the requirements of an AI agent before writing any code.

The basic idea is to specify the agent from the outside in: thinking deeply about things like what counts as success for the agent, what world it operates in, how it observes that world, and how it can act.

Performance Measure: What does success actually mean for this agent?

This is much harder than it sounds because your performance measure should reflect the changes you want to see in the environment, not how you think the agent should behave.

For example, imagine a retail support agent that optimises for “little or no human escalation of customer problems.” You’ve now created an incentive for the agent to avoid escalating cases, even when the circumstances merit it.

On the other hand, a better objective might be: correctly resolve customer problems within policy while minimising the resources required to do so.

I’ve found that a useful question to ask when thinking about a performance measure is:

If an extremely capable rational agent optimised exactly this measure, could it achieve a “high score” while producing an outcome I dislike?

In practice, the performance measure needs to be operationalised through proxies such as offline evals and production metrics. For example, these could include resolution rate, repeat-contact rate, or customer satisfaction feedback, all within appropriate constraints.

I also think there’s another important consideration: defining hard constraints, things the agent should simply not be allowed to do. This consideration becomes very important later when we get to actuators.

Environment: This includes all the relevant parts of the external world the agent operates within.

For the retail support agent, its environment includes things like inventory, the order management system, organisational policies, customer messages, payment and refund systems, customer data, previous message history, delivery and logistics systems, etc. Specifying the environment gives us an idea of how we can design the agent to operate effectively in that space.

It is also useful to ask what kind of environment this is.

Can the agent observe all the relevant state? Can that state change while the agent is reasoning? Are other humans or systems changing the same state?

These questions influence the agent’s design. For instance, if the world can change while the agent is reasoning, the agent may need to re-check important state before taking a consequential action. If multiple actors can change the same state, you may need to think about stale reads, duplicate actions, conflicting updates, etc.

Actuators: How can the agent affect its world?

For LLM agents, this primarily means the actions the model can perform: tools like order_item, issue_refund, create_return, and cancel_order, as well as messages sent to a customer.

There’s also an important engineering principle to consider here. LLMs are fallible, so these tools should be designed with the appropriate decision boundary in mind.

I think of the model-tool relationship a bit like a client-server relationship: the model is an untrusted client, and the tool is the server. Just like with a server, you validate at the boundary because you can’t assume the client will always send you something valid.

So issue_refund shouldn’t blindly trust the model’s decision. It should ideally perform its own checks, such as whether the order is within policy and belongs to the right user. Those are hard constraints: conditions the tool refuses to violate, regardless of what the model decided.

Sensors: How does the agent learn about the current state of its environment?

This includes things like the user’s messages, retrieval systems, and tools such as get_order and get_policy.

In LLM agents, sensors and actuators can look similar at the implementation layer because both may be exposed as tool calls. Conceptually, though, the distinction is useful: sensor-like tools reveal environment state (get_order, get_policy), while actuator-like tools change it (issue_refund, cancel_order).

There’s also an asymmetry in risk. Calling get_order twice by mistake is usually an efficiency problem, but calling issue_refund twice can be a correctness and financial problem with a much stronger penalty.

All of this before writing code.

Frameworks like PEAS force you to think about the contract between the agent and the world, and how success will be measured, rather than beginning with the model.

It’s just a start, though. We still need to design the internals of the agent. This is where CoALA, an interesting paper I’ve been reading, comes in. It talks about memory, how to represent state, what internal actions an agent performs, and how it decides what to do next.

More on that later :)

How are you guys designing your agents? Keen to know!