Most guides to creating an AI agent start with the framework. That is the wrong end of the problem. The framework is the least consequential decision you will make; the task you point it at, and the reliability layer you wrap around it, determine whether it ever leaves your laptop.
This is the sequence we use on client builds, in the order that actually reduces risk.
Step 1 — Choose a Task Worth Automating
Before any code, answer three questions about the candidate task:
- Does it happen often? Under roughly a hundred times a month and the engineering rarely pays back.
- Is there a right answer? If two competent people would disagree about the correct output, you cannot evaluate the agent, and you cannot improve what you cannot measure.
- What does being wrong cost? Cheap and reversible means you can move fast. Expensive or irreversible means a human stays in the loop regardless of how good the model gets.
The tasks that pass all three are usually unglamorous — document triage, ticket routing, data enrichment. That is the point. Our AI agent examples covers ten that clear this bar.
The most common mistake at this stage is choosing the most interesting problem rather than the most repetitive one. Interesting problems make good demos and bad first projects.
Step 2 — Build the Evaluation Set First
This is the step teams skip, and skipping it is why so many agents stall in pilot.
Before you build anything, assemble 50 to 200 real historical examples with known-correct answers. Real ones, from your actual systems, including the messy edge cases — not synthetic examples you wrote to be representative.
That set is what lets you answer the only question that matters during development: did that change make it better or worse? Without it you are tuning by vibes, and you will not be able to justify a production deployment to anyone who asks a hard question.
It also front-loads a useful discovery. Teams frequently find, while assembling the set, that they disagree internally about what the correct answer is. Better to find that in week one than after launch.
Step 3 — Give It the Minimum Tools
Tools are the agent. The model is largely interchangeable and improves for free every few months; what the agent can see and do is what makes it useful or useless.
Start with read-only access. An agent that can look things up is dramatically easier to ship, test and trust than one that can change things. Add each write capability individually, and put an approval step on it until it has earned removal.
For each tool, be explicit about:
- What it returns — structured data beats prose the agent has to re-parse.
- What it costs — a tool that takes eight seconds shapes the whole interaction.
- What happens when it fails — timeouts and empty results are normal, and unhandled they become confident fabrication.
That last point matters more than it sounds. A great many "hallucinations" are actually an agent receiving an empty result and filling the gap rather than reporting that it found nothing.
Step 4 — Ground It in Your Own Data
An agent answering from model memory is guessing with excellent grammar. Retrieval grounding fixes this: the agent searches your documents, and the answer is built from what it found, with citations back to the source.
The citation is not a nicety. It is the audit trail. An answer with a link can be checked in five seconds; an answer without one has to be trusted or independently verified, and neither scales.
Practical notes from builds that went well:
- Chunk on structure, not character count. Split on headings and sections. Arbitrary 500-character chunks cut sentences in half and retrieve badly.
- Keep metadata. Source document, section, last-updated date. "Last updated" is what lets an agent flag that it is quoting a three-year-old policy.
- Retrieval quality beats model quality. A smaller model with excellent retrieval consistently outperforms a frontier model searching badly.
Step 5 — Add the Reliability Layer
This is where the real engineering lives, and where the budget goes.
Allow-lists. Define exactly which actions are permitted. Not "the agent shouldn't delete records" in the prompt — the delete tool simply is not available to it. Prompts are guidance; permissions are enforcement.
Human checkpoints. On anything expensive, irreversible, or customer-visible. Design these as approve/reject on a drafted action, so the human is reviewing rather than authoring.
Logging and tracing. Every run: what was asked, what was retrieved, which tools were called with what arguments, what came back, what was produced. When something goes wrong — it will — this is the difference between a fix and a shrug.
Regression testing on the evaluation set. Run it before every deployment. Model providers update their models underneath you, and a prompt that worked in March can behave differently in June.
Step 6 — Deploy Narrow, Then Widen
Do not launch to all traffic. Take one team, one document type, or one ticket category. Run it alongside the existing process and compare outputs for a fortnight.
This does two things. It surfaces the edge cases your evaluation set missed, and — more importantly — it builds organisational trust incrementally. Trust is the actual constraint on AI adoption inside companies, it is spent in a single visible incident, and unlike the models it does not improve on its own.
What This Costs, Honestly
A prototype against real data: two to three weeks. Production-ready, with the reliability layer above: another six to ten weeks for a single bounded workflow.
The prototype is rarely the bottleneck. Most calendar time goes to two things: the reliability engineering, and getting access to the systems the agent needs — which is an organisational problem far more often than a technical one. Budget for the access conversations.
The Shortest Version
If you take one thing from this: pick a boring, high-volume task with a checkable answer, build the evaluation set before the agent, start read-only, and earn write access one action at a time.
Everything else — framework choice, model provider, orchestration pattern — is a detail you can change later. Those four decisions are the ones that determine whether the thing ships.
For how much autonomy to grant in the first place, our guide to agentic AI vs AI agents covers the spectrum and what each rung costs.
We build AI agents as fixed-fee engagements: a scoping call, a working prototype against your real data in two to three weeks, then the guardrails and evaluation that make it production-grade. Book a scoping call or read more about our AI automation practice.