Three AI systems we built are in production today. A research assistant for a US client, running on AWS Bedrock with an orchestrator and ten specialised sub-agents, twenty-five tools and retrieval over an indexed corpus of grants and publications. Bob, our own multi-tenant assistant, with conversational memory, document retrieval, voice and an email integration. And the question generator inside tenSurvey, which writes survey questions in more than thirty languages.
None of them started with a model. They started with a spreadsheet of real cases and a question: does this work well enough, on our data, to be worth building? This article is the method we use to answer that question and to turn a yes into a feature that survives contact with users.
Step 1: the spike, two weeks on real data
Every engagement starts with a spike, and the spike starts with data. We ask for fifty to two hundred real examples of the task: the documents that need summarising, the tickets that need classifying, the questions people actually ask. Not the clean ones. The ones with typos, missing fields and the edge case that only the person who has done the job for five years knows about.
Then we build the thinnest possible pipeline and measure. For a document extraction task that means: model, prompt, structured output, a script that runs all two hundred cases and compares them to what a human said. For the research assistant it meant: can retrieval find the right passage in the corpus, and does the answer cite it? The output of the spike is a table with accuracy, latency and cost per case, and a recommendation that is sometimes "do not build this yet".
That recommendation has happened. A classification task where the model agreed with the human reviewers 82% of the time sounds impressive until you learn the reviewers agreed with each other 84% of the time. The task was ambiguous, not the model. We spent the second week helping the client sharpen the categories instead of tuning prompts, and the eventual feature was better for it.
Step 2: the golden dataset, before the architecture
The two hundred cases from the spike become the golden dataset: inputs paired with what "correct" means, agreed with the client. It lives in the repository next to the code, and it runs in CI on every change to a prompt, a model version, a retrieval setting or a tool definition.
This is the single most important artefact in an AI project, and it is the one most teams skip. Without it, every prompt change is a vibe. With it, a change either passes the gate or it does not, and the number is visible to everyone in the pull request. When Anthropic ships a new model, we run the golden set against it the same afternoon and know within an hour whether to switch. We did exactly that when Fable 5.1 replaced Fable 5.
Step 3: architecture that assumes the model will be wrong
Once the spike says yes, we design the system around one assumption: the model will sometimes be wrong, and the architecture has to make that safe and visible.
Model access. On AWS we use Claude through Bedrock: the data stays in the client's account and region, nothing is used for training, and IAM controls who can call what. Off AWS we use the Anthropic API directly. When data cannot leave a private network at all, we have run open-weight models on the client's own hardware; our notes on Mistral's open mixture-of-experts release and on running Claude Code against a local model come from that work.
Structured outputs, validated by code. Every model response that feeds a system is a typed object, validated before it is used. A summary is a string; an extracted invoice is a schema with amounts that must add up. If validation fails, the code retries with the error, then falls back to a human queue. The model never writes directly to a database.
Retrieval with citations. For anything answered from documents, the answer carries the passages it was built from, and the user interface shows them. Hybrid search (keyword plus embeddings) on PostgreSQL with pgvector covers most corpora; OpenSearch when the corpus is large or the query patterns are complex. We tune chunking and re-ranking on the golden set, not on a benchmark.
Agents that ask before acting. The research assistant uses an orchestrator that plans and delegates to specialised sub-agents, each with a narrow set of tools exposed through MCP servers with explicit scopes. Reading is free; anything that changes state outside the system waits for a human click. The audit log records every tool call with its inputs, so "why did it do that" always has an answer. We wrote about the platform behind it in the Agent Factory platform and about the general pattern in enterprise AI agents at scale.
Prompt injection is an input problem. Documents, emails and web pages fed to a model are untrusted. Instructions found inside them are data, never commands. Guardrails check inputs for injection patterns and outputs for leaked secrets or off-policy content, and the sub-agents that read external content have no tools that write.
Step 4: cost you can put in a budget
During the spike we already know the tokens per case. Before the build starts, the client gets a cost per document, per conversation or per ticket, at the expected volume, for two or three model choices. This is usually the moment where the smallest model that passes the evals wins, and where prompt caching and batching are designed in rather than bolted on.
In production, every call carries the token counts into the tracing system. There are budgets per feature and alerts when a day's spend exceeds the forecast. We have seen a single retry loop double a monthly bill in a system we inherited; a budget alert would have caught it in an hour. For our own tooling costs we did the same exercise publicly in what $200 of Claude really buys.
Step 5: operate it like software, because it is
The feature ships in stages: one percent of traffic, then ten, then everyone, with the evals and the tracing watching each stage. Prompts are versioned in the repository, not in a dashboard. Model versions are pinned and upgraded deliberately, through the golden set. Failures from production are triaged weekly, and the interesting ones become new golden cases, which is how the dataset grows from two hundred cases to two thousand without anyone writing test data by hand.
This is also where we are honest about what AI does not fix. A broken process with a model in front of it is a faster broken process. We turned down more than one project this year for that reason and said why in AI won't fix your broken app.
What we use, and what we do not
We build on Claude because it is the model that has performed best on our clients' evals for agentic and document-heavy work, and because Bedrock makes the enterprise questions (residency, training, IAM) easy to answer. We have used the Strands agent framework on Bedrock for multi-agent systems, MCP for tools, pgvector and OpenSearch for retrieval, and plain PostgreSQL for everything the model produces. We have no reseller relationship with anyone; when an open model passes the evals at a fraction of the cost, we say so.
What we do not do is demos. If a feature cannot be measured on real data, we will not pretend it works. If you have a language-heavy process you suspect a model could take over, send us fifty real examples and we will tell you in two weeks whether it can.