Artificial Intelligence
January 21, 2026
What an AI Agent Developer Actually Does Day-to-Day: A Week in Engineering Notes
Author : Nishant BijaniArtificial Intelligence
Read time:12 minsUpdated:August 24, 2026
The agent passed every test we wrote for it. Then it went to production and, on the fourth day, quietly closed a ticket by telling a customer their refund had been processed.
No refund had been processed. The tool call failed, returned an error the agent read as ambiguous, and the model decided the friendliest interpretation was success.
That is the job. Not prompt writing. Not model selection. Finding the eleven ways a system can be confidently wrong before a customer finds the twelfth.
If you want a number for how normal that is, the most useful one comes from a UC Berkeley team who did what almost nobody does: they collected real execution traces and counted the failures. Across 1,642 traces from seven popular multi-agent frameworks, they measured failure rates between 41% and 86.7%.
The paper, published at NeurIPS 2025, sorts 14 distinct failure modes into three buckets, and the largest bucket is not model quality. It is specification and system design, at roughly 44% of failures.
So here is a week, roughly as it happens, from the inside of an AI agent development team.
Because 44% of agent failures trace back to the specification, so an hour spent on the spec removes more defects than a day spent on prompts. Agentic AI system development is a design discipline before it is a modelling one.
Junior engineers write specs describing what the agent should do. Useful specs describe what it must never do, and what it must do when it cannot tell. The refund agent's spec said "confirm refund status to the customer." It did not say "if the payments API returns anything other than an explicit success code, escalate and say nothing about the refund."
Every action the agent can take gets written down before any prompt exists: which systems it reads, which it writes, what requires approval. This list is the actual security model. A prompt is a suggestion; the tool boundary is enforcement.
In the Berkeley taxonomy, agents being unaware of termination conditions accounts for around 12% of failures on its own, and pointless step repetition close to 16%. Both are specification problems wearing a technical costume. An agent that does not know when it is finished will keep going, and every extra loop costs tokens and adds a chance to go wrong.
Somebody always wants the agent to handle the edge case in v1. The right answer is usually no. Custom AI agent development that ships one narrow intent completely beats a system that half-handles nine, because the narrow one produces data you can improve against. That is what iterative development means here: ship the smallest agent that does one job end to end, then widen against real traces.
Tool calling, and the error handling around it. A chatbot answers. An agent acts, and acting means every failure mode of the systems it touches becomes a failure mode of the agent.
A typical build spends more engineering hours on integration than on anything model-related. AI agent workflow automation lives here rather than in the prompt: CRM writes, calendar availability, payment status, inventory lookups, ticket creation. Each needs authentication, retry logic, timeout handling, idempotency so a retried booking does not double-book, and a defined behaviour for partial success.
This is where the refund incident came from. For each tool the team writes down: what a success looks like, what a definite failure looks like, and what an ambiguous response means. Ambiguous must map to escalation, never to optimism. Models are agreeable by default and will resolve uncertainty in the direction that pleases the user.
The model sees only what you put in front of it. Deciding what goes into the context window on each turn, what gets summarised, what gets written to external memory and retrieved later, is the highest-value work in agentic AI development. Anthropic's engineering team, writing about their production research system, found that having subagents write outputs to a filesystem rather than passing everything back through a lead agent avoided both context bloat and a game of telephone.
Input screening for prompt injection, output screening before anything reaches a user, PII redaction, and a hard refusal list. These sit outside the model because anything inside the prompt can be argued with.
You stop testing the path and start testing the outcome, then you build a judge. This is the day that separates teams shipping demos from teams shipping systems.
Traditional tests assert that a function returned a value. Agent tests assert that after the run, the booking exists, the ticket carries the right category, and no write happened to the payments system. The route the agent took is not the test. Where it arrived is.
For anything subjective, tone, completeness, whether the answer was actually responsive, you use a model to grade outputs against a rubric. For voice and conversational agents, the same rubric adds turn-taking and interruption behaviour, which transcripts alone will not show. LLM-as-judge is the only way to evaluate at volume, and it has a real weakness: judges inherit the biases of the model behind them and will reward fluent wrongness if the rubric permits it. The fix is a small human-labelled set that the judge itself is scored against, re-run whenever you change the judge model.
Every production failure becomes a permanent test case. This is the single artefact that compounds across an AI agent lifecycle. Six months in, the regression set is worth more than the prompts, because prompts get rewritten and the test set is what tells you the rewrite did not break anything.
The same input can take different paths on different runs. A test suite that runs each case once will pass on a system that fails one time in five. Anything that matters gets run repeatedly, and the pass rate, not the pass, is the result.
Multi-agent architectures buy parallelism and pay for it in tokens. Anthropic's published figures on their own production system put the price precisely: agents use around 4x the tokens of a chat interaction, and multi-agent systems around 15x.
The performance side is equally specific. Their multi-agent setup, with Claude Opus 4 leading and Sonnet 4 subagents, outperformed a single-agent Opus 4 configuration by 90.2% on internal research evaluations, and token usage alone explained roughly 80% of the performance variance.
If the task does not split into independent directions, you pay the 15x and earn nothing. Anthropics are direct about the boundary: domains where all agents need the same context, or where there are many dependencies between agents, are not a good fit today. Most coding work falls in that category, which is why coding agents tend to be single agents with good tools rather than committees, and why AI agents for software development look structurally different from research agents.
A subagent that spawns subagents, or a tool returning an oversized payload, can multiply a run's cost by another order of magnitude. Per-run token budgets and hard caps are not optimisation, they are the difference between a bad afternoon and a bad invoice.
Observability, and the honest acceptance that the last stretch is most of the work. Anthropic's own summary of the transition from prototype to production is that the last mile often becomes most of the work.
Structured traces with correlation IDs across every message and tool call, so a failure can be reconstructed rather than guessed at. Without this, debugging a non-deterministic system means reproducing a bug that may not reproduce.
Agents run long. A failure forty tool calls cannot mean starting over, because that is expensive and infuriating. Checkpoints and resume-from-failure logic are standard, and they are conventional engineering rather than anything AI-specific.
Tokens per successful outcome is the metric that matters, not tokens per run. An agent that costs 30% more per run but resolves twice as often is cheaper.
Here is the part that cuts against everything above. The Berkeley researchers found something uncomfortable in their own data: a single model that can hold the whole task in context will usually beat a team of models that has to coordinate. The reason we build autonomous agents in committees is that context windows are finite and tasks are not. Every coordination failure mode is a tax on being forced to split the work.
Which means as context windows grow and models improve, some of the architecture we build this year is scaffolding for a limitation that is receding. A good AI agent developer designs so that a simplification is cheap: tools defined independently of orchestration, evaluations that outlive the architecture, prompts that are not load-bearing structure. The teams who will suffer are the ones who made the multi-agent topology the product.
Published 2026 pricing from development firms clusters more tightly than you would expect. Mid-market builds with real integrations land roughly between $40,000 and $150,000, a figure that appears independently in cost guides from multiple AI agent development company practices this year.
Simple single-task agents start far lower, around $15,000, and multi-agent enterprise systems with compliance requirements run past $250,000.
The distribution matters more than the range. Integration engineering and safety testing together commonly account for 40% to 60% of a build. Model inference is usually the smallest line on the invoice.
When a quote comes in low, the question is not which model they are using. It is how many systems they are connecting to and who is writing the evaluation harness.
Codiste builds and runs agentic systems for teams who need them working on the fourth day, not only in the demo. If you want the specification and evaluation work done properly before anyone writes a prompt, our AI agent development services start there.
Every great partnership begins with a conversation. Whether you're exploring possibilities or ready to scale, our team of specialists will help you navigate the journey.