Most write-ups of “how we built our AI agent” stop at the demo: a chat box, a clever prompt, a screenshot of it answering one question correctly. That’s the easy 20%. The hard 80% is everything that keeps the thing alive in production — the retrieval that has to be right every time, the action tools that touch a real PSA, the evals that let you change a prompt without a prayer, and the dashboard that stops the spend from surprising you.

This is the engineering walkthrough of AVA, the assistant our own MSP runs every day on Autotask. The case study over on /work covers why we built it and what it does; this piece is the build itself — the decisions, the wiring, and the parts that were genuinely hard. If you run an MSP or you’re scoping a custom agent, this is the shape of the work.

The five boxes, and the order they run in

AVA is not one model call. It’s a small pipeline where each stage has a narrow job and a clear boundary. Here is the whole thing on one page.

The two orange boxes are the ones that earn their keep: the agent core that decides what to do, and the human gate that stands between the model and anything that mutates the PSA. Everything else exists to feed those two and to watch them.

The retrieval layer: years of tickets, ranked honestly

AVA’s first job on every turn is to find the right history. An MSP that has run on Autotask for a decade is sitting on a few hundred thousand tickets, and the answer to “have we seen this before” is almost always yes — buried in a resolution note nobody can find.

We treat this as a RAG problem, not a memory problem. The model never tries to recall facts about our environment. It reasons over facts we hand it on the turn. Three decisions made retrieval actually work:

We embed ticket bodies at semantic chunk boundaries, not whole tickets and not titles. Titles are too short and too inconsistent. Whole-ticket embeddings rank on length, not relevance. We chunk the body — description plus resolution notes plus the comments that carry signal — at paragraph, list, and code-block edges, embed each chunk, rank chunks, then roll the scores back up to the parent ticket. That single change was the biggest precision lift in the whole build.

We rank by similarity, then re-weight by recency and resolution success. A perfect semantic match to a ticket that was closed “customer stopped responding” is worse than a slightly looser match to one that was closed with a documented fix. The ranking knows the difference.

Every retrieved claim cites its source ticket. No source, no answer. This is not a nicety — it’s the guardrail that makes an engineer trust the suggestion enough to click through, and the thing that lets them catch a wrong retrieval in two seconds instead of acting on a hallucination. It is the same posture we’d want from a junior engineer: show your work.

The hard part here was not the embeddings. It was the data. Ten years of tickets includes inconsistent categorization, copy-pasted boilerplate resolution notes, and PII that has no business going into a vector store. The cleaning and redaction pipeline took longer than the retrieval code. Budget for the data, not the model.

The action layer: small, audited, and bounded on purpose

A chatbot answers. An agent acts. AVA’s value is that an engineer can wrap a call and have the note drafted, the time logged, and the status moved without leaving the chat. That means AVA holds keys to a production PSA, which is exactly where most “agent” projects quietly become dangerous.

So the tool set is deliberately tiny:

Tool	What it does	Can it write to Autotask?
search_tickets	semantic search over the corpus	No — read only
get_ticket	fetch one ticket’s full detail	No — read only
get_user_history	the end user’s recent tickets and trends	No — read only
draft_ticket_note	compose a note in our standard format	No — produces a draft
post_ticket_note	publish the approved note	Yes — gated
log_time_entry	record billable time on the ticket	Yes — gated
set_status_or_assignment	move status, change owner	Yes — gated

Every tool we considered adding faced the same two questions: does it make a decision an engineer would otherwise make, and is the cost of being wrong bounded? Most candidates failed. “Close the ticket automatically” failed — the cost of a wrongly closed ticket is a frustrated client and a billing dispute, not a 30-second fix. “Email the client directly” failed for the same reason. We rejected more tools than we shipped, and the Anthropic notes on building effective agents make the case for that restraint better than we will.

The read tools and the write tools live behind different permission scopes. AVA can read broadly. It can write only the four bounded things above, and only through the gate.

The human gate: approvals are the product, not a speed bump

Every write action AVA proposes lands in front of a person before it touches Autotask. The engineer sees the drafted note, the time entry, or the status change, and approves, edits, or discards it with one click.

We resisted the urge to “graduate” trusted actions to auto-execute. The temptation is real — if AVA drafts a good note 95% of the time, why not auto-post? Because the 5% is where a wrong time entry hits a client invoice, and there is no clean way to un-send that. The approval click costs an engineer half a second. It is the cheapest insurance we ship.

The gate is also where the model learns its place. AVA proposes; the human disposes. That keeps the chat honest about what it is — a fast first draft from something that has read every ticket you’ve ever closed — rather than an autonomous coworker nobody hired.

The eval harness: the deliverable, not an afterthought

Before AVA was a product, it was a folder of test cases on disk: real engineer questions, real client messages, with known good retrievals and known good draft notes. Every prompt edit, model swap, and chunking change runs against that set before it merges. This is the single practice that separates an agent that survives contact with production from one that works in week three and silently rots by week eight.

We follow the pattern in Anthropic’s guidance on evaluating Claude applications: write the eval before you tune the prompt, and treat a regression in the eval the way you’d treat a failing unit test. It blocks the merge.

The eval set is not static. When AVA gets something wrong in production — a bad retrieval, an off-format note — that real case gets added to the harness so the same mistake can never ship twice. The harness grows with the product. That is the whole point.

Cost control: a dashboard, not a surprise

An agent’s running cost varies with the model and the amount of context each call carries. Retrieval-heavy turns are expensive turns. We instrument every AVA run for token count and dollar cost, broken down per task and per surface, and we watch it on a live dashboard.

Two findings from that dashboard changed the build. First, most of the cost was context, not generation — we were stuffing too many retrieved chunks into the prompt “to be safe.” Tightening retrieval to the top few well-ranked chunks cut per-turn cost meaningfully with no measurable drop in answer quality. Second, prompt caching on the stable parts of the system prompt paid for itself immediately. Without the dashboard, we’d have been guessing at both.

The fallback path: what happens when the model is down

AVA’s default is a frontier reasoning model. But “the provider is rate-limited” is not an acceptable answer to an engineer in the middle of a client call. So there’s a tested fallback to a second provider that kicks in on rate-limit or provider-side failure, and the fallback is exercised in CI on every change — not just configured and hoped for. A fallback you’ve never run is not a fallback; it’s a comment in a config file.

The four marks of a shipped agent

Across every agent project we scope, we hold the line on the same four marks. If a build doesn’t have all four, we call it a demo, not a shipped agent:

1. An eval harness — a growing set of real cases that gate every change.
2. A cost dashboard — per-task spend, visible, so the bill is never a mystery.
3. A fallback path — a tested answer to “what happens when the model is wrong or down.”
4. A named human owner inside the org who is accountable for the thing.

AVA has all four. Most of the agent projects other Bay Area shops have shown us are missing two or three, which is exactly why they break in production. The marks aren’t a maturity model to grow into. They’re the entry fee.

Common questions

How is AVA different from putting ChatGPT in a chat window? A chat window answers from whatever the model already knows. AVA answers from retrieved, cited Autotask history and can take four bounded, human-approved actions inside the PSA. The model is a component; the retrieval, the action tools, and the gate are the product.

Why retrieve over tickets instead of fine-tuning a model on them? Tickets change daily, contain PII, and need source attribution. Retrieval keeps the data fresh, keeps redaction in one controllable place, and lets every answer cite the ticket it came from. Fine-tuning bakes stale, un-attributable knowledge into weights. For an MSP, RAG is the right tool here, and it isn’t close.

Does AVA close tickets or email clients on its own? No. It can draft notes, log time, and propose status or assignment changes, and a human approves each one before it touches Autotask. We deliberately kept destructive or client-facing actions out of the tool set. The cost of being wrong there is unbounded.

What was the hardest part of the build? The data, not the model. Ten years of inconsistent categories, boilerplate resolution notes, and PII meant the cleaning and redaction pipeline took longer than the retrieval code. We’d budget more for that next time and start it sooner.

Can you build this kind of agent for our shop? Yes — that’s the work. If you run an MSP on Autotask, AVA is licensable and you can skip the integration plumbing. If you’re an SMB or public-sector IT team, the architecture above generalizes even when the use case doesn’t. The honest first step is a discovery call to figure out which 30 questions your team asks most and whether your data is clean enough to retrieve over.

We didn’t build AVA to write a blog post about agents. We built it because our own engineers were paying a tax every day to re-discover what the team already knew, and we got tired of paying it. Everything in this piece — the chunking we got wrong twice, the tools we refused to ship, the approval click we refused to remove — is what it actually took to make the thing trustworthy enough to run our own business on. That’s the bar. If an agent isn’t good enough to run your shop, it isn’t good enough to sell, and we’d rather show you the wiring than the demo.