How to Build an AI Agent That Talks to Customers

This is the setup guide I wish I’d had before building an AI agent that talks to real people. Not theory-first. Just the pieces you actually need, the order I’d build them in, and a few mistakes we made the expensive way.

Everything here comes from one real system: an agent we built and run in production today, still in alpha.

What we built: an AI agent that texts new job applicants for a high-volume recruiting operation with offices across the country — hundreds of applicants a month. It starts the conversation, answers questions about pay and the role, pulls live interview slots from the scheduling system, and books the interview.

Humans still handle the interviews and the training. That’s on purpose. The agent takes the repeatable front part of the funnel so people can spend time where people matter most.

What it’s doing so far: about 30 to 60 cents per scheduled interview. The layer it replaces: a call center worker doing the same texting and follow-up. We’re still iterating.

If you want the theory version first — what an agent is, workflow vs. agent — OpenAI and Anthropic have both written solid pieces on that. I’d start with Anthropic’s Building Effective Agents, then read OpenAI’s A Practical Guide to Building Agents. This write-up assumes you already want to build one and just want to get your hands dirty.

Step zero: use the AI for as little as possible

This is the choice that affects everything else: give the AI the smallest job you can get away with.

If your process only moves one direction — send this, wait, send that — you don’t need an agent. You need a workflow. That’s what our old system was: the entire message sequence queued up front, with fixed waits in between. Cheap. Predictable. Boring, which in software is a compliment.

Its real problem: it couldn’t react. If an applicant asked a question, the queued sequence just kept plowing ahead.

That’s the test:

• If the process doesn’t need to respond to what the other person does: use a workflow.
• If it does: you need a loop, and now you’re building an agent.

The eight components

Every one of these exists in our production system. None of them are decorative.

1. A goal

The agent needs to know what “done” means, in writing. In our case, that’s a literal config field it reads every turn: get the applicant scheduled for an interview.

Two things became clear once I actually wrote that goal down:

• Bound the job. Our prompt tells the agent that everything after the interview belongs to a human, not to it. Without that boundary, it keeps going.
• Command the action, don’t just describe the state. “The goal is met when the interview is booked” didn’t do much. “The moment the interview is booked: send one wrap-up text, close the conversation, report goal met” worked.

2. A stop condition

Easier to forget than the goal, and just as important: how does the system end when things don’t work?

Ours stops when:

• the applicant opts out
• the applicant goes quiet for more than 48 hours
• the conversation needs a human

Every turn, the agent has to report one outcome: continue, goal met, or stopped. If it reports nothing five turns in a row, the platform force-stops it.

One early model version scheduled a wake-up for itself every five minutes, forever, for applicants who had ghosted. That turned into hundreds of pointless runs per contact per day until we caught it. An agent without a stop condition is an agent that runs forever.

3. Tools

Tools are how the agent touches reality: send a text, read the applicant record, update a field, fetch live interview slots.

A few rules learned the hard way:

• Tell the model when to use each tool, not just what it does. If a tool doesn’t say “use this when…,” the model may act like it’s invisible. We watched that happen.
• Expect the failure point to be routing, not execution. When our model picked the right tool, the tool worked. The real coin flip was whether it picked the right one in the first place. That’s where most of the design work goes.
• Don’t let the model supply an identifier it can get wrong. Our agent never decides which contact it’s texting. The server resolves that. No parameter, no hallucinated ID.

4. The loop

The react loop is what makes this an agent: look at the situation, decide, act, repeat.

Ours ended up being two loops.

• Inner loop: one turn. The model can take up to ten steps, call tools, and keep working until it decides the turn is complete.
• Outer loop: the real agent. Turns happen over days.

Three things can wake it up:

• the applicant replies
• a scheduled wake-up fires
• a human nudges it

One of its tools is literally “schedule my next wake-up.”

The mental model that helped me most: it’s not really a chatbot. It’s a proactive evaluator. Most of the time it’s asleep. Then it wakes up, looks around, does one thing, reports the outcome, and goes back to sleep.

5. Durable memory

If an agent works across days, it can’t keep its memory in one chat session.

Ours is split, and the split that matters is fact vs. interpretation.

• Fact: the actual text thread, rebuilt from the database every turn from the source of truth
• Interpretation: the model’s chat history and logged decisions — its own account of what happened

We learned the difference when the model wrote a closing note that echoed an old message into its own history. On the next run, it trusted that false diary over the real thread and sent the message again.

The fix: anchor memory to reality. What goes into history should be what the system actually did, not what the model claims it did.

6. The prompt

We assemble the prompt fresh on every turn from a few layers:

• a global platform prompt
• the operator’s instructions
• the goal and stop conditions
• the decisions made so far
• a footer of hard rules nothing can override

That last layer holds rules like:

• “your reply text is never delivered; the only way to send a message is the send-message tool”
• “never re-send a message”
• “silence is allowed”

The layering matters because different people own different parts. The platform owns the safety rails. The operator owns the persona and the script. The rules you absolutely can’t afford to lose need to live in the layer the day-to-day editor can’t touch.

7. Logging

This is a probabilistic system talking to real people, which means “it did something weird on Tuesday” has to be answerable.

So we log:

• every turn with an ID
• every tool call tied to that ID
• token usage for every model step
• the agent’s own decisions as structured, queryable events

When something breaks, we can reconstruct what the agent saw, what it did, and what it cost, per request. Otherwise you’re debugging fog.

Put logging in before the first real conversation, not after the first incident.

8. A testing cycle

This was the piece basically everyone, us included, underestimated. It also paid off the most. Two big parts:

• A test console. We ran a test copy of the agent through the full real loop, but suppressed actual sends at the very last step. We also added a simulated clock so we could fast-forward and watch 48-hour behavior in seconds instead of waiting two days. That’s what let us iterate fast, and iteration speed was the whole game.
• Version the instructions like software. Every prompt change got a real version number. We haven’t rolled back; we’ve always rolled forward. But knowing we could roll back made it much easier to change things. Same mental safety net version control gives you with code.

The process that got us to production

Ship fast and iterate. Don’t spend weeks guessing what’s needed. Make the most reasonable choice in front of you, get it running, and let real behavior show you what to build next. Every guardrail above came from watching the agent do something wrong, not from predicting it in a design doc.

Start with a smart model; downsize later. We switched to a frontier model in the middle of the build, and it was noticeably better in the judgment moments that actually matter. It cost more per call, and early on that’s fine — you don’t know yet where you truly need the intelligence and where a smaller, cheaper model will do the job. Overpay for smarts first, then optimize once the behavior shows you where the smarts matter.

Keep taking work away from the model. The clearest example was scheduling. In testing, the agent offered interview times that didn’t exist. The scheduling tool had passed it raw timestamps and let it write the message. Most runs were fine. One run just made stuff up.

The fix wasn’t a better prompt or a bigger model. The server now turns the slots into finished, human-readable strings and decides which ones to offer. The model’s job is just: “copy this exactly.” Nothing left to invent.

We made the same move in other places too:

• pay details go out as approved copy the model can’t edit
• the model is deliberately timezone-blind because it once turned a 9 a.m. local follow-up into 9:00 UTC
• follow-up timing moved out of the model’s judgment and back into plain scheduled timers

An agent can do everything however it thinks is best. Most of the time, though, I don’t want “however it thinks is best.” I want one specific thing done one specific way.

So spend the model’s judgment where judgment actually helps:

• reading a confused reply
• figuring out which lane the conversation is in
• deciding whether the goal has been met

And use boring, deterministic code for everything else.

The build didn’t get worse as we took capability away from the model. It got shippable.

What’s next on the roadmap

Two things I’m working on now. I’ll write about each once I’ve actually figured them out.

• Giving the agent a way to ask for help. Instead of failing quietly or guessing, it should be able to raise its hand and pull a human into the conversation mid-stream.
• Reducing cost. We deliberately overpaid for a frontier model to get the behavior right. Now the work is figuring out which decisions can move to smaller, cheaper models without losing the judgment that matters.

Very open to feedback and ideas on both.

30 to 60 cents per scheduled interview. Hundreds of applicants a month. Humans doing the parts humans are good at. Still in alpha. Still taking work away from the model.