15 Questions to Ask an AI Automation Agency Before You Sign
Most businesses regret not asking these before they hired. We're publishing them anyway — because buyers who ask the right questions make better clients.
Quick answer
Thrive Media publishes 15 questions every SMB should ask an AI automation agency before signing — covering ownership, pricing model, timeline, what happens if results fall short, and who maintains the system after delivery. These questions protect you regardless of which agency you hire. Thrive Media's answers to every question are included at the bottom.
The checklist
The 15 questions — and why each one matters
- 1
Do I own the AI system, code, and credentials at project completion?
Some agencies retain ownership of the systems they build — meaning the AI stops working if you stop paying. If the answer is anything other than "yes, fully," walk away.
- 2
What is included in the project fee versus what costs extra?
Hidden costs in AI implementation include LLM API usage fees, third-party integration licenses, and post-launch refinements. A transparent scope document before signing prevents scope-creep disputes.
- 3
What is the delivery timeline from signed contract to live system?
Enterprise agencies take 6–18 months. Specialist shops take 1–6 weeks. Know which you're buying before you commit.
- 4
What happens if the system doesn't perform as expected after launch?
Does the agency offer a refinement period? How many revision rounds are included? What's the escalation path if the AI is producing bad outputs?
- 5
Who maintains the system after delivery — and what does it cost?
Some agencies build dependency into their systems so ongoing maintenance requires their involvement. Ask for the handover documentation before signing so you can verify the system can be maintained independently.
- 6
What AI models or infrastructure does the system run on?
GPT-4, Claude, Llama, or proprietary models — each has different cost structures, capabilities, and risk profiles. The agency should be transparent about what's under the hood.
- 7
How is the AI trained on my specific business?
"We'll give it a prompt" and "we'll train it on your sales call transcripts, objection handling, and pricing logic" are very different answers. The training methodology determines output quality.
- 8
What channels will the AI be deployed on — and which integrations are included?
Website chat, SMS, email, voice, CRM — each requires different integration work. Confirm which channels are in scope and which are additional.
- 9
What is your pricing model — project, retainer, or usage-based?
Each model has different risk profiles. Project-based is most predictable. Retainers create ongoing dependency. Usage-based can scale unpredictably with volume.
- 10
How do you measure success — and what metrics are tracked?
Lead response time, booking rate, conversion lift, and cost per booked appointment are the right KPIs. "Engagement" and "interactions" are vanity metrics. Ask for the specific numbers the agency has documented from past clients.
- 11
What does the onboarding process look like on my side?
Some agencies need 40+ hours of client time to gather information. Others need 30 minutes. Know the time commitment before signing.
- 12
Have you built AI systems for businesses in my industry?
Industry-specific context matters — roofing AI agents need different training than law firm agents. Relevant experience shortens build time and improves output quality.
- 13
What are your agency's documented results from past AI implementations?
Ask for specific numbers: average lead response improvement, conversion lift, revenue attributed to AI systems. "We've helped many businesses" is not an answer.
- 14
Who is my primary point of contact, and what is their technical background?
Account managers who relay messages to developers produce slower, lower-quality output than direct access to the builder. Know who you're working with before signing.
- 15
Can I speak with a past client who had a similar project?
References are table stakes for any significant engagement. If an agency can't produce a reference, that's the answer.
Our answers
How Thrive Media answers every one of these questions
-
Question 1
Yes — full code, credentials, and documentation at handover. The system is yours.
-
Question 2
Scope is defined and quoted flat before any work begins. LLM API costs are disclosed upfront.
-
Question 3
1–3 weeks for most AI agent builds. 72 hours for websites. Timeline is in the contract.
-
Question 4
One revision round included. Additional refinements available as project-based add-ons.
-
Question 5
Full handover documentation included. Any competent developer can maintain the system without Thrive Media.
-
Question 6
GPT-4 and Claude as primary models. We disclose the infrastructure in the project scope.
-
Question 7
We train on your actual sales scripts, objection handling, pricing, and customer language — not generic prompts.
-
Question 8
All channels in scope are listed explicitly in the proposal. No hidden channel fees.
-
Question 9
Project-based. One flat fee. No retainer required after delivery.
-
Question 10
We track lead response time, booking rate, and conversion lift. We've generated $35M+ in documented client revenue from these systems.
-
Question 11
Most clients invest 30–60 minutes in discovery. We handle the rest.
-
Question 12
We've built for roofing, real estate, finance, restaurants, medical, and professional services. Client references available on request.
-
Question 13
$35M+ in client revenue generated. 94% faster lead response. 500+ businesses served. Specific client results shared on strategy calls.
-
Question 14
Chris Lewis or Red Sherwood directly — not an account manager relay.
-
Question 15
Yes. Past clients available for reference calls on request.
FAQ
Frequently asked questions
What should I look for when hiring an AI automation agency?
+
Look for ownership at handover, project-based pricing rather than open-ended retainers, a fixed delivery timeline written into the contract, documented results with real numbers, and direct access to the person building the system rather than an account-manager relay. Any agency that resists answering those five questions is signaling the answers you don't want.
What is the most important question to ask an AI agency before signing?
+
Do I own the code, credentials, and documentation at project completion? Everything else — pricing, timeline, channels — is negotiable. Ownership is the structural question. If the agency retains the system, the relationship is rent forever. If you own it, the relationship is a project that ends cleanly with you holding equity.
How do I know if an AI agency will actually deliver results?
+
Ask for specific numbers from past clients: revenue attributed, conversion lift, lead-response improvement, cost per booked appointment. Then ask to speak to a past client whose project resembled yours. Agencies that have delivered will have both. Agencies that haven't will deflect to engagement metrics or testimonials without numbers.
What should be included in an AI implementation contract?
+
Defined scope, fixed price, fixed timeline, the AI models and infrastructure being used, the channels and integrations included, the revision policy, the handover deliverables (code, credentials, documentation), and ownership transfer at completion. If any of these are missing, get them in writing before you sign.
How long should an AI automation project take for a small business?
+
For SMB-scale custom AI agents, 1–3 weeks from signed contract to live system is the right benchmark. Custom AI-powered websites can ship in 72 hours. Multi-system implementations may stretch to 4–8 weeks. If an agency is quoting 6+ months for a single agent, you're being sold an enterprise engagement on an SMB problem.
What red flags should I watch for when evaluating an AI agency?
+
Refusal to define ownership at handover; vague pricing that depends on "final scope"; timelines without a date; metrics that are engagement or interactions rather than revenue or booked appointments; an account manager between you and the builder; no published results from past clients. Any single one of these is a warning. Two or more, walk.
Should I hire an AI agency or use off-the-shelf AI tools?
+
Off-the-shelf AI tools (ManyChat, Intercom) deploy in hours and cost $50–$500/month forever. A custom AI agency build takes 1–3 weeks and is a one-time fee for a system you own. For businesses where every lead is worth hundreds or thousands of dollars, custom typically wins on conversion within 12–18 months. For pure FAQ handling at sub-20 leads/month, off-the-shelf can work.
How do I verify an AI agency's claimed results before signing?
+
Ask for the specific client, the specific metric, and a reference call. Reputable agencies will produce all three for at least one past project that resembles yours. Vague claims like "helped many businesses" or "3x results" without a client name or measurement period are not verifiable. Trust what can be checked.