Questions to ask before buying an AI chatbot
Short answer: Before signing up for an AI chatbot, ask the vendor how your customers' data is used and stored, where the bot's answers come from and how you correct them, how it hands over to a person, what you can export if you leave, how pricing scales, and how they protect against misuse. Ask to test it on your own content with your own awkward questions before committing. A vendor who can't answer these clearly is telling you something.
AI chatbots for business websites are easy to buy and hard to evaluate. Demos are polished; your real customers are not. Many Australian businesses only discover the important details, like who can see the conversations or what happens when a customer asks something the bot doesn't know, after launch. Use this checklist in sales calls and trials.
Data: what happens to your customers' messages
- ☐ Are conversations used to train AI models? Many vendors build on large model providers. Ask both what the vendor does and what its model provider does. For context: OpenAI's API documentation states that, since 1 March 2023, data sent to its API isn't used to train or improve its models unless the customer opts in. A vendor using an API like that should still confirm its own policy.
- ☐ How long are conversations kept, and where? Ask about retention at the vendor and at its providers. OpenAI's documentation, for example, describes abuse-monitoring logs that by default may include prompts and responses and are retained for up to 30 days, with stricter retention controls available to eligible, approved customers.
- ☐ Who at the vendor can read conversations, and for what purposes?
- ☐ Can you delete conversations, individually and in bulk?
- ☐ What personal information does the bot ask for, and can you switch that off? Check the rules that apply to customer data where you operate.
Answers: where they come from and how you fix them
- ☐ What does the bot answer from? Your website, uploaded documents, a help centre, product data? Can you restrict it to those sources only?
- ☐ What happens when the answer isn't in your content? You want "I don't know, here's how to reach us", not a confident guess.
- ☐ Can you see the source behind an answer to check it?
- ☐ How do you correct a wrong answer? Edit the source, add an override, or block a topic? How quickly does the change take effect?
- ☐ Can you set the scope and tone, including topics it must refuse?
Safety and misuse
- ☐ How does it handle people trying to trick it into ignoring its rules? Anthropic's guidance on guardrails describes measures such as screening user input with a lightweight model, validating input against known injection patterns, system prompts that state how to refuse, and responding to repeat offenders. Ask which of these, or equivalents, the vendor uses.
- ☐ Is there moderation on inputs and outputs?
- ☐ Can the bot take actions (bookings, refunds, account changes)? If so, what confirmation and limits exist?
Handover to people
- ☐ How does a customer reach a human? Live chat, callback, email, ticket?
- ☐ What triggers a handover? Customer request, repeated failure, certain topics?
- ☐ Does the person receiving it see the full conversation?
- ☐ What happens outside business hours?
Integration
- ☐ Which systems does it connect to? Your booking system, shop, CRM, help desk?
- ☐ Does it work on your website platform without slowing pages? Ask how the widget loads.
- ☐ Mobile experience: test the widget on a phone, not only a laptop.
Reporting and improvement
- ☐ Can you read transcripts and search them?
- ☐ Does it report unanswered questions so you can fill content gaps?
- ☐ Can you tag or rate conversations to track quality?
Exit and ownership
- ☐ What can you export? Transcripts, the knowledge base you built, configuration, in a usable format.
- ☐ What happens to your data when you cancel, and how long until it's deleted?
- ☐ Is there a minimum contract term?
Pricing model
- ☐ What's the unit? Per seat, per conversation, per resolved conversation, per message?
- ☐ What counts as a conversation or resolution, and who decides?
- ☐ What happens if usage spikes, for example during a sale or an outage?
- ☐ Which features cost extra? Integrations, languages, removing branding, extra sources.
Ask for these answers in writing. A per-resolution model can look cheap until you see how resolutions are counted.
The trial: test it like a customer
- ☐ Load your real content.
- ☐ Ask 30 or so real customer questions, including ones your content doesn't cover.
- ☐ Ask off-topic and tricky questions.
- ☐ Test handover end to end.
- ☐ Have someone who knows your policies review every answer.
Illustrative example: a hypothetical Adelaide bike shop might find a bot answers stock questions well but invents repair turnaround times, which the trial reveals before any customer sees it.
Comparing vendors side by side
Put the answers into a simple table with one row per question and one column per vendor, and mark each answer as clear, vague or missing. Patterns appear quickly. One vendor may have excellent answers on data but weak handover; another may be cheap per conversation but count every greeting as a conversation. Weight the rows by what matters to your business: a clinic will care most about data handling and refusals, a shop about integrations and peak-time pricing, a trades business about after-hours handover.
Then score the trial results the same way: correct answers, wrong answers, refusals that should have been answers, and handovers that worked. A vendor that looks best on paper but gives more wrong answers in your trial is not the best vendor for you. Keep the table; it becomes your checklist when the contract comes up for renewal.
FAQ
Is a cheaper chatbot good enough for a small business?
It can be, if it answers only from your content, hands over cleanly and treats data responsibly. Price matters less than those three.
Should we build our own instead?
Building makes sense when you need deep integration with your own systems or unusual rules. Otherwise, a well-chosen product is faster.
How long should a trial be?
Long enough to see real traffic, often a few weeks, with someone reviewing transcripts.
What's the biggest red flag?
Vague answers about data use, or no way to see where an answer came from.
If you're comparing chatbot vendors and want a second opinion on their answers, send us your shortlist and their replies and we'll help you weigh them up.