Lorixen.
EN

Measuring a Website Chatbot: The Numbers That Matter

Short answer: Track a handful of numbers that answer three questions: is the bot actually solving problems (resolution rate, not just conversations handled), is it handing off well when it can't (handoff rate and what happens next), and is it doing anything for the business (leads, bookings or orders it starts). Then read real conversations every month, because the numbers tell you where to look and the transcripts tell you what to fix.

Once a chatbot is live, the dashboard usually shows a big "conversations" figure and a percentage that looks impressive. Neither tells you whether customers got what they came for. This checklist sorts the metrics worth tracking from the vanity ones, with a review routine a small team in Calgary or Halifax can keep up with.

First, know how your tool defines its numbers

Chatbot platforms use similar words for different things, so check the definitions before comparing.

Intercom's documentation for its Fin AI agent, for example, separates three rates (as of November 2026):

  • Involvement rate: conversations where the bot actively participates, out of all new conversations.
  • Resolution rate: conversations the bot resolved, out of the ones it was involved in.
  • Automation rate: conversations resolved by the bot, out of all new conversations.

A bot that's involved in few conversations can have a high resolution rate and still handle little of your volume. That's why the automation rate exists.

Zendesk's documentation on automated resolutions shows another detail that matters: when a conversation counts as finished. For messaging, it says a conversation ends by default two hours after the first message, for email 72 hours after the first email, and that only conversations that weren't escalated are evaluated as automated resolutions. So "resolved" often means "the customer didn't come back or ask for a person within a window", not "the customer confirmed it was solved".

Write down your tool's definitions in a shared note so everyone reads the dashboard the same way.

The metrics checklist

Is it solving problems?

  • [ ] Resolution rate (as your tool defines it), tracked monthly.
  • [ ] Coverage: the share of all conversations the bot attempts. Low coverage with high resolution means it's only answering the easy questions.
  • [ ] Unanswered or "I don't know" responses: a list, not just a count. This is your content to-do list.
  • [ ] Repeat contact: customers who come back about the same issue within a few days, if your tool can show it. A "resolved" conversation followed by a phone call wasn't resolved.

Is it handing off well?

  • [ ] Handoff rate: how often the bot passes to a person.
  • [ ] Handoff reasons: grouped (customer asked for a human, bot couldn't answer, sensitive topic, account-specific request).
  • [ ] Time to human reply after handoff: a handoff into a queue nobody watches is worse than no bot.
  • [ ] Out-of-hours handoffs: what customers see when nobody's available, and whether they're contacted the next day.

Do customers like it?

  • [ ] Rating after conversations: thumbs up/down or a short score. Intercom's CX Score, for example, is the share of positive ratings for conversations handled by the bot alone.
  • [ ] Comments on negative ratings: read every one in the early months.

Is it doing anything for the business?

  • [ ] Leads captured (contact details or quote requests started in the chat).
  • [ ] Bookings or orders started from a chat link.
  • [ ] Questions about price, availability or delivery: these are buying signals; check how many end with a next step.

Website analytics can carry this part. Google's GA4 documentation explains that events measure user interactions on a site and can be sent with the Google tag's gtag() function or through Google Tag Manager, with recommended and custom events. Typical custom events for a chatbot: chat opened, handoff requested, link to booking clicked, lead form submitted inside the chat. Many chatbot tools can fire these for you; otherwise a developer can add them.

Is it worth what it costs?

  • [ ] Cost per resolved conversation: your monthly bot cost divided by resolved conversations. Check how your plan bills; some platforms charge per resolution, and Zendesk's page explains how its billing unit is counted.
  • [ ] Staff time saved: estimated honestly, by asking the team what changed, not by multiplying conversations by an assumed number of minutes.

Metrics to treat with caution

  • Total conversations: grows when the bot pops up more often, not when it helps more.
  • Average conversation length: short could mean quick answers or customers giving up.
  • "Containment" without context: keeping people away from staff isn't a goal on its own if they leave unhappy.

The monthly review routine

Set aside an hour a month, more in the first three months.

1. Check the numbers (10 minutes). Resolution, coverage, handoff rate, rating and leads, compared with last month. Note anything that moved sharply.

2. Read conversations (30 minutes). Pick about 20: ten with negative ratings or handoffs, five marked resolved, five at random. For each, ask: did the customer get the right answer? Was anything wrong or outdated? Should this have gone to a person sooner?

3. Fix the content (15 minutes). Most problems are missing or outdated information, not the bot itself. Update the help article, the price page or the bot's instructions. Add the top unanswered questions.

4. Record what changed (5 minutes). A simple log of "what we fixed and why" lets you see next month whether it worked.

Each quarter, review the bigger picture: are handoff reasons shifting, is the bot covering new topics you've launched, and is the cost per resolution reasonable for your volume?

FAQ

What's a good chatbot resolution rate?

It depends entirely on how your tool defines resolution and on what customers ask. Track your own trend over time rather than comparing with other businesses' published figures.

Should I count a handoff as a failure?

No. A fast handoff for an account-specific or sensitive question is good service. Count handoffs that happen because the bot lacked basic information as content gaps.

Can I measure chatbot leads in Google Analytics?

Yes, if the chat sends events (chat opened, lead submitted, booking link clicked) to your analytics. Mark the meaningful ones as key events.

How many conversations should I read each month?

Around 20 is enough for a small business to spot patterns; read more after major changes or when ratings drop.

To set up chatbot reporting your team will actually use, tell us which chatbot and analytics tools you run.

Sources

WhatsAppLet’s talk