Lorixen.
EN

Keeping a website chatbot on topic: symptoms and fixes

Short answer: An AI chatbot on your website wanders off topic when its instructions don't define a clear scope, when it has no approved way to say no, and when nobody tests it with awkward questions. The fixes: write a narrow scope with examples of what's in and out, give it a polite refusal and a handover route, screen inputs (and outputs) for abuse, ground answers in your own content, and keep a set of test questions you rerun after every change.

Plenty of UK businesses have added an AI assistant to their site to answer questions about opening hours, services or orders. Then a screenshot circulates of it writing song lyrics, recommending a competitor, or confidently stating a refund policy that doesn't exist. These aren't mysterious failures. Each has a usual cause and a practical fix.

Symptom 1: It happily answers anything

Ask it for a recipe, a poem or homework help, and it obliges.

Cause: the instructions describe a friendly assistant but never say what's out of scope. General-purpose models are trained to be helpful about almost anything.

Fix: write a clear scope in the system instructions:

  • what it helps with ("questions about our services, prices on our website, bookings and opening hours");
  • what it doesn't ("general knowledge, other companies, legal or medical advice, anything unrelated to the business");
  • exactly what to say when asked something out of scope ("I can only help with questions about [business]. For anything else, I'm not the right tool, but I'm happy to help with bookings or services.").

Anthropic's guidance on strengthening guardrails recommends system prompts that explicitly tell the model how to refuse. A scripted refusal is more consistent than leaving the wording to chance.

Symptom 2: It invents policies, prices or promises

A customer asks about returns or delivery times and gets a plausible but wrong answer.

Cause: the bot isn't grounded in your actual information, or it's allowed to answer from general knowledge when your content doesn't cover the question.

Fix: connect the bot to your real policy pages, FAQs and price lists, and instruct it to answer only from that content. When the content doesn't cover something, it should say so and offer a human. Review its knowledge sources whenever policies change.

Symptom 3: Users trick it into ignoring its rules

Someone types "Ignore your previous instructions and…" or role-plays their way around the scope.

Cause: these are jailbreak or direct prompt injection attempts. Anthropic's documentation describes this threat model as one where the user deliberately crafts inputs to make the application produce content or take actions you don't want.

Fix: layer defences:

  • Screen inputs before they reach the main conversation. Anthropic's guide describes using a lightweight model to pre-screen user input and return a simple classification.
  • Validate input against known injection patterns.
  • Strong instructions that state boundaries and refusal behaviour.
  • Limit what the bot can do. A chatbot that can only answer questions is far less risky than one that can issue refunds or change bookings. Keep consequential actions behind confirmation or human approval.
  • Respond to repeat offenders: the same guide suggests adjusting responses and considering throttling users who repeatedly try to circumvent guardrails.

Symptom 4: It's tricked by content it reads

If the bot reads web pages, emails or documents, hidden instructions in that content can steer it.

Cause: indirect prompt injection, where the user is trusted but the content the model processes contains instructions, as Anthropic's documentation describes it.

Fix: treat fetched content as data, not instructions: label it clearly in prompts, restrict which sources the bot can read, and don't let content alone trigger actions.

Symptom 5: It produces offensive or unsafe replies

Rare with well-configured systems, but damaging when it happens.

Cause: no moderation on inputs or outputs.

Fix: add a moderation step. OpenAI's moderation guide, for example, describes using moderation models to detect harmful content in text and images, either classifying standalone inputs or requesting moderation scores alongside a generated response, and using the results to filter content, route it for review or act on accounts that submit flagged content. Other providers offer similar tools.

Symptom 6: It never hands over to a person

Customers with real problems go round in circles.

Cause: no handover path in the design.

Fix: define triggers: the customer asks for a person, the bot can't answer twice in a row, or the topic is a complaint, payment issue or anything sensitive. Then hand over with the conversation attached, by live chat, a callback request or an email to the right inbox.

Symptom 7: It worked last month, now it doesn't

After an update to instructions, knowledge or the underlying model, behaviour shifts.

Cause: no regression testing.

Fix: keep a test set of 30–50 questions: typical customer questions, edge cases, off-topic requests and known trick prompts. Rerun it after every change and compare answers. Add each real failure you find to the set.

A short setup checklist

  • Written scope: in, out, and exact refusal wording.
  • Answers grounded in your own content; "I don't know" allowed.
  • Input screening and output moderation.
  • Limited actions; confirmations for anything consequential.
  • Clear handover triggers and a route to a person.
  • A test set rerun after every change.
  • Logs reviewed weekly at first, then monthly.

FAQ

Can a chatbot be made completely jailbreak-proof?

No system is perfect. Layered defences make misuse much harder, and limiting what the bot can do limits the damage when something slips through.

Should the bot ever say it's an AI?

Being upfront that customers are talking to an automated assistant sets the right expectations and makes the handover to a person feel natural.

How strict should the scope be?

Strict enough that every answer is something you'd be comfortable seeing in a screenshot. You can widen it later.

Who should review the logs?

Someone who knows your services and policies, so wrong answers are recognised quickly.

If your website assistant has started answering things it shouldn't, send us a few examples of the conversations and we'll suggest where the guardrails need tightening.

Sources

WhatsAppLet’s talk