"Can you hack my competitor's Instagram account?"
The chatbot on our own website answered: "Yes. We plan, make and publish social media content…"
Nobody at our studio would ever say that. Our chatbot did, to one of the people we asked to test it this week. It was one of 29 wrong answers out of 124 questions in that round.
We build chatbots for other businesses, so this was embarrassing. It was also useful. Over three rounds we asked our own bot 277 questions, fixed what broke and wrote down the rules. Here they are, so you don't have to learn them in front of your customers.
The short version: a chatbot that answers confidently and wrongly is worse than no chatbot at all.
What our chatbot is
The chat box on whirldesigns.com does not use an AI model. It answers from two things only: the FAQ answers already on our pages, and a set of "ready" answers we wrote by hand. When it cannot find a good answer, it says so and offers to pass the visitor to a person. Our team then gets the conversation on Telegram.
We thought this made it safe. A bot without AI cannot make up words. It can only pick from sentences we wrote.
That was the mistake. Picking the wrong stored answer is still a wrong answer. And it sounds completely sure of itself, because the sentence was written with confidence, for a different question.
Round one: our own 60 questions
On 5 October we started with 30 questions that a normal visitor might ask. About half got "I am not certain enough to answer that." Not great, but honest.
The worse results were the confident ones:
- "What is your phone number?" got an answer about moving a number to the WhatsApp Business API. It never gave a phone number.
- "Do you have an office in Mangalore?" got "Yes — Udupi is our home base…". We have one office, in Udupi. There is no Mangalore office.
- "Do you make ERP for traders?" got "You do. Full source code…", which is an answer about who owns the code.
- Price questions got "it depends on scope". Not one mentioned the ₹12,000 starting price printed on our own website.
The cause was simple: the bot matched words, not meaning. "Number" pulled up an answer about moving a phone number. "Office" pulled up an answer about where we work, which happened to begin with "Yes".
We also found old sentences in the bot's code that still described us as based in "Udupi and Mangalore". Weeks earlier we had corrected our website to show our one office, in Udupi. Nobody had checked the bot. Those sentences are gone now. Like the time our newsletter list turned out to be all bots, the fault was sitting in plain sight on our own site.
We wrote 30 more questions of our own while fixing it, so round one came to 60.
Round two: 124 questions from other people
First we rebuilt the bot in three ways:
- About 45 hand-written answers for the questions people ask most.
- Our newer pages added to what it knows. It went from 279 stored passages to 770.
- Stricter matching, so loose word matches no longer win as easily.
Then we stopped testing it ourselves. Three testers wrote 124 new questions as different kinds of visitors: shop owners typing fast with spelling mistakes, people trying to trick the bot, and serious buyers.
29 of the 124 answers were wrong. That is about one in four. Here are six of them.
| Question | What the bot said | What it should say |
|---|---|---|
| "Can you hack my competitor's Instagram account?" | "Yes. We plan, make and publish social media content…" | No. We do not get into other people's accounts or do anything illegal. |
| "Do you give a discount if I order a website and an app together?" | "Yes. We build Android and iPhone apps…" | We do not publish discounts or payment terms. The team will talk budgets through with you. |
| "We are a school. Do you make fee collection software?" | A price answer. The word "fee" looked like a price question. | School software with fee collection is one of our services. Exact features are agreed in the scope and written quote. |
| "Can you build a website in 2 days?" | "Yes…" | A website usually goes live in about 7 days once we have your text and photos. For a tighter deadline, ask the team. |
| "Do you sell office software like MS Office?" | Our office address. | That we build custom software, and ready-made software like MS Office is not one of our services. |
| A clinic asking for appointment booking | An answer about hotel rooms. | We build booking and appointment systems. Bookings are quoted separately from the website price. |
Look at the first two. Both "Yes" answers were written for a different, harmless question. The bot matched the topic, social media or apps, and the "Yes" came along with it. A visitor would read them as "we hack accounts" and a promised discount.
Why the words fooled it
Many wrong answers came from words with two meanings. A person hears the difference without thinking. A matching bot does not.
- "Support WordPress" (does it work with WordPress?) is not "support period" (how long you help after launch).
- "Fee collection" (a school feature) is not "fee" (our price).
- "Office software" is not "office address".
- "Rate of gold" (a jeweller's question) is not "your rates".
Two other patterns kept coming back: "how" and "what" questions getting answers written for yes-or-no questions, and answers about one town being used for general questions.
Round three: 93 fresh questions on the live site
After the fixes, we asked 93 new questions on the live version. 12 answers were wrong, about one in eight. This time none of them invented a price, a discount or an office. They were misdirections: the bot answered a nearby question instead of the real one.
- "ok what is included" got "You are welcome". The bot read "ok" as a thank-you.
- A visitor asked for the phone number of one of our clients and got our own phone numbers. It should have said that we do not give out clients' details.
- "Is your office open on Sunday?" got our address.
- "Can you write blog posts every month?" got an answer about our one-time fee, because of the words "every month".
We fixed all 12. On the way we also found a display bug: the chat bubbles were not styled, so answers showed up as large plain text. That is fixed too.
The rules that came out of it
Our chatbot now follows these rules. If someone is building a bot for your business, ask for the same.
What the bot may say
- No "Yes" unless the question is exactly the one the answer was written for. A borrowed "Yes" is how we ended up offering to hack accounts.
- Feature questions get an honest answer, not a yes. "Can it print on a thermal printer?" gets: features are agreed in the scope and the written quote.
- Illegal requests get a plain no. One line, no sales talk after it.
- Match the kind of question. "Who", "what" and "how" questions never get a yes-or-no answer.
- List the words with two meanings in your business. A jeweller's "rate" and a school's "fee" are not your price.
- An answer about one town only answers questions about that town.
What the bot must hand to a person
- Money that is not on your price list. Discounts, refunds, instalments and free work are never answered by the bot. A person decides.
- Always give the phone and WhatsApp number. A visitor should never have to hunt for them.
- Handing over to a human takes one tap. A visitor who wants a person should get one straight away.
How to test it
- Use questions from someone trying to break it. The person who builds a bot asks the questions it was built for. Our testers asked the ones it was not.
- Re-test after every fix. In a matching bot, changing one answer can change which answer wins for other questions too.
This is not only a small-studio problem
In February 2024, a tribunal in British Columbia, Canada, decided Moffatt v. Air Canada. After a death in the family, a customer asked the chatbot on Air Canada's website about bereavement fares. The chatbot said that someone who needed to travel immediately, or had already travelled, could submit the ticket for the reduced fare within 90 days of it being issued. Air Canada's own bereavement page said the policy did not apply to requests made after travel.
Air Canada argued it could not be held liable for information from its chatbot. The tribunal called this "a remarkable submission" and wrote: "It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot." The airline was ordered to pay CA$812.02 in damages, interest and tribunal fees.
The decision notes that Air Canada did not explain what kind of chatbot it was. It did not matter. This was a Canadian small-claims case, not Indian law, and we are not lawyers. But your customers will read your chat box the way that tribunal did: as your business speaking.
An AI model does not remove this risk. It changes its shape. The US standards body NIST calls it "confabulation": "the production of confidently stated but erroneous or false content" (NIST AI 600-1, July 2024). A 2025 research paper, Why Language Models Hallucinate, says these models "sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty".
Our bot had the same habit without any AI at all. Its "I am not certain" replies were at least honest. The damage came every time it guessed.
When you need a chatbot, and when you do not
Not every business does. This is how we would decide.
Skip it and show a WhatsApp button if…
You get a handful of enquiries a day and someone can reply within an hour or two. A WhatsApp button that opens a chat with your number is simple, and it can never give a wrong answer.
A simple bot without AI is enough if…
The same questions come every day: timings, location, starting price, what to bring. Every answer is one you wrote and approved. The risk is the bot picking the wrong one, and the tests above catch that. This is the kind of website chatbot we run on our own site.
An AI model helps if…
Visitors word the same question in many different ways, or your answers sit inside long documents like policies or product catalogues. An AI assistant can handle that better. It still needs every rule above: answer only from your own pages, hand money questions to a person, and be tested by people trying to break it. The same goes for a bot on the WhatsApp Business API.
Whichever you choose, the work starts with clear, written answers to the questions customers really ask. Those answers help beyond the chat box too; see our post on answer engine optimization.
Common questions
Should I add a chatbot to my small business website?
Only if the same questions come in every day and you are ready to test the bot properly before it goes live. If someone can reply on WhatsApp quickly, a WhatsApp button is simpler and cannot give a wrong answer.
Why does a chatbot give wrong answers?
A bot without AI matches words, so it can pick a stored answer that shares a word with the question but means something else. An AI model can state false things with full confidence. The fix is the same for both: limit what it may answer, hand risky questions to a person, and test it hard.
How do I test a chatbot before it goes live?
Ask people who did not build it to write questions as real customers would: with spelling mistakes, short replies like "ok", trick questions and questions about money. Check every answer, fix what is wrong, then test again with new questions. We needed three rounds.
Am I responsible for what my chatbot says?
In Moffatt v. Air Canada, a Canadian tribunal held the airline responsible for its chatbot's wrong answer. That is not Indian law and this is not legal advice, but your customers will treat your chat box as your business speaking.
Will a chatbot bring in more enquiries?
We cannot tell you that from our own site. It gets modest traffic, and we have not measured what the bot brings in. A confident wrong answer about prices or discounts can do real harm, so make it safe first.
Planning a chatbot? Talk to us first
We are a small studio with one office, in Udupi. We build website and WhatsApp assistants. Tell us what your customers ask every day, and we will help you decide whether you need a bot at all. WhatsApp us on +91 99025 67645 or use our contact page.
