What is an AI hallucination?

Hallucination (AI), explained

Language models generate text by predicting likely next words. When a question falls outside what the model has been given, it does not naturally say “I don't know”; it produces the most plausible-sounding answer. That is a hallucination: plausible, specific and wrong.

In support, hallucinations cluster around specifics. A customer asks whether a sale price applies to an item, whether international returns are free, or when an order will arrive, and the bot fills the gap with a confident number. Because the tone matches correct answers, customers rarely doubt it.

The main defences are structural. Ground answers in retrieved sources and instruct the model to answer only from them. Show the source so the claim can be checked. Define a fallback that says the bot does not have that information and offers a person. Keep high-stakes facts such as order status coming from real system lookups, not from the model's memory.

Then measure. Review a sample of conversations each week, mark answers that are not supported by a source, and fix the content or the rules that let them through.

How to measure it

Score from a human-reviewed sample; automated checks can help triage but should not be the only judge.

Common mistake

How SuperBot relates

SuperBot answers from your approved knowledge first, falls back to a constrained reply when no source matches, and hands off to a human when confidence is low. Citations are shown when matched knowledge supports the answer, though not on every reply.

• AI chatbot evaluation checklist
• Knowledge gaps: the AI support metric that matters

Frequently asked questions

Related terms

• Grounding and citationsGrounding is the practice of making an AI system base its answer on specific, approved source material rather than its general training data. Citations make that grounding visible by showing which source supports the answer. Together they let customers and reviewers verify a reply and make wrong or unsupported answers easier to spot.
• Retrieval-augmented generation (RAG)Retrieval-augmented generation (RAG) is a technique in which an AI system first searches a trusted knowledge source for passages relevant to a question, then gives those passages to a language model to write the answer. In customer support, RAG keeps replies tied to the business's current policies and help content instead of the model's general training data.
• Fallback rateFallback rate is the percentage of chatbot messages or conversations where the bot could not produce a confident answer and fell back to a default response, such as “I'm not sure” or an offer to contact a person. It measures how often customers ask things the bot's knowledge or rules do not cover.
• Knowledge gapsKnowledge gaps are the customer questions that a support team's content, and therefore its AI agent, cannot answer. They show up as fallbacks, low-confidence answers, escalations and repeated rephrasing. Tracking them tells you exactly which help articles to write or fix next, making them one of the most actionable metrics in AI support.
• Prompt injectionPrompt injection is an attack in which text supplied to an AI system, typed by a user or hidden in content it reads, tries to override the system's instructions, for example to reveal confidential information or take unintended actions. OWASP lists it as the top risk for applications built on large language models.