Skip to main content
PromptQuorumBuilt for humans. Structured for AI.
Home/Prompt Engineering
Fundamentals

AI Hallucinations: Why AI Makes Things Up — and How to Stop Them

·12 min readBy · Founder of PromptQuorum · Discovery engine for open-weight & open-source AI

AI hallucinations are fluent but false statements that language models produce because they predict likely text instead of looking up facts. Detect them by checking sources and comparing answers; reduce them with grounding, clear constraints and human review.

Key Takeaways

  • What hallucinations are: Confident false statements generated by LLMs that violate reality—invented sources, wrong facts, fabricated details—delivered fluently as if accurate
  • Why they happen: LLMs predict text patterns rather than retrieve facts; they lack a database to verify against and no internal confidence mechanism to signal uncertainty
  • How to detect them: Check every citation, name, date and number against a primary source; re-ask the question or compare several models (disagreement is a warning sign); ask the model to verify its own draft with separate fact-check questions
  • Consensus scoring: Send the same prompt to several independent models; a claim that appears in only one response is suspect and needs verification
  • Mitigation strategies: Use RAG (grounding in verified documents), provide explicit sources, set constraints ("only use information from this document"), use multi-model consensus, and require human review for high-risk claims

Large language models state false things fluently and with confidence. These errors, called hallucinations, range from invented citations to fabricated facts.

Visual Summary: AI Hallucinations: Why AI Makes Things Up — and How to Stop Them

Prefer slides over reading? Click through this interactive presentation covering all key concepts, settings, and use cases — then save as PDF for reference.

The slide deck below covers: why LLMs hallucinate (probabilistic token prediction, not comprehension), 7 hallucination types with detection patterns, prompt techniques that reduce hallucination, and a comparison of model behavior. Download the PDF as a hallucination reduction reference card.

Download AI Hallucinations: Why AI Makes Things Up — and How to Stop Them Reference Card (PDF)

What Are AI Hallucinations?

📍 In One Sentence

Hallucinations are fluent false statements that language models generate because they predict text patterns rather than retrieve facts from a reliable store.

💬 In Plain Terms

The model is not lying — it has no idea it is wrong. It is finishing your sentence the way the training data suggests, and a confident-sounding wrong answer scores just as well as a right one. That is why fabricated sources read exactly like real ones.

An AI hallucination is a factually false or fabricated statement generated by an LLM with apparent confidence. The model delivers text that violates reality—wrong names, invented sources, impossible dates, fictional URLs—using the same fluent language as accurate information.

This differs fundamentally from a model expressing uncertainty. Hallucinations are characterized by confident, detailed assertions about things that don't exist or events that didn't happen. A model might cite a paper published in a journal that doesn't exist, invent biographical details, state a historical date that is off by centuries, or describe a product feature that was never built. The user reads it, assumes the clear language signals accuracy, and acts on it—only to discover the information is fabricated.

Why Language Models Hallucinate

LLMs work by predicting the next word in a sequence. They don't consult a database or verify facts against ground truth. They calculate probabilities based on patterns in training data. This core design—highly effective for language tasks—inherently creates pressure to hallucinate.

The Core Mechanisms

  • Next-token prediction, not truth retrieval. The model's architecture is optimized for language generation, not fact verification. When a prompt asks a question, the model's goal is to produce a coherent, plausible continuation of text. Coherence and truthfulness are not the same thing. A false statement can be far more coherent than an admission of uncertainty.
  • Training and evaluation reward guessing. A 2025 research paper argues that hallucinations persist because model training and the benchmarks used to rank models reward a plausible guess more than an honest "I don't know" (Kalai et al., 2025).
  • Training data gaps and expiration. Models train on data with a specific end date. Information gaps—topics the model never encountered during training, recent events after the training cutoff, specialized knowledge in narrow domains—create voids. When asked about these gaps, the model lacks truthful patterns to predict from. It invents plausible-sounding details rather than saying "I don't have this information."
  • No explicit confidence mechanism. Models don't generate a confidence score alongside each output. They produce text without an internal signal that says *"I'm 30% sure about this claim."* The pressure to fill the page with output overrides the option to signal doubt or refuse the request.
  • Pressure from prompts that demand answers. Prompts like "Explain everything about topic" or "List all the reasons reason statement" implicitly communicate: *you must answer, even if unsure*. The model responds by inventing details to satisfy the request. It treats the pressure to be helpful as more important than the risk of being inaccurate.
  • Limited context window and information loss. LLMs can hold only a finite amount of context in memory. Long documents or conversations cause earlier details to fade. The model may forget what was said in an earlier section, invent or misremember it, and confidently assert the fabrication as if it were consistent with the earlier context. See our guide on context windows for why this happens and how token limits affect output reliability.
  • Confabulation in multi-step reasoning. For problems requiring multiple reasoning steps, the model can lose track of intermediate results. It may invent a supporting step to justify a conclusion, or skip a step and jump to a false conclusion while generating text that sounds logically sound. Understanding how tokens and costs scale with long reasoning chains helps you balance accuracy against efficiency.

Common Types of Hallucinations

Hallucinations come in recognizable patterns. Identifying the type helps you target mitigation strategies.

Type
Example
Why It Happens
Severity
Invented sourcesCitation to a peer-reviewed paper that doesn't exist; fake author names and publication yearsModel was trained on millions of citations and likely learned citation-like patterns, then invents new onesVery High
Wrong facts (dates, numbers, names)Stating a historical event in the wrong year; incorrect biographical detailsTraining data is incomplete or conflicting; model picks a plausible-sounding numberVery High
Fabricated URLs and emailsProviding a link or email address that doesn't resolve or belong to the claimed organizationModel learned URL and email patterns and generates new ones that sound realistic but are fictionalHigh
Lost contextAnswering a question as if the model understood earlier context, when it actually lost track of itContext window is finite; long documents cause earlier details to fade from the model's attentionHigh
Role driftStarting in one role (analyst) and gradually shifting to another (storyteller), inventing details to fill gapsModel loses track of the initial instruction and defaults to pattern-matching on text aloneMedium
Overconfident generalizationStating "All X do Y" when only the specific training examples show thisModel generalizes too broadly from limited training data without confidence-checkingMedium
Internal contradictionStating opposing facts within the same responseModel doesn't have a mechanism to track consistency across multiple sentencesMedium

The seven types of hallucinations are: invented sources, wrong facts, fabricated URLs and emails, lost context, role drift, overconfident generalization, and internal contradictions.

Researchers also separate intrinsic hallucinations (the output contradicts the source text or prompt the model was given) from extrinsic ones (the output adds claims the source cannot support) (Maynez et al., 2020). The NIST Generative AI Profile (NIST AI 600-1) lists the same behavior as a risk under the name "confabulation".

How Prompt Design Affects Hallucination Risk

Your prompts either encourage or discourage hallucinations. Prompt wording changes how often a model guesses; test it on your own task.

Prompts that increase hallucination risk:

  • "Tell me everything about topic" — no limits, no permission to say "I don't know"
  • "Make sure to include many details" — explicit pressure to fill space with invented information
  • "Write as if you're a leading expert" — encourages authoritative-sounding assertions, even for uncertain claims
  • "Answer even if you're not completely sure" — removes the brake on hallucination

Prompts that reduce hallucination risk:

  • "You may say 'I don't know' if unsure" — explicit permission to admit knowledge gaps
  • "Only use information from the context below" — constrains the answer to provided data, not invented knowledge
  • "Separate facts from opinions. Mark uncertain claims UNCERTAIN" — forces the model to differentiate
  • "Cite your source for any factual claim" — makes invented citations obviously visible
  • "If you can't verify this claim, do not include it" — explicit constraint on unverified assertions

Good Prompt Structure

Good prompts combine four elements: a clear role or context (what frame should the model adopt?), a specific task (what output do I need?), real input data (what information is provided?), and explicit constraints (what must the model NOT do?). This structure reduces the pressure to hallucinate by removing ambiguity about what the model is supposed to do. Our guide on the 5 building blocks every prompt needs covers each element with examples.

See the full definition of prompt engineering fundamentals for a deeper exploration of how structure affects output reliability.

AI Hallucination Prevention: Techniques to Reduce Hallucinations

Three complementary approaches reduce hallucinations: - Prompt-level: Add constraints and permission to admit uncertainty in your prompts - System-level: Use RAG, function calling, or retrieval to ground answers in real data - Model-level: Run the same prompt across multiple independent models to detect hallucinations by consensus

1. Explicit Permission to Say "I Don't Know"

Tell the model: "If you're unsure or don't have information, say so. Do not guess."

This removes the pressure to invent answers. Many models are trained to be helpful and will attempt to answer even when completely unsure. Explicitly releasing them from that expectation gives them permission to decline.

2. Ask for Sources or Evidence

Request: "Cite the source for any factual claim" or "Provide the URL and publication date for each reference."

This makes invented citations obvious (they won't resolve or will point to nonexistent sources) and forces the model to be more careful about asserting facts. It also gives you a way to fact-check the output: click each link, verify each source.

3. Self-Critique and Contradiction Checking

Ask the model to review its own output:

> "After you finish your answer, review it for any contradictions or claims that contradict something you said earlier. Point out any inconsistencies you find."

Self-review can catch some inconsistencies, but a model may also repeat its own mistake, so check the result against a source.

4. Use Negative Instructions

Explicitly state what the model must NOT do:

  • "Do not invent sources, URLs, or author names under any circumstances"
  • "Do not guess about dates if you're unsure—leave the date blank instead"
  • "Do not add information that is not in the provided context"

Negative framing sometimes works better than positive framing for preventing specific errors.

5. Step-by-Step Reasoning with Verification

For complex tasks, ask:

> "Work through this step by step. After each step, verify that the previous step is correct before proceeding to the next step."

Breaking the task into smaller chunks with verification steps gives the model chances to catch inconsistencies before they compound.

6. Structured Output Format with Evidence Section

Ask the model to separate answer, reasoning, and evidence into distinct sections:

``` ANSWER: Direct response

REASONING: How you arrived at this

EVIDENCE: Sources, facts, or citations supporting this

CONFIDENCE: How sure are you, and why? ```

This structure makes hallucinations easy to spot: unsupported claims will have empty or vague EVIDENCE sections, and low CONFIDENCE values.

System-Level Strategies Beyond Prompt Design

Prompts alone are not sufficient for high-stakes work. Add these tools and workflows.

  • Retrieval-Augmented Generation (RAG). Feed the model a specific document, knowledge base, or dataset and ask it to answer only using that content. This grounds answers in real data instead of the model's training data and reduces hallucinations about information the model would otherwise have to guess; it does not remove them, because the model can still misread or misquote the retrieved text. Tools like LangChain and vector databases implement this pattern. See our full guide on RAG: how to ground AI answers in real data.
  • Tool calling and function use. Let the model call external functions for calculations, database lookups, or fact-checking. Instead of inventing a statistic, the model calls a function to retrieve it. This removes the need to guess the facts the function returns, but the model can still misread or misuse the result.
  • Human review and expert verification. For critical decisions—medical, legal, financial, safety-critical—always have a human (preferably an expert) verify AI-generated answers. No prompt technique replaces expert judgment.
  • Automated fact-checking workflows. Run model outputs through automated systems (fact-checking APIs, URL validation, citation verification) before showing them to users. This catches hallucinations at scale without manual review of every output.

How to Identify and Detect AI Hallucinations

No single check catches every hallucination, so combine several. Warning signs are very specific details with no source, citations that look right but do not resolve, and answers that change when you rephrase the question.

Use these methods, from cheapest to most thorough:

  • Verify the checkable details. Open every cited URL and paper, and check names, dates and numbers against a primary source. A source you cannot find is a strong sign of invention.
  • Re-ask and compare. Ask the same question several times or to different models. Facts a model knows tend to repeat; invented details tend to vary. The SelfCheckGPT method scores this consistency automatically (Manakul et al., EMNLP 2023).
  • Make the model fact-check its draft. In Chain-of-Verification, the model drafts an answer, writes verification questions, answers them separately and then revises; the authors report fewer hallucinations on the tasks they tested (Dhuliawala et al., 2023). Example prompt: "Write three questions that would expose errors in your draft, answer each one separately, then revise the draft."
  • Match quotes to the source. When the model answers from a supplied document, require direct quotes and check by text search that each quote appears in the document.
  • Test in production. Validate URLs and package names automatically, keep prompts that have failed before as regression tests, and route high-stakes answers to a human reviewer.

Multiple Models and Consensus Detection

A single model can hallucinate confidently. But when you ask several independent models the same question, they often disagree on hallucinated claims.

If several models independently give the same answer, that answer is more likely to be correct than if one model answers alone. If only one model claims something and the others do not mention it, treat that claim as suspect and verify it.

This is the principle behind consensus scoring: sending the same prompt to models from different vendors and analyzing where they agree and disagree.

Consensus Checking Across Models

Consensus checking applies the re-ask idea across different models: the same prompt goes to several independent models and you compare the answers. Claims that every model repeats are more likely to be right; a claim only one model makes needs verification.

Agreement is a signal, not proof. Models trained on similar data can share the same mistake, so check anything high-stakes against a primary source.

PromptQuorum automates the dispatch and comparison: send one prompt to multiple AI models at once and see which claims agree and which diverge.

Hallucinations and Regulation

Regulators increasingly expect organizations to manage inaccurate AI output. In the EU, Article 15 of the AI Act requires high-risk AI systems to reach an appropriate level of accuracy, robustness and cybersecurity throughout their lifecycle. In the US, the NIST Generative AI Profile (NIST AI 600-1) lists confabulation among the risks organizations should manage.

Which rules apply depends on your system and use case; this is general information, not legal advice. Regardless of region, the core techniques (grounding, consensus checking, human review) remain applicable.

Real-World Cases of AI Hallucinations

Hallucinations have already had legal and security consequences. Three documented examples:

  • Invented court citations (2023). In Mata v. Avianca, lawyers filed a brief citing court opinions that ChatGPT had made up. On June 22, 2023, a federal judge in New York fined the two lawyers and their firm $5,000 (LawNext report).
  • Wrong chatbot answer about a refund policy (2024). Air Canada's website chatbot told a customer he could claim a bereavement fare retroactively, which the airline's policy did not allow. A British Columbia tribunal held the airline responsible for its chatbot's answer in Moffatt v. Air Canada (decision 2024 BCCRT 149).
  • Software packages that do not exist. A study of 576,000 code samples from 16 models found that, on average, at least 5.2% of the packages suggested by commercial models and 21.7% of those suggested by open-source models did not exist; an attacker could register such a name and ship malicious code (Spracklen et al., 2024).
  • Invented SQL and API details. The same failure can occur in SQL and API code: tables, columns or functions that sound plausible but are not in your schema. Paste the real schema or API reference into the prompt and run generated code against a test database first.

When Hallucinations Are Most Dangerous

Hallucinations risk significant harm in specific domains. This article is general information, not medical, legal or financial advice; ask a qualified professional before acting on AI output in these areas. Be especially cautious with:

  • Medical and health decisions — Wrong drug names, dosages, or symptom interpretations can harm patients
  • Legal and compliance — Invented case law, regulatory requirements, or precedent can lead to costly mistakes or violations
  • Financial advice — False market data, incorrect tax information, or fabricated performance metrics mislead high-stakes decisions
  • Safety-critical systems — Hallucinations in code review, architecture decisions, or security analysis can introduce vulnerabilities or bugs
  • Public attribution — Anything published under your name or brand must be fact-checked; hallucinations damage credibility

Critical principle: Even with perfect prompts and consensus checking, human verification remains essential for high-stakes decisions. Use AI as a time-saver and first-pass tool, not as a replacement for expert judgment or primary source verification.

Learn how self-critique techniques can further reduce errors in complex reasoning tasks.

Practical Checklist: Before You Send a Critical Prompt

Use this checklist before sending a prompt you'll rely on for decisions or public-facing output:

  • Does the prompt explicitly permit "I don't know"? Add: "You may say 'I don't know' if unsure."
  • Is there real context or data in the prompt? Vague prompts invite invention. Provide specific documents, examples, or input data.
  • Are constraints explicit? State what the model must NOT do, especially: "Do not invent sources, URLs, or citations."
  • Is the output format structured? Separate Answer / Reasoning / Evidence / Confidence. This makes unsupported claims obvious.
  • Are you asking for sources? For any factual claim, require: "Cite the source for this fact."
  • Is the task specific, not open-ended? "List five marketing strategies *for a B2B SaaS product targeting finance professionals*" is better than "Tell me about marketing."
  • Have you asked the model to self-check? Add: "Review your answer for contradictions before submitting it."
  • For high-stakes decisions, are you cross-checking? Send the same prompt to multiple models and compare answers.

Before / After Prompt Example

Bad Prompt

Why this fails: Open-ended, no constraints, no permission to admit uncertainty. The model will invent dates, misattribute breakthroughs, confidently state outdated information, and potentially cite papers that don't exist.

Tell me about the history of artificial intelligence. Include major breakthroughs and important researchers.

Good Prompt

Using only the following timeline, summarize the major breakthroughs in AI from 1950 to 1990: {TIMELINE DATA INSERTED} Format your answer as: BREAKTHROUGH: {Name} YEAR: {Year — only if stated in timeline} SIGNIFICANCE: {What it enabled} SOURCE: {Which document from the timeline mentions this?} Do not add information not in the timeline. If you're unsure whether something is in the timeline, skip it rather than guessing.

Why this works:

  • Real data instead of invention: The model works from provided context, not from training data gaps
  • Structured output: The format makes missing sources immediately obvious
  • Negative instruction: "Do not add information not in the timeline" is explicit
  • Permission to omit: "If unsure, skip it" releases pressure to invent details
  • Source accountability: Every claim requires citing which document it came from

How to Reduce AI Hallucinations: 6 Prompt Techniques

  1. 1
    Give explicit permission to say "I don't know": Add to every factual prompt: "If you are not certain, say 'I don't know' rather than guessing." This reduces one class of confident fabrications, but does not remove them.
  2. 2
    Ask for sources or evidence: Instruct the model to cite a source for each claim, or flag claims it cannot verify. Example: "After each fact, note either the source name or 'unverified'."
  3. 3
    Add a self-critique step: Ask the model to review its own output before finalising. Example: "Before answering, list any claims you are less than 90% confident in."
  4. 4
    Use negative instructions: Explicitly prohibit hallucination-prone behaviour. Example: "Do not invent names, titles, or statistics. Do not extrapolate beyond the provided context."
  5. 5
    Require step-by-step reasoning: Chain-of-thought prompting forces the model to show its work, making errors detectable before they reach the final answer.
  6. 6
    Dispatch to multiple models with PromptQuorum: Send the same prompt to GPT-5.6, Claude, and Gemini simultaneously. Facts confirmed by all three are more likely to be right (agreement is a signal, not proof); diverging answers flag claims needing verification.

Frequently Asked Questions

Can hallucinations be completely eliminated?

No. Hallucinations are inherent to how language models work—they predict text patterns rather than retrieve facts from a verified store. You can reduce them significantly with good prompt design, tools like RAG, and multi-model consensus, but eliminating them entirely is not possible given current LLM architecture. Human verification remains necessary for high-stakes decisions.

Why does the model sound so confident when it's wrong?

Language models are trained to generate fluent, coherent text. Confidence is a side effect of linguistic coherence. A false statement can be far more coherent and well-articulated than an honest admission of uncertainty. The model has no built-in mechanism to express doubt—it produces text with the same fluent confidence regardless of accuracy.

Do newer, larger models hallucinate less?

Not reliably. Hallucination rates depend on the model, the task and how they are measured, and published comparisons disagree, so test your own use case instead of assuming a newer or larger model is safer. Newer models often follow instructions such as "say I don't know if unsure" better, but none removes the problem.

Does connecting a model to the internet remove hallucinations?

Partially. Real-time web access helps with current events and recent facts, but it doesn't solve the core problem: the model still invents details, misinterprets search results, or hallucinates about content it didn't actually retrieve. Internet access is a tool that reduces one class of hallucinations, not a cure.

How can I quickly check whether an answer is hallucinated?

Check sources: Click URLs or search for cited papers. If they don't exist, treat the answer as unreliable and verify it. Verify facts: Check dates, names, and numbers against trusted sources. Compare multiple models: Ask the same question to different models. Sharp disagreement suggests at least one is hallucinating. Apply domain expertise: If you know the field, read critically for subtle implausibility—hallucinations often have tell-tale signs to expert readers.

Should I stop using AI if it hallucinates?

No. Use AI strategically with verification. For brainstorming, drafting, and exploratory work, hallucinations are a minor annoyance. For fact-critical work (research, compliance, medical decisions, financial advice), use AI as a starting point, then verify everything with trusted sources or expert review.

What's the difference between a hallucination and a genuine mistake?

A hallucination is confident and false. If a model says "I'm not sure, but it might be X," that's honest uncertainty, not hallucination. If it says "The capital of France is Berlin" with full confidence, that's a hallucination—the model stated something false as if it were fact. The hallmark is confident assertion of something untrue.

What should I do when an AI gives a confident answer without evidence?

Treat it as potentially incorrect and verify it against reliable evidence before you use or publish it. A confident tone says nothing about accuracy. Ask for sources, open each one, and compare the claim with a primary source or a second model.

Is "confabulation" the same as hallucination?

In practice, yes: both words describe a model producing fluent content that is false or unsupported. "Hallucination" is the common term; the NIST Generative AI Profile (NIST AI 600-1) uses "confabulation" for the same risk.

Why does AI invent code, package names and SQL?

The model predicts code that looks like the code it has seen, so it can produce function calls, packages or table names that fit the pattern but do not exist. In a 2024 study of 16 models, on average at least 5.2% of packages suggested by commercial models and 21.7% of those suggested by open-source models did not exist. Give the model your real schema or API reference, install only packages you have checked, and test generated code before use.

Sources

  • Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). "Why Language Models Hallucinate." arXiv — argues that training and evaluation reward guessing over admitting uncertainty.
  • Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020). "On Faithfulness and Factuality in Abstractive Summarization." ACL Anthology — empirical study of hallucination in neural summarization; source of the intrinsic and extrinsic distinction.
  • Manakul, P., Liusie, A., & Gales, M. J. F. (2023). "SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models." arXiv — detects hallucinations by checking consistency across sampled responses.
  • Dhuliawala, S., Komeili, M., Xu, J., et al. (2023). "Chain-of-Verification Reduces Hallucination in Large Language Models." arXiv — a draft, verify and revise method that reduced hallucinations in the authors' tests.
  • Spracklen, J., Wijewickrama, R., Sakib, A. H. M. N., et al. (2024). "We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs." arXiv — measures how often code-generating models suggest packages that do not exist.
  • Wei, J., Wang, X., Schuurmans, D., et al. (2022). "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." arXiv — shows that step-by-step reasoning prompts improve multi-step reasoning performance.
  • National Institute of Standards and Technology (2024). "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1)." DOI — lists confabulation among the risks of generative AI.
  • European Union (2024). "Artificial Intelligence Act, Article 15: Accuracy, robustness and cybersecurity." AI Act Service Desk — accuracy, robustness and cybersecurity requirements for high-risk AI systems.
  • LawNext (2023). "Court imposes sanctions on lawyers who filed bogus cases after relying on ChatGPT for legal research." LawNext — report on the June 22, 2023 sanctions in Mata v. Avianca.
  • Civil Resolution Tribunal of British Columbia (2024). "Moffatt v. Air Canada, 2024 BCCRT 149." Decision — tribunal decision in Moffatt v. Air Canada.

Apply these techniques with a local LLM or your own API keys — PromptQuorum works with any backend.

Try PromptQuorum free →

← Back to Prompt Engineering

MCP