AI Invented a Law That Doesn't Exist: Hallucinations Explained

"AI Invented a Law That Doesn't Exist" ⚖️😱

A lawyer uses ChatGPT to prepare a plea. The subject is technical: a labor law dispute, with complex procedural nuances. The lawyer, convinced that AI can accelerate their research, asks directly: "Cite precedents for this case."

ChatGPT produces 3 Court of Cassation rulings, complete with dates, chamber numbers, and detailed summaries. The references are presented with total confidence, in impeccable legal format. The lawyer is relieved: this is exactly what they needed.

They include them in their brief, submit them to the judge, and plead with confidence.

The judge checks.

The rulings don't exist. They never existed. They are fictional constructs generated by a language model incapable of distinguishing the true from the plausible.

The lawyer is humiliated, their credibility destroyed, and their career seriously jeopardized. This reported incident illustrates a reality that now affects all professions using AI for documentary research: AI hallucinations inventing non-existent laws are not a rare anomaly. They are a structural feature of large language models.

Why Do AI Hallucinations Represent a Critical Risk for Your Business?

AI doesn't "lie" in the moral sense. It has no intention to deceive. It does something far more dangerous: it completes sentences following the statistical logic of what it has learned. It's trained to be plausible, not truthful. And when the most probable continuation of a sentence resembles a legal reference — because it has read thousands during training — it generates that reference with the same confidence as a real one.

This distinction is fundamental. An AI that gets something wrong doesn't know it's wrong. It has no internal signal indicating "warning, this information is invented." It produces text. That text can be true or false, and it delivers both with exactly the same assurance.

For a business, this means you cannot trust a "raw" AI — without a verification system — for tasks where factual precision is critical. The legal domain is the most extreme example, but the risk exists everywhere: financial data, medical information, product history, contractual clauses.

The Mechanism of Hallucinations: Why LLMs Invent Legal References

To understand why AI hallucinations involving non-existent laws occur, you must understand what a large language model actually is. An LLM is a system trained to predict the next word in a sequence, based on billions of texts. It has no internal database of "truths." It has statistical weights encoding the probability that one word follows another, given a specific context.

When you ask it to cite a case ruling, it doesn't consult Westlaw or LexisNexis. It calculates which tokens (word fragments) form a statistically coherent sequence with your request. The problem is that legal texts have a very specific and recognizable structure: "Smith v. Jones, 2nd Cir. 2021, No. 20-1234." This structure has been seen thousands of times during training. The model can reproduce it perfectly, with a number that resembles a real case number, a plausible date, a court that makes sense — without any real case existing behind it.

This is what researchers call confabulation: the model produces a response that is coherent on the surface, constructed by interpolation between patterns it has memorized, with no anchor in verifiable reality. The more structured and technical the domain (law, medicine, finance), the more convincing the hallucinations, because the model has learned to imitate that structure with precision.

There is also a phenomenon researchers call the need for "sounding authoritative." LLMs are trained with human feedback (RLHF) that rewards responses that seem confident and complete. A response that says "I don't know" is rated lower by human evaluators than a response providing a precise reference, even if that reference is invented. The model has learned that being precise and confident is rewarded — even when that precision is fictitious.

Flowchart illustrating how an LLM moves from a legal query to an invented legal reference, and how RAG breaks this cycle

Mata v. Avianca: The Case That Shook the Global Legal Profession

On June 22, 2023, federal judge P. Kevin Castel issued a landmark ruling in Mata v. Avianca, Inc. (No. 22-cv-1461, S.D.N.Y.). Attorney Steven Schwartz, of the firm Levidow, Levidow & Oberman, had used ChatGPT to research legal precedents in an in-flight injury case.

ChatGPT had provided multiple case references, presented with complete citations and detailed summaries. These references included cases with perfectly plausible names and numbers, such as "Varghese v. China Southern Airlines," "Shaboon v. EgyptAir," and "Martinez v. Delta Air Lines." Schwartz had submitted these citations in his court filings without verifying them.

When opposing counsel and the court tried to locate these cases, they proved to be nonexistent. Judge Castel summoned the attorneys, asked them to explain their approach, and imposed a $5,000 sanction along with a severe public reprimand. In his ruling, he wrote that the attorneys had "abdicated their responsibility to the court" and had introduced "entirely fictitious" citations into official court records.

What makes this case particularly instructive is Schwartz's initial reaction: when questioned about the authenticity of the cited cases, he had re-confirmed their existence with ChatGPT, which once again asserted they were real. The model couldn't admit its error — it simply had no way of knowing it had hallucinated.

Similar Incidents Reported Across Multiple Countries

The Mata v. Avianca case is not isolated. Since 2023, incidents involving AI-invented legal references have been reported across multiple jurisdictions. Regulators in Australia, the United Kingdom, and Canada have issued formal warnings to legal practitioners regarding unsupervised use of LLMs in legal research.

In Australia, the Bar Council issued an urgent guidance note in 2024 after several reported cases of practitioners citing nonexistent decisions in proceedings. Professional associations in the United Kingdom issued similar recommendations, stressing that AI use in legal contexts must always be subject to independent human verification.

In France, while cases brought before courts remain little publicized, the Conseil National des Barreaux has begun formulating guidelines on AI use by lawyers. This challenge is not uniquely French — it's a systemic problem linked to the fundamental nature of LLMs.

Sequence diagram showing the verification process for an AI-provided legal reference, from the request to validation or hallucination alert

The 3 Types of Hallucinations That Kill Businesses

Before diving into solutions, it's important to understand that not all hallucinations look the same. Three distinct families can be identified, with different danger levels.

❌ Pure Invention

This is the most serious type: AI creates from scratch information that has no real basis whatsoever. In law, this produces nonexistent rulings, laws never enacted, fictional authors for real scholarly articles. In commerce, it can take the form of an invented product with its specifications, a nonexistent price, or a return policy never defined.

An e-commerce chatbot that invents a product "iPhone 15 Pro Max Ultra" at €999 when this model doesn't exist — and a customer orders based on that information — exposes the company to commercial claims and severe trust damage.

❌ Fact Confusion

Less spectacular but just as dangerous: AI mixes real information from different sources. It associates the wrong client with the wrong order, confuses two similar laws covering different matters, or blends two case rulings into an incorrect synthesis.

In a professional context, this type of hallucination can lead to sending confidential data to the wrong person, or drafting a contract with clauses drawn from case law applicable to a completely different sector.

❌ False Logic

This type has become rarer with recent models but persists in complex calculations or multi-step legal reasoning. AI may chain together propositions that appear logically coherent but whose conclusion is incorrect — for example, misinterpreting a contractual clause by ignoring a statutory exception.

Hallucination vs. Factual Error: A Critical Distinction

It's important to distinguish hallucination from simple factual error, because mitigation strategies differ.

A factual error occurs when AI gets a real, existing fact wrong — a slightly incorrect historical date, an approximate city population, the name of an executive who has changed since the model's training cutoff. These errors are often correctable by a simple search, and the model can acknowledge its error when provided with the correct information.

A hallucination, by contrast, is the invention of information that doesn't exist. The fundamental difference is that you cannot "correct" a hallucination by searching for it: you'll find nothing, because there is nothing to find. And if you ask AI to verify its claim, it may confirm its hallucination with the same confidence it produced it.

The Mata v. Avianca case perfectly illustrates this distinction: when the attorney asked ChatGPT to confirm the existence of the cited cases, the model responded positively. Not out of malice, but because it had no internal mechanism to detect that its own outputs were fictitious.

The Legal Domain: Ideal Ground for Hallucinations

Why is law particularly vulnerable to AI hallucinations? Because it combines several characteristics that favor LLM confabulation.

First, the highly formalized structure of legal documents. A Supreme Court ruling always follows the same format: court, chamber, date, number. An LLM that has ingested thousands of such documents can reproduce this format perfectly, filling in the fields with plausible but invented values.

Second, the density of technical information. Law is a domain where authority rests on the precision of references. A citation that resembles a real citation will be treated as such by a non-vigilant practitioner, especially under time pressure.

Third, the difficulty of verification for non-specialists. A client receiving a legal document with case citations generally lacks the tools or skills to verify those references. Verification falls on the practitioner — who may themselves trust the AI.

Fourth, asymmetric consequences. A medical error can kill a person. A legal error can ruin a company, cost millions in damages, or lead to an unjust conviction. The cost of a legal hallucination is disproportionate to the time saved.

Bar chart illustrating the domains where LLMs hallucinate most frequently — law and case law rank highest

When NOT to Use AI for Legal Work Without Supervision

The golden rule is simple: AI without supervision is unacceptable for any task where a factual error or inaccurate reference can have significant legal, financial, or professional consequences.

Specifically, never use AI alone for: citing jurisprudence precedents (unless connected to a verified database), interpreting statutory texts and their scope, drafting contractual clauses without review by a qualified jurist, providing legal advice to a client, or preparing materials for court proceedings.

On the other hand, AI can be used with relative confidence for: analyzing documents you have yourself provided (contracts, meeting minutes), reformulating existing texts, generating first drafts that will be entirely reviewed and validated, or summarizing documents whose authenticity you have already verified.

The key distinction is this: AI is a powerful tool for processing information you give it, but it becomes dangerous when it must produce information from its training memory in high-stakes domains.

Decision flowchart: which legal tasks can be delegated to AI and with what precautions

Legal and Professional Consequences

The consequences of an undetected legal hallucination can be devastating at multiple levels.

On the disciplinary level, bar associations in many countries have begun examining the responsibility of lawyers who use AI without sufficient verification. Citing knowingly — or through negligence — false references constitutes a breach of fundamental professional duties. In most jurisdictions, the rules of professional conduct impose a duty of diligence that includes verifying sources used in procedural documents.

On the civil level, if a client suffers harm due to a legal error caused by an AI hallucination, the practitioner's liability may be engaged. The defense "AI made a mistake" is not legally recognized: the professional bears responsibility for their work regardless of the tool used.

On the criminal level, in extreme cases — particularly if false references are used to mislead a court — prosecutions for contempt or evidence tampering are not excluded in certain jurisdictions.

And beyond the strict legal framework, there is reputation. The Mata v. Avianca case was covered by hundreds of media outlets worldwide. The names of the sanctioned attorneys are now permanently associated with this error. In a profession where credibility is the primary asset, this can be a worse punishment than the fine.

The Professional Solution: Grounding

You cannot prevent an LLM from hallucinating as a fundamental mechanism. But you can prevent hallucinations from reaching the final output by implementing robust verification systems. This is called grounding — anchoring the model in verified sources.

Comparison between raw AI (invents answers) and grounded AI (RAG + self-verification + source citation): the difference between business risk and total reliability

✅ 1. Limit Context (RAG)

RAG (Retrieval Augmented Generation) is the most effective technique against legal hallucinations. The principle is simple: before generating a response, the AI searches a verified document base and only draws on those documents to build its answer. If the information is not in the base, it says it doesn't know.

For law, this means connecting AI directly to Westlaw, LexisNexis, EUR-Lex, or up-to-date case law databases. The model first searches these sources, then generates its response solely from the documents found, citing their exact source. The hallucination rate for cited references then drops to near zero.

Implementation:

prompt = f"""
Available context:
{context}

Question: {question}

Instructions:
- Answer ONLY using the context above
- If the answer is not in the context, answer "I don't know"
- Never invent information
"""

✅ 2. Verification (Self-Reflection)

A second layer of protection involves asking the model to verify its own response. After generating an answer, you submit the source context and the produced response, asking whether each stated fact is actually present in the context. This self-verification doesn't replace human verification, but it filters out some of the most glaring hallucinations.

Implementation:

# First response
response = llm.generate(question, context)

# Verification
verification = llm.generate(f"""
Verify this response:
{response}

Source context:
{context}

Is each fact present in the context? YES or NO.
""")

if "NO" in verification:
    return "I cannot answer with certainty."

✅ 3. Source Citation

Requiring the model to cite its source for every assertion is a simple but powerful technique. When AI must indicate where it found the information, it cannot hallucinate without this becoming immediately visible during verification. If it cites "Page 12 of document X" and you look at page 12 of that document, you'll immediately see whether the information is real or invented.

Implementation:

prompt = f"""
Context:
[Page 12] Delivery time is 3-5 business days.
[Page 45] Returns are free within 30 days.

Question: {question}

Answer by citing the exact source: [Page X]
"""

The Truth About Hallucination Rates

Raw AI hallucinates 15-20% of the time on precise facts. A properly grounded AI reaches 0% error rate (bank client case: zero errors in 6 months)

Marketing demos show you a perfect AI. Reality is different. A "raw" AI — without RAG, without verification, without context constraints — hallucinates approximately 15% to 20% of the time on precise facts, dates, references, or figures. This percentage may seem low until you realize what it means in practice.

If you use an ungrounded legal chatbot to research ten precedents, two or three of them may be fictitious. If you draft a contract relying on ten AI-generated clauses, one or two may contain nonexistent legal references. In a sector where precision is non-negotiable, a 15% error rate is catastrophic.

The RAG (Retrieval Augmented Generation) technique is the most effective solution to reduce this rate. To learn more about AI hallucinations and how to prevent them at the enterprise level, also see our complete guide on avoiding AI hallucinations.

Reducing this rate to 0% for a business doesn't happen in 5 minutes. It requires architecture work, fine-tuning, and continuous validation. But it is the sine qua non condition for seriously using AI in sensitive professional contexts. An AI that lies once in a hundred is unusable in business. A grounded AI is a decisive asset.

Real Case: A Bank That Eliminated 100% of Errors

To illustrate the effectiveness of grounding techniques, here is a production deployment example. A customer support chatbot for a banking institution had initially been deployed without RAG or context constraints.

At first: It invented attractive interest rates. It cited promotional offers that didn't exist. It gave contradictory information on credit terms. Customers who relied on this information to make financial decisions encountered unpleasant surprises when signing. The reputational and legal impact was potentially severe. For more real-world cases, read our analysis of the 99% of companies making the AI supervision mistake.

After implementing Grounding:

→ It responds only with official rates extracted from the real-time database.

→ If it cannot find the exact information in the database, it escalates to a human advisor rather than improvising.

→ Every response cites the exact source of the information provided.

Zero errors in 6 months. Not because AI became perfect, but because the system is designed so that its potential errors can never reach the final customer.

AI Act and Liability: The New Rules of the Game

The European Union adopted the AI Act in 2024, with progressive enforcement through 2027. This legislation introduces significant obligations for AI systems used in high-risk contexts — and the legal domain is explicitly among them.

For high-risk AI applications, the AI Act mandates transparency (informing users they are interacting with AI), traceability (documenting system decisions), robustness (ensuring the system functions correctly even with unexpected inputs), and human oversight (providing mechanisms for humans to control and correct system outputs).

For companies using AI in legal, administrative, or medical processes, these requirements align directly with grounding best practices. A RAG system with human verification and source traceability is well positioned to meet AI Act obligations. A "raw" chatbot generating references without verification is, conversely, structurally non-compliant.

The question of liability is also clarified in new European directives: even if AI is at the origin of an error, it is the system operator who is responsible toward users. "The AI hallucinated" will never be a valid legal defense for a company that deployed a system without sufficient precautions.

For businesses using AI tools in their daily work, the financial risk is very real. As illustrated in our article on the hidden cost of unsecured AI templates, a single error can cost tens of thousands of euros.

Professional Best Practices: The 5-Step Verification Protocol

Faced with the risks of AI hallucinations in legal and high-stakes professional contexts, here is a verification protocol every organization should adopt.

Step 1: Never use raw AI for high-stakes tasks. Any AI used in a sensitive professional context must be configured with RAG, context constraints, and explicit instructions to signal missing information rather than invent it.

Step 2: Verify all references. Every legal citation, statistic, or factual claim produced by AI must be verified in a primary source before use. This rule admits no exceptions, regardless of how confident the AI appears.

Step 3: Require sources. Configure the prompt so that AI is required to cite its sources for every assertion. If it cannot cite a source, it must say it doesn't know.

Step 4: Systematic human supervision. Every document produced with AI assistance in a legal, medical, or financial context must be reviewed and validated by a human expert before use.

Step 5: Train users. Employees using AI must understand hallucination mechanisms and verification protocols. Blind trust in AI outputs is the primary risk factor.

These best practices don't significantly slow down work — they secure it. And in a context where regulators are beginning to require proof of due diligence in AI use, they also constitute a competitive advantage and legal protection.

The Competitive Advantage of Trustworthy AI

Beyond risk management, there is a significant competitive dimension to getting AI reliability right. Law firms, consulting firms, and professional services organizations that can credibly demonstrate their AI systems are hallucination-resistant gain a meaningful edge over competitors who cannot.

Think of it this way: two firms offer the same service at similar prices. One uses raw AI that might occasionally fabricate references. The other has implemented a full grounding pipeline with human verification and source traceability. Which one would a sophisticated client choose — especially a client who has read about Mata v. Avianca?

The trust premium attached to verified, reliable AI is real and growing. As AI becomes ubiquitous in professional services, the differentiator will not be who uses AI, but who uses it responsibly. The organizations that invest now in grounding infrastructure, verification protocols, and staff training are building a moat that raw AI adopters will struggle to cross later.

This is equally true for smaller businesses and freelancers. If you use AI in your client deliverables — content, research, analysis, legal documents — proactively disclosing your verification process and showing clients that your AI outputs are systematically checked builds credibility that competitors who blindly trust AI outputs cannot match. In professional services, the question is not "do you use AI?" but "can I trust what your AI produces?"

The answer to that question depends entirely on the systems and protocols you have in place — not on which AI model you chose.

---

Additional Resources:

🛡️ Complete Guide: AI for Everyone I explain how to configure these safeguards: anti-hallucination prompts, verification architecture, reliability tests. 👉 Access the Complete Guide

---

Has Your AI Ever Lied to You? 👇