Glue on Pizza Law in Pieces - When Everyday AI Blunders Escape the Sandbox
Sources
How short answers and sloppy guardrails turned AI hallucinations into court sanctions, corporate liability, and medical risk
The judge starts reading, and the cases look authoritative—until they don’t exist. In another room, a customer service bot invents a refund rule, and the airline insists the bot is its own legal persona. In medicine, a research model nonchalantly confuses two different parts of the human brain and marches on as if nothing happened. None of this is sci-fi. It’s paperwork, customer support, and clinical workflows in 2024–2025—and the receipts are public.
The courtroom doesn’t care that your bot was “confident”
A new, public database compiled by legal analyst Damien Charlotin tracks judicial decisions around the world where courts concluded that generative AI fabricated citations, quotes, or “authorities” submitted in filings. By mid-2025, his tally had passed 120 decisions, with a steep acceleration this year—corroborated by independent reporting that also highlights rising penalties and the shift from pro se litigants to credentialed lawyers as the primary offenders. In short: this isn’t a handful of funny goofs; it’s a measurable, global pattern with consequences.
The canonical origin story is Mata v. Avianca, where a Manhattan federal judge sanctioned two lawyers and their firm for submitting a brief laced with six imaginary cases generated by ChatGPT. The $5,000 fine was the headline; the lasting lesson was the court’s disgust at how easily fake jurisprudence slid into the record. That was 2023. Two years later, judges are still catching fabrications—now at major firms. In May, a special master hit K&L Gates and Ellis George with $31,100 in sanctions for “bogus AI research,” a figure later echoed across national outlets as proof that “everyone does it” is not a defense.
If you’re wondering whether these decisions are cherry-picked, Charlotin’s tracker (and analyses referencing it) show the breadth: U.S. federal and state courts, plus rulings from the U.K., South Africa, Israel, Australia, and Spain. And while ChatGPT is the most frequently named tool, judges increasingly note that even legal-tuned systems can produce fabricated citations when users offload verification to a machine that predicts text rather than retrieves law. Stanford researchers, for example, found general-purpose chatbots hallucinated between 58% and 82% of the time on legal queries; later work on legal-oriented tooling cautioned that even retrieval-augmented systems can still invent “correct-looking” citations.
The “be brief” tax: why asking for short answers often makes models less factual
Here’s the uncomfortable twist: new testing from the Paris-based AI safety company Giskard indicates that instructing chatbots to “keep it short” correlates with more hallucinations. Their multilingual benchmark and analysis—covered by independent tech press—found that concise prompts push models toward overly decisive, single-shot claims with less self-checking, nudging them to swap nuance for fluency. The outputs sound cleaner while the factual error rate rises. If you write policy, UI text, or prompt templates, that is not a small thing.
Why would brevity hurt truthfulness? At a systems level, reinforcement and decoding choices reward answers that lookcomplete within a tight token budget; hedges, citations, and counter-evidence get pruned first. At a human level, users treat concise answers as authoritative, which reduces follow-up scrutiny—especially in domains (law, health, finance) where over-precision should trigger more doubt, not less. Giskard’s result doesn’t mean every short answer is wrong; it means your defaults can quietly bias a system toward attractive certainty.
Outside the courthouse: when a hallucination becomes your policy
The Air Canada case is the cleanest consumer-law line in the sand so far. A tribunal ordered the airline to compensate a passenger after its website chatbot invented a retroactive bereavement refund policy that didn’t exist. The airline argued the bot was a “separate” entity. The tribunal described that argument as “remarkable” and held the company liable for what its own interface had told customers. If your customer bot features your logo at the top, it is you. Courts will not let you outsource accountability to a stochastic parrot.
Search has its own showcase of confident nonsense. Google’s AI Overviews famously suggested using glue to keep cheese on pizza, a viral emblem of how a product built to summarize “the web” can misread jokes, satire, or junk as guidance. Google pulled back, tuned, and defended; rivals even ran ads mocking the fiasco, but the broader issue remains: when you algorithmically compress messy information into a single box at the top of the screen, you convert errors into instructions. That’s not a UX quirk—it’s a liability surface.
Healthcare is the high-risk domain where “mostly right” is still dangerous. Peer-reviewed work in JAMA Network Open and other venues documents hallucinations and omission errors in clinical contexts, and a recent investigation spotlighted a Google health model that hallucinated an anatomic structure in a research example—an error later waved away as a “misspelling.” Even when models perform well on average, the tail risks are not abstract; they appear as fabricated details in notes, mis-summarized studies, or phantom facts inserted during transcription.
Defamation law is now being tested to its limits. In Georgia, radio host Mark Walters sued OpenAI after ChatGPT falsely described him as a defendant in a nonexistent case; OpenAI ultimately prevailed, but the court let the claim mature before dismissal, and other suits—like Robby Starbuck’s against Meta—signal that disclaimers alone won’t preempt every claim if plaintiffs can show negligence. The legal standard is tough, but the docket is filling.
What’s really causing this—and why the fixes keep failing
Hallucinations aren’t bugs the way a typo is a bug; they’re emergent behavior in systems that predict the next token by pattern, not by ground truth. Train them on vast textual probability and they will produce fluent, plausible content even when the underlying world has shifted, the source is wrong, or the prompt demands data that isn’t actually retrieved. Polishing the style often worsens the effect because it makes the prediction look like knowledge. This is why legal “AI assistants” can still cite cases that never existed, and why “are you sure?” guardrails help—but do not eliminate—the tendency to invent. The Stanford work quantifying error rates on legal queries is sobering precisely because it measured ordinary use, not adversarial edge cases.
There’s also a governance story. In both law and health, the riskiest failures come when organizations conflate “assistive drafting” with “authoritative source.” A retrieval-augmented system can reduce fabrication if it shows provenance and you actually verify it. A summarizer with a corporate badge that hides its chain of custody will do the opposite: concentrate error and launder it through your brand voice. Courts, regulators, and clinical ethicists are starting to speak the same language on this point, and the Air Canada ruling made the business doctrine plain: if it’s on your site, it’s your statement.
How to make the hallucination problem your competitor’s problem instead of yours
If you build or buy AI, design for verification first and eloquence second. That starts with prompts that invite sourcing rather than suppress it. Giskard’s “concise answer” effect is a quiet killer because many enterprise templates hard-code brevity as a virtue. In high-stakes contexts, that’s a vice. Ask for the chain of evidence. Budget tokens for citations, caveats, and known-unknowns. If your UX makes the model sound certain but shows no receipts, you’re engineering overconfidence—for users and auditors alike.
Then make provenance visible and testable. In law, it requires links to primary sources and runs automated Shepardization/KeyCite checks on every cited authority before a human touches “file.” In consumer UX, treat anything the bot states as policy unless it is explicitly fenced off with verifiable references and human escalation paths. In healthcare, treat LLMs as drafting tools behind the glass, never as decision engines at the bedside; evaluate them with the same discipline you’d use for a device or drug, not a clever autocomplete. The literature doesn’t promise zero hallucinations; it tells you how to box the risk in.
Finally, align incentives. The sanctions data suggest that where lawyers and firms put speed over verification, courts respond with fines, public orders, and reputational harm. That’s the point. Make diligence the fast path inside your organization—automate the checks that humans forget, and make it cheaper to do the right thing than to roll dice with a judge, a regulator, or a patient.
The verdict for Chatbots Behaving Badly readers
The Mashable takeaway—that courts have already caught well over a hundred AI hallucination incidents, and that forcing chatbots to be terser can increase error rates—is supported by primary sources and independent reporting. The legal system is establishing a pattern: fabricated authorities will cost you money, time, and credibility, regardless of how friendly your interface appears when it creates them. And outside the courts, customer service and medicine show why “confident and concise” can be the most dangerous style of being wrong.