Finnect AI Insights

India AI · · 5 min read

Can Government Lawyers Trust AI When It Fabricates Supreme Court Judgments?

Analyse the MP High Court's warning against AI-generated legal fiction and its implications for due process and AI adoption in business.

Paperwork (6475944)

A government lawyer recently faced a high-stakes petition. They needed a quick legal response. So, they used an AI tool for research.

This was not a hypothetical scenario. Reports from Hindustan Times and Live Law detailed a recent incident in Madhya Pradesh. The state government's legal team filed a reply. It cited a Supreme Court judgment that simply did not exist.

Justice Vivek Kumar Singh of the MP High Court reacted strongly. He cautioned the state government for "callous and irresponsible" drafting. He observed they had "blindly resorted" to AI tools.

The case involved police sub-inspector Piyush Mani Tiwari. He was dismissed from service within 24 hours of a sexual harassment complaint. The court quashed his dismissal, highlighting the state's disregard for constitutional due process.

But this incident is far more than a procedural error in one case. It exposes a critical vulnerability. AI tools are becoming common in professional settings, especially in sensitive sectors like law.

The court's strong rebuke raises serious questions. How reliable is AI in legal work? What is the professional duty of verification?

That raises three questions.

Question 1

What Exactly Happened in the Madhya Pradesh High Court Case Involving AI?

Live Law reported the story of police sub-inspector Piyush Mani Tiwari. He served in Bhopal. A sexual harassment complaint was filed against him on May 26, 2024. He was suspended that day. His dismissal followed on May 27, 2024. This rapid action occurred within 24 hours.

This swift dismissal used Article 311(2)(b) of the Constitution. This provision allows termination without a full inquiry under certain circumstances. Tiwari challenged this. He argued constitutional due process was bypassed.

What exactly did the state government cite? When the state government filed its reply, it cited a Supreme Court judgment. The case was "State of Andhra Pradesh vs S.S. Mohammad Nisar." Its citation was (2008) 5 SCC 230. (SCC stands for Supreme Court Cases, a widely used legal reporter.)

Justice Vivek Kumar Singh investigated. He found no such judgment existed. The actual citation (2008) 5 SCC 230 belongs to a different case. It is "Sidhartha Vashisht alias Manu Sharma vs State (NCT of Delhi)," a criminal appeal. The High Court found the state had "blindly resorted" to AI tools. Justice Singh described this practice as "reckless" and "actively misleading the Court."

The High Court quashed Tiwari's dismissal. It ordered his immediate reinstatement. The state can still conduct a regular departmental inquiry. This ruling sends a clear message. Accuracy and diligence are crucial in legal submissions. This leads us to our next question.

Question 2

How Do AI Tools 'Hallucinate' Legal Judgments and Why is This a Major Problem for the Justice System?

How do these AI tools 'hallucinate'? When we talk about AI 'hallucinating,' we are not implying it tries to mislead. Generative AI models, like those used for text, are essentially prediction machines. They learn from vast amounts of data. They predict the most plausible next word or phrase in a sequence.

These models excel at creating grammatically correct and coherent text. But their primary function is not to verify facts or ensure accuracy. Unlike human understanding, which relies on factual recall and logical reasoning, AI models operate on statistical patterns. They do not 'know' facts; they predict what *looks* right based on their training data.

If the training data contains ambiguities, or if the AI is prompted for synthesis over strict factual recall, it can invent information. This sounds entirely convincing but is utterly false.

Think of a seasoned cab driver. You ask for the fastest route to a rarely visited city area. If their mental map has a gap, they might confidently suggest a non-existent shortcut. It sounds plausible but is fake. For the justice system, this 'hallucination' is catastrophic. The MP High Court observed, "Pleadings filed before a Constitutional Court are solemn documents."

The casual insertion of fictitious case laws wastes precious judicial time. It also undermines the very foundation of legal credibility and due process. This risks actively misleading the court. It could lead to unjust outcomes if not caught. This problem is significant. India's Supreme Court in July struck down a tribunal ruling based on non-existent judgments. A US judge in June disqualified lawyers for similar fabricated citations. This is not an isolated incident; it is a growing global concern. So, what does this mean for you and your business?

Question 3

What Does This Incident Mean for Businesses and Government Bodies Adopting AI Tools in Critical Functions?

What does this mean for your business? This case is a powerful cautionary tale. It is for any organisation: a manufacturing giant, a healthcare provider, or a government department. Are you considering or already using AI in critical functions? The temptation to leverage AI for speed and efficiency is strong. But the risks of unverified output are profound.

What should businesses do to mitigate these risks? First, robust human oversight is paramount. AI should be treated as an assistant, not an autonomous decision-maker or fact-checker. Every piece of information generated by AI needs rigorous human verification. This is especially true in legal, financial, or medical contexts.

Second, organisations need clear policies and training for AI tool usage. Employees must understand AI's limitations. They must know the potential for 'hallucinations.' They also have a professional responsibility to cross-check all AI-generated content. The Advocate General's office was asked to ensure thorough verification of legal propositions and judicial citations.

Finally, liability and reputation are at stake. A business might make a critical decision. It could issue a public statement. Or file a legal document. If these are based on AI-generated falsehoods, it faces consequences. These include legal repercussions and severe damage to its credibility and public trust.

This incident underscores that AI offers immense potential for productivity. However, its deployment in areas demanding absolute factual accuracy requires caution. It demands diligence and human accountability. More than perhaps initially imagined.

The Bigger Picture

This incident from Madhya Pradesh is a stark reminder. We integrate powerful AI tools into our professional lives. But the human element of critical thinking, verification, and accountability remains irreplaceable.

The promise of AI is immense. But its integration into critical sectors like law demands caution. It needs robust verification. It requires a clear understanding of its limitations. Businesses and governments must establish safeguards. Efficiency gains cannot come at the cost of accuracy or due process.

Ultimately, technology serves us. It is up to us to ensure it is used wisely. The human mind remains the ultimate arbiter of truth.

Quick Answers

What happened in the MP High Court AI case?

The MP High Court cautioned the state government for citing a non-existent Supreme Court judgment, found to be generated by AI, in a reply to a petition regarding a police sub-inspector's dismissal.

Why is AI hallucination a problem in legal contexts?

AI hallucinations can invent plausible but false information, wasting judicial time, undermining credibility, and risking constitutional due process if unverified. This can lead to unjust outcomes and erode trust in the legal system.

What should businesses consider when using AI for legal work?

Businesses must implement robust verification protocols, ensure human oversight for all AI-generated content, and train staff on AI's limitations. This helps avoid legal repercussions, reputational damage, and ensures accuracy in critical functions.

Sources

Facts are credited to the outlets above. The analysis, calculations and Signal Check are by the Finnect AI Insights desk.