Published On: June 4th 2026
Authored By: Mankirat Singh
Incoming Law Student At
Symbiosis Law School
Abstract
AI hallucinations (instances where artificial intelligence systems fabricate legal citations, generate non-existent case law, or distort legal propositions) represent a profound and growing threat to the integrity of judicial proceedings. This paper critically examines the phenomenon of AI hallucinations in legal practice, analysing their implications for lawyer accountability, evidentiary integrity, and the constitutional right to a fair trial. Through a comparative analysis of judicial and legislative responses in the United States, the United Kingdom, and the European Union, the paper identifies critical legislative and regulatory lacunae in the Indian legal framework. It argues that the Advocates Act 1961 and Bar Council of India Rules 1975 are structurally ill-equipped to address the novel risks posed by generative AI. India urgently requires targeted legislative and institutional reform. The paper concludes with concrete reform proposals grounded in comparative legal reasoning.
I. Introduction
The integration of generative artificial intelligence (AI) into legal practice has proceeded with remarkable speed and minimal regulatory oversight. Lawyers across jurisdictions are increasingly deploying AI tools for legal research, case summarisation, and the drafting of pleadings and written submissions. While these tools offer genuine efficiencies, they carry a systemic and alarming deficiency: the tendency to generate plausible but entirely fabricated legal content, a phenomenon widely referred to as “AI hallucination.” The landmark decision in Mata v Avianca Inc brought this risk to global attention when lawyers submitted ChatGPT-generated briefs containing six entirely fictitious case citations.[1] Since then, the documented incidence of AI hallucinations in court filings has grown exponentially, with a comprehensive international database tracking over 160 cases across multiple jurisdictions as of 2026.[2]
India, home to one of the world’s most overburdened judiciaries, stands at a critical and underappreciated crossroads. While AI tools are already being adopted in Indian legal practice for research, drafting, and case management, no statute, bar council rule, or judicial practice direction specifically governs their use. This legislative silence, when assessed against rapidly evolving international developments, represents an urgent governance failure.
This paper advances three central arguments: first, that AI hallucinations constitute a fundamental threat to the lawyer’s duty of candour and the right to a fair trial; second, that India’s existing legal framework is structurally inadequate to address this threat; and third, that India must enact targeted legislative and regulatory reform that is calibrated to, but not uncritically imported from, comparative international models. The paper engages with case law, professional conduct rules, constitutional provisions, and comparative legislation to develop a coherent reform agenda.
II. The Anatomy of AI Hallucinations in Legal Proceedings
To appreciate the full legal implications, it is necessary to understand the technical basis of AI hallucinations. Large language models (LLMs) operate by predicting statistically probable sequences of text drawn from vast training datasets. They do not retrieve verified information from curated legal databases; they generate text that structurally and stylistically resembles legal material present in their training corpus. When prompted about legal precedents, an LLM may produce a case name, citation, court identifier, and judicial holding, each element convincingly formatted, yet none of which may actually exist. The National Center for State Courts has defined AI hallucinations in the legal context as instances where “legal AI tools generate fabricated case citations, distorted holdings, or false procedural information that appears authentic but does not exist or is factually incorrect.”[3]
This definition reveals three distinct dimensions of legal risk. The first is wholesale fabrication: citations to cases that have never existed. The second is misrepresentation: the accurate citation of a real case paired with a fabricated or distorted holding. The third is procedural falsification: the generation of non-existent court rules or statutory provisions that mislead courts and parties. Each category carries distinct but overlapping implications for professional responsibility and due process.
What makes AI hallucinations peculiarly dangerous in legal proceedings is their confidence and structural plausibility. Unlike an obvious factual error, a hallucinated case citation mirrors the format of genuine legal authority, complete with volume numbers, reporter abbreviations, page references, and even passages of seemingly coherent judicial reasoning. In Mata v Avianca, Judge P. Kevin Castel noted that the fabricated decisions contained “internal citations and quotes” that lent them an appearance of authenticity, and described one of the fabricated legal analyses as “gibberish” only upon close analytical examination.[4] It is precisely this superficial plausibility that makes AI hallucinations so corrosive to judicial integrity.
III. Legal Accountability: Professional Duty and Evidentiary Integrity
3.1 The Duty of Candour and Professional Responsibility
The submission of fabricated legal authority to a court strikes at the heart of the lawyer’s foundational duty of candour, among the most non-negotiable obligations in any common law jurisdiction. In India, this duty is embedded in Part VI of the Bar Council of India Rules 1975, framed under the Advocates Act 1961. Rule 9 of Chapter II expressly provides that an advocate shall not knowingly make a false statement to the court or allow a false statement to remain on the record uncorrected.[5] Rule 15 further requires an advocate to refuse to act on instructions that involve the fabrication of evidence.[6]
The critical jurisprudential question is whether the submission of an AI-hallucinated citation, where the lawyer is unaware of the fabrication, constitutes a violation of these rules. This paper argues affirmatively. The duty of verification is inseparable from the duty of candour. A lawyer who cites a case to a court implicitly certifies its existence and authenticity. The delegation of research to an AI tool cannot extinguish this verification obligation; on the contrary, the widely known propensity of LLMs to generate plausible falsehoods elevates it. Ignorance of the AI’s capabilities is not a defence; it is itself a dereliction of the duty of competence.
This reasoning is directly supported by the approach adopted in Mata v Avianca, where the court held that the lawyers’ failure to verify ChatGPT-generated citations constituted a breach of Rule 11 of the Federal Rules of Civil Procedure, which requires that representations to the court be founded on reasonable inquiry.[7] The court’s finding that “ChatGPT was not supplementing the attorney’s research — it was his research” is an apposite encapsulation of the professional responsibility failure at stake.[8]
In India, the analogous disciplinary mechanism is Section 35 of the Advocates Act 1961, which empowers the State Bar Council to initiate proceedings against an advocate for professional misconduct.[9] The submission of unverified AI-generated content to a court, particularly where the advocate was aware or ought to have been aware of the hallucination risk, would readily satisfy the threshold of professional misconduct within the meaning of this provision. The absence of judicial precedent addressing this specific scenario does not preclude its application; it merely confirms the urgency of legislative and regulatory clarification.
3.2 Evidentiary Integrity and the Right to a Fair Trial
Beyond professional responsibility, AI hallucinations raise profound constitutional concerns. The right to a fair trial, protected under Article 21 of the Constitution of India,[10] comprehends not merely procedural fairness but substantive legitimacy in the adjudicatory process. A court that unknowingly relies upon a fabricated precedent in reaching its decision does not merely make an error of law; it issues a judgment that rests, in part, upon a fiction.
The implications for evidentiary integrity are equally significant. Under the Bharatiya Sakshya Adhiniyam 2023, which replaced the Indian Evidence Act 1872, electronic records are accorded evidentiary recognition subject to prescribed procedural conditions.[11] However, the Act contains no provision specifically addressing the reliability or admissibility of AI-generated legal research or AI-assisted pleadings. While such material is not “evidence” in the technical sense, it directly shapes the legal arguments and judicial reasoning upon which decisions turn. The absence of any reliability standard or disclosure obligation for AI-generated content in submissions constitutes a material lacuna in the evidentiary framework.
IV. Comparative Judicial and Regulatory Responses
4.1 United States of America
The United States has been the most jurisprudentially active jurisdiction in developing responses to AI hallucinations in legal proceedings. The foundational decision remains Mata v Avianca Inc, 678 F Supp 3d 443 (SDNY 2023),[12] in which Judge Castel imposed sanctions of USD 5,000 each against two attending attorneys and their firm under Rule 11 of the Federal Rules of Civil Procedure, following the submission of a brief containing six ChatGPT-fabricated case citations in a personal injury action. The court’s 46-page opinion has since been widely described as a regulatory blueprint that state bar associations and ethics committees have relied upon in issuing professional guidance.[13]
The American Bar Association responded formally on 29 July 2024, issuing Formal Opinion 512 on Generative Artificial Intelligence Tools,[14] affirming that the duty of competence under Model Rule 1.1 requires lawyers to understand the limitations of AI tools they deploy, and that the duty of candour under Model Rule 3.3 is violated where AI-generated research outputs are submitted without adequate verification.[15] This normative framework, grounding AI verification obligations in pre-existing professional responsibility rules rather than bespoke legislation, represents one approach to governance. Its limitation, as this paper argues, is that it depends on voluntary compliance and ex post disciplinary enforcement rather than ex ante structural safeguards.
4.2 United Kingdom
The United Kingdom has developed its response through a sequence of judicial decisions of escalating gravity. In Harber v HMRC [2023] UKFTT 1007 (TC), a litigant submitted nine fictitious AI-generated decisions to the First-tier Tribunal Tax Chamber.[16] The Tribunal explicitly endorsed the warnings articulated in Mata v Avianca and emphasised the many institutional harms flowing from AI hallucinations, including the corruption of genuine judicial precedent and the erosion of public confidence in the legal system.
The issue was elevated to the High Court in Ayinde v London Borough of Haringey; Al-Haroun v Qatar National Bank [2025] EWHC 1383 (Admin), in which lawyers had cited fabricated AI-generated authorities in High Court proceedings, including, remarkably, a citation attributed to the very judge presiding over the case.[17] The High Court emphasised that lawyers bear a personal and non-delegable professional responsibility to verify all cited authorities, and that the use of an AI tool does not attenuate this obligation.[18] The matter was referred to the relevant professional regulatory bodies. Taken together, these decisions demonstrate a clear judicial trajectory: the common law courts are developing, through iterative adjudication, an increasingly robust accountability standard for AI-assisted legal work.
4.3 European Union: A High-Risk Regulatory Framework
The structurally most significant regulatory development is the EU Artificial Intelligence Act (Regulation (EU) 2024/1689), which entered into force on 1 August 2024 and approaches full implementation on 2 August 2026.[19] The Act’s architects adopted a risk-stratified approach to AI governance. Critically, Annex III, Section 8(a) classifies AI systems intended to assist judicial authorities in researching or interpreting facts and law as “high-risk” systems.[20] This classification triggers a range of mandatory obligations: accuracy documentation, transparency requirements, human oversight mechanisms, and requirements to log interactions for audit purposes.
Recital 61 of the Act provides the normative underpinning for this classification, acknowledging that AI deployed in the administration of justice may have “potentially significant impact on democracy, the rule of law, individual freedoms as well as the right to an effective remedy and to a fair trial.”[21] This framing, treating judicial AI as a constitutional rather than merely technical governance concern, is the most jurisprudentially sophisticated regulatory approach to date. It recognises that the risks of AI in judicial proceedings are not reducible to professional misconduct by individual lawyers but represent a systemic threat to the legitimacy of adjudication itself.
4.4 India: A Conspicuous Legislative Silence
In stark and troubling contrast to developments in the United States, United Kingdom, and European Union, India has produced no reported judicial decision, bar council rule, court practice direction, or legislative provision specifically addressing AI hallucinations in legal proceedings. This silence is particularly alarming given the scale and pace of AI adoption in Indian legal practice and the well-documented burden on India’s judicial system.
The existing Indian framework (the Advocates Act 1961, Bar Council of India Rules 1975, Bharatiya Sakshya Adhiniyam 2023, and Information Technology Act 2000) was designed for a pre-generative-AI legal landscape. The Digital Personal Data Protection Act 2023, while significant in the broader AI governance landscape, addresses personal data protection and does not engage with AI use in judicial proceedings.[22] India’s draft AI governance discussions have similarly failed to specifically address the intersection of AI and judicial proceedings.
V. Legislative Gaps and Reform Proposals
The comparative analysis in the preceding section reveals four critical and interrelated legislative gaps in India’s legal framework, each requiring targeted remediation.
First, there is a complete absence of professional conduct rules governing AI use in legal research and drafting. The Bar Council of India Rules 1975 must be amended to incorporate an explicit provision, analogous to Rule 11 of the Federal Rules of Civil Procedure and the ABA’s Formal Opinion 512, requiring advocates to independently verify the accuracy of all AI-generated citations and legal propositions prior to submission to any court or tribunal. The amendment should additionally require mandatory disclosure to the court when AI tools have been used in the preparation of submissions, enabling judges to apply appropriate scrutiny to AI-assisted arguments.
Second, there are no judicial practice directions addressing AI use by practitioners. The Supreme Court of India and the various High Courts possess inherent rule-making powers under Articles 145 and 225 of the Constitution of India respectively and should exercise them promptly. Practice directions should establish verification obligations, disclosure requirements, and sanctions for submitting AI-generated content without due verification. These directions should draw on the standards articulated in the case law analysed above, while remaining calibrated to the specific conditions of Indian legal practice.
Third, the Bharatiya Sakshya Adhiniyam 2023 should be amended to address the reliability of AI-generated legal analysis in formal legal submissions. While AI-generated research does not constitute evidence in the traditional sense, the Act could be amended to introduce a dedicated provision rendering AI-generated legal arguments subject to mandatory verification certification by the submitting advocate, thereby rendering unverified AI submissions procedurally defective rather than merely subject to ex post professional sanction.
Fourth, and most fundamentally, India requires a sector-specific legislative framework for AI in judicial and quasi-judicial proceedings. Any future AI regulation legislation, whether a dedicated AI Act modelled on the EU framework or amendments to the IT Act 2000, must specifically designate judicial AI as a domain of heightened risk. This would trigger enhanced accountability obligations: mandatory human oversight, pre-deployment accuracy testing, audit trails for AI-assisted submissions, and civil liability for legal practitioners who submit AI-generated content without adequate verification. The EU AI Act’s Annex III, Section 8 approach, classifying judicial AI as high-risk and imposing structural rather than merely voluntary safeguards, offers a normatively sound model that India can adapt to its constitutional and institutional context.
VI. Conclusion
AI hallucinations in judicial proceedings represent a fundamental threat to legal accountability, evidentiary integrity, and the constitutional right to a fair trial. The trajectory from Mata v Avianca in 2023 to over 160 documented cases globally by 2026 reflects a governance crisis that will deepen as AI adoption in legal practice accelerates. The courts of the United States, United Kingdom, and the regulatory architecture of the European Union have each responded, through sanctions, professional guidance, and high-risk regulatory classification, with increasing urgency and rigour.
India has not responded at all. The Advocates Act 1961 and Bar Council of India Rules 1975 impose a general duty of candour but provide no specific mechanism for the novel accountability questions raised by AI-generated legal research. The Bharatiya Sakshya Adhiniyam 2023 fails to engage with AI-assisted submissions. No Indian court has yet issued a reported decision on the question, and no bar council or judicial body has issued guidance.
The reform agenda is both clear and urgent. Targeted amendments to the Bar Council of India Rules should mandate verification and disclosure obligations for AI-generated content. The Supreme Court should issue practice directions establishing a framework applicable across all courts and tribunals. The Bharatiya Sakshya Adhiniyam should be amended to introduce a reliability and disclosure standard for AI-assisted pleadings. Any forthcoming Indian AI regulation framework must treat judicial AI as a high-risk domain warranting the most exacting governance standards.
The deeper significance of this issue transcends professional responsibility. The legitimacy of judicial adjudication rests upon the integrity of the legal arguments upon which courts act. When those arguments are generated, without adequate verification, by systems that confabulate with confident fluency, the epistemic foundations of the rule of law itself are placed at risk. India’s regulatory response must be commensurate with the scale of that risk.
Bibliography
Primary Sources: Cases
Ayinde v London Borough of Haringey; Al-Haroun v Qatar National Bank [2025] EWHC 1383 (Admin)
Harber v HMRC [2023] UKFTT 1007 (TC)
Mata v Avianca Inc, 678 F Supp 3d 443 (SDNY 2023)
Primary Sources: Legislation and Regulatory Instruments
Advocates Act 1961 (India)
Bar Council of India Rules 1975 (India)
Bharatiya Sakshya Adhiniyam 2023 (India)
Code of Civil Procedure 1908 (India)
Constitution of India 1950
Digital Personal Data Protection Act 2023 (India)
Federal Rules of Civil Procedure (US), Rule 11
Information Technology Act 2000 (India)
Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence [2024] OJ L 1689
Secondary Sources
American Bar Association, Formal Opinion 512: Generative Artificial Intelligence Tools (Standing Committee on Ethics and Professional Responsibility, 29 July 2024)
Charlotin D, ‘AI Hallucination Cases’ (Database, 2023–2026) <https://www.damiencharlotin.com/hallucinations/> accessed 22 April 2026
Council on Criminal Justice, ‘The Implications of AI for Criminal Justice’ (October 2024)
TRI/NCSC AI Policy Consortium for Law and Courts, ‘A Legal Practitioner’s Guide to AI and Hallucinations’ (National Center for State Courts, 16 February 2026)
Footnotes
[1] Mata v Avianca Inc, 678 F Supp 3d 443 (SDNY 2023).
[2] Damien Charlotin, ‘AI Hallucination Cases’ (Database, 2023–2026) <https://www.damiencharlotin.com/hallucinations/> accessed 22 April 2026.
[3] TRI/NCSC AI Policy Consortium for Law and Courts, ‘A Legal Practitioner’s Guide to AI and Hallucinations’ (National Center for State Courts, 16 February 2026).
[4] Mata v Avianca Inc (n 1) 452–453.
[5] Bar Council of India Rules 1975, Part VI, Chapter II, r 9.
[6] ibid r 15.
[7] Federal Rules of Civil Procedure (US), r 11(b)(2) and (c).
[8] Mata v Avianca Inc (n 1) 457: ‘ChatGPT was not supplementing [the attorney’s] research — it was his research’ (per Castel J).
[9] Advocates Act 1961 (India), s 35.
[10] Constitution of India 1950, art 21.
[11] Bharatiya Sakshya Adhiniyam 2023 (India), ss 61–65 (electronic records).
[12] Mata v Avianca Inc (n 1).
[13] Claire, ‘ChatGPT Legal Work Dangers: Mata v Avianca $5,000 Sanction and Data Exposure Risks’ (LetsAskClaire, 25 February 2026) <https://www.letsaskclaire.com/legal/chatgpt-legal-work-dangers> accessed 22 April 2026.
[14] American Bar Association, Formal Opinion 512: Generative Artificial Intelligence Tools (Standing Committee on Ethics and Professional Responsibility, 29 July 2024).
[15] ibid; see also ABA Model Rules of Professional Conduct, rr 1.1 (competence), 3.3 (candour toward the tribunal).
[16] Harber v HMRC [2023] UKFTT 1007 (TC) [24].
[17] Ayinde v London Borough of Haringey; Al-Haroun v Qatar National Bank [2025] EWHC 1383 (Admin).
[18] ibid [47].
[19] Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence [2024] OJ L 1689 (EU AI Act), entering into force 1 August 2024 with full implementation 2 August 2026.
[20] EU AI Act (n 19), Annex III, s 8(a): AI systems intended to assist a judicial authority in researching and interpreting facts and the law and applying the law to a concrete set of facts are classified as high-risk.
[21] ibid Recital 61.
[22] Digital Personal Data Protection Act 2023 (India).
