Introduction
The fast development of generative technology has put the Indian court system at a point where new innovations face a different kind of digital trickery. As the legal field moves from using old paper books to relying on computer programs for research, the risk of confusing authentic legal information with AI generated inaccuracies is increasing.
AI hallucinations is when large language models[1] like ChatGPT, Claude or Gemini, make up information that sounds real and believable, even though it’s not true or doesn’t exist. In legal world, this can be especially be risky. It shows up as making up fake curt cases, made-up laws, or pretend legal decisions that are seen real but are not based on anything true. These “ghost” precedents can appear similar to genuine judicial decisions, making it difficult for untrained individuals to distinguish them from authentic legal authorities.
In today’s world of modern government, having accurate data is essential for making sure the law is fair and works properly. As courts around the world start using more digital tools, lawyers are using AI help more often to handle large number of cases. The fairness of the justice system relies completely on truthfulness of the documents and sources being used[2]. If false information, like made up data, gets added to court records or court decisions, it puts at risk the trust people have in the law and fair results from court cases. India’s journey towards a digital court system has been both big and fair-reaching. This journey includes: The [3]e-Courts Mission Mode Project is the Key initiative aimed at digitizing court records and procedures. The National Judicial Data Grid (NJDG) serves as a large scale, open access data base that holds a vast amount of case related information. It is designed to offer transparency by making detailed case data easily available. Recently, new tools such as SUPACE, which is a Supreme Court Portal For Assistance in Courts Efficiency, and several private legal research platforms have been introduced. These improvements are meant to cut down the large number of unresolved cases, but the quick spread of these changes have gone much faster than the development of rules to control them.
The core legal issue in the conflict between getting things done quickly and making sure everyone is responsible. The use of AI in Indian Legal Practice raises important questions about:
- Professional Accountability: Can an Advocate be responsible for an error made by a machine?
- Article 21 Rights: Does using made up AI data during a legal case take away a persons right to a fair trial?
The duty of candour means that an advocate be truthful with the court. This duty is broken when false information created by AI is treated as real and presented as facts.
AI is a strong tool for Indian courts, but the problem of fake court decision and made up legal ideas is a big risk to the fairness of the courts and the basic rules of law.
AI hallucination – origin and nature
AI language models work by predicting the next word in a sentence based on patterns they have learned, rather than by searching through a database or looking things up like an encyclopedia or a library. They use probability to make guesses about what words might come next. When a model creates a response, it’s basically figuring out the chances of one word coming from another, based on the huge number of patterns it learned during its training. This process is usually called as Stochastic Parroting[4]. [5]The “hallucination” happens because the AI is focused on being fluent and helpful, not always checking if the information is true. If the model’s training data includes the general structure of a legal argument but not the specific case you are asking about, the AI will create a response by putting together information in a way that sounds correct based on what it has learned. Because it doesn’t have a way to check if its answers are correct against real information as it goes, it creates responses that sound correct and well written, but are made up completely.
The legal field is particularly at risk from false information created by AI because of how legal work is done and written:
[6]Legal writing depends on very specific and repeated ways of expressing ideas, such as “The court held that…” or “Pursuant to the doctrine of…” AI is very good at sounding formal and serious, which can make a lawyer feel overconfident about how accurate a document is. The way legal citations are written follows a fixed and consistent structure, such as “Name v. Name, Volume, Reporter, Page, Year.” Because this format is so predictable, AI can easily create a “placeholder” citation that looks completely real. It often combines real judge names with real court names but describes a case that never actually happened. Lawyers are always under pressure to bill hours and meet deadlines. Because of this, they may use AI to quickly summarize complex legal cases or draft legal documents. This creates what can be called an “efficiency trap”, where the need to save time leads to “verification gap”, meaning proper human checking is ignored. This legal system depends heavily on past court decisions, known as ‘stare decisis’[7], where courts follow earlier rulings. If a fake case is mentioned in a legal document and the opposing lawyer or even the judge’s assistant does not verify it, that false case might end up being used in an actual court decision. This can make the legal record inaccurate and harmful. This integration of Generative Artificial Intelligence (AI) into legal research and drafting has introduced a new challenge to the fairness of judicial proceedings, known as “hallucinations”. These ae situations where AI system generate information that sounds correct but is completely false, such as fake case laws, citations or legal interpretation. Even though this problem is modern, the legal principles needed to deal with it already exist in Indian law. By combining ideas like the doctrine of “clean hands”, professional misconduct, and the idea that judicial time is a public resource, it becomes clear that using unverified AI- generated content is a modern violation of long established ethical rules.
The doctrine of “clean hands” is a basic principle of the Indian judicial system, which requires complete honesty from every litigant and lawyer. In the important case of [8]Dalip Singh v. State of U.P. (2010), the Supreme Court of India stated that the “guiding star” of the judicial process must be truth. The court also warned about a growing group of people who try to mislead the curt to get favourable outcomes. This principle applies directly to the modern use of Generative AI. When a lawyer submits a petition that contains “hallucinated” citations, that means references that look real but do not actually exist, it damages the integrity of the court. Even if the lawyer claims that there was no intention (mens rea) and that they relied on automated tools, the judgment in Dalip Singh suggests that failing to verify information is still a breach of trust. Just as a case can be dismissed in limine for hiding important facts, a legal argument based on AI- generated false data should also be rejected. The responsibility lies entirely with the lawyer to ensure that the information presented to court is truthful and reliable.
The ethical standards of the legal profession demand more than just technical correctness ; they require a strong commitment to justice. In [9]D.P. Chadha v. Triyugi Narain Mishra (2000), the Supreme Court set a strict standard for professional conduct by punishing a lawyer who used blank signed papers to create a false compromise petition. The Court made it clear that a lawyer is an officer of the court and must ensure that justice is never obstructed. Submitting an AI generated document without proper human checking is similar to the misconduct seen in the D.P. Chadha case. When an AI creates a fake case name and a lawyer presents it as a real precedent, it is essentially the same as creating false evidence to mislead the court. The judgement in D.P. Chadha clearly states that the legal profession is not just a business but a noble profession where truth is most important. Therefore, lack of technical knowledge is not an excuse. The use of AI does not reduce liability when the final submitted document contains false or fabricated information. The source of the false information, whether human or machine, does not reduce the responsibility. The impact of AI hallucinations is not just limited to ethical concerns; it also wastes judicial time, which is a valuable public resource. In [10]Kishore Samrite v. State of U.P. (2012), The Supreme Court emphasized that the discovery of truth is the main goal of the judiciary. The Court pointed out the presenting false information is not just a personal mistake but a misuse of the judicial system that wastes valuable court time. When a lawyer cites a non-existent case, judges and opposing lawyers must spend time searching for a case that does not exist. This results in unnecessary delays and misuse of resources. The judgment in Kishore Samrite allows courts to impose “exemplary costs” on those who mislead the judiciary. This provides a clear way to punish careless use of AI. AI hallucinations are not harmless errors; they actively interfere with the process of justice. By imposing heavy penalties, courts can discourage lawyers from avoiding their responsibility by blaming technology.
AI in the Indian judicial system
[11]The digital transformation of the Indian judiciary is mainly driven by the e-Courts Mission Mode Project, which aims to improve efficiency through technology. This project has developed into three stages. The first two stages focused on computerising court systems, including setting up hardware and enabling online filing of documents and electronic payment of court fees. The third stage focuses on moving towards cloud-based systems, paperless courts, virtual hearings, and the use of advanced technologies like AI to assist judicial work. At the centre of this system is the [12]National Judicial Data Grid (NJDG), which is a large public database that tracks millions of cases in real time across the country. It helps identify delays and improve the use of judicial resources. To further improve efficiency, the Supreme Court of India has created a special AI committee to study how technology can assist judicial work. This has led the development of two important tools. [13]SUPACE (Supreme Court Portal for Assistance in Court Efficiency) is an AI based legal research assistance tool developed to help judges and court staff analyse legal information efficiently. [14]SUVAS (Supreme Court Vidhik Anuvaad Software) is a machine learning tool that translates legal documents and judgments from English into regional Indian languages, making the law more accessible to people. The Indian legal profession has moved from traditional physical libraries to modern digital systems. This has changed legal research from a slow manual process to a faster and more efficient one. This development can be divided into three stages.
[15]The first stage was the introduction of specialised Indian platforms like SCC Online and Manupatra. These platforms are very important in a country like India, where there is a huge number of legal cases. They use AI to understand the meaning of legal terms and provide features like judgment summaries and headnotes, allowing lawyers to quickly understand long judgments. Most importantly, these platforms use verified legal records, so the risk of of incorrect information is very low. The second stage involved advanced analytics tools introduced by global platforms like Lexis + India. [16]These tools use data analysis to show patterns in legal decisions and help lawyers understand how courts have ruled on similar issues over time. This allows lawyers to predict outcomes and develop stronger legal strategies based on data rather than guesswork. The third stage is the rise of generative AI tools like ChatGPT, Google Gemini, and Microsoft Copilot. Unlike earlier tools that only retrieve information, these tools create new content. Lawyers use them to draft documents, summarise large amounts of information, and generate legal ideas. While these tools are fast and helpful, they also carry serious risks. These AI systems are trained on [17]global data and may not fully understand the specific details of Indian law. When they do not have enough information, they may create fake case names, incorrect legal interpretations, or misleading summaries. Unlike traditional databases, these tools focus on producing smooth and natural language rather than ensuring complete accuracy.
Traditional legal research tools work as “closed loop systems”, meaning they rely only on verified legal records. In contrast, generative, AI systems work as “predictive systems”. They do not retrieve information in the usual way but instead predict the next word based on patterns in their training data, a process known as next-token prediction. While this makes their output sound convincing, it does not guarantee accuracy. When the AI lacks proper knowledge about a specific legal issue, it may create information that appears real but is completely false. The effect of these AI hallucinations is already visible. Courts have started noticing use of fake legal references generated by AI. For example, in the case of [18]Gummadi Usha Rani in 2026, the Supreme Court examined a lower court decision that relied on AI generated fake judgments and highlighted the serious risks involved. Similarly, in early 2026, the Punjab and Haryana High Court instructed judges not to rely on AI while writing judgments, stating that technology cannot replace human reasoning. These developments how the urgent need for proper guidelines and strict human verification.
Legal and constitutional framework in india
The use of AI in the judicial system must also be examined under the Constitution of India, especially Article 21, which guarantees the right to life and personal liberty. This includes the right to fair trial and access to justice. In a digital system, this also means that the information used in court decisions must be accurate and reliable. If a court or judicial authority relies on a fake AI generated citation, the decision may undermine the constitutional guarantee of fair procedure and access to justice under Article 21. Lawyers also have rights under Article 19 (1)(a), which protects freedom of speech and expression. However, in the legal context, this right comes with the duty of truthful advocacy. Lawyers are officers of the court and must ensure that their statements are honest and do not mislead the court. Using unverified AI generated content, especially when it contains false AI generated information, the process becomes unfair. While AI tools can help improve efficiency, their careless use can cause delays, as courts must spend time correcting mistakes. This goes against the right to speedy justice. The legal profession in India is regulated by the [19]Advocates Act, 1961, and the rules of the Bar Council of India. Section 35 of the Act deals with professional misconduct and allows action against lawyers who fail to meet professional standards. In the modern context, this includes negligence in using technology. Submitting false AI generated information can be treated as serious violation of professional duty and may even amount to constructive fraud, where a person misleads the court or gains an unfair advantage even without intentionally trying to provide false information.
The Bar Council of India Rules also set clear standards of conduct. [20]Rule 11 states that a lawyer must not knowingly present false information or hide important facts. In the age of AI, this means that lawyers must carefully verify all AI generated content before using it. Courts may interpret “knowing” to include situations where a lawyer ignores the risks of using unreliable technology. Therefore, lawyers remain the final authority responsible for ensuring that all information presented in court is accurate.
The judgment in [21]Maneka Gandhi v. Union of India, is one of the most important cases in Indian constitutional law. In this case, the Supreme Court held that any law or procedure that affects a person’s life or personal liberty must be fair, just, and reasonable and not arbitrary. This principle has a direct connection with the issue of AI hallucinations in the judicial system. When courts rely on legal information, it is essential that such information is accurate and verified. If a lawyer submits AI generated case citations or legal arguments that are false or do not actually exist, it affects the fairness of the entire legal process. A decision based on incorrect or fabricated information cannot be considered just or reasonable. Therefore, the use of unverified AI generated content in court proceedings goes against the standard or procedural fairness established in this case. It shows that technological tools must be used carefully, and human verification is necessary to protect constitutional rights.
Similarly, in [22]Hussainara Khatoon v. State of Bihar, the Supreme Court recognized that the right to speedy justice is a fundamental right under Article 21. The Court emphasized that justice delayed is justice denied, and the legal system must ensure that cases are decided without unnecessary delay. In the modern context, AI tools are often used to speed up legal research and make the work of lawyers and courts more efficient. However, the problem of AI hallucination creates a serious challenge to this goal. When AI tools generate false or misleading legal information, it leads to confusion and errors in legal proceedings. Courts may need to spend extra time verifying whether the cited cases actually exist or whether the legal principle mentioned are correct. This increases the workload of judges and delays the progress of cases. Instead of helping the system, unverified AI use can slow it down. Therefore, while AI has the potential to improve efficiency, careless reliance on it can violate the fundamental right to speedy justice.
Further, in Re: Artificial Intelligence in Courts (Suo Motu/Committee Reports), [23]the Supreme Court has shown awareness of the growing use of AI in the judicial system. Through its committees and discussions, the court has highlighted both the benefits and risks of using AI in legal processes, it has been made clear that AI tools are meant to assist judges and lawyers, not replace human decision making[24]. The court has also raised concerns about the reliability of AI generated outputs and the absence of proper regulations to control their use. This is especially important in the context of AI hallucinations, where the system may generate information that appears correct but is actually false. The observations made by the Court show that there is a need for caution, proper guidelines, and accountability while using AI in courts. It also reflects that the judiciary in India is beginning to recognize the risks associated with AI and is taking initial steps to address them. These judicial principles clearly support the idea that while technology like AI can be useful in the legal system, its use must always follow constitutional values and professional responsibility. As mentioned earlier, lawyers have a duty to present truthful and verified information before the court, and courts must ensure that decisions are based on accurate legal material. The case laws further strengthen this position by showing that fairness, accuracy, and timely justice are essential parts of the legal process. When AI generated content is used without proper verification, it not only violates professional ethics but also affects fundamental rights guaranteed under the Constitution. Therefore, the issue of AI hallucination is not just a technical problem but also a serious legal concern that requires careful regulation and responsible use within the judicial system.
Documented incidents and risks in indian courts
The increasing use of Artificial Intelligence in legal drafting and research has created a serious challenge for the Indian judicial system. Courts depend entirely on accurate facts, verified precedents and authentic legal reasoning. However, generative AI tools are capable of producing legal material that appears genuine even when it has no real legal basis. This includes imaginary case citations, incorrect statutory references and fabricated judicial observations. When such material enters court proceedings, it affects the reliability of the justice delivery system and weakens confidence in judicial institutions, The danger becomes greater because legal writing follows a structural and formal style. AI systems can imitate this format very convincingly by generating case names, citation patterns, bench compositions and legal language that closely resemble authentic judgments. To an advocate, litigant or even a junior researcher these outputs may appear trustworthy at first glance. Without careful verification from reliable databases such as SCC Online, AIR or Manupatra, inaccurate material may unknowingly become part of legal pleadings and arguments before the court.
A. Types of AI Generated Fake Legal Content
One of the most common forms of AI misuse is the creation of non-existent case citation. In such situations, the AI generate case names, citations, and court details that look legally valid but are not found in any recognised legal database. These false references are often presented in a highly professional format, making detection difficult unless independently verified. Another growing concern is fabricated judicial reasoning. AI tools sometimes produce detailed legal interpretations and attribute them to courts even though no such ruling was ever delivered. This creates confusion regarding the actual position of law and may mislead lawyers who rely heavily on automated research tools. AI systems may also generate incorrect statutory provisions by inventing subsections, explanations, or amendments that do not exist within the original legislation. Such errors are particularly dangerous because they directly affect legal arguments and interpretations of rights and liabilities. In addition, some AI tools create false academic commentary by producing imaginary textbook extracts, law review opinions, or scholarly discussions that cannot be traced to any genuine source. AI may incorrectly describe appeal stages, procedural orders, or earlier judicial findings in a case. This distorts the factual background of litigation and may create a completely misleading understanding of the matter before the court.
B. Documented incidents in Indian Courts
[25]Although India does not yet maintain an official reporting system specifically for AI related legal errors, several incidents involving questionable AI generated legal material have already been noticed in litigation practice. Courts and legal professions have identified pleadings containing citations that could not be traced in recognised legal databases. In some matters, AI assisted drafting resulted in incorrect statutory references and inaccurate leal propositions being misplaced before the court. There have also been concerns regarding petitions prepared using automated tools that contained exaggerated or unsupported factual claims. Such practices create difficulties not only for judges but also for opposing parties who must spend additional time verifying whether the legal material relied upon is genuine. This increases procedural delays and places unnecessary pressure on an already burdened judicial system. Unlike certain jurisdictions in the United States, where courts in regions such as Texas, New York, and the Northern District of California have introduced disclosure requirements regarding AI assisted filings, India currently lacks a specific framework governing the use of AI in litigation. This absence of regulation creates uncertainty regarding accountability and professional responsibility.
Comparative perspective
The rapid growth of Artificial Intelligence in the legal profession has created both opportunities and dangers for judicial systems around the world. Courts and lawyers are increasingly using AI tools for legal research, drafting, translation, document review and case management. These technologies are capable for improving efficiency and reducing the burden on overworked judicial systems. However, the rise of generative AI has also introduced the serious problem of AI hallucinations, where AI systems produce false legal citations, imaginary judgments and incorrect legal reasoning that appear convincing but are completely fabricated. Different countries have responded to this challenge in different ways. Some countries have started introducing ethical guidelines and mandatory disclosure rules, while others are still in the early stages of regulation. A comparison between the United Kingdom, the United States, and India helps in understanding how legal systems are trying to balance technological advancement with judicial accountability. This comparison is important because India is rapidly digitising its courts yet it still lacks a clear legal framework governing the use of AI in legal proceedings.
A. United Kingdom
The United Kingdom has adopted a cautious and professionally regulated approach towards the use of AI in legal practice. The UK legal system places great importance on the principle of professional responsibility and the duty of candour. Under this principle, lawyers are expected to present only truthful and verified information before courts. Any misleading statement or false legal material submitted before the court can amount to professional misconduct, even if the mistake was caused by technology. The Uk legal profession is regulated by bodies such as the Solicitors Regulation Authority and the Bar Standards Board. In recent years, these bodies have recognised that AI tools are becoming increasingly common in legal work. As a result, the Solicitors Regulation Authority issued guidance in 2023 regarding the responsible use of AI technologies in legal services. [26]The guidance did not prohibit lawyers from using AI tools but it strongly warned them about the risks associated with the unverified AI generated content. [27]The SRA guidance clearly stated hat lawyers remain personally responsible for all work submitted in court, regardless of whether AI tools were involving in preparing the documents. This means that a lawyer cannot escape responsibility by claiming that an AI system generated incorrect information. Lawyers are expected to verify all citations, legal principles and factual statements before presenting them in court. The responsibility for maintaining accuracy continues to remain with the human advocate and not the machine. This system has a strong emphasis on professional ethics. The legal profession in the United Kingdom us treated as public service that requires honesty, competence and integrity. Courts expect advocates to assist in the administration of justice rather than simply win cases at any cost. Because of this tradition the misuse of AI generated material is treated seriously. If a lawyer submits false authorities generated by AI it may be viewed as negligence or professional misconduct.
The United Kingdom approach mainly focuses on controlled adoption rather than unrestricted use of AI. Courts and regulatory authorities understand that AI can help lawyers manage heavy workloads and improve access to legal information. At the same time, they recognise that generative AI systems are not always reliable because they produce answers based on probability rather than unverified truth. Therefore human supervision remains essential. The United Kingdom also benefits from a comparatively smaller litigation burden when compared to India. This allows lawyers and courts more time to verify legal materials before filing documents or delivering judgments. Additionally, legal research in the UK is heavily dependent on verified databases and authoritative sources, reducing the chances of fabricated citations entering court proceedings. Thus, the Uk model reflects a balanced system where innovation is permitted but closely linked with professional accountability. The central idea is that AI may assist legal professionals, but it cannot replace human judgment, ethical responsibility or verification.
B. United States
[28]The United States has become one of the most important examples in discussions relating to AI hallucinations in courts because several major incidents involving fake AI generated citations have already occurred in American courts. Unlike theoretical concerns, the dangers of AI misuse became a real judicial issue in the United States when lawyers submitted fabricated authorities created by generative AI systems. One of the most famous cases is [29]Mata v. Avianca. In this case, lawyers used ChatGPT to prepare legal research and submitted a brief containing several case citations that appeared genuine. However when the court examined the authorities, it discovered that many of the cited judgments did not actually exist. The AI system had created imaginary case names, fake quotations and fabricated legal principles. The lawyers initially relied on AI generated material without proper verification and submitted it before the court. When questioned by the judge, the lawyers admitted that they had used ChatGPT and were unaware that the cases were fabricated. The court treated the matter very seriously because false citations directly affect the integrity of judicial proceedings. The lawyers were sanctioned and fined for misleading the court. The Mata case became internationally significant because it demonstrated how generative AI can create highly convincing but completely false legal content. It also showed that even trained legal professionals may sometimes fail to recognise fabricated authorities when relying excessively on AI systems. The case served as a warning to courts and lawyers worldwide regarding the dangers of unverified AI use in legal practice.
Another important case is [30]Park v. Kim. In this case, an attorney submitted an appeal brief containing AI generated citations to non existent cases. The court later found that several authorities mentioned in the brief were fabricated. As a result, the attorney’s appeal was dismissed and the court criticised the careless reliance on generative AI tools without proper verification, holding that the conduct violated [31]Rule 11 of the Federal Rules of Civil Procedure, which requires lawyers to reasonably verify the accuracy and authenticity of authorities cited before the court. The Park v. Kim case further strengthened the growing judicial concern regarding AI generated legal misinformation. Unlike ordinary clerical mistakes, fabricated citations waste judicial time and damage the reliability of legal proceedings. Judges and opposing lawyers are forced to spend additional time verifying authorities that do not actually exist. This delays court proceedings and increases the burden on the judicial system. Following these incidents, several federal courts in the United States began introducing formal disclosure requirements relating to AI use. Some courts now require lawyers to certify that either no generative AI was used in preparing filings or that all AI generated content has been independently verified by a human lawyer. These disclosure rules are intended to ensure accountability and reduce careless dependence on AI systems. The American Bar Association also issued guidance in 2023 explaining that lawyers have an ethical duty to understand the risks and limitations of AI technologies. [32]According to the ABA, legal competence in the modern era includes technological competence. This means that lawyers must understand how AI systems function, recognise their limitations and ensure that all legal work remains accurate and reliable.
Several states bar associations in the United States have similarly warned lawyers that careless use of AI may violate professional ethics rules. Lawyers are expected to maintain confidentiality, competence and honesty even while using technological tools. The American approach therefore combines technological innovation with strict professional responsibility. An important feature of the United States approach is its quick institutional response. Once courts identified the problem of AI hallucinations, discussions regarding regulation, ethical obligations and disclosure rules began almost immediately. This reflects a more proactive legal culture where emerging technological risks are addressed through judicial orders, professional ethics opinions and court rules. At the same time the American legal system continues to encourage technological development. AI tools are widely used in document review, predictive analytics, contract analysis and legal drafting. The issue is not the use of AI itself but rather the careless use of AI without verification. Therefore, the American system does not reject AI technology but instead seeks to regulate it through accountability and transparency.
C. India
India currently stands at a transitional stage in relation to Artificial Intelligence in the legal system. On one hand, the Indian judiciary is rapidly adopting digital technologies to improve efficiency and reduce delays. On the other hand, India still lacks a regulatory framework specifically governing the use of generative AI in legal practice. The Indian judicial system faces one of the largest litigation burdens in the world, with the more than fifty million pending cases across various courts. This enormous backlog creates strong pressure to adopt technological tools capable of increasing efficiency. Projects such as the e-Courts Mission Mode Project, the National Judicial Data Grid, online filing systems, and virtual hearings have already transformed many aspects of judicial administration. AI based tools such as SUPACE have also been introduced to assist with legal translation and case management. Because of this pressure for efficiency, lawyers and courts in India may increasingly depend on AI tools for legal drafting, summarisation and research. However, unlike the United States India has not yet developed detailed professional rules specifically addressing AI generated legal content. There is currently no mandatory requirement for lawyers to disclose AI usage in court filings.[33] Similarly, there are no official judicial standards explaining how AI generated material should be verified before submission.
The Bar Council of India has also not yet issued comprehensive ethical guidelines dealing specifically with generative AI. While the Advocates Act, 1961 and Bar Council Rules impose duties of honesty and professional conduct, these rules were created before the rise of modern AI systems. As a result, there remains uncertainty regarding how existing professional ethics rules apply to AI generated hallucinations. Indian courts have nevertheless started recognising the risks associated with AI. In some recent observations, judges have warned against excessive reliance on AI systems for judicial reasoning. Courts have emphasised that technology may assist judges and lawyers but cannot replace human application of mind. This reflects the growing awareness about the dangers of fabricated legal information. Compared to UK, India lacks specialised regulatory guidance similar to the SRA’s AI rules. Compared to US, India has not yet introduced mandatory disclosure norms or detailed judicial protocols regarding AI generated filings. This places India in an under regulated middle position where AI adoption is increasing faster than legal safeguards. Another major challenge for India is digital inequality. While large law firms and urban advocates may have access to reliable legal databases and technological resources, smaller practitioners may depend on freely available public AI tools that are more prone to hallucinations. This increases the risk of fake citations entering judicial proceedings especially in lower courts where verification systems may be weaker. India’s judicial system also depends heavily on precedent. If fabricated authorities generated by AI enter court records and remain undetected, they may create confusion in future litigation. Since Indian courts frequently rely on earlier judgments, the circulation of false precedents could seriously damage the integrity of the legal system. Therefore, India urgently requires a balanced regulatory framework that encourages innovation while protecting constitutional values and judicial integrity. The experiences of the United Kingdom and the United States show that responsible AI regulation must include professional accountability, disclosure requirements, ethical guidance and strict verification standards. India can learn from these systems while developing rules suited to its own judicial realities. The comparative study clearly shows that AI hallucination is no longer a hypothetical or future problem. Countries across the world are already facing real incidents involving fabricated legal citations and misleading AI generated material. While the UK focuses on professional responsibility and the USA emphasises disclosure and sanctions, India still lacks a fully developed regulatory response. This makes legal reform extremely important. Without proper safeguards, the increasing use of AI in Indian courts may weaken public trust in the justice system and threaten the fundamental principle that judicial decisions must always be based on truth and verified law.
Policy framework for regulating AI in the judiciary
The growing use of Artificial Intelligence in courts and legal practice requires a structured regulatory framework to prevent misuse, protect judicial integrity and maintain public confidence in the justice system. While AI can improve efficiency and reduce delays and, unregulated or careless use of AI generated content may lead to false citations, biased outcomes and erosion of trust in judicial process. Therefore, instead of treating safeguards as isolated recommendations, a clear policy framework should be developed that follows a sequence of regulation, implementation, monitoring and accountability.
- Establishment of a Regulatory and Ethical Framework
The first step should be the creation of a formal regulatory framework governing the use of AI in courts and legal practice. The Bar Council of India, together with the judiciary and the legislature should formulate uniform ethical guidelines defining the permissible scope of AI usage. These guidelines should clarify professional responsibilities, confidentiality obligations, standards of verification and liabilities arising from misuse of AI generated material. Establishing clear rules at the outset is necessary to ensure consistency and accountability across the legal system.
- Mandatory Disclosure and Verification Requirements
After establishing regulatory standards, the next stage should focus on ensuring transparency in the use of AI tools. Lawyers and legal professionals should be required to disclose whenever generative AI has been used in preparing pleadings, legal research or submissions before the court. In addition, all AI generated citations and legal authorities must be independently verified through reliable legal databases such as SCC Online, AIR, Manupatra, Indian Kanoon, or official court websites. Mandatory verification mechanisms would reduce the risk of fabricated or inaccurate citations entering judicial proceedings.
- Institutional Adoption of Verified AI Systems
Once verification standards are introduced, courts should permit the use only of verified AI systems connected to authentic legal databases and official judicial records. Limiting judicial use to approved and supervised AI tools would minimise hallucinated citations and unreliable legal information. A certification mechanism may also be introduced to evaluate whether AI tools satisfy standards of accuracy, transparency and data security before being used within courts.
- Capacity Building and Training Programmes
Effective implementation of AI regulation requires adequate institutional training. Judges, lawyers and court staff should therefore receive regular training regarding the functioning, limitations and ethical risks of AI systems. AI literacy programmes would help legal professionals identify fabricated authorities, understand algorithmic bias and avoid excessive dependence on automated systems. Training is essential to ensure that AI remains an assistive tool rather than a substitute for professional judgment.
- Monitoring and Oversight Mechanisms
A separate institutional mechanism should be established to continuously monitor the use of AI in the judicial sector. An independent regulatory authority or supervisory body may oversee compliance with ethical standards, examine risks arising from AI assisted legal work and periodically review the effectiveness of safeguards. Continuous monitoring would allow timely identification of misuse and ensure that regulatory standards evolve alongside technological developments.[34]
- Enforcement and Accountability Measures
No regulatory framework can function effectively without enforcement mechanisms. Courts should therefore impose strict penalties where false or misleading AI generated citations are knowingly or negligently used in legal proceedings. Depending on the seriousness of the misconduct, penalties may include fines, disciplinary proceedings, adverse judicial observations or suspension from legal practice. Accountability measures are necessary to discourage irresponsible use of AI technologies in the justice system.
- Public Awareness and Protection of Judicial Integrity
Finally, public awareness initiatives should be undertaken to educate citizens about both the benefits and limitations of AI generated legal information. Individuals should be encouraged to verify important legal advice with qualified legal professionals before relying upon AI generated responses. At the same time, courts must ensure that AI does not replace human judicial reasoning. Judicial decisions must continue to be based on fairness, constitutional principles, human reasoning and procedural justice. AI should function only as a supportive tool that enhances efficiency while preserving the integrity and credibility of the judiciary. Through a phased and structured policy framework consisting of regulation, verification, implementation, monitoring, enforcement and public awareness, India can responsibly integrate AI into its legal system while protecting judicial fairness, accountability and public trust.
Conclusion
The study shows that AI hallucinations are not just technical mistakes but a serious threat to judicial fairness and legal accountability. While India is rapidly modernising its judiciary through projects such as e-Courts, NJDG, SUPACE, SUVAS the use of generative AI has exposed a major gap between technological advancement and legal regulation. Unlike traditional legal databases, generative AI systems create responses through prediction rather than verified legal reasoning, which increases the risk of fabricated citations and false legal arguments entering court proceedings. The research further demonstrates that existing constitutional principles and professional ethics already place responsibility on advocates to ensure truthfulness before the court. Cases such as Dalip Singh, D.P. Chadha and Kishore Samrite show that misleading the court even through negligent use of technology can damage judicial integrity and waste valuable judicial time. Comparative analysis with the United Kingdom and the United States also reveals that other jurisdictions are moving towards disclosure requirements, ethical guidelines, and stronger accountability mechanisms, whereas India still lacks a clear regulatory framework for AI use in litigation. Therefore, the central issue is not whether AI should be used in courts, but how it should be regulated and verified. AI can improve efficiency, accessibility and legal research but it cannot replace human judgment, constitutional values or professional responsibility. The future of Indian judiciary depends on creating a balanced framework here technology supports justice without compromising truth, fairness and public confidence in the legal system.
*****
Footnotes
[1] Matthew Dahl et al., Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models, 16 J. Legal Analysis 237 (2024).
[2] Varun Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, J. Empirical Legal Stud. (2025), https://law.stanford.edu/wp-content/uploads/2024/05/Legal_RAG_Hallucinations.pdf.
[3] e-Committee, Supreme Court of India, e-Courts Mission Mode Project, Gov’t of India, https://ecourts.gov.in/ecourts_home/static/about-us.php (last visited May 20, 2026).
[4] Emily M. Bender et al., On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?, in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency 610 (2021).
[5] AI Law Librarians, What the Science Says About Hallucinations in Legal Research, AI L. Librarians (Feb. 19, 2026), https://www.ailawlibrarians.com/2026/02/19/what-the-science-says-about-hallucinations-in-legal-research/
[6] Varun Nair et al., AI Hallucinations in the Legal Field: Present Experiences, Future Considerations, Observer Rsch. Found. (Aug. 12, 2025), https://www.orfonline.org/expert-speak/ai-hallucinations-in-the-legal-field-present-experiences-future-considerations.
[7]Phantom Precedents: The Rise of AI-Generated Case Law in Indian Courts, Live Law (Mar. 17, 2026), https://www.livelaw.in/articles/phantom-precedents-ai-generated-case-law-indian-courts-526665.
[8] Dalip Singh v. State of U.P., (2010) 2 SCC 114 (Ind.)
[9] D.P. Chadha v. Triyugi Narain Mishra, (2000) 2 SCC 221 (Ind.)
[10] Kishore Samrite v. State of U.P., (2012) 2 SCC 398 (Ind.)
[11] e-Courts Mission Mode Project Enters Phase III: Transforming Indian Judiciary with Digital Solutions, Press Info. Bureau, Gov’t of India (Dec. 17, 2024), https://www.pib.gov.in/PressReleaseIframePage.aspx?PRID=2085127.
[12] National Judicial Data Grid (NJDG), Nat’l Informatics Ctr., https://www.nic.gov.in/project/national-judicial-data-grid/ (last visited May 21, 2026).
[13] Artificial Intelligence in the Indian Judiciary: SUPACE, SUVAS, and the Limits of Assistive Automation, Dr. Syama Prasad Mookerjee Rsch. Found. (Feb. 12, 2026), https://spmrf.org/artificial-intelligence-in-the-indian-judiciary-supace-suvas-and-the-limits-of-assistive-automation.
[14] Prabhanu Kumar Dason, Explained: How Indian Courts Are Using AI, MediaNama (Dec. 4, 2025), https://www.medianama.com/2025/12/223-supreme-court-ai-white-paper-indian-judiciary/.
15 Shardul Amarchand Mangaldas & Co., AI in Courtrooms and How It Will Change the Landscape of Litigation, AMS Shardul (May 15, 2025), https://www.amsshardul.com/insight/ai-in-courtrooms-and-how-it-will-change-the-landscape-of-litigation/.
[16] Artificial Intelligence and Judicial Bias, Ctr. for L. & Pol’y Rsch. (Aug. 28, 2021), https://clpr.org.in/blog/artificial-intelligence-and-the-courts/.
[17] Damien Charlotin, AI Hallucination Cases Database, https://www.damiencharlotin.com/hallucinations/ (last visited May 20, 2026).
[18] Gummadi Usha Rani v. Sure Mallikarjuna Rao & Ors., (2026) SCC Online SC 341 (Ind.)
[19] The Advocates Act, No. 25 of 1961, §35 (Ind.)
[20] Bar Council of India Rules, Rule 11 (Ind.)
[21] Maneka Gandhi v. Union of India, (1978) 1 SCC 248 (Ind.)
[22] Hussainara Khatoon v. State of Bihar, (1980) 1 SCC 81 (Ind.)
[23] Use of AI in Supreme Court Case Management, Press Info. Bureau, Gov’t of India (2025), https://www.pib.gov.in/PressReleasePage.aspx?PRID=2113224.
[24] Won’t Let Artificial Intelligence Do Decision Making; Judges’ Autonomy & Discretion Will Be Retained: CJI Bobde, Live Law (Apr. 7, 2021), https://www.livelaw.in/top-stories/supreme-court-artificial-intelligence-portal-supace-chief-justice-sa-bobde-172220.
[25]Sonal Khetarpal, AI Hallucination in Court: ITAT Recalls Order Over Fake ChatGPT-Generated Case Laws, CounselVise (Feb. 2025), https://counselvise.com/blogs/ai-hallucination-itat-buckeye-trust; Risks of AI in Indian Courts: Justice Gavai Flags Fake Citations, MediaNama (Mar. 11, 2025), https://www.medianama.com/2025/03/223-justice-gavai-flags-ai-risks-when-chatgpt-gets-legal-facts-wrong; Phantom Precedents: The Rise of AI-Generated Case Law in Indian Courts, Live Law (Mar. 17, 2026), https://www.livelaw.in/articles/phantom-precedents-ai-generated-case-law-indian-courts-526665; Prabhanu Kumar Dason, SC Says Citing AI-Generated Fake Case Laws Is Misconduct, MediaNama (Mar. 3, 2026), https://www.medianama.com/2026/03/223-supreme-court-ai-fake-case-laws-misconduct.
[26] Generative AI — The Essentials, The L. Soc’y, https://www.lawsociety.org.uk/topics/ai-and-lawtech/generative-ai-the-essentials (last visited May 20, 2026).
[27]Risk Outlook Report: The Use of Artificial Intelligence in the Legal Market, Solicitors Regulation Auth. (2023), https://www.sra.org.uk/sra/research-publications/artificial-intelligence-legal-market/..
[28] Standing Order of Judge Brantley Starr, U.S. Dist. Ct. for the N. Dist. of Tex. (May 2023), https://www.txnd.uscourts.gov/judge/judge-brantley-starr; Navigating AI Disclosure Rules in New York Courts, Greenberg Traurig LLP (Nov. 21, 2025), https://www.gtlaw.com/en/insights/2025/11/navigating-ai-disclosure-rules-in-new-york-courts; AI Disclosure Requirements for the Northern District of California, AI Vortex (Apr. 2026), https://www.aivortex.io/legal/ai-disclosure/northern-district-californ.
[29] Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023).
[30] Park v. Kim, 91 F.4th 610 (2d Cir. 2024).
[31] Fed. R. Civ. P. 11.
[32] Formal Op. 512, Generative Artificial Intelligence Tools, Am. Bar Ass’n Standing Comm. on Ethics & Pro. Responsibility (July 29, 2024), https://www.americanbar.org/content/dam/aba/administrative/professional_responsibility/ethics-opinions/aba-formal-opinion-512.pdf.
[33] Paste in Haste: The Fallout of AI Hallucinations in Court Filings, Att’y Registration & Disciplinary Comm’n (Jan. 29, 2026), https://iardc.org/Files/TheFalloutOfAIHallucinationsInCourtFilings.pdf.
[34] SRA Innovate: Compliance Tips for Solicitors, Solicitors Regulation Auth., https://www.sra.org.uk/solicitors/resources/innovate/compliance-tips-for-solicitors/ (last visited May 20, 2026).