New research from Irish academics has raised concerns about the potential impact of artificial intelligence on access to justice, fairness and reliability within legal systems, finding that AI models acting as decision-makers can be significantly influenced by the quality of arguments presented to them.
The study, Persuadability and LLMs as Legal Decision Tools, was conducted by Oisin Suttle of Maynooth University’s School of Law and Criminology and David Lillis of UCD’s School of Computer Science. Described as the first systematic examination of how persuadable large language models are when confronted with difficult legal questions drawn from real cases, the research was presented at the 21st International Conference on Artificial Intelligence and Law in Singapore.
The researchers sought to answer a simple but important question: can an AI judge be persuaded into reaching a particular verdict? Their conclusion was that it can, and in some instances to a remarkable extent.
The study comes as the Irish judiciary prepares guidance on the use of artificial intelligence by practitioners and litigants. Commenting on the findings, Suttle noted that while judges must be open to persuasion for the adversarial process to function, they should not be so susceptible to advocacy that outcomes depend more on the skill of the advocate than on the law and facts of the case. The results, he said, indicate that AI systems used in legal settings may be highly persuadable, raising concerns about both fairness and reliability.
Lillis emphasised the importance of rigorous scientific assessment of AI systems, particularly where they may be deployed in sensitive areas such as legal decision-making. He cautioned against accepting claims about AI capabilities without careful empirical evaluation.
To conduct the research, the authors designed a series of experiments using real appellate court decisions from Ireland, England and Wales, and the United States. In each scenario, AI models were assigned the roles of opposing advocates and judge. One model argued in favour of one side, another argued for the opposing side, while a third model acted as the decision-maker. By varying the models assigned to each advocacy role and measuring which side prevailed, the researchers were able to assess the influence of argument quality on outcomes.
The study found that every AI model tested was measurably persuadable. Across twenty different configurations involving leading AI systems developed by Anthropic, Google, OpenAI, Mistral and DeepSeek, the stronger advocate prevailed in between 58 per cent and 71 per cent of cases on average. In the most pronounced examples, the stronger advocate succeeded in more than 90 per cent of cases.
The researchers observed that courts, tribunals and administrative bodies in several jurisdictions are already experimenting with AI tools for functions ranging from legal research and case triage to judgment drafting. Despite this growing use, they noted that ‘persuadability’ has received little attention as a factor in assessing the suitability of such systems for legal decision-making.
According to the study, the implications are particularly significant for access to justice. If AI decision-makers are heavily influenced by the sophistication of legal arguments, parties with access to experienced lawyers or more advanced AI advocacy tools may enjoy a systematic advantage over those with fewer resources.
Although larger and more capable models tended to be somewhat less persuadable than smaller systems, the researchers cautioned that this should not be viewed as reassuring. Even the least persuadable models displayed significant sensitivity to the quality of arguments presented. Furthermore, the apparent resistance of some smaller models appeared to stem not from stronger independent judgment but from difficulties in evaluating competing arguments effectively.
While the authors do not argue against the use of AI in legal contexts, they contend that persuadability should become a standard metric when assessing AI legal decision tools. They recommend that the extent to which a system can be influenced by advocacy should be measured, disclosed and taken into account before such tools are deployed in judicial or quasi-judicial settings.
The study concludes that understanding how AI systems respond to persuasion is essential if they are to be used fairly and responsibly in legal processes, particularly where outcomes may affect fundamental rights and access to justice.
Click here to read the report.