LSE - Small Logo
LSE - Small Logo

Zubair Abbasi

October 11th, 2024

Can generative AI master Islamic inheritance law?

2 comments | 4 shares

Estimated reading time: 4 minutes

Zubair Abbasi

October 11th, 2024

Can generative AI master Islamic inheritance law?

2 comments | 4 shares

Estimated reading time: 4 minutes

In this article, Zubair Abbasi outlines his comparative study of the accuracy of ChatGPT-4, Co-Pilot, and Gemini, to assess the accuracy of generative AI in interpreting Islamic inheritance law. 

Generative AI (Artificial Intelligence) models like ChatGPT-4 represent a qualitative leap forward in natural language processing, but unlike numerical calculators, the information they generate is often not verifiable in fields characterised by subjectivity and expert opinion. These models, often described as “dangerous but convincing tools,” can produce human-like responses that, while confident, may spread oversimplified and non-representative knowledge.

Despite these risks, the advancements in AI—particularly in ChatGPT-4—are undeniably impressive. The latest version of ChatGPT exhibits remarkable capabilities across a range of domains, including law, and even demonstrates traits of human-like intelligence. However, can we truly measure the intelligence of these models, and, if so, what benchmarks should we use?

In my recent research project, I sought to assess whether ChatGPT-4, alongside other models like Gemini and Co-Pilot, could handle the complexities of Islamic inheritance law. Islamic inheritance law, with its precise rules rooted in the Qur’an, presents a unique opportunity to evaluate the accuracy of AI-generated information. The rules are often described as predictable with “mathematical precision,” and tools such as online Islamic inheritance calculators already exist. Thus, the question arises: can AI go beyond basic calculations and generate legally sound responses based on intricate legal principles?

To explore this, I posed ten questions related to Islamic inheritance law to ChatGPT-4, Gemini, and Co-Pilot. These questions spanned a variety of issues, including differences between Sunni and Shia interpretations, as well as gender-inclusive legal reforms from countries like Pakistan, Egypt, and Iran. I developed a benchmark for evaluation based on accuracy and authenticity. Accuracy referred to the correctness and comprehensiveness of the answers, while authenticity involved verifying the references provided by each model to ensure they were genuine and not fabricated.

The results were revealing. ChatGPT-4 outperformed both Gemini and Co-Pilot, achieving a 67.5% score in accuracy and 70% in authenticity. The model synthesised vast amounts of legal information with impressive speed, explaining intricate principles of Islamic inheritance law in a clear and accessible manner. However, it still fell victim to “hallucinations”—a term used when AI generates plausible but incorrect or fabricated information. For example, it created a fake title for a book, Islamic Law: A Guide by Wael Hallaq, which resembled a genuine title, but did not actually exist. Similarly, it incorrectly cited statutes, for example, mistakenly referring to Egypt’s Law No. 71 of 1946 as Law No. 77 of 1943, and fabricated citations to reported judgements that appeared genuine but were, in fact, fake.

Chart A

Intrigued by the initial findings, I decided to test the upgraded paid version, ChatGPT Turbo. The expectation was that with greater access to resources, the paid model would yield significantly improved results. Indeed, ChatGPT Turbo demonstrated a marked improvement in authenticity, with its score rising to 80%. The paid model avoided fabricated references altogether and cited classical Arabic sources of Islamic inheritance law, covering both Sunni and Shia traditions. In contrast, the free version had relied primarily on English-language textbooks.

However, the increased access to legal resources did not translate into improved accuracy. Surprisingly, ChatGPT Turbo’s accuracy score dropped sharply to 40%, a significant decrease from the free version’s 67.5%. This suggests an inverse relationship between access to resources and accuracy—a finding that is particularly concerning in an Islamic legal context where truthful and accurate dissemination of religious knowledge is crucial. It is worth noting that the accuracy score reflects the model’s initial responses to prompts. In several instances, ChatGPT Turbo did eventually provide correct answers, but only after being prompted with specific legal details or when relevant documents were uploaded to its portal.

Chart B

Another issue with ChatGPT Turbo was its lack of transparency. While it cited accurate legal sources, the information it generated did not always align with those references, making it difficult to verify the validity of its responses. A potential solution would be to upload the correct data or documents, as ChatGPT Turbo showed partial success in extracting accurate information from these sources. Further improvements in its ability to process uploaded material would likely enhance both its accuracy, authenticity, and accountability.

In conclusion, while ChatGPT Turbo demonstrated improvements in authenticity and resource access, the sharp decline in accuracy is a concern. The results suggest that more resources do not necessarily lead to better outcomes in terms of accuracy. AI shows potential in supporting legal research, particularly with human guidance, but it is clear that continued refinement is needed to balance resource access and accuracy.

Photo by Dmitrij Paskevic 

 


Note: This article gives the views of the author, not the position of LSE Religion and Global Society nor the London School of Economics and Political Science.  


📨 Click here to sign up to the LSE Religion and Global Society newsletter.


 

About the author

Zubair Abbasi

Dr Muhammad Zubair Abbasi is an academic lawyer with expertise in family, corporate, commercial, comparative and Islamic law. He holds a DPhil in Law from Oxford University and an LL.M (Corporate Governance) from Manchester University. Currently based at the School of Law and Social Sciences, Royal Holloway, University of London, his research focuses on the integration of generative artificial intelligence into legal education, lawyering, and adjudication.

Posted In: Featured

2 Comments