Exploring Transformer-based LLMs Models, Applications, and Challenges in Law: A Survey

Abstract

Transformer-based Large Language Models (LLMs) have rapidly advanced the state of natural language processing, offering powerful capabilities in understanding text, and automating drafting. Given that law is a text-driven domain, such models transform workflows in legal practice. This survey provides a comprehensive overview of 80 publications on Transformer-based LLM classes and their datasets in the legal domain. It simplifies the categorization of these models into three groups: general Transformer models, legal (domain-specific) LLMs, and task-specific models. For general models, we examine the encoder-only, decoder-only, and encoder–decoder structure, as well as domain-specific and commercial legal LLMs. We also provide legal datasets for LLM training and evaluation and identify open challenges and research directions that enhance deployment. By reviewing current Transformer model applications and limitations, this survey provides a task-based view of legal LLM classes and performance, in order to better understand the application implications for natural language processing and legal researchers, and future directions for legal LLMs to become effective and reliable tools for client-centred service and judicial economy.