BERT-Law: Information Extraction Model for Legal Texts

Model Description

BERT-Law is a fine-tuned version of BERT (Bidirectional Encoder Representations from Transformers), focusing on information extraction from legal documents. The model is specifically trained on a custom dataset called UTE_LAW, which consists of approximately 30,000 pairs of legal questions and related documents. The main goal of this model is to extract relevant information from legal text while reducing the costs associated with using third-party APIs.

Additionally, the model supports Retrieval-Augmented Generation (RAG), which enhances its ability to handle smaller context windows, thereby optimizing API costs for third-party integrations. RAG is especially useful in scenarios where processing large legal documents in a single request might be inefficient or expensive.

Key Features

• Base Model: The model is built on top of google-bert/bert-base-multilingual-cased, which is a pre-trained multilingual BERT model.
• Fine-tuning: It has been fine-tuned with the UTE_LAW dataset, focusing on extracting relevant information from legal texts.
• Model Type: BERT-based model for question-answering tasks.
• Task: The model is optimized for information extraction tasks, specifically designed to handle legal documents.
• RAG Support: Enhanced ability to process smaller context windows, improving cost-efficiency when using third-party APIs.

Model Specifications

Specification	Description
Maximum Sequence Length	512 tokens
Language	Primarily focused on Vietnamese legal texts
Task	Question-answering, Information extraction
RAG Support	Yes
LLMS Generate	Yes

References

• Zaib, Munazza and Tran, Dai Hoang and Sagar, Subhash and Mahmood, Adnan and Zhang, Wei E. and Sheng, Quan Z. (2021). BERT-CoQAC: BERT-based Conversational Question Answering in Context. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Link
• Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Link

Usage

This model is suitable for applications in legal domains, such as:

• Legal document analysis: Extracting relevant information from legal texts.
• Question answering: Providing answers to legal questions based on the content of legal documents.

The model aims to reduce reliance on third-party APIs, which can incur higher costs, by providing a locally deployable solution for legal document processing. With the integration of RAG, it further optimizes the extraction process by handling smaller context windows, improving efficiency and reducing costs when dealing with large or complex legal documents.