Model Card for Legal Document Summarizer

This model is fine-tuned to convert legal documents into human-readable summaries using Llama 3 8B Instruct as the base model. It was trained using QLoRA/LoRA techniques for efficient fine-tuning.

Model Details

Model Description

This is a fine-tuned version of NousResearch/Meta-Llama-3-8B-Instruct, optimized for summarizing legal documents in plain English. The model uses Parameter-Efficient Fine-Tuning (PEFT) methods, specifically LoRA, to achieve performance comparable to full fine-tuning while using significantly fewer computational resources.

Developed by: jcbthnflrs
Model type: Causal Language Model (LLaMA 3 Architecture)
Language(s): English
License: [Base model license applies]
Finetuned from model: NousResearch/Meta-Llama-3-8B-Instruct

Model Sources

Base Model: NousResearch/Meta-Llama-3-8B-Instruct
Training Code: Based on LLM Engineering Challenge from AI Makerspace

Uses

Direct Use

This model is designed for converting legal documents, terms of service, and other legal content into plain English summaries that are easier for general audiences to understand. It can be used directly through the Hugging Face API or interface.

Downstream Use

The model can be integrated into:

Legal document processing systems
Terms of service simplification tools
Contract analysis applications
Legal document management systems

Out-of-Scope Use

The model should not be used as a replacement for legal advice or professional legal interpretation. It is meant to assist in understanding legal documents but not to provide legal guidance.

Training Details

Training Data

The model was trained on the Plain English Summary of Contracts dataset, which contains pairs of legal documents (EULA, TOS, etc.) and their natural language summaries. The dataset was split into:

Training set: 68 examples
Test set: 9 examples
Validation set: 8 examples

Training Procedure

Preprocessing

Input text is formatted using a specific template following Llama 3's chat format
Special tokens are used to mark legal document boundaries
Maximum sequence length: 2048 tokens

Training Hyperparameters

Training regime: 4-bit quantization using QLoRA
Optimizer: AdamW
Learning rate: 2e-4
Batch size: 1 per device
Training steps: 500
Warmup steps: 30
Evaluation steps: 25
Learning rate scheduler: Linear
LoRA rank (r): 16
LoRA alpha: 32
LoRA dropout: 0.1

Hardware and Software

Hardware Requirements

GPU: T4 or better
Memory: Optimized for consumer-level resources through QLoRA

Software Requirements

transformers library
PEFT library
bitsandbytes for quantization
TRL for supervised fine-tuning

Evaluation

Training metrics show:

Starting training loss: ~1.52
Final training loss: ~0.0006
Final validation loss: ~2.74

Model Card Authors

@jcbthnflrs

Model Card Contact

https://x.com/jcbthnflrs