allenai

/

OLMo-7B

like 629

Follow

Ai2 1,449

TypeError: OLMoForCausalLM.forward() got an unexpected keyword argument 'inputs_embeds'.

That appears when I run:

trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = dataset,
dataset_text_field = "text",
max_seq_length = max_seq_length,
#data_collator=collator,
dataset_num_proc = 32,
packing = False, # Can make training 5x faster for short sequences.
args = TrainingArguments(
num_train_epochs = 1,
save_total_limit = 3,
save_steps = 50,
per_device_train_batch_size = 20,
gradient_accumulation_steps = 4,
warmup_steps = 5,
learning_rate = 2e-4,
fp16 = not torch.cuda.is_bf16_supported(),
bf16 = torch.cuda.is_bf16_supported(),
logging_steps = 1,
optim = "adamw_8bit",
weight_decay = 0.01,
lr_scheduler_type = "linear",
seed = 3407,
output_dir = "olmo7b_outputs",
),
)

trainer_stats = trainer.train()