Training on corpus of text (astronomy) - without templates
#11
by
demetera
- opened
I have an astronomical corpus of text from Wikipedia. One article = one line of text without instructions.
The question is about how to fine tune the model properly?
I will be very pleased if somebody will give some examples, cos currently the training loss is around 2.2-2.4,
and i can't get it lower. Additionally, the model generates non-sense.
this is chat bot model. You provide "question" to model, model answer your question. In fact it generates next tokens after your tokenized question. So you have to prepare dataset in format, discussed here
https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0/discussions/16
(try system role = 'astronomer')
and then fine tune pretrained model on your dataset.