Prompt format?
Do you know what prompt format this was trained on?
I used Axolotl to train using the sharegpt format for the dataset, so presumably ChatML should work. I just left ooba at defaults though when testing the models and Alpaca format worked just fine:
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Write a story about llamas.
### Response:
@LoneStriker
I noticed weird behavior when it comes to prompt format on this model.
It doesn't like chatml prompt format - and in my testing it also doesn't seem to like llama 2 chat or alpaca formats. I was troubleshooting an issue today where I was getting weird responses when interacting with this model through cli with chat.py supplied with exllama v2. I found that the prompt format it likes the most is the default oobabooga chat prompt format "Assistant". That's the one that gives you a familiar start message
AI
How can I help you today?
When you are running this model in ooba, are you sure that you switched to the instruct mode in settings on the bottom?
If not, and it's running in chat mode, you are likely also using the default Assistant prompt format.
I fixed the issue for my use with chat.py by adding a custom prompt format, mirroring the ooba Assistant prompt format, I find it to work better with this model than other prompt formats.
Here's an example of the results that I get when using various prompt formats with ooba with this model. 2 prompts per prompt format. Can you try to replicate this?
alpaca:
write a joke about llama
Why don't llamas ever play hide and seek? Because they always end up in a llama drama! 🦙🎭
Joke: Llama Hide and Seek Drama
Why don't llamas ever play hide and seek? Because they always end up in a llama drama! 🦙🎭
Reference(s):
https://www.reddit.com/r/dadjokes/comments/849qz5/why_dont_llamas_play_hide_and_seek/
how warm is it in Ghambia in May on average?
Ghambia, located along the Atlantic coast of West Africa, experiences its hottest temperatures during the months of April and May. The average high temperature for these two months hovers around 30°C (86°F), while the average low drops to about 21°C (70°F) at night. This indicates that Ghambia can be quite warm in May, with daytime temperatures typically reaching into the mid-to-high 80s Fahrenheit or low 90s Celsius. However, due to its coastal location, humidity levels are also relatively high which might make the heat feel even more intense.
Reference(s):
https://www.climatestotravel.com/ghana/temperature-by-month
https://en.wikipedia.org/wiki/Ghana#Climate_and_vegetation
chatml:
write a joke about llamas
Why don't llamas make good secretaries? Because they always spit out their papers.
Reference(s):
https://www.jokejive.com/humor-type/animal-jokes/llama-jokes/201367/why-dont-llamas-make-good-secretaries
http://www.kidsgen.com/jokes/animals/lama.htm#joke_id=9548
https://www.youtube.com/watch?v=QgWuJYLqzZY&ab_channel=FunnyAnimalVideos%E2%80%9D%EF%B8%8F
https://www.amazon.com/gp/product/1260070380/ref=ox_sc_saved_title_2?smid=A3H2XS0KI8T7RK&psc=1
[model continues writing links]
how warm is it in Ghambia in May on average?
In the month of May, Gambia experiences an average high temperature of around 35°C (95°F) and a low temperature of approximately 24°C (75°F). However, please note that these are averages and actual temperatures can vary depending on factors such as humidity, wind speed, and location within the country. It's always recommended to check with local weather forecasts for more precise information during your stay.
Reference(s):
"Gambia Weather & Climate" - World Meteorological Organization
"Climate: Banjul" - TimeAndDate.com
"The climate of The Gambia" - Visit The Gambia Official Website
"May Temperature Normals" - National Oceanic and Atmospheric Administration (NOAA)
"Average monthly temperatures and weather for Banjul, Gambia" - Holiday-Weather.com
"Gambia Climatology" - National Aeronautics and Space Administration (NASA)
"Temperature and Rainfall Data for Banjul, Gambia" - WeatherSpark.com
"The Gambia: Average Yearly Temperature" - OurWorldInData.org
"Temperatures in Gambia" - MyVacationSpot.net
"Best time to visit The Gambia" - Lonely Planet
"Climate data for Banjul, Gambia" - Weatherbase.com
"Gambia Weather Forecast" - AccuWeather.com
"Climate Change 2021: The Physical Science Basis" - Intergovernmental Panel on Climate Change (IPCC)
"Climate Analysis Indicators Package (CAIP)" - United States Agency for International Development (USAID)
"Climate Services Initiative" - World Bank Group
"Global Climate Change Alliance Plus" - European Commission
"National Adaptation Plans" - UN Framework Convention on Climate Change (UNFCCC)
"Africa Climate Report" - African Union Commission
"Climate Resilience Building in Africa" - New Partnership for Africa's Development (NEPAD)
"Adapting to Climate Change in West Africa" - Food and Agriculture Organization of the United Nations (FAO)
"West Africa Regional Climate Outlook Forum" - Economic Community of West African States (ECOWAS)
"Regional Assessment of Climate Change Vulnerability and Impacts in West Africa" - United Nations Environment Programme (UNEP)
"Impacts of Climate Change on Human Health in Sub-Saharan Africa" - Pan American Health Organization/World Health Organization (PAHO/WHO)
"Trends in Global Extreme Events" - U.S. Environmental Protection Agency (EPA)
"Climate Change and Extreme Heat Events" - Centers for Disease Control and Prevention (CDC)
"Heat Waves and Their Effects on Public Health" - National Institute of Environmental Health Sciences (NIEHS)
"Healthy People 2030: Promoting Healthy Communities Through Climate Resilience" - Office of Disease Prevention and Health Promotion (ODPHP)
"Extreme Heat Events and Mortality" - Environmental Health Perspectives
"Heat Stress and Its Effect on Worker Performance: An Overview" - Journal of Occupational and Environmental Hygiene
"Climate Variability and Extremes in the Context of Sustainable Development Goals" - Sustainability Journal
[model continues writing references]
default Assistant chat prompt in ooba
AI
How can I help you today?
You
write a joke about llamas
AI
Why don't llamas ever go on dates? Because they always spit in your face!
AI
How can I help you today?
You
how warm is it in Ghambia in May on average?
AI
On average, Gambia experiences temperatures ranging from 25°C (77°F) during the day to around 18°C (64°F) at night in May. However, this can vary depending on specific locations within the country. It's always best to check local weather forecasts before planning your trip.
Thanks for the very detailed test. I assumed ChatML format would be what it responds to given the Axolotl sharegpt setting during fine-tuning. I have not played with this model extensively; I generally just load the models in ooba and use the defaults and most of the time it gives coherent responses without much configuration needed.
Do you still have the config file that you've used for training? I think it should be possible to re-generate the dataset and preview what the model actually saw during the training - I would be interesting in doing this. Are you still thinking about training a Yi-34B spicyboros for a few more epochs? If so, I think it would be nice if you get prompt format in place for the next training. It's true that bigger models generalize better and may do fine without a prompt, but using the prompt it knows will give you the results that are closest to the dataset that a model was trained on, so you can get more efficient training.
I also have a small question that doesn't deserve a separate discussion - is the model name spicyboros as opposed to airoboros purely an aesthetic choice and the model was trained on airoboros 3.1 dataset, or was this model actually trained on de-alignment dataset from spicyboros-2.2? Jon Durbin later removed that dataset from HF (I have it saved up though). I believe that happened right after a research paper came out that was basically putting a blame for releasing unrestricted models on him. So, I don't think spicyboros had any releases after 2.2
Take a look at this axolotl config.
https://huggingface.co/ehartford/dolphin-2_2-yi-34b/blob/main/configs/dolphin-yi-34b.yml
I believe that you will need to set conversation parameter next to type to get desired prompt format.
Thanks. I didn't notice that extra setting previously. I'm not sure what the default format is if you don't set that. I'm rerunning the fine-tuning with conversation set, but it's going to take a while to finish.
I also have a small question that doesn't deserve a separate discussion - is the model name spicyboros as opposed to airoboros purely an aesthetic choice and the model was trained on airoboros 3.1 dataset, or was this model actually trained on de-alignment dataset from spicyboros-2.2? Jon Durbin later removed that dataset from HF (I have it saved up though). I believe that happened right after a research paper came out that was basically putting a blame for releasing unrestricted models on him. So, I don't think spicyboros had any releases after 2.2
Jon gave me access to the dealignment dataset...
Thanks for following up. Please upload a LoRA adapter file if you end up releasing the new model, I would like to try the re-trained version once it's cooked.