databricks/dolly-v2-7b · extremely poor repsonses

Feb 28

•

Could someone please help me understand what might be wrong with Dolly 7b (or with the way i am trying to use that)

Dolly 7b model provides almost irrelevant responses to some queries (actually almost all queries I tried). I am not sure if i am doing something very wrong or if the model has a very limited area of applications.

some examples below. many online 7b models I tried provided significantly accurate answers to these ones.

what is the FRL level of 600mm concrete?
response: 3.2 pounds
what is the fire resistance level of 600mm concrete?
response: slippery when wet.

so, a less technical query here, with context;

jane had a Paw Patrol toy yesterday. She gave it to her cousin this morning. who has the paw patrol toy today?
response: jane has peppa pig.
jane had a Paw Patrol toy yesterday. She gave the Paw Patrol toy to her cousin this morning. who has the paw patrol toy today?
response: jane has Paw patrol doll.

srowen

Databricks org Feb 28

Don't use Dolly, there are much better models at this size now.

However, it doesn't look like you are using its prompting format. See the model card for how you call the model, if you're manually feeding text into the model without its model class.

tomenks

Feb 28

Thank you so much for the response. I will have a look at the correct way of using prompts. I used langchain too, with similar success. Would you mind giving a couple of examples of the other potential models which could be used for commercial purposes, I mean with exactly same licensing as dolly would be based on?

Thanks again for the repsonse.

srowen

Databricks org Feb 28

If you're OK with the LLama 2 terms, it's pretty permissive unless you're a big company. LLama 2 models are a reasonable choice
https://llama.meta.com/llama-downloads/

tomenks

Feb 28

Thanks Srowen. I am an indvidual, so should be OK. However, I really wanted to use Dolly due to its fully open nature. Would you know if 12b version would be good enough to meet the levels of more state of the art models? sorry if I am making this discussion interafce look like a dialog. I could not find a personal message section.