The checkpoint you are trying to load has model type `bunny-qwen` but Transformers does not recognize this architecture.

#3
by catworld1212 - opened

I am getting this error when I try to run this model.

The checkpoint you are trying to load has model type bunny-qwen but Transformers does not recognize this
architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

Cognitive Computations org

@catworld1212 you need to pass trust_remote_code = True into AutoModelForCausalLM.from_pretrained

@qnguyen3 I did still same error please check my code:

    import torch
        import transformers
        from transformers import AutoModelForCausalLM, AutoTokenizer
        from PIL import Image
        import warnings
        import requests
        # disable some warnings
        transformers.logging.set_verbosity_error()
        transformers.logging.disable_progress_bar()
        warnings.filterwarnings('ignore')

        # set device
        torch.set_default_device('cuda')  # or 'cpu'

        model_name = 'cognitivecomputations/dolphin-vision-7b'

        # create model
        model = AutoModelForCausalLM.from_pretrained(
            model_name,
            torch_dtype=torch.float16,
            device_map='auto',
            trust_remote_code=True)
        tokenizer = AutoTokenizer.from_pretrained(
            model_name,
            trust_remote_code=True)

        # text prompt
        prompt = 'Describe this image in detail'

        messages = [
            {"role": "user", "content": f'<image>\n{prompt}'}
        ]
        text = tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=True
        )

        print(text)

        text_chunks = [tokenizer(chunk).input_ids for chunk in text.split('<image>')]
        input_ids = torch.tensor(text_chunks[0] + [-200] + text_chunks[1], dtype=torch.long).unsqueeze(0)

        # image, sample images can be found in images folder
        image = Image.open(requests.get(
            "https://uploads4.wikiart.org/images/paul-klee/death-for-the-idea-1915.jpg!Large.jpg", stream=True).raw)
        image_tensor = model.process_images([image], model.config).to(dtype=model.dtype)

        # generate
        output_ids = model.generate(
            input_ids,
            images=image_tensor,
            max_new_tokens=2048,
            use_cache=True)[0]

        print(tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True).strip())
This comment has been hidden

hi @qnguyen3 can you pls test the model?

Cognitive Computations org

yes i will try to test it tonight

Cognitive Computations org

@catworld1212 should be fixed, could you give it a check?

Works for me too :3
GettyImages-9261821821-5c69c1b7c9e77c0001675a49.jpg

The image portrays a contemporary living room bathed in natural light from a large window on the left. Dominating the center of the room is a beige sofa, comfortably flanked by a blue armchair and a wooden side table, each adorned with a plant, adding a touch of greenery to the space.

Above the sofa, four framed pictures of insects are neatly arranged on the white wall, creating a striking contrast against the plain background. To the right of the sofa stands a wooden bookshelf, its top decorated with an array of pottery and other objects, adding a personal touch to the room.

A wooden floor stretches out across the room, leading to a black and white striped rug that lies on the wood, adding a patterned element to the otherwise neutral palette. The room is a harmonious blend of modern design and personal style, with each piece of furniture and decor contributing to the overall aesthetic.

Cognitive Computations org

Can I add this to the model card? This is a killer example

Cognitive Computations org

@catworld1212 should be fixed, could you give it a check?

Thank you for fixing it!

Can I add this to the model card? This is a killer example

Sure!
It reminds me a lot of howlmms-lab/llava-next-interleave-qwen-7b describes an image

But this 7b dolphin model can't rate things on a scale of 1/10 it says as AI it can't do that even with a system prompt.

Sign up or log in to comment