task_categories:text-generation task_categories:question-answering task_categories:conversational language:zh language:en language:ja language:de license:gpl-3.0 alpaca llama guanaco doi:10.57967/hf/0570 region:us
---
license: gpl-3.0
task_categories:
- text-generation
- question-answering
- conversational
language:
- zh
- en
- ja
- de
tags:
- alpaca
- llama
- guanaco
---
# GuanacoDataset

**News: We're heading towards multimodal VQA, with blip2-flan-t5-xxl Alignment to Guannaco 7B LLM.**

Still under construction: [GuanacoVQA weight](https://huggingface.co/JosephusCheung/GuanacoVQA) & [GuanacoVQA Dataset](https://huggingface.co/datasets/JosephusCheung/GuanacoVQADataset) 

**Notice: Effective immediately, the Guanaco and its associated dataset are now licensed under the GPLv3.**

Released weights: 

- [Guanaco α](https://huggingface.co/JosephusCheung/Guanaco) 

The dataset for the [Guanaco model](https://guanaco-model.github.io/) is designed to enhance the multilingual capabilities and address various linguistic tasks. It builds upon the 175 tasks from the Alpaca model by providing rewrites of seed tasks in different languages and adding new tasks specifically designed for English grammar analysis, natural language understanding, cross-lingual self-awareness, and explicit content recognition. The dataset comprises a total of 534,530 entries, generated at a low cost of $6K.

- Free chat dialogues without System input: 32,880 entries (recent update) - in English zh-Hans zh-Hant-TW Japanese Deutsch

  *To test 0-shot tasks of Japanese & Deutsch on original 175 tasks with finetuning on chat only.*

- Chat dialogues with System input: 16,087 entries (recent update) - in English zh-Hans zh-Hant-TW zh-Hant-HK

**A new additional dataset is released, separated and larger dataset is available for different languages.**

The original 175 tasks were translated into 4 versions and regenerated independently:

Below is the details of **mixed data**:

- Japanese (Ja-JP - recently updated) 7,485 entries

- Simplified Chinese (zh-Hans): 27,808 entries

- Traditional Chinese (Taiwan) (zh-Hant-TW): 21,481 entries

- Traditional Chinese (Hong Kong) (zh-Hant-HK): 19247 entries

- English: 20K+ entries, not from Alpaca

Besides, a mini version of 52K multi-lang dataset is released with:

- Japanese (Ja-JP - recently updated) 7,485 entries

- Simplified Chinese (zh-Hans): 5,439 entries

- Traditional Chinese (Taiwan) (zh-Hant-TW): 9,322 entries

- Traditional Chinese (Hong Kong) (zh-Hant-HK): 9,954 entries

- English: 20,024 entries, not from Alpaca

The mini version is included in the full non-chat dataset.

**Additional dataset** *separated by language （temporary）*:

*This additional dataset should only be used for additional training if using mixed data did not yield good results. Using it directly will not produce good results.*

This part of the data will be merged into the main dataset at the appropriate time.

- Chinese: 117,166 entries
  - Simplified Chinese (zh-Hans): 92,530 entries
  - Traditional Chinese (Taiwan) (zh-Hant-TW): 14,802 entries
  - Traditional Chinese (Hong Kong) (zh-Hant-HK): 9,834 entries

- Japanese (Ja-JP - recently updated) 60,772 entries

In addition to the language-specific tasks, the dataset includes new tasks that aim to improve the model's performance in English grammar analysis, natural language understanding, cross-lingual self-awareness, and explicit content recognition. These new tasks ensure that the Guanaco model is well-rounded and capable of handling a wide range of challenges in the field of natural language processing.

By incorporating this diverse and comprehensive dataset into the Guanaco model, we aim to provide researchers and academics with a powerful tool for studying instruction-following language models in a multilingual context. The dataset's design encourages the development of more robust and versatile models capable of addressing complex linguistic tasks across different languages and domains.

**Additional dataset** *Paper/General-QA*:

The Paper/General-QA dataset is a collection of questions and answers constructed for AI-generated papers or general texts in English, Chinese, Japanese, and German. The question dataset contains 106,707 questions, and the answer dataset contains 99,292 answers. The purpose of this dataset is to generate paragraph-level answers to questions posed about lengthy documents such as PDFs. Similar questions are combined to form a tree-like structure, and graph theory algorithms are used to process user questions, content summaries, and contextual logic.

*It is worth noting that some ChatGPT applications claim to be able to read PDFs, but they do not actually read the entire article. Instead, they compare the user's input question with segmented paragraphs of the article, select the most similar paragraph, and insert it as the answer. This is not true language model reading, but rather a form of deception.*

**Note: I intentionally mixed up entries and languages to prevent anyone from solely selecting certain language entries for finetuning. This is not only unhelpful for the community, but also because some tasks are 0-shot in specific languages, please use the complete dataset for finetuning.**

## To-Do List:

- Expand language support in the dataset:
  
  Incorporate additional languages such as Japanese, German, and more into the dataset. This expansion should include task examples that cover advanced grammar analysis and dialogue understanding for these languages.

- Create a dialogue-oriented Chatbot dataset: 

  Develop a dataset specifically designed for conversation-based applications, containing examples that facilitate the model's ability to engage in interactive and dynamic dialogues with users.

- Add Toolformer-supporting tasks: 

  Introduce tasks that train the model to autonomously call external APIs using Toolformer, allowing the model to access and utilize various web services and data sources, thereby enhancing its problem-solving capabilities.

- Develop tasks for rapid integration of external knowledge: 

  Design tasks that encourage the model to quickly incorporate knowledge from external sources such as search engines and artificial intelligence knowledge engines. These tasks would be particularly beneficial for smaller models with limited knowledge reserves, enabling them to efficiently utilize external information to respond to user queries.

## Recent News

We've noticed a recent entrant in the field, the QLoRa method, which we find concerning due to its attempt to piggyback on the reputation of Guanaco. We strongly disapprove of such practices. QLoRa, as far as we can tell, lacks mathematical robustness and its performance significantly trails behind that of GPTQ and advancements such as PEFT fine-tuning, which have been successful in improving upon it.

Guanaco has been diligent, consistently releasing multilingual datasets since March 2023, along with publishing weights that are not only an enhanced version of GPTQ but also support multimodal VQA and have been optimized for 4-bit. Despite the substantial financial investment of tens of thousands of dollars in distilling data from OpenAI's GPT models, we still consider these efforts to be incremental.

We, however, aim to move beyond the incremental:

1. We strive to no longer rely on distillation data from OpenAI: We've found that relying on GPT-generated data impedes significant breakthroughs. Furthermore, this approach has proven to be disastrous when dealing with the imbalances in multilingual tasks.

2. We're focusing on the enhancement of quantization structure and partial native 4-bit fine-tuning: We are deeply appreciative of the GPTQ-Llama project for paving the way in state-of-the-art LLM quantization. Its unique qualities, especially at the 7B size, are facilitating significant progress in multilingual and multimodal tasks.

3. We plan to utilize visual data to adjust our language models: We believe this will fundamentally address the issues of language imbalance, translation inaccuracies, and the lack of graphical logic in LLM.

While our work is still in the early stages, we're determined to break new ground in these areas. Our critique of QLoRa's practices does not stem from animosity but rather from the fundamental belief that innovation should be rooted in originality, integrity, and substantial progress.
  