README.md

metadata

language:
  - en
license: apache-2.0
library_name: transformers
base_model:
  - HuggingFaceH4/mistral-7b-anthropic
  - ajibawa-2023/Code-Mistral-7B
  - Undi95/BigL-7B
model-index:
  - name: autocodit
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: AI2 Reasoning Challenge (25-Shot)
          type: ai2_arc
          config: ARC-Challenge
          split: test
          args:
            num_few_shot: 25
        metrics:
          - type: acc_norm
            value: 66.38
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=adowu/autocodit
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: HellaSwag (10-Shot)
          type: hellaswag
          split: validation
          args:
            num_few_shot: 10
        metrics:
          - type: acc_norm
            value: 84.82
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=adowu/autocodit
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU (5-Shot)
          type: cais/mmlu
          config: all
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 65.09
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=adowu/autocodit
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: TruthfulQA (0-shot)
          type: truthful_qa
          config: multiple_choice
          split: validation
          args:
            num_few_shot: 0
        metrics:
          - type: mc2
            value: 59.95
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=adowu/autocodit
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: Winogrande (5-shot)
          type: winogrande
          config: winogrande_xl
          split: validation
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 80.51
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=adowu/autocodit
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GSM8k (5-shot)
          type: gsm8k
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 60.65
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=adowu/autocodit
          name: Open LLM Leaderboard

AUTOCODIT

Description

This model represents an innovative fusion of three cutting-edge language models: BigL-7B, Code-Mistral-7B, and mistral-7b-anthropic, leveraging the strengths of each to create a more powerful and versatile tool. The integration process employs the TIES merge method, meticulously combining these models to enhance performance and adaptability across a broad spectrum of natural language processing tasks.

Creation Process

The model was crafted through a strategic merging process, utilizing the TIES merge method. This approach was chosen for its effectiveness in preserving the unique capabilities of each constituent model while ensuring seamless interoperability. The base model for this fusion was HuggingFaceH4/mistral-7b-anthropic, selected for its robust architecture and performance.

The merge parameters were carefully calibrated to achieve the optimal balance between the models, with the following configuration:

• BigL-7B was integrated with a density of 0.9 and a weight of 0.8, contributing its extensive language understanding and generation capabilities.
• Code-Mistral-7B was incorporated with a density of 0.7 and a weight of 0.7, enhancing the model's proficiency in code-related tasks and technical language comprehension.
• mistral-7b-anthropic served as the foundation, with its parameters set to a density of 0.9 and a weight of 0.8, ensuring the model's general language processing abilities remained at the forefront.

Features

• Model Type: MistralForCausalLM
• Vocabulary Size: 32,000 tokens, encompassing a wide array of linguistic elements for comprehensive language coverage.
• Maximum Position Embeddings: 32,768, facilitating the processing of extended passages of text.
• Hidden Size: 4,096, enabling the model to capture complex patterns and nuances in the data.
• Num Attention Heads: 32, allowing for detailed attention to various aspects of the input.
• Num Hidden Layers: 32, providing depth to the model's understanding and generation capabilities.

Applications This model is adept at a wide range of natural language processing tasks, including but not limited to text generation, language translation, code synthesis, and more. Its unique blend of features from BigL-7B, Code-Mistral-7B, and mistral-7b-anthropic makes it particularly effective in scenarios requiring a deep understanding of both human and programming languages.

---

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	69.57
AI2 Reasoning Challenge (25-Shot)	66.38
HellaSwag (10-Shot)	84.82
MMLU (5-Shot)	65.09
TruthfulQA (0-shot)	59.95
Winogrande (5-shot)	80.51
GSM8k (5-shot)	60.65