alexmarques's picture
Update README.md
9feb924 verified
metadata
datasets:
  - theblackcat102/evol-codealpaca-v1
language:
  - en
pipeline_tag: text-generation

SparseLlama-2-7b-evolcodealpaca-pruned_50.2of4

Model Overview

  • Model Architecture: Llama-2
    • Input: Text
    • Output: Text
  • Model Optimizations:
    • Pruned: 50% 2:4
  • Release Date: 7/2/2024
  • Version: 1.0
  • Model Developers: Neural Magic

Compressed version of Llama-2-7b specialized for code-generation. This model was obtained by fine-tuning the Sparse Foundational model SparseLlama-2-7b-pruned_50.2of4 on the evol-codealpaca-v1 dataset. SquareHead knowledge distillation was used with Llama-2-7b-evolcodealpaca as teacher. It achieves HumanEval pass@1 of 34.58%, whereas the dense Llama-2-7b-evolcodealpaca model achieves 32.03%.

This model was produced as part if Neural Magic's Sparse Foundational Models initiative, and demostrates the capability of Sparse Foundational Models to transfer to the code-generation domain.

Model Optimizations

This model is derived from the Sparse Foundational model Sparse-Llama-2-7b-pruned_50.2of4, which was obtained by applying the SparseGPT algorithm to prune Llama-2-7b to 50% sparsity with a 2:4 mask. This optimization reduces the number of parameters by 50%, reducing the disk size and FLOPs by the same level.

Evaluation

This model was evaluated in the HumanEval benchmark using the bigcode-evaluation-harness.

Accuracy

Model HumanEval pass@1 Recovery
Llama-2-7b-evolcodealpaca 32.03% --
SparseLlama-2-7b-evolcodealpaca-pruned_50.2of4 34.58% 108%