# Multilingual LLAMA Bot General (m-llama-bot-general)

### Method
 - Promt-Tuning/Adapter-tuning
 - Parameters

| Parameters | No.                  |
|------------|----------------------|
| All        | 6739645440 (100.00%) |
| Trainable  | 6537216 (0.02%)      |
| Freezed    | 1229824 (99.98%)     |


<img src="https://huggingface.co/anhdungitvn/m-llama-bot-general-7b/resolve/main/adapters.png"  width="800">


### Model
Adapter
```
LAYER NAME                      #PARAMS      RATIO       MEM(MB)
--model:                      1,229,824    100.00%          4.69
  --layers:                       1,024      0.08%          0.00
    --0           
      --attention 
        --gate:                      32      0.00%          0.00
    --1           
      --attention 
        --gate:                      32      0.00%          0.00
    --2           
      --attention 
        --gate:                      32      0.00%          0.00
    --3           
      --attention 
        --gate:                      32      0.00%          0.00
    --4           
      --attention 
        --gate:                      32      0.00%          0.00
    --5           
      --attention 
        --gate:                      32      0.00%          0.00
    --6           
      --attention 
        --gate:                      32      0.00%          0.00
    --7           
      --attention 
        --gate:                      32      0.00%          0.00
    --8           
      --attention 
        --gate:                      32      0.00%          0.00
    --9           
      --attention 
        --gate:                      32      0.00%          0.00
    --10          
      --attention 
        --gate:                      32      0.00%          0.00
    --11          
      --attention 
        --gate:                      32      0.00%          0.00
    --12          
      --attention 
        --gate:                      32      0.00%          0.00
    --13          
      --attention 
        --gate:                      32      0.00%          0.00
    --14          
      --attention 
        --gate:                      32      0.00%          0.00
    --15          
      --attention 
        --gate:                      32      0.00%          0.00
    --16          
      --attention 
        --gate:                      32      0.00%          0.00
    --17          
      --attention 
        --gate:                      32      0.00%          0.00
    --18          
      --attention 
        --gate:                      32      0.00%          0.00
    --19          
      --attention 
        --gate:                      32      0.00%          0.00
    --20          
      --attention 
        --gate:                      32      0.00%          0.00
    --21          
      --attention 
        --gate:                      32      0.00%          0.00
    --22          
      --attention 
        --gate:                      32      0.00%          0.00
    --23          
      --attention 
        --gate:                      32      0.00%          0.00
    --24          
      --attention 
        --gate:                      32      0.00%          0.00
    --25          
      --attention 
        --gate:                      32      0.00%          0.00
    --26          
      --attention 
        --gate:                      32      0.00%          0.00
    --27          
      --attention 
        --gate:                      32      0.00%          0.00
    --28          
      --attention 
        --gate:                      32      0.00%          0.00
    --29          
      --attention 
        --gate:                      32      0.00%          0.00
    --30          
      --attention 
        --gate:                      32      0.00%          0.00
    --31          
      --attention 
        --gate:                      32      0.00%          0.00
  --adapter_query 
    --weight:                 1,228,800     99.92%          4.69
```


Model
```
LAYER NAME                      #PARAMS      RATIO       MEM(MB)
--model:                  6,739,645,440    100.00%      12854.85
  --tok_embeddings     
    --weight:               131,072,000      1.94%        250.00
  --layers:               6,476,268,544     96.09%      12352.50
    --0:                    202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --1:                    202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --2:                    202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --3:                    202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --4:                    202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --5:                    202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --6:                    202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --7:                    202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --8:                    202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --9:                    202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --10:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --11:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --12:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --13:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --14:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --15:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --16:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --17:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --18:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --19:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --20:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --21:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --22:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --23:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --24:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --25:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --26:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --27:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --28:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --29:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --30:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
    --31:                   202,383,392      3.00%        386.02
      --attention:           67,108,896      1.00%        128.00
        --gate:                      32      0.00%          0.00
        --wq:                16,777,216      0.25%         32.00
        --wk:                16,777,216      0.25%         32.00
        --wv:                16,777,216      0.25%         32.00
        --wo:                16,777,216      0.25%         32.00
      --feed_forward:       135,266,304      2.01%        258.00
        --w1:                45,088,768      0.67%         86.00
        --w2:                45,088,768      0.67%         86.00
        --w3:                45,088,768      0.67%         86.00
      --attention_norm 
        --weight:                 4,096      0.00%          0.01
      --ffn_norm       
        --weight:                 4,096      0.00%          0.01
  --norm               
    --weight:                     4,096      0.00%          0.01
  --output             
    --weight:               131,072,000      1.94%        250.00
  --adapter_query      
    --weight:                 1,228,800      0.02%          2.34
```
### References
 - LLAM: <a href="https://ai.facebook.com/blog/large-language-model-llama-meta-ai/" download>**LLaMA: A foundational, 65-billion-parameter large language model**</a>
 - Prompt Tuning: <a href="https://arxiv.org/abs/2104.08691" download>**The Power of Scale for Parameter-Efficient Prompt Tuning**</a>
 - Prompt Tuning v2: <a href="https://arxiv.org/abs/2110.07602" download>**P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks**</a>
{"license": "other"}
arxiv:2104.08691 arxiv:2110.07602 license:other region:us
TAGS
#arxiv-2104.08691 #arxiv-2110.07602 #license-other #region-us 

Multilingual LLAMA Bot General (m-llama-bot-general)
====================================================


### Method


* Promt-Tuning/Adapter-tuning
* Parameters



<img src="URL width="800">


### Model


Adapter


Model


### References


* LLAM: <a href="URL download>LLaMA: A foundational, 65-billion-parameter large language model
* Prompt Tuning: <a href="URL download>The Power of Scale for Parameter-Efficient Prompt Tuning
* Prompt Tuning v2: <a href="URL download>P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks



### Method


* Promt-Tuning/Adapter-tuning
* Parameters



<img src="URL width="800"> ### Model


Adapter


Model ### References


* LLAM: <a href="URL download>LLaMA: A foundational, 65-billion-parameter large language model
* Prompt Tuning: <a href="URL download>The Power of Scale for Parameter-Efficient Prompt Tuning
* Prompt Tuning v2: <a href="URL download>P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
TAGS
#arxiv-2104.08691 #arxiv-2110.07602 #license-other #region-us 
 ### Method


* Promt-Tuning/Adapter-tuning
* Parameters



<img src="URL width="800"> ### Model


Adapter


Model ### References


* LLAM: <a href="URL download>LLaMA: A foundational, 65-billion-parameter large language model
* Prompt Tuning: <a href="URL download>The Power of Scale for Parameter-Efficient Prompt Tuning
* Prompt Tuning v2: <a href="URL download>P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks