BigMaid-20B-v2.0

This is a merge of pre-trained language models created using mergekit. FP32 version

Tests

• model retains qualities of BigMaid-20B-v1.0 and it's also more creative and coherent.

Merge Details

Merge Method

• This model was merged using the passthrough merge method.
• Merge parameters were inspired by work of DavidAU and jukofyork
• thread
• thread

Models Merged

The following models were included in the merge:

• KatyTheCutie_EstopianMaid-13B

Configuration

The following YAML configuration was used to produce this model:

slices:
  - sources:
    - model: "./KatyTheCutie_EstopianMaid-13B"
      layer_range: [0, 16]
      parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
  - sources:
    - model: "./KatyTheCutie_EstopianMaid-13B"
      layer_range: [8, 24]
      parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
  - sources:
    - model: "./KatyTheCutie_EstopianMaid-13B"
      layer_range: [17, 32]
      parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
  - sources:
    - model: "./KatyTheCutie_EstopianMaid-13B"
      layer_range: [25, 40]
      parameters:
      scale:
        - filter: q_proj
          value: 0.7071067812
        - filter: k_proj
          value: 0.7071067812
        - value: 1
merge_method: passthrough
dtype: float32

All comments are greatly appreciated, download, test and if you appreciate my work, consider buying me my fuel: