Edit model card

output-model

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the breadcrumbs merge method using microsoft/Phi-3-mini-4k-instruct as a base.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:

slices:
  - sources:
      - model: microsoft/Phi-3.5-mini-instruct
        layer_range: [0, 32]
      - model: microsoft/Phi-3-mini-4k-instruct
        layer_range: [0, 32]  # Limité aux 36 premières couches
merge_method: breadcrumbs
base_model: microsoft/Phi-3-mini-4k-instruct
parameters:
  weight: 0.5  # Remplacer 'relative' par une valeur numérique
  normalize: true
  density: 0.9
  gamma: 0.01
  #t:
  #  - filter: self_attn
  #    value: [0.1, 0.25, 0.5, 0.75, 1]  # Ajustement des poids pour self_attn
  #  - filter: mlp
  #    value: [1, 0.75, 0.5, 0.25, 0.1]  # Ajustement des poids pour mlp
  #  - value: 0.5  # Valeur par défaut pour les autres couches

dtype: bfloat16
tokenizer_source: base
Downloads last month
15
Safetensors
Model size
3.82B params
Tensor type
BF16
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for Marsouuu/breadcrumbs3B