Edit model card

Model Card for Model ID

Model Details

Model Description

MoM: Mixture of Mixture

This Model is a first test to combine Jamba architecture with mixture of attention head and mixture of depth.

Mamba and attention layers are in bf16 precision and the rest is in 1.58bits precision

107M over a total of 1025M parameters are in bf16 precision ~ 10% of the parameters are in bf16

The goal is to developpe and test if this kind of architectures have not too much quality loss for a fast inference.

  • Model type: Mixture of attention head mixture of depth and mixture of expert with 1.58bits linear layer for MLP
  • License: Apache licence 2.0

Model Sources [optional]

How to Get Started with the Model

If you want to test this model please look at this repo at this commit

Training Details

Training Data

We use the first 100k data of Locutusque/UltraTextbooks to train this model

Training Procedure

We use adam-8 bits with default betas and epsilon values

Preprocessing [optional]

The data fit the model max length i.e. 512 tokens

Training Hyperparameters

Please look at the wandb meta data or the train.py in the repo to see the hyperparameters

Technical Specifications [optional]

Compute Infrastructure

Hardware

  • one 4070 ti GPU

Software

  • pytorch, transformers etc
Downloads last month
9
Safetensors
Model size
1.03B params
Tensor type
BF16
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Dataset used to train Ostixe360/MoMv3-M-A-mixed-precision