Edit model card

E5-Mistral-7B-Instruct-Embedding-GGUF

Original Model

intfloat/e5-mistral-7b-instruct

Run with LlamaEdge

  • LlamaEdge version: v0.8.2 and above

  • Prompt template

    • Prompt type: embedding
  • Context size: 4096

  • Run as LlamaEdge service

    wasmedge --dir .:. --nn-preload default:GGML:AUTO:e5-mistral-7b-instruct-Q5_K_M.gguf \
      llama-api-server.wasm \
      --prompt-template embedding \
      --ctx-size 4096 \
      --model-name e5-mistral-7b-instruct
    

Quantized GGUF Models

Name Quant method Bits Size Use case
e5-mistral-7b-instruct-Q2_K.gguf Q2_K 2 2.72 GB smallest, significant quality loss - not recommended for most purposes
e5-mistral-7b-instruct-Q3_K_L.gguf Q3_K_L 3 3.82 GB small, substantial quality loss
e5-mistral-7b-instruct-Q3_K_M.gguf Q3_K_M 3 3.52 GB very small, high quality loss
e5-mistral-7b-instruct-Q3_K_S.gguf Q3_K_S 3 3.16 GB very small, high quality loss
e5-mistral-7b-instruct-Q4_0.gguf Q4_0 4 4.11 GB legacy; small, very high quality loss - prefer using Q3_K_M
e5-mistral-7b-instruct-Q4_K_M.gguf Q4_K_M 4 4.37 GB medium, balanced quality - recommended
e5-mistral-7b-instruct-Q4_K_S.gguf Q4_K_S 4 4.14 GB small, greater quality loss
e5-mistral-7b-instruct-Q5_0.gguf Q5_0 5 5.00 GB legacy; medium, balanced quality - prefer using Q4_K_M
e5-mistral-7b-instruct-Q5_K_M.gguf Q5_K_M 5 5.13 GB large, very low quality loss - recommended
e5-mistral-7b-instruct-Q5_K_S.gguf Q5_K_S 5 5.00 GB large, low quality loss - recommended
e5-mistral-7b-instruct-Q6_K.gguf Q6_K 6 5.94 GB very large, extremely low quality loss
e5-mistral-7b-instruct-Q8_0.gguf Q8_0 8 7.7 GB very large, extremely low quality loss - not recommended
e5-mistral-7b-instruct-f16.gguf f16 8 14.5 GB very large, extremely low quality loss - not recommended

Quantized with llama.cpp b2334

Downloads last month
1,961
GGUF
Model size
7.24B params
Architecture
llama

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for second-state/E5-Mistral-7B-Instruct-Embedding-GGUF

Quantized
(1)
this model