|
--- |
|
license: apache-2.0 |
|
tags: |
|
- merge |
|
--- |
|
|
|
# openmixtral-6x7b-merged_v2 |
|
|
|
openmixtral-6x7b-merged_v2 is a merge of the following models: |
|
|
|
## 🧩 Configuration |
|
|
|
```yaml |
|
base_model: mlabonne/Marcoro14-7B-slerp |
|
experts: |
|
- source_model: openchat/openchat-3.5-1210 |
|
positive_prompts: |
|
- "chat" |
|
- "assistant" |
|
- "tell me" |
|
- "explain" |
|
- source_model: Weyaxi/Einstein-v4-7B |
|
positive_prompts: |
|
- "physics" |
|
- "biology" |
|
- "chemistry" |
|
- "science" |
|
- source_model: BioMistral/BioMistral-7B |
|
positive_prompts: |
|
- "medical" |
|
- "pubmed" |
|
- "healthcare" |
|
- "health" |
|
- source_model: beowolx/CodeNinja-1.0-OpenChat-7B |
|
positive_prompts: |
|
- "code" |
|
- "python" |
|
- "javascript" |
|
- "programming" |
|
- "algorithm" |
|
- source_model: maywell/PiVoT-0.1-Starling-LM-RP |
|
positive_prompts: |
|
- "storywriting" |
|
- "write" |
|
- "scene" |
|
- "story" |
|
- "character" |
|
- source_model: WizardLM/WizardMath-7B-V1.1 |
|
positive_prompts: |
|
- "reason" |
|
- "math" |
|
- "mathematics" |
|
- "solve" |
|
- "count" |
|
tokenizer_source: union |
|
``` |
|
## 💻 Usage |
|
```python |
|
|
|
# install llamacpp see here: https://github.com/ggerganov/llama.cpp |
|
# or other GGUF tool like llamacpp-python: https://github.com/abetlen/llama-cpp-python |
|
|
|
MODEL_REPO="openmixtral-6x7b-merged_v2-GGUF" |
|
MODEL_NAME="openmixtral-6x7b-merged_v2" |
|
method="Q4_K_M" |
|
|
|
prompt="why the sky is blue" |
|
|
|
qtype = f"{MODEL_REPO}/{MODEL_NAME.lower()}.{method.upper()}.gguf" |
|
!./llama.cpp/main -m {qtype} -n 128 --color -ngl 0 -p "{prompt}" |
|
``` |
|
Log Result |
|
``` |
|
Log start |
|
main: build = 2382 (621e86b3) |
|
main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu |
|
main: seed = 1710306347 |
|
ggml_init_cublas: GGML_CUDA_FORCE_MMQ: no |
|
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes |
|
ggml_init_cublas: found 1 CUDA devices: |
|
Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9, VMM: yes |
|
llama_model_loader: loaded meta data with 25 key-value pairs and 803 tensors from openmixtral-6x7b-merged_v2-GGUF/openmixtral-6x7b-merged_v2.Q4_K_M.gguf (version GGUF V3 (latest)) |
|
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. |
|
llama_model_loader: - kv 0: general.architecture str = llama |
|
llama_model_loader: - kv 1: general.name str = . |
|
llama_model_loader: - kv 2: llama.context_length u32 = 32768 |
|
llama_model_loader: - kv 3: llama.embedding_length u32 = 4096 |
|
llama_model_loader: - kv 4: llama.block_count u32 = 32 |
|
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336 |
|
llama_model_loader: - kv 6: llama.rope.dimension_count u32 = 128 |
|
llama_model_loader: - kv 7: llama.attention.head_count u32 = 32 |
|
llama_model_loader: - kv 8: llama.attention.head_count_kv u32 = 8 |
|
llama_model_loader: - kv 9: llama.expert_count u32 = 6 |
|
llama_model_loader: - kv 10: llama.expert_used_count u32 = 2 |
|
llama_model_loader: - kv 11: llama.attention.layer_norm_rms_epsilon f32 = 0.000010 |
|
llama_model_loader: - kv 12: llama.rope.freq_base f32 = 10000.000000 |
|
llama_model_loader: - kv 13: general.file_type u32 = 15 |
|
llama_model_loader: - kv 14: tokenizer.ggml.model str = llama |
|
llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32000] = ["<unk>", "<s>", "</s>", "<0x00>", "<... |
|
llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32000] = [0.000000, 0.000000, 0.000000, 0.0000... |
|
llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32000] = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... |
|
llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 1 |
|
llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 2 |
|
llama_model_loader: - kv 20: tokenizer.ggml.unknown_token_id u32 = 0 |
|
llama_model_loader: - kv 21: tokenizer.ggml.padding_token_id u32 = 1 |
|
llama_model_loader: - kv 22: tokenizer.ggml.add_bos_token bool = true |
|
llama_model_loader: - kv 23: tokenizer.ggml.add_eos_token bool = false |
|
llama_model_loader: - kv 24: general.quantization_version u32 = 2 |
|
llama_model_loader: - type f32: 65 tensors |
|
llama_model_loader: - type f16: 32 tensors |
|
llama_model_loader: - type q4_K: 593 tensors |
|
llama_model_loader: - type q6_K: 113 tensors |
|
llm_load_vocab: special tokens definition check successful ( 259/32000 ). |
|
llm_load_print_meta: format = GGUF V3 (latest) |
|
llm_load_print_meta: arch = llama |
|
llm_load_print_meta: vocab type = SPM |
|
llm_load_print_meta: n_vocab = 32000 |
|
llm_load_print_meta: n_merges = 0 |
|
llm_load_print_meta: n_ctx_train = 32768 |
|
llm_load_print_meta: n_embd = 4096 |
|
llm_load_print_meta: n_head = 32 |
|
llm_load_print_meta: n_head_kv = 8 |
|
llm_load_print_meta: n_layer = 32 |
|
llm_load_print_meta: n_rot = 128 |
|
llm_load_print_meta: n_embd_head_k = 128 |
|
llm_load_print_meta: n_embd_head_v = 128 |
|
llm_load_print_meta: n_gqa = 4 |
|
llm_load_print_meta: n_embd_k_gqa = 1024 |
|
llm_load_print_meta: n_embd_v_gqa = 1024 |
|
llm_load_print_meta: f_norm_eps = 0.0e+00 |
|
llm_load_print_meta: f_norm_rms_eps = 1.0e-05 |
|
llm_load_print_meta: f_clamp_kqv = 0.0e+00 |
|
llm_load_print_meta: f_max_alibi_bias = 0.0e+00 |
|
llm_load_print_meta: n_ff = 14336 |
|
llm_load_print_meta: n_expert = 6 |
|
llm_load_print_meta: n_expert_used = 2 |
|
llm_load_print_meta: pooling type = 0 |
|
llm_load_print_meta: rope type = 0 |
|
llm_load_print_meta: rope scaling = linear |
|
llm_load_print_meta: freq_base_train = 10000.0 |
|
llm_load_print_meta: freq_scale_train = 1 |
|
llm_load_print_meta: n_yarn_orig_ctx = 32768 |
|
llm_load_print_meta: rope_finetuned = unknown |
|
llm_load_print_meta: ssm_d_conv = 0 |
|
llm_load_print_meta: ssm_d_inner = 0 |
|
llm_load_print_meta: ssm_d_state = 0 |
|
llm_load_print_meta: ssm_dt_rank = 0 |
|
llm_load_print_meta: model type = 7B |
|
llm_load_print_meta: model ftype = Q4_K - Medium |
|
llm_load_print_meta: model params = 35.43 B |
|
llm_load_print_meta: model size = 19.96 GiB (4.84 BPW) |
|
llm_load_print_meta: general.name = . |
|
llm_load_print_meta: BOS token = 1 '<s>' |
|
llm_load_print_meta: EOS token = 2 '</s>' |
|
llm_load_print_meta: UNK token = 0 '<unk>' |
|
llm_load_print_meta: PAD token = 1 '<s>' |
|
llm_load_print_meta: LF token = 13 '<0x0A>' |
|
llm_load_tensors: ggml ctx size = 0.31 MiB |
|
llm_load_tensors: offloading 0 repeating layers to GPU |
|
llm_load_tensors: offloaded 0/33 layers to GPU |
|
llm_load_tensors: CPU buffer size = 20441.87 MiB |
|
.................................................................................................... |
|
llama_new_context_with_model: n_ctx = 512 |
|
llama_new_context_with_model: freq_base = 10000.0 |
|
llama_new_context_with_model: freq_scale = 1 |
|
llama_kv_cache_init: CUDA_Host KV buffer size = 64.00 MiB |
|
llama_new_context_with_model: KV self size = 64.00 MiB, K (f16): 32.00 MiB, V (f16): 32.00 MiB |
|
llama_new_context_with_model: CUDA_Host input buffer size = 10.01 MiB |
|
llama_new_context_with_model: CUDA_Host compute buffer size = 114.52 MiB |
|
llama_new_context_with_model: graph splits (measure): 1 |
|
|
|
system_info: n_threads = 12 / 24 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | |
|
sampling: |
|
repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000 |
|
top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800 |
|
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000 |
|
sampling order: |
|
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature |
|
generate: n_ctx = 512, n_batch = 512, n_predict = 128, n_keep = 1 |
|
|
|
|
|
why the sky is blue |
|
|
|
# Why The Sky is Blue: Rayleigh Scattering |
|
|
|
## Introduction to Rayleigh Scattering |
|
|
|
Have you ever wondered why the sky appears blue during a clear day? You might think that it's because there are more blue molecules in the Earth's atmosphere, or maybe that the sun emits blue light. However, neither of these explanations is accurate. The true reason behind the blue color of the sky has to do with a phenomenon called Rayleigh scattering. |
|
|
|
Rayleigh scattering is a fundamental process in physics and atmospheric science that occurs when sunlight interacts with mole |
|
llama_print_timings: load time = 1605.68 ms |
|
llama_print_timings: sample time = 14.83 ms / 128 runs ( 0.12 ms per token, 8632.32 tokens per second) |
|
llama_print_timings: prompt eval time = 312.86 ms / 6 tokens ( 52.14 ms per token, 19.18 tokens per second) |
|
llama_print_timings: eval time = 21083.26 ms / 127 runs ( 166.01 ms per token, 6.02 tokens per second) |
|
llama_print_timings: total time = 21439.97 ms / 133 tokens |
|
Log end |
|
``` |
|
|
|
## Quantized Model Response |
|
can you spot the difference? |
|
``` |
|
#### User Prompt |
|
why the sky is blue |
|
|
|
#### Q2_K Response |
|
The sky appears blue because of a phenomenon called Rayleigh scattering. This process occurs when sunlight (which is made up of many different colors or wavelengths of light) interacts with particles in the atmosphere like air molecules, water vapor and dust. The shorter wavelengths of light, such as violet and blue, are more easily scattered than longer wavelengths like red and yellow, due to their smaller size compared to the wavelengths they're interacting with. |
|
When we look up at the sky, most of these scattered shorter-wavelength lights (violet, blue) are |
|
|
|
#### Q3_K_M Response |
|
In the late 19th century, an English scientist named Lord Rayleigh and a Scottish scientist named William Thomson (later Lord Kelvin) embarked on a quest to solve one of the most enduring mysteries in science: why is the sky blue? Their findings, which are now widely accepted, can be attributed to a combination of physics principles and molecular behavior. |
|
|
|
The color we perceive in the sky is actually the result of how sunlight interacts with various gases and particles in our atmosphere. Sunlight is composed of multiple colors, or wavelengths, ranging from red at one end |
|
|
|
#### Q4_K_M Response |
|
The reason why we see the sky as blue is due to a natural phenomenon known as Rayleigh scattering. When sunlight enters the Earth’s atmosphere, it is made up of numerous colors or wavelengths, including blue, green, yellow, orange, and red. |
|
|
|
Rayleigh scattering occurs when the shorter-wavelength light, such as blue and violet colors, interacts with the molecules (mainly nitrogen and oxygen) in the atmosphere. These tiny particles scatter these short-wave lengths more effectively than longer ones like red or yellow, which results in the sky appearing blue to our eyes. |
|
|
|
#### Q5_K_M Response |
|
the blue color of the sky, a phenomenon that has been observed and admired since ancient times, can be explained through the principles of physics and atmospheric science. |
|
|
|
Sunlight, which appears white to us, is actually made up of many colors of light, each with a different wavelength. These colors include red, orange, yellow, green, blue, indigo, and violet; together, they are known as the visible light spectrum. |
|
|
|
When sunlight enters Earth's atmosphere, it encounters various gases, such as nitrogen and oxygen, which scatter the shorter wavelengths of light. |
|
|
|
#### Q6_K Response |
|
In order to understand why the sky appears blue, we must first understand how light behaves and interacts with air molecules. Visible light is a form of electromagnetic radiation that spans wavelengths between approximately 400 nanometers (violet) and 700 nanometers (red). When sunlight reaches Earth's atmosphere, it encounters various gases and particles that scatter or absorb different wavelengths of light. |
|
|
|
The scattering of light is due to the interaction between electromagnetic waves and free-moving electrons within air molecules, particularly nitrogen and oxygen...etc |
|
|
|
#### Q8_0 Response |
|
I’m just curious, it seems like a lot of people know this but I can’t recall ever being taught. Why exactly is the sky blue? I know it reflects sunlight and so forth but how does that lead to the blue color? |
|
|
|
The reason the sky appears blue has to do with a combination of Rayleigh scattering and Earth's atmosphere. Here's a brief explanation: |
|
|
|
1. Sunlight consists of multiple colors (wavelengths) including red, orange, yellow, green, blue, indigo, and violet. These different colors are represented by their respective wavelengths in...etc |
|
``` |
|
|
