llama_model_loader: loaded meta data with 36 key-value pairs and 197 tensors from Phi-3.5-mini-instruct-IMat-GGUF/Phi-3.5-mini-instruct.Q8_0.gguf.hardlink.gguf (version GGUF V3 (latest)) llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output. llama_model_loader: - kv 0: general.architecture str = phi3 llama_model_loader: - kv 1: general.type str = model llama_model_loader: - kv 2: general.name str = Phi 3.5 Mini Instruct llama_model_loader: - kv 3: general.finetune str = instruct llama_model_loader: - kv 4: general.basename str = Phi-3.5 llama_model_loader: - kv 5: general.size_label str = mini llama_model_loader: - kv 6: general.license str = mit llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/microsoft/Phi-... llama_model_loader: - kv 8: general.tags arr[str,3] = ["nlp", "code", "text-generation"] llama_model_loader: - kv 9: general.languages arr[str,1] = ["multilingual"] llama_model_loader: - kv 10: phi3.context_length u32 = 131072 llama_model_loader: - kv 11: phi3.rope.scaling.original_context_length u32 = 4096 llama_model_loader: - kv 12: phi3.embedding_length u32 = 3072 llama_model_loader: - kv 13: phi3.feed_forward_length u32 = 8192 llama_model_loader: - kv 14: phi3.block_count u32 = 32 llama_model_loader: - kv 15: phi3.attention.head_count u32 = 32 llama_model_loader: - kv 16: phi3.attention.head_count_kv u32 = 32 llama_model_loader: - kv 17: phi3.attention.layer_norm_rms_epsilon f32 = 0.000010 llama_model_loader: - kv 18: phi3.rope.dimension_count u32 = 96 llama_model_loader: - kv 19: phi3.rope.freq_base f32 = 10000.000000 llama_model_loader: - kv 20: general.file_type u32 = 7 llama_model_loader: - kv 21: phi3.attention.sliding_window u32 = 262144 llama_model_loader: - kv 22: phi3.rope.scaling.attn_factor f32 = 1.190238 llama_model_loader: - kv 23: tokenizer.ggml.model str = llama llama_model_loader: - kv 24: tokenizer.ggml.pre str = default llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,32064] = ["", "~~", "~~", "<0x00>", "<... llama_model_loader: - kv 26: tokenizer.ggml.scores arr[f32,32064] = [-1000.000000, -1000.000000, -1000.00... llama_model_loader: - kv 27: tokenizer.ggml.token_type arr[i32,32064] = [3, 3, 4, 6, 6, 6, 6, 6, 6, 6, 6, 6, ... llama_model_loader: - kv 28: tokenizer.ggml.bos_token_id u32 = 1 llama_model_loader: - kv 29: tokenizer.ggml.eos_token_id u32 = 32000 llama_model_loader: - kv 30: tokenizer.ggml.unknown_token_id u32 = 0 llama_model_loader: - kv 31: tokenizer.ggml.padding_token_id u32 = 32000 llama_model_loader: - kv 32: tokenizer.ggml.add_bos_token bool = false llama_model_loader: - kv 33: tokenizer.ggml.add_eos_token bool = false llama_model_loader: - kv 34: tokenizer.chat_template str = {% for message in messages %}{% if me... llama_model_loader: - kv 35: general.quantization_version u32 = 2 llama_model_loader: - type f32: 67 tensors llama_model_loader: - type q8_0: 130 tensors llm_load_vocab: special tokens cache size = 14 llm_load_vocab: token to piece cache size = 0.1685 MB llm_load_print_meta: format = GGUF V3 (latest) llm_load_print_meta: arch = phi3 llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32064 llm_load_print_meta: n_merges = 0 llm_load_print_meta: vocab_only = 0 llm_load_print_meta: n_ctx_train = 131072 llm_load_print_meta: n_embd = 3072 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 32 llm_load_print_meta: n_rot = 96 llm_load_print_meta: n_swa = 262144 llm_load_print_meta: n_embd_head_k = 96 llm_load_print_meta: n_embd_head_v = 96 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: n_embd_k_gqa = 3072 llm_load_print_meta: n_embd_v_gqa = 3072 llm_load_print_meta: f_norm_eps = 0.0e+00 llm_load_print_meta: f_norm_rms_eps = 1.0e-05 llm_load_print_meta: f_clamp_kqv = 0.0e+00 llm_load_print_meta: f_max_alibi_bias = 0.0e+00 llm_load_print_meta: f_logit_scale = 0.0e+00 llm_load_print_meta: n_ff = 8192 llm_load_print_meta: n_expert = 0 llm_load_print_meta: n_expert_used = 0 llm_load_print_meta: causal attn = 1 llm_load_print_meta: pooling type = 0 llm_load_print_meta: rope type = 2 llm_load_print_meta: rope scaling = linear llm_load_print_meta: freq_base_train = 10000.0 llm_load_print_meta: freq_scale_train = 1 llm_load_print_meta: n_ctx_orig_yarn = 4096 llm_load_print_meta: rope_finetuned = unknown llm_load_print_meta: ssm_d_conv = 0 llm_load_print_meta: ssm_d_inner = 0 llm_load_print_meta: ssm_d_state = 0 llm_load_print_meta: ssm_dt_rank = 0 llm_load_print_meta: model type = 3B llm_load_print_meta: model ftype = Q8_0 llm_load_print_meta: model params = 3.82 B llm_load_print_meta: model size = 3.78 GiB (8.50 BPW) llm_load_print_meta: general.name = Phi 3.5 Mini Instruct llm_load_print_meta: BOS token = 1 '' llm_load_print_meta: EOS token = 32000 '<|endoftext|>' llm_load_print_meta: UNK token = 0 '' llm_load_print_meta: PAD token = 32000 '<|endoftext|>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_print_meta: EOT token = 32007 '<|end|>' llm_load_print_meta: max token length = 48 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes llm_load_tensors: ggml ctx size = 0.21 MiB llm_load_tensors: offloading 32 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 33/33 layers to GPU llm_load_tensors: CPU buffer size = 99.81 MiB llm_load_tensors: CUDA0 buffer size = 3772.59 MiB ..................................................................................... llama_new_context_with_model: n_ctx = 512 llama_new_context_with_model: n_batch = 512 llama_new_context_with_model: n_ubatch = 512 llama_new_context_with_model: flash_attn = 0 llama_new_context_with_model: freq_base = 10000.0 llama_new_context_with_model: freq_scale = 1 llama_kv_cache_init: CUDA0 KV buffer size = 192.00 MiB llama_new_context_with_model: KV self size = 192.00 MiB, K (f16): 96.00 MiB, V (f16): 96.00 MiB llama_new_context_with_model: CUDA_Host output buffer size = 0.12 MiB llama_new_context_with_model: CUDA0 compute buffer size = 83.00 MiB llama_new_context_with_model: CUDA_Host compute buffer size = 7.01 MiB llama_new_context_with_model: graph nodes = 1286 llama_new_context_with_model: graph splits = 2 system_info: n_threads = 25 / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | compute_imatrix: tokenizing the input .. compute_imatrix: tokenization took 153.372 ms compute_imatrix: computing over 151 chunks with batch_size 512 compute_imatrix: 0.37 seconds per pass - ETA 0.92 minutes [1]5.3486,[2]4.0265,[3]3.9456,[4]4.4244,[5]4.9154,[6]5.0722,[7]4.5718,[8]5.0680,[9]5.2837, save_imatrix: stored collected data after 10 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [10]5.6407,[11]5.6331,[12]5.1979,[13]5.1047,[14]5.3431,[15]5.7433,[16]5.8627,[17]6.1666,[18]6.3235,[19]6.4728, save_imatrix: stored collected data after 20 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [20]6.6095,[21]6.8323,[22]6.5489,[23]6.2512,[24]6.3336,[25]6.3962,[26]6.3401,[27]6.2382,[28]6.3150,[29]6.5252, save_imatrix: stored collected data after 30 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [30]6.6455,[31]6.6179,[32]6.7537,[33]6.8668,[34]7.0645,[35]7.0811,[36]7.0455,[37]6.7483,[38]6.5585,[39]6.4600, save_imatrix: stored collected data after 40 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [40]6.3490,[41]6.2733,[42]6.1949,[43]6.0610,[44]5.9909,[45]5.9037,[46]5.8628,[47]5.8688,[48]5.9398,[49]6.0382, save_imatrix: stored collected data after 50 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [50]6.0530,[51]6.2392,[52]6.4036,[53]6.5880,[54]6.7713,[55]6.8770,[56]6.8103,[57]6.7257,[58]6.7474,[59]6.8060, save_imatrix: stored collected data after 60 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [60]6.8965,[61]6.7923,[62]6.8092,[63]6.8602,[64]6.9373,[65]6.9981,[66]7.0318,[67]7.0837,[68]7.1447,[69]7.1363, save_imatrix: stored collected data after 70 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [70]7.1706,[71]7.1757,[72]7.1942,[73]7.1419,[74]7.0657,[75]7.0505,[76]7.0985,[77]7.0956,[78]7.0611,[79]7.0497, save_imatrix: stored collected data after 80 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [80]7.0528,[81]7.0176,[82]6.9958,[83]6.9644,[84]6.9635,[85]6.9685,[86]6.9597,[87]6.9553,[88]6.9445,[89]6.9314, save_imatrix: stored collected data after 90 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [90]6.9158,[91]6.9244,[92]6.8846,[93]6.8809,[94]6.8512,[95]6.8094,[96]6.8196,[97]6.8018,[98]6.8086,[99]6.7802, save_imatrix: stored collected data after 100 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [100]6.7700,[101]6.7767,[102]6.7408,[103]6.7024,[104]6.6940,[105]6.7131,[106]6.7226,[107]6.7524,[108]6.7814,[109]6.7431, save_imatrix: stored collected data after 110 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [110]6.7002,[111]6.6627,[112]6.6245,[113]6.5826,[114]6.5350,[115]6.5023,[116]6.4703,[117]6.4417,[118]6.4560,[119]6.4623, save_imatrix: stored collected data after 120 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [120]6.5134,[121]6.5610,[122]6.6194,[123]6.6698,[124]6.7480,[125]6.8185,[126]6.8314,[127]6.8347,[128]6.7740,[129]6.7699, save_imatrix: stored collected data after 130 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [130]6.7391,[131]6.7160,[132]6.6748,[133]6.6303,[134]6.6421,[135]6.6622,[136]6.6570,[137]6.6561,[138]6.6647,[139]6.6783, save_imatrix: stored collected data after 140 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [140]6.6920,[141]6.6934,[142]6.6948,[143]6.6962,[144]6.6739,[145]6.6924,[146]6.7249,[147]6.7671,[148]6.8082,[149]6.8491, save_imatrix: stored collected data after 150 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat [150]6.8893,[151]6.9351, save_imatrix: stored collected data after 151 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat llama_print_timings: load time = 1340.15 ms llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second) llama_print_timings: prompt eval time = 40993.32 ms / 77312 tokens ( 0.53 ms per token, 1885.97 tokens per second) llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second) llama_print_timings: total time = 42407.77 ms / 77313 tokens Final estimate: PPL = 6.9351 +/- 0.08916