Hugging Face's logo

legraphista
/
Phi-3.5-mini-instruct-IMat-GGUF

Text Generation
GGUF
multilingual
nlp
code
quantized
GGUF
quantization
imat
imatrix
static
16bit
8bit
6bit
5bit
4bit
3bit
2bit
1bit
conversational
Model card Files Community
legraphista commited on
Commit
740cf45
•
1 Parent(s): 3b5b8fd

Upload imatrix.log with huggingface_hub

Browse files
Files changed (1) hide show
  1. imatrix.log +162 -0
imatrix.log ADDED
@@ -0,0 +1,162 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ llama_model_loader: loaded meta data with 36 key-value pairs and 197 tensors from Phi-3.5-mini-instruct-IMat-GGUF/Phi-3.5-mini-instruct.Q8_0.gguf.hardlink.gguf (version GGUF V3 (latest))
2
+ llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
3
+ llama_model_loader: - kv 0: general.architecture str = phi3
4
+ llama_model_loader: - kv 1: general.type str = model
5
+ llama_model_loader: - kv 2: general.name str = Phi 3.5 Mini Instruct
6
+ llama_model_loader: - kv 3: general.finetune str = instruct
7
+ llama_model_loader: - kv 4: general.basename str = Phi-3.5
8
+ llama_model_loader: - kv 5: general.size_label str = mini
9
+ llama_model_loader: - kv 6: general.license str = mit
10
+ llama_model_loader: - kv 7: general.license.link str = https://huggingface.co/microsoft/Phi-...
11
+ llama_model_loader: - kv 8: general.tags arr[str,3] = ["nlp", "code", "text-generation"]
12
+ llama_model_loader: - kv 9: general.languages arr[str,1] = ["multilingual"]
13
+ llama_model_loader: - kv 10: phi3.context_length u32 = 131072
14
+ llama_model_loader: - kv 11: phi3.rope.scaling.original_context_length u32 = 4096
15
+ llama_model_loader: - kv 12: phi3.embedding_length u32 = 3072
16
+ llama_model_loader: - kv 13: phi3.feed_forward_length u32 = 8192
17
+ llama_model_loader: - kv 14: phi3.block_count u32 = 32
18
+ llama_model_loader: - kv 15: phi3.attention.head_count u32 = 32
19
+ llama_model_loader: - kv 16: phi3.attention.head_count_kv u32 = 32
20
+ llama_model_loader: - kv 17: phi3.attention.layer_norm_rms_epsilon f32 = 0.000010
21
+ llama_model_loader: - kv 18: phi3.rope.dimension_count u32 = 96
22
+ llama_model_loader: - kv 19: phi3.rope.freq_base f32 = 10000.000000
23
+ llama_model_loader: - kv 20: general.file_type u32 = 7
24
+ llama_model_loader: - kv 21: phi3.attention.sliding_window u32 = 262144
25
+ llama_model_loader: - kv 22: phi3.rope.scaling.attn_factor f32 = 1.190238
26
+ llama_model_loader: - kv 23: tokenizer.ggml.model str = llama
27
+ llama_model_loader: - kv 24: tokenizer.ggml.pre str = default
28
+ llama_model_loader: - kv 25: tokenizer.ggml.tokens arr[str,32064] = ["<unk>", "<s>", "</s>", "<0x00>", "<...
29
+ llama_model_loader: - kv 26: tokenizer.ggml.scores arr[f32,32064] = [-1000.000000, -1000.000000, -1000.00...
30
+ llama_model_loader: - kv 27: tokenizer.ggml.token_type arr[i32,32064] = [3, 3, 4, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
31
+ llama_model_loader: - kv 28: tokenizer.ggml.bos_token_id u32 = 1
32
+ llama_model_loader: - kv 29: tokenizer.ggml.eos_token_id u32 = 32000
33
+ llama_model_loader: - kv 30: tokenizer.ggml.unknown_token_id u32 = 0
34
+ llama_model_loader: - kv 31: tokenizer.ggml.padding_token_id u32 = 32000
35
+ llama_model_loader: - kv 32: tokenizer.ggml.add_bos_token bool = false
36
+ llama_model_loader: - kv 33: tokenizer.ggml.add_eos_token bool = false
37
+ llama_model_loader: - kv 34: tokenizer.chat_template str = {% for message in messages %}{% if me...
38
+ llama_model_loader: - kv 35: general.quantization_version u32 = 2
39
+ llama_model_loader: - type f32: 67 tensors
40
+ llama_model_loader: - type q8_0: 130 tensors
41
+ llm_load_vocab: special tokens cache size = 14
42
+ llm_load_vocab: token to piece cache size = 0.1685 MB
43
+ llm_load_print_meta: format = GGUF V3 (latest)
44
+ llm_load_print_meta: arch = phi3
45
+ llm_load_print_meta: vocab type = SPM
46
+ llm_load_print_meta: n_vocab = 32064
47
+ llm_load_print_meta: n_merges = 0
48
+ llm_load_print_meta: vocab_only = 0
49
+ llm_load_print_meta: n_ctx_train = 131072
50
+ llm_load_print_meta: n_embd = 3072
51
+ llm_load_print_meta: n_layer = 32
52
+ llm_load_print_meta: n_head = 32
53
+ llm_load_print_meta: n_head_kv = 32
54
+ llm_load_print_meta: n_rot = 96
55
+ llm_load_print_meta: n_swa = 262144
56
+ llm_load_print_meta: n_embd_head_k = 96
57
+ llm_load_print_meta: n_embd_head_v = 96
58
+ llm_load_print_meta: n_gqa = 1
59
+ llm_load_print_meta: n_embd_k_gqa = 3072
60
+ llm_load_print_meta: n_embd_v_gqa = 3072
61
+ llm_load_print_meta: f_norm_eps = 0.0e+00
62
+ llm_load_print_meta: f_norm_rms_eps = 1.0e-05
63
+ llm_load_print_meta: f_clamp_kqv = 0.0e+00
64
+ llm_load_print_meta: f_max_alibi_bias = 0.0e+00
65
+ llm_load_print_meta: f_logit_scale = 0.0e+00
66
+ llm_load_print_meta: n_ff = 8192
67
+ llm_load_print_meta: n_expert = 0
68
+ llm_load_print_meta: n_expert_used = 0
69
+ llm_load_print_meta: causal attn = 1
70
+ llm_load_print_meta: pooling type = 0
71
+ llm_load_print_meta: rope type = 2
72
+ llm_load_print_meta: rope scaling = linear
73
+ llm_load_print_meta: freq_base_train = 10000.0
74
+ llm_load_print_meta: freq_scale_train = 1
75
+ llm_load_print_meta: n_ctx_orig_yarn = 4096
76
+ llm_load_print_meta: rope_finetuned = unknown
77
+ llm_load_print_meta: ssm_d_conv = 0
78
+ llm_load_print_meta: ssm_d_inner = 0
79
+ llm_load_print_meta: ssm_d_state = 0
80
+ llm_load_print_meta: ssm_dt_rank = 0
81
+ llm_load_print_meta: model type = 3B
82
+ llm_load_print_meta: model ftype = Q8_0
83
+ llm_load_print_meta: model params = 3.82 B
84
+ llm_load_print_meta: model size = 3.78 GiB (8.50 BPW)
85
+ llm_load_print_meta: general.name = Phi 3.5 Mini Instruct
86
+ llm_load_print_meta: BOS token = 1 '<s>'
87
+ llm_load_print_meta: EOS token = 32000 '<|endoftext|>'
88
+ llm_load_print_meta: UNK token = 0 '<unk>'
89
+ llm_load_print_meta: PAD token = 32000 '<|endoftext|>'
90
+ llm_load_print_meta: LF token = 13 '<0x0A>'
91
+ llm_load_print_meta: EOT token = 32007 '<|end|>'
92
+ llm_load_print_meta: max token length = 48
93
+ ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
94
+ ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
95
+ ggml_cuda_init: found 1 CUDA devices:
96
+ Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
97
+ llm_load_tensors: ggml ctx size = 0.21 MiB
98
+ llm_load_tensors: offloading 32 repeating layers to GPU
99
+ llm_load_tensors: offloading non-repeating layers to GPU
100
+ llm_load_tensors: offloaded 33/33 layers to GPU
101
+ llm_load_tensors: CPU buffer size = 99.81 MiB
102
+ llm_load_tensors: CUDA0 buffer size = 3772.59 MiB
103
+ .....................................................................................
104
+ llama_new_context_with_model: n_ctx = 512
105
+ llama_new_context_with_model: n_batch = 512
106
+ llama_new_context_with_model: n_ubatch = 512
107
+ llama_new_context_with_model: flash_attn = 0
108
+ llama_new_context_with_model: freq_base = 10000.0
109
+ llama_new_context_with_model: freq_scale = 1
110
+ llama_kv_cache_init: CUDA0 KV buffer size = 192.00 MiB
111
+ llama_new_context_with_model: KV self size = 192.00 MiB, K (f16): 96.00 MiB, V (f16): 96.00 MiB
112
+ llama_new_context_with_model: CUDA_Host output buffer size = 0.12 MiB
113
+ llama_new_context_with_model: CUDA0 compute buffer size = 83.00 MiB
114
+ llama_new_context_with_model: CUDA_Host compute buffer size = 7.01 MiB
115
+ llama_new_context_with_model: graph nodes = 1286
116
+ llama_new_context_with_model: graph splits = 2
117
+
118
+ system_info: n_threads = 25 / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
119
+ compute_imatrix: tokenizing the input ..
120
+ compute_imatrix: tokenization took 153.372 ms
121
+ compute_imatrix: computing over 151 chunks with batch_size 512
122
+ compute_imatrix: 0.37 seconds per pass - ETA 0.92 minutes
123
+ [1]5.3486,[2]4.0265,[3]3.9456,[4]4.4244,[5]4.9154,[6]5.0722,[7]4.5718,[8]5.0680,[9]5.2837,
124
+ save_imatrix: stored collected data after 10 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
125
+ [10]5.6407,[11]5.6331,[12]5.1979,[13]5.1047,[14]5.3431,[15]5.7433,[16]5.8627,[17]6.1666,[18]6.3235,[19]6.4728,
126
+ save_imatrix: stored collected data after 20 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
127
+ [20]6.6095,[21]6.8323,[22]6.5489,[23]6.2512,[24]6.3336,[25]6.3962,[26]6.3401,[27]6.2382,[28]6.3150,[29]6.5252,
128
+ save_imatrix: stored collected data after 30 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
129
+ [30]6.6455,[31]6.6179,[32]6.7537,[33]6.8668,[34]7.0645,[35]7.0811,[36]7.0455,[37]6.7483,[38]6.5585,[39]6.4600,
130
+ save_imatrix: stored collected data after 40 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
131
+ [40]6.3490,[41]6.2733,[42]6.1949,[43]6.0610,[44]5.9909,[45]5.9037,[46]5.8628,[47]5.8688,[48]5.9398,[49]6.0382,
132
+ save_imatrix: stored collected data after 50 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
133
+ [50]6.0530,[51]6.2392,[52]6.4036,[53]6.5880,[54]6.7713,[55]6.8770,[56]6.8103,[57]6.7257,[58]6.7474,[59]6.8060,
134
+ save_imatrix: stored collected data after 60 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
135
+ [60]6.8965,[61]6.7923,[62]6.8092,[63]6.8602,[64]6.9373,[65]6.9981,[66]7.0318,[67]7.0837,[68]7.1447,[69]7.1363,
136
+ save_imatrix: stored collected data after 70 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
137
+ [70]7.1706,[71]7.1757,[72]7.1942,[73]7.1419,[74]7.0657,[75]7.0505,[76]7.0985,[77]7.0956,[78]7.0611,[79]7.0497,
138
+ save_imatrix: stored collected data after 80 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
139
+ [80]7.0528,[81]7.0176,[82]6.9958,[83]6.9644,[84]6.9635,[85]6.9685,[86]6.9597,[87]6.9553,[88]6.9445,[89]6.9314,
140
+ save_imatrix: stored collected data after 90 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
141
+ [90]6.9158,[91]6.9244,[92]6.8846,[93]6.8809,[94]6.8512,[95]6.8094,[96]6.8196,[97]6.8018,[98]6.8086,[99]6.7802,
142
+ save_imatrix: stored collected data after 100 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
143
+ [100]6.7700,[101]6.7767,[102]6.7408,[103]6.7024,[104]6.6940,[105]6.7131,[106]6.7226,[107]6.7524,[108]6.7814,[109]6.7431,
144
+ save_imatrix: stored collected data after 110 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
145
+ [110]6.7002,[111]6.6627,[112]6.6245,[113]6.5826,[114]6.5350,[115]6.5023,[116]6.4703,[117]6.4417,[118]6.4560,[119]6.4623,
146
+ save_imatrix: stored collected data after 120 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
147
+ [120]6.5134,[121]6.5610,[122]6.6194,[123]6.6698,[124]6.7480,[125]6.8185,[126]6.8314,[127]6.8347,[128]6.7740,[129]6.7699,
148
+ save_imatrix: stored collected data after 130 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
149
+ [130]6.7391,[131]6.7160,[132]6.6748,[133]6.6303,[134]6.6421,[135]6.6622,[136]6.6570,[137]6.6561,[138]6.6647,[139]6.6783,
150
+ save_imatrix: stored collected data after 140 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
151
+ [140]6.6920,[141]6.6934,[142]6.6948,[143]6.6962,[144]6.6739,[145]6.6924,[146]6.7249,[147]6.7671,[148]6.8082,[149]6.8491,
152
+ save_imatrix: stored collected data after 150 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
153
+ [150]6.8893,[151]6.9351,
154
+ save_imatrix: stored collected data after 151 chunks in Phi-3.5-mini-instruct-IMat-GGUF/imatrix.dat
155
+
156
+ llama_print_timings: load time = 1340.15 ms
157
+ llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
158
+ llama_print_timings: prompt eval time = 40993.32 ms / 77312 tokens ( 0.53 ms per token, 1885.97 tokens per second)
159
+ llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
160
+ llama_print_timings: total time = 42407.77 ms / 77313 tokens
161
+
162
+ Final estimate: PPL = 6.9351 +/- 0.08916