Some of you might be interested in my 'silly' experiment.
I created a program to randomize the weights of a model. The program has 2 parameters: the percentage of weights to modify and the percentage of the original value to randmly apply to each weight.
At the end I check the resulting GGUF file for binary differences.
In this example I set to modify 100% of the weights of Mistral 7b Instruct v0.3 by a maximum of 15% deviation.
Since the deviation is calculated on the F32 weights, when quantized to Q8_0 this changes.
So, in the end I got a file that compared to the original has:
Bytes Difference percentage: 73.04%
Average value divergence: 2.98%
The cool thing is that chatting with the model I see no apparent difference and the model still works nicely as the original.
Since I am running everything on CPU, I could not run perplexity scores or anything computing intensive.
As a small test, I asked the model a few questions (like the history of the roman empire) and then fact check its answer using a big model. No errors were detected.
Update: all procedure tested and created on COLAB.
Result (one of the many you can get): Meta-Llama-3.1-8B-Instruct-SILLY
Very interesting. Reminds me the stories about people who had injuries that removed parts of their brain and survived. In some cases there was no readily noticeable affect but after some time they notice some cognitive impairment or their families report personality changes.
Very interesting. Reminds me the stories about people who had injuries that removed parts of their brain and survived. In some cases there was no readily noticeable affect but after some time they notice some cognitive impairment or their families report personality changes.
Yep. and sometimes new abilities surfaced. That's the point of the "silly" experiment!