Alternate quantizations.
These are my own quantizations (updated almost daily).
The difference with normal quantizations is that I quantize the output and embed tensors to f16.
and the other tensors to 15_k,q6_k or q8_0.
This creates models that are little or not degraded at all and have a smaller size.
They run at about 3-6 t/sec on CPU only using llama.cpp
And obviously faster on computers with potent GPUs
Ok?
what is this model based on? which version of phi-3? it seems small to be the medium and even the mini seems bigger..
I've been in excruciating pain for two days due to toothache. This time, I used Lora for adjustment, and this Lora is suitable for all versions of Phi-3. For writing lyrics, the mini version will suffice.
I've been in excruciating pain for two days due to toothache. This time, I used Lora for adjustment, and this Lora is suitable for all versions of Phi-3. For writing lyrics, the mini version will suffice.
I can relate... I had teeth problems too (and still have sometimes).