Instruction template?
#2
by
AIGUYCONTENT
- opened
Do we need to manually set the Instruction Template to Llama-V3?
Depends on what tool you're using i suppose? The instruct template should be properly embedded, but if you're using for example LM Studio you probably want to make sure to manually select the correct template
I use Ooobabooga.
Is LM Studio recommended over Oobabooga for mutli-gpu use?
I tried to get VLLM and OpenWeb UI running and it was a complete disaster. I think the same for Aprhodite. I have heard both of these are the best for multi-GPU use....but also heard that they can't run GGUF and EXL2 quants (which are the two quants I run the most). You for GGUF and FailSpy for EXL2. : )
I have 120GB of VRAM self-hosted and use it for personal and education.