xodoh74984@lemmy.worldtoFree Open-Source Artificial Intelligence@lemmy.world•What open-source LLMs are you using in 2024?English
8·
10 months agoOf all of the code specific LLMs I’m familiar with Deepseek-Coder-33B is my favorite. There are multiple pre-quantized versions available here:
https://huggingface.co/TheBloke/deepseek-coder-33B-base-GGUF/tree/main
In my experience a minimum of 5-bit quantization performs best.
This one is only 7B parameters, but it punches far above its weight for such a little model:
https://huggingface.co/berkeley-nest/Starling-LM-7B-alpha
My personal setup is capable of running larger models, but for everyday use like summarization and brainstorming, I find myself coming back to Starling the most. Since it’s so small, it runs inference blazing fast on my hardware. I don’t rely on it for writing code. Deepseek-Coder-33B is my pick for that.
Others have said Starling’s overall performance rivals LLaMA 70B. YMMV.