Qwen 27B has way less “world knowledge” than GPT-5. Ask it random trivia without internet search access, and GPT would know waaay more.
This is generally true of small vs large models.
…But honestly, Qwen 27B is better at tool use or agentic stuff. It’s hyper optimized for just that and coding assistance, basically.
This is often true of old vs new. Most newer models have hyper focused on agents/coding, often to the detriment of other use cases.
Quantization for practically running Qwen 27B also has an impact. A off-the-shelf Q4_K_M is not the same as the unquantized weights in real-world use, or even an “optimized” quantization like a custom exl3.
If you run it with RAG (retrieval augmented generation), where the model includes web search, you get pretty decent results. I often include the phrase “search the web extensively for up to date information” in my prompts.
I mean baking knowledge into a model isn’t really all that useful to begin with. Just download wikipedia locally and have it access it through tool use, it’s way more efficient and more accurate. And yeah, I find Q6 tends to be the sweet spot where it’s close enough to full 16 bit in performance, but doesn’t chew up too much memory.
What version do you use and how do you run Qwen3.6? I’ve played around a bit with the Q4 version in LM studio +Zed, but I was not happy with the results. It looses track very often and often enters infinite loops or just stops…
I use an exl3, with 4 bit MLPs but higher bit depth attention layers. And I force some custom sampling so I can lower the temperature a bit while keeping it out of loops.
This won’t work in LM Studio though. You have to run such a thing in TabbyAPI or some other backend that supports exllamav3.
It’s a bit misleading.
Qwen 27B has way less “world knowledge” than GPT-5. Ask it random trivia without internet search access, and GPT would know waaay more.
This is generally true of small vs large models.
…But honestly, Qwen 27B is better at tool use or agentic stuff. It’s hyper optimized for just that and coding assistance, basically.
This is often true of old vs new. Most newer models have hyper focused on agents/coding, often to the detriment of other use cases.
Quantization for practically running Qwen 27B also has an impact. A off-the-shelf Q4_K_M is not the same as the unquantized weights in real-world use, or even an “optimized” quantization like a custom exl3.
If you run it with RAG (retrieval augmented generation), where the model includes web search, you get pretty decent results. I often include the phrase “search the web extensively for up to date information” in my prompts.
I mean baking knowledge into a model isn’t really all that useful to begin with. Just download wikipedia locally and have it access it through tool use, it’s way more efficient and more accurate. And yeah, I find Q6 tends to be the sweet spot where it’s close enough to full 16 bit in performance, but doesn’t chew up too much memory.
This is a pretty good idea I hadn’t thought about. The whole wikipedia is about 37Gb.
yeah, it’s not a completely insane amount of data, and a db like postgres can do fast text search on that too with fuzzy matching
What version do you use and how do you run Qwen3.6? I’ve played around a bit with the Q4 version in LM studio +Zed, but I was not happy with the results. It looses track very often and often enters infinite loops or just stops…
I use an exl3, with 4 bit MLPs but higher bit depth attention layers. And I force some custom sampling so I can lower the temperature a bit while keeping it out of loops.
This won’t work in LM Studio though. You have to run such a thing in TabbyAPI or some other backend that supports exllamav3.
Ok, thanks! Thaf sounds quite advanced, but I’ll have a read afterwards :)
I sort of get it because of you know you’re only using the model when you have Internet and want it to search then just let it.