Those models are for general use. If you have business use case and data related to it you can finetune model for specific use that will outperform all of frontier models and run at fraction of cost.
How can you run a local model for better coding support than fable/opus?
that’s the other huge advantage of open models you can run locally
what kinda hardware do you need to “run this on your desktop”?
You need a GPU with around 16gb vram at a minimum to run qunatized version.
I run Qwen 3.6 27B quantized down to Q4_K_M or even a variant Q5_K_S on my 8gb VRAM entry level AMD GPU, with support of my CPU and 32gb system RAM. Yes, I also limit the Context Length heavily to something like 18k. It’s slow. But the point is, you don’t need necessarily 16gb VRAM.
But it’s better to use faster models for this type of hardware anyway. The MoE type of models (such as 26B A4B, or 35B A3B) are vastly, vastly faster for normal usage, but they are a bit worse in some cases. Also Googles QAT trained model versions also have less RAM requirements without losing much quality.
I’m just saying that, so others are not discouraged too much. It’s not the same experience as the full version off course, but you can use them with some tricks on weak hardware too.
I’m fairly optimistic that people will figure out how to optimize the models a lot further going forward. One obvious path is to try and separate the reasoning network from the trivia that gets baked into the model, and some work is being done in this area. If you could have a context free reasoning engine and then feed the facts it needs to know on the fly based on the context you’re running it in, then you could likely have a much smaller model that’s very capable.
You can run it on systems with as little as 32 GBs of RAM.
You don’t want to run these models in RAM. I started using them on an RTX 3060-12Gb VRAM, and quickly added another, for 24Gb RAM. I also have 64 Gb RAM. Now it works well. Not lightning fast, but quite useable. VRAM is the key.
Sure, but with a model of that size you are unloading layers in any case. I’m just saying you need 32 GBs of RAM at the minimum to be able to run it.
deleted by creator
Gotta wait till Qwen 4 for proper font scaling.
Gotta wait till Qwen 4 for proper font scaling.
Ok, but all of them are far too inefficient for what they deliver.
Ok, but that’s a completely nonsensical statement. If you ever used Qwen in an agentic loop, you’d know that it delivers working code, and it takes about same resources as playing a modern game, and I don’t see anybody whinging that game are too inefficient for what they deliver.
Sure they are. Especially with moores law dead. And especially since 2020; the more million dollars behind it, the more they run like one from a inexperienced single-dev on itch.
Not sure what Moore’s law has to do with anything here to be honest. The models you can run locally on a consumer desktop can do real work, and their resource usage is no different from any other software like games that you’d run.
Which one of them made these overlapping bar labels?
Probably alibaba with Qwen
Hey, a lot of people sucked making matplotlib plots before LLMs existed.
Kimi is better. Waiting for it to appear on Ollama Cloud.
The difference is that you can run Qwen completely local though.
It’s a bit misleading.
Qwen 27B has way less “world knowledge” than GPT-5. Ask it random trivia without internet search access, and GPT would know waaay more.
This is generally true of small vs large models.
…But honestly, Qwen 27B is better at tool use or agentic stuff. It’s hyper optimized for just that and coding assistance, basically.
This is often true of old vs new. Most newer models have hyper focused on agents/coding, often to the detriment of other use cases.
Quantization for practically running Qwen 27B also has an impact. A off-the-shelf Q4_K_M is not the same as the unquantized weights in real-world use, or even an “optimized” quantization like a custom exl3.
If you run it with RAG (retrieval augmented generation), where the model includes web search, you get pretty decent results. I often include the phrase “search the web extensively for up to date information” in my prompts.
I mean baking knowledge into a model isn’t really all that useful to begin with. Just download wikipedia locally and have it access it through tool use, it’s way more efficient and more accurate. And yeah, I find Q6 tends to be the sweet spot where it’s close enough to full 16 bit in performance, but doesn’t chew up too much memory.
This is a pretty good idea I hadn’t thought about. The whole wikipedia is about 37Gb.
yeah, it’s not a completely insane amount of data, and a db like postgres can do fast text search on that too with fuzzy matching
What version do you use and how do you run Qwen3.6? I’ve played around a bit with the Q4 version in LM studio +Zed, but I was not happy with the results. It looses track very often and often enters infinite loops or just stops…
I use an exl3, with 4 bit MLPs but higher bit depth attention layers. And I force some custom sampling so I can lower the temperature a bit while keeping it out of loops.
This won’t work in LM Studio though. You have to run such a thing in TabbyAPI or some other backend that supports exllamav3.
Ok, thanks! Thaf sounds quite advanced, but I’ll have a read afterwards :)
I sort of get it because of you know you’re only using the model when you have Internet and want it to search then just let it.
So which one tried to cheat its way into the test?
Llama is one for sure
[citation needed]
This is for qwen 3.5, not 3.6, but the vram requirements are a little out there for most:
https://willitrunai.com/blog/qwen-3-5-27b-vram-requirements
16 to 55GB…
I bought two RTX 3060 with 12 Gb each for a total of 24Gb VRAM. Sionce the cards are not top shelf anymore for gaming, they can be had used in the 200€ range, so my setup was 400€ over the cost of my existing PC. My PC, while not last gen was already pretty beefy, with a Ryzen 9 and 64 Gb RAM. I think 400€ to have a pretty good local AI setup is not bad at all.
I very much regret getting a 3070, with only 8GB of ram… Only reason I am hesitant about 3060s is that dual cards will use more power, and the economics are already pretty dicey when the hosted models are being sold below cost.
I learnt a few years back that the most important driver in GPU. or any other non upgradable memory device obsolescence, is RAM. I got a banger of a deal a few years on a RX 480, but it was 4Gb, adequate at the time. The card was obsolete way faster than the 8GB. As for the power draw, are you planning to have your AI setup running 24/7? Nvtop (linux GPU monitoring app) reports 50-80% of the cards top power draw. There is also the fact that my data doesn’t leave my setup.
At the time, it wasn’t clear that VRAM beyond 8GB was beneficial. Most cards at the time had basically capped at 8 for ages, and I wanted performance for ray tracing. In hindsight it was a bad call, but for gaming it made sense.
Not 24/7, I’d just start my PC when I need it, but its still more power than I’d like.
Keeping data local does make sense, which is why I have been toying with it. I can run relatively stepped on gemma4 models in 8GB, so for now, I’m happy, but im definitely looking at a Nvidia P40 as a second card.
I can run both the 27B and 35B version of Qwen3.6 on my 10GB VRAM 3080, and they run ok. You don’t need to put the entire model in VRAM, even if it’s probably beneficial.
Full quantisation? I’ve only got a 8GB 3070, but I’ll give it a go
Edit: Tried the unsloth/qwen3.6 with llama.CPP, and it failed to allocate a 26GB Vulcan buffer and died. Dunno what magic your using, no luck for me though :(
I run it using LM Studio, which defaults to Q4 quantization, I think. I was able to put about 10 layers on the GPU with 64k token context. That put me at about 9.1 GB VRAM usage, leaving some room for Video playback xD
I’ll give LM studio a go, thanks.
I mean that’s high but not unreasonable, especially with unified ram machines
and it’s even lower with MTP https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF
Little outside my budget personally, but I’ve definitely been itching to pull the trigger on a nVidia P40
