Full quantisation? I’ve only got a 8GB 3070, but I’ll give it a go
Edit: Tried the unsloth/qwen3.6 with llama.CPP, and it failed to allocate a 26GB Vulcan buffer and died. Dunno what magic your using, no luck for me though :(
I run it using LM Studio, which defaults to Q4 quantization, I think. I was able to put about 10 layers on the GPU with 64k token context. That put me at about 9.1 GB VRAM usage, leaving some room for Video playback xD
Full quantisation? I’ve only got a 8GB 3070, but I’ll give it a go
Edit: Tried the unsloth/qwen3.6 with llama.CPP, and it failed to allocate a 26GB Vulcan buffer and died. Dunno what magic your using, no luck for me though :(
I run it using LM Studio, which defaults to Q4 quantization, I think. I was able to put about 10 layers on the GPU with 64k token context. That put me at about 9.1 GB VRAM usage, leaving some room for Video playback xD
I’ll give LM studio a go, thanks.