The Kaitchup – AI on a Budget

The Kaitchup – AI on a Budget

Qwen3.6 27B vs Qwen3.5 27B vs Gemma 4 31B: Accuracy, Latency, Memory, and Token Efficiency Tested

Qwen3.6 improves on Qwen3.5, but Gemma 4 remains surprisingly competitive.

Benjamin Marie's avatar
Benjamin Marie
May 05, 2026
∙ Paid

In a previous article, I found Gemma 4 31B to be superior or comparable to Qwen3.5 27B in most areas, with similar or better accuracy and lower latency.

Gemma 4 31B vs Qwen3.5 27B: Inference Speed, Token-Efficiency, Accuracy, and Memory Consumption

Gemma 4 31B vs Qwen3.5 27B: Inference Speed, Token-Efficiency, Accuracy, and Memory Consumption

Benjamin Marie
·
Apr 15
Read full story

However, the Qwen3.6 update likely changes that conclusion. On benchmarks, it appears to be significantly stronger than Qwen3.5. The best accuracy in this model class is therefore probably now achieved by Qwen3.6, but at what cost? And how does that trade-off vary depending on the task?

The Kaitchup – AI on a Budget is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

I’ll answer these questions in this article. I measured accuracy, latency, and token efficiency, both with thinking enabled and disabled, and compared the results with the numbers I previously obtained for Qwen3.5 27B and Gemma 4 31B. I used the exact same setup to ensure that all results are directly comparable.

Acknowledgments

This article would not have been possible without the compute sponsorship generously provided by Verda, whose B200 and RTX Pro 6000 GPUs I used throughout this work.

Verda provides access to high-end GPUs such as the B200 and B300, with GB300 support coming soon, as well as smaller GPUs such as the RTX 6000 Ada, which are among the most affordable per hour on the market.

Verda is a European, AI-focused cloud and GPU infrastructure provider with sovereignty, sustainability, data privacy, and performance at its core.

You can check them out here.

Qwen3.6 27B: Much Better Accuracy than Gemma 4 and Qwen3.5?

Before diving into the results, I should note that all the benchmarks I used are non-agentic and do not use tool calls. This setup is not particularly favorable to Qwen3.6, which appears to have been fine-tuned specifically for stronger agentic performance.

And indeed, on several of the benchmarks I ran, Qwen3.6 performs behind both Qwen3.5 and Gemma 4, while for a few benchmarks, it’s significantly better.

Hard math questions: AIME

On hard math questions, as measured by AIME, Qwen3.6 significantly outperforms both Qwen3.5 and Gemma 4.

It also performs better than Qwen3.5 on simpler math benchmarks such as Math 500.

Single-turn coding: LiveCodeBench

On single-turn coding tasks, measured with LiveCodeBench, Qwen3.6 improves over Qwen3.5 but still slightly underperforms Gemma 4.

World knowledge: MMLU Pro

Qwen3.6 also shows stronger world knowledge. On MMLU Pro, it answers more accurately than both Qwen3.5 and Gemma 4.

The strange results

Some results were more surprising.

First, Qwen3.6 is significantly worse than Qwen3.5 at following instructions, as measured by IFBench.

Second, Qwen3.6 is also significantly worse on GPQA Diamond. This was unexpected, since Qwen reports a 2.3-point improvement over Qwen3.5 on this benchmark.

Was something wrong with my setup? Whenever I doubt my own results, I cross-check them against Artificial Analysis, since they also run some of these benchmarks. In this case, they found the same thing: Qwen3.5 performs better than Qwen3.6 on GPQA Diamond.

This likely means Qwen ran something different: different hyperparameters, a different version of the benchmark, different post-processing, or some other variation in the evaluation setup. It is a useful reminder that benchmark scores published by different groups are not directly comparable.

Overall, Qwen3.6 is only slightly better than Qwen3.5 on average, and it still underperforms Gemma 4 on this type of non-agentic task.

Of course, benchmark scores only tell part of the story. I ran additional analyses to better understand these results. One analysis I now run systematically is CoDeC scoring, which identifies benchmarks where a model appears more comfortable, suggesting possible special training or fine-tuning on benchmark-like data.

Did the Model See the Benchmark During Training? Detecting LLM Contamination

Did the Model See the Benchmark During Training? Detecting LLM Contamination

Benjamin Marie
·
Feb 2
Read full story

My CoDeC run confirms that Qwen3.6 is much more comfortable with AIME-style benchmarks.

Image

It reaches a score above 62, which is very rare. By contrast, AIME-like data appears to be much newer to Gemma 4 31B IT.

This helps explain why Qwen3.6 improves so much over Qwen3.5 on AIME.

I also examined how Qwen3.6 behaves on LiveCodeBench, since this is one of the benchmarks where Gemma 4 remains significantly ahead.

Interestingly, the accuracy gap can be closed by leveraging random sampling, that is, by looking at pass@k rather than pass@1. At pass@4, Gemma 4 31B and Qwen3.6 27B achieve nearly the same accuracy.

How to Reduce LLM Inference Cost and Improve Accuracy with Pass@k and Majority Voting

How to Reduce LLM Inference Cost and Improve Accuracy with Pass@k and Majority Voting

Benjamin Marie
·
Apr 27
Read full story

However, as we will see later, this does not make Qwen3.6 the better choice in practice. Because of its much higher cost, using Gemma 4 31B remains substantially cheaper when targeting the same level of coding accuracy.

Qwen3.6 27B: Token Efficiency and Latency

This post is for paid subscribers

Already a paid subscriber? Sign in
© 2026 The Kaitchup · Privacy ∙ Terms ∙ Collection notice
Start your SubstackGet the app
Substack is the home for great culture