Current project / statistical report

Qwen Family Preliminary Benchmark Study

A full factorial local-inference experiment across Qwen-family model sizes, quantisation settings, KV-cache configurations, and MTP draft-token values. The test set covered 2,400 completed runs across 20 enterprise-style use cases.

2,400factorial combinations completed exactly once

0.6696overall pass rate across all runs

0.8155average semantic score

15primary Pareto frontier configurations

Introduction to the tests

The experiment was designed to answer a practical question: which small local Qwen-family models are credible candidates for scheduled, mundane enterprise automation jobs? The benchmark varied model size, quantisation, KV-cache configuration, and MTP draft-token count, then scored the outputs against deterministic rubrics for semantic quality and pass/fail outcomes.

The model sizes tested were qwen3.5-0.8b, qwen3.5-2b, qwen3.5-4b, and qwen3.5-9b. Quantisation covered Q4_K_M, Q6_K, and Q8_0. KV-cache settings compared q8 against mixed_q8k_q4nlv. MTP draft tokens ranged from 0 to 4.

Summary

The clearest result is that model size drives most of the quality and pass-rate difference. The 4b and 9b models separate sharply from the 0.8b and 2b models, with the 4b model offering the most useful balance of quality, latency, and throughput for this workload. Quantisation and KV-cache settings matter much less for semantic score and pass rate, although they still affect speed and memory behaviour.

MTP is not a universal quality knob. It changes latency and speed, and the best draft-token value depends on the model and quantisation. Globally, MTP=1 produced the best semantic-points-per-second score, but model-specific sweet spots differed: 0.8b preferred MTP=3, 2b preferred MTP=2, 4b preferred MTP=3 by semantic points/s, and 9b preferred MTP=4.

The strongest practical pattern is that qwen3.5-4b repeatedly appears on the Pareto frontier. qwen3.5-9b can match or slightly exceed quality, but the latency cost is harder to justify for routine scheduled tasks unless the use case needs the larger model's additional robustness.

Recommendations

Use qwen3.5-4b Q8_0 mixed_q8k_q4nlv MTP=1 as the default balanced configuration: semantic 0.9125, pass 0.8500, elapsed 2.1977s.
Use qwen3.5-4b Q8_0 mixed_q8k_q4nlv MTP=0 when quality is the priority and the extra latency is acceptable: semantic 0.9150, pass 0.8500, elapsed 2.7165s.
Use qwen3.5-4b Q6_K q8 MTP=2 as the fastest acceptable configuration: semantic 0.8733, pass 0.8000, elapsed 1.5266s.
Treat qwen3.5-9b as a specialist option rather than the default: its best run matched the top pass rate, but with higher average latency.
Avoid qwen3.5-0.8b Q4_K_M mixed_q8k_q4nlv MTP=1 for this benchmark: semantic 0.6079, pass 0.3500.
Do not generalise these findings too far yet. Further testing across context size, prompt family, use-case type, long-context behaviour, and KV-cache/MTP interactions is still needed.

Best Configurations

The top deployment-efficiency rows from the report, shown before the full tables for quick scanning.

model	quantization	KV cache config	MTP draft tokens	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	1	20	0.9125	0.85	2.1977	60.55	1126.9
qwen3.5-9b	Q8_0	q8	4	20	0.9125	0.85	2.7197	51.61	1008.8
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	0	20	0.915	0.85	2.7165	43.70	1183.4
qwen3.5-4b	Q4_K_M	q8	1	20	0.8967	0.8	1.5285	88.39	1668.1
qwen3.5-4b	Q6_K	q8	1	20	0.8942	0.8	1.8037	78.12	1549.6
qwen3.5-4b	Q8_0	q8	2	20	0.8917	0.8	1.6557	76.11	1481.0
qwen3.5-4b	Q6_K	q8	3	20	0.8825	0.8	1.6206	87.24	1524.5
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	1	20	0.9025	0.85	3.1701	38.49	845.6
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	0	20	0.9042	0.8	2.395	62.31	1306.3
qwen3.5-9b	Q6_K	q8	2	20	0.9025	0.8	2.2967	59.51	1078.7
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	0	20	0.9008	0.85	3.8691	39.83	1024.0
qwen3.5-9b	Q6_K	q8	0	20	0.9021	0.85	3.5504	34.58	1198.1

12 of 120 rows shown.

Main Effects

The report's main-effect tables, transferred from the statistical analysis outputs.

Model main effect

model	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s
qwen3.5-0.8b	600	0.7309	0.5117	0.9659	144.8	3434.1
qwen3.5-2b	600	0.772	0.615	1.3081	108.4	2738.8
qwen3.5-4b	600	0.8775	0.775	1.8913	74.36	1384.8
qwen3.5-9b	600	0.8818	0.7767	3.7623	51.41	1012.5

Quantisation main effect

quantization	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s
Q4_K_M	800	0.7985	0.6388	1.7881	103.2	2211.7	0.5623	0.4651	4.4061
Q6_K	800	0.8239	0.6913	1.9415	96.64	2123.6	0.5766	0.4418	4.3786
Q8_0	800	0.8243	0.6787	2.216	84.53	2094.7	0.583	0.4436	3.9542

KV-cache main effect

KV cache config	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s
mixed_q8k_q4nlv	1200	0.8163	0.6675	2.0858	88.54	1802.5	0.5752	0.4473	3.997
q8	1200	0.8148	0.6717	1.878	101.0	2483.6	0.5727	0.453	4.4956

MTP draft-token main effect

MTP draft tokens	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s
0	480	0.8192	0.675	2.6227	95.54	2374.2			4.2019
1	480	0.8162	0.6667	1.9261	101.4	2119.6	0.7592	0.2545	4.4552
2	480	0.8126	0.6687	1.6865	99.78	2090.0	0.6134	0.4009	4.3495
3	480	0.8152	0.6646	1.638	92.85	2085.3	0.5053	0.5249	4.2546
4	480	0.8145	0.6729	2.036	84.27	2048.6	0.418	0.6204	3.9702

Function-area main effect

function area	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s
Customer Support	120	0.6333	0.2167	1.4037	94.72	2060.9	0.5403	0.4444	3.38
Data/Analytics	240	0.7935	0.5042	2.7715	90.67	2079.8	0.5068	0.495	3.3198
Finance	360	0.7069	0.5389	3.6445	95.21	2660.5	0.5944	0.3628	3.7239
HR	240	0.8917	0.8	0.2368	98.36	1917.7	0.6707	0.4423	8.8507
IT	240	0.8719	0.8708	1.1954	96.61	1955.5	0.5666	0.4411	5.4666
Knowledge Management	240	0.9156	0.7542	1.9914	89.56	1870.4	0.4823	0.4708	2.7336
Legal	240	0.8108	0.725	2.3235	93.32	2058.2	0.5566	0.452	3.287
Operations	480	0.77	0.6146	1.7889	94.54	2042.5	0.5688	0.4822	3.5426
Sales	240	0.9549	0.8958	1.5538	99.98	2450.0	0.6574	0.4697	4.4443

Prompt main effect

prompt name	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s
Action Item Extraction	120	0.3167	0	1.0588	102.5	1958.8	0.6926	0.4816	3.1811
CRM Note Enrichment	120	0.9403	0.9833	1.4146	101.6	2390.9	0.6778	0.4657	4.7857
Contract Clause Extraction	120	0.8867	0.9167	0.969	100.8	1992.9	0.6866	0.433	5.1383
Contract Risk Flagging	120	0.735	0.5333	3.6779	85.83	2123.5	0.4267	0.471	1.4357
Customer Complaint Triage	120	0.6333	0.2167	1.4037	94.72	2060.9	0.5403	0.4444	3.38
Email Classification	120	0.8967	0.9167	0.9485	96.51	1855.7	0.5618	0.4375	6.181
Employee Handbook QA	120	0.9958	0.9917	0.2157	101.1	1873.4	0.7151	0.4417	10.67
Expense Claim Policy Check	120	0.5042	0	7.4629	86.81	3059.4	0.448	0.4341	1.2195
HR Leave Policy QA	120	0.7875	0.6083	0.2579	95.59	1962.0	0.6263	0.4429	7.0285
IT Incident Ticket Classification	120	0.8472	0.825	1.4422	96.71	2055.4	0.5713	0.4446	4.7521
Invoice Extraction	120	0.875	0.875	1.7956	100.5	1763.8	0.6778	0.3222	5.0636
Meeting Summary	120	0.975	0.9917	4.4513	87.54	2309.4	0.4645	0.4894	2.0157
Policy Contradiction Detection	120	0.9542	0.8417	2.8862	87.30	1740.7	0.4606	0.475	1.5962
Product Feedback Categorization	120	0.9354	0.775	0.8607	94.69	2198.5	0.5702	0.4649	5.1945
Purchase Order Extraction	120	0.7417	0.7417	1.6751	98.22	3165.0	0.6574	0.332	4.8884
SQL Generation From Schema	120	0.985	0.9917	0.9559	99.25	1433.9	0.6239	0.4936	6.0319
Sales Call Summary	120	0.9694	0.8083	1.6931	98.38	2509.1	0.637	0.4737	4.1028
Support Knowledge Base Answer	120	0.8771	0.6667	1.0967	91.82	2000.0	0.5041	0.4665	3.8711
Table To Insight Analysis	120	0.6021	0.0167	4.587	82.10	2725.8	0.3896	0.4964	0.6077
Tool Calling Intent Selection	120	0.8528	0.6917	0.7848	93.39	1703.2	0.5478	0.4929	3.7789

ANOVA And Effect Sizes

Type-II OLS ANOVA tables from the report. The model factor dominates semantic score, elapsed time, generation speed, and overall pass rate.

Semantic score ANOVA

effect	F	p value	eta squared	partial eta squared
C(model)	52.86	0	0.0624	0.0628
C(quantization)	2.6788	0.0689	0.0021	0.0023
C(kv_cache_config)	0.0226	0.8806	0	0
C(mtp_draft_tokens_factor)	0.0438	0.9964	0.0001	0.0001
C(model):C(mtp_draft_tokens_factor)	0.2968	0.99	0.0014	0.0015
C(model):C(quantization)	0.9042	0.4908	0.0021	0.0023
C(kv_cache_config):C(mtp_draft_tokens_factor)	0.1925	0.9424	0.0003	0.0003
Residual			0.9316

Elapsed seconds ANOVA

effect	F	p value	eta squared	partial eta squared
C(model)	48.58	0	0.0565	0.058
C(quantization)	1.9579	0.1414	0.0015	0.0017
C(kv_cache_config)	1.3488	0.2456	0.0005	0.0006
C(mtp_draft_tokens_factor)	3.8883	0.0038	0.006	0.0065
C(model):C(mtp_draft_tokens_factor)	3.4208	0.0001	0.0159	0.017
C(model):C(quantization)	0.8223	0.5525	0.0019	0.0021
C(kv_cache_config):C(mtp_draft_tokens_factor)	0.4781	0.7519	0.0007	0.0008
Residual			0.9169

Generation tokens/sec ANOVA

effect	F	p value	eta squared	partial eta squared
C(model)	7508.1	0	0.721	0.905
C(quantization)	537.0	0	0.0344	0.3123
C(kv_cache_config)	710.3	0	0.0227	0.231
C(mtp_draft_tokens_factor)	166.2	0	0.0213	0.2194
C(model):C(mtp_draft_tokens_factor)	321.6	0	0.1235	0.62
C(model):C(quantization)	5.5278	0	0.0011	0.0138
C(kv_cache_config):C(mtp_draft_tokens_factor)	2.8373	0.0231	0.0004	0.0048
Residual			0.0757

Overall pass ANOVA

effect	F	p value	eta squared	partial eta squared
C(model)	48.04	0	0.0571	0.0574
C(quantization)	2.8622	0.0573	0.0023	0.0024
C(kv_cache_config)	0.0496	0.8239	0	0
C(mtp_draft_tokens_factor)	0.0426	0.9966	0.0001	0.0001
C(model):C(mtp_draft_tokens_factor)	0.2197	0.9976	0.001	0.0011
C(model):C(quantization)	0.8986	0.4949	0.0021	0.0023
C(kv_cache_config):C(mtp_draft_tokens_factor)	0.2032	0.9367	0.0003	0.0003
Residual			0.9371

MTP And Interaction Effects

MTP draft tokens improved speed/latency characteristics in some settings, but the best setting is model- and configuration-dependent.

Global MTP sweet spot

MTP draft tokens	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s	selection rule
1	480	0.8162	0.6667	1.9261	101.4	2119.6	0.7592	0.2545	4.4552	within 2pp pass and 0.02 semantic, then semantic points/s, rejection, lower draft

MTP sweet spot by model

model	MTP draft tokens	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s	selection rule
qwen3.5-0.8b	3	120	0.7563	0.5417	1.0986	125.0	3345.6	0.2186	0.7837	5.6533	within 2pp pass and 0.02 semantic, then semantic points/s, rejection, lower draft
qwen3.5-2b	2	120	0.7781	0.6333	1.2151	117.1	2663.2	0.5831	0.4466	4.9031	within 2pp pass and 0.02 semantic, then semantic points/s, rejection, lower draft
qwen3.5-4b	3	120	0.8702	0.775	1.7952	82.42	1352.6	0.6493	0.4077	3.9874	within 2pp pass and 0.02 semantic, then semantic points/s, rejection, lower draft
qwen3.5-9b	4	120	0.8825	0.7917	3.7843	60.05	982.3	0.6512	0.4496	2.9258	within 2pp pass and 0.02 semantic, then semantic points/s, rejection, lower draft

MTP sweet spot by quantisation

quantization	MTP draft tokens	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s	selection rule
Q4_K_M	0	160	0.8208	0.6813	2.1841	107.4	2428.3			4.5482	within 2pp pass and 0.02 semantic, then semantic points/s, rejection, lower draft
Q6_K	2	160	0.8338	0.7125	1.4938	102.6	2068.8	0.6244	0.3848	4.6952	within 2pp pass and 0.02 semantic, then semantic points/s, rejection, lower draft
Q8_0	1	160	0.8306	0.7	2.0884	90.51	2070.4	0.7667	0.2518	4.1275	within 2pp pass and 0.02 semantic, then semantic points/s, rejection, lower draft

MTP sweet spot by KV-cache

KV cache config	MTP draft tokens	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s	selection rule
mixed_q8k_q4nlv	3	240	0.816	0.6708	1.7118	87.10	1760.2	0.5079	0.5241	3.998	within 2pp pass and 0.02 semantic, then semantic points/s, rejection, lower draft
q8	1	240	0.8117	0.6708	1.8576	108.4	2446.8	0.761	0.2567	4.7209	within 2pp pass and 0.02 semantic, then semantic points/s, rejection, lower draft

Model by draft-token interaction

model	MTP draft tokens	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s
qwen3.5-0.8b	0	120	0.7419	0.5167	0.7636	182.3	3808.5			7.2598
qwen3.5-0.8b	1	120	0.7134	0.5	0.809	165.8	3404.8	0.557	0.4174	6.6824
qwen3.5-0.8b	2	120	0.7153	0.4917	0.9511	143.1	3354.3	0.3122	0.6748	5.8235
qwen3.5-0.8b	3	120	0.7563	0.5417	1.0986	125.0	3345.6	0.2186	0.7837	5.6533
qwen3.5-0.8b	4	120	0.7275	0.5083	1.207	108.0	3257.4	0.1598	0.8352	4.9396
qwen3.5-2b	0	120	0.7643	0.6083	1.2039	109.0	3059.8			4.8306
qwen3.5-2b	1	120	0.7793	0.6083	1.4203	118.8	2702.6	0.7444	0.2987	5.1551
qwen3.5-2b	2	120	0.7781	0.6333	1.2151	117.1	2663.2	0.5831	0.4466	4.9031
qwen3.5-2b	3	120	0.7603	0.6083	1.3079	105.0	2651.6	0.4243	0.6102	4.4808
qwen3.5-2b	4	120	0.7779	0.6167	1.3932	92.01	2616.8	0.3249	0.6995	4.2282
qwen3.5-4b	0	120	0.8835	0.7833	2.3499	56.11	1517.5			2.9099
qwen3.5-4b	1	120	0.8894	0.7833	1.8556	73.74	1374.2	0.8674	0.139	3.588
qwen3.5-4b	2	120	0.8743	0.7583	1.6964	82.38	1343.0	0.758	0.2512	3.907
qwen3.5-4b	3	120	0.8702	0.775	1.7952	82.42	1352.6	0.6493	0.4077	3.9874
qwen3.5-4b	4	120	0.8702	0.775	1.7595	77.00	1338.0	0.5361	0.4973	3.7872
qwen3.5-9b	0	120	0.8873	0.7917	6.1736	33.94	1093.1			1.8074
qwen3.5-9b	1	120	0.8828	0.775	3.6196	47.47	996.8	0.868	0.1629	2.3954
qwen3.5-9b	2	120	0.8826	0.7917	2.8834	56.51	999.4	0.8004	0.2309	2.7644
qwen3.5-9b	3	120	0.874	0.7333	2.3503	58.95	991.4	0.7289	0.298	2.8968
qwen3.5-9b	4	120	0.8825	0.7917	3.7843	60.05	982.3	0.6512	0.4496	2.9258

Quantisation by draft-token interaction

quantization	MTP draft tokens	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s
Q4_K_M	0	160	0.8208	0.6813	2.1841	107.4	2428.3			4.5482
Q4_K_M	1	160	0.7973	0.6375	1.6576	110.9	2188.0	0.7493	0.2687	4.6519
Q4_K_M	2	160	0.7772	0.625	1.4888	107.8	2158.3	0.5972	0.4151	4.3051
Q4_K_M	3	160	0.7952	0.6188	1.4675	99.05	2161.4	0.4923	0.5314	4.365
Q4_K_M	4	160	0.8017	0.6312	2.1424	90.61	2122.6	0.4104	0.6454	4.1604
Q6_K	0	160	0.8131	0.6813	2.568	96.79	2379.6			4.2086
Q6_K	1	160	0.8206	0.6625	2.0323	102.9	2100.3	0.7616	0.243	4.5863
Q6_K	2	160	0.8338	0.7125	1.4938	102.6	2068.8	0.6244	0.3848	4.6952
Q6_K	3	160	0.8226	0.6937	1.5438	95.40	2060.7	0.5033	0.5166	4.3383
Q6_K	4	160	0.8294	0.7063	2.0698	85.52	2011.7	0.417	0.6228	4.0644
Q8_0	0	160	0.8237	0.6625	3.1161	82.40	2314.7			3.849
Q8_0	1	160	0.8306	0.7	2.0884	90.51	2070.4	0.7667	0.2518	4.1275
Q8_0	2	160	0.8268	0.6687	2.077	88.96	2042.8	0.6186	0.4027	4.0482
Q8_0	3	160	0.8279	0.6813	1.9027	84.10	2033.8	0.5203	0.5267	4.0604
Q8_0	4	160	0.8124	0.6813	1.8959	76.68	2011.6	0.4265	0.593	3.6859

KV-cache by draft-token interaction

KV cache config	MTP draft tokens	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s
mixed_q8k_q4nlv	0	240	0.8231	0.6833	2.966	88.34	1947.3			3.9142
mixed_q8k_q4nlv	1	240	0.8207	0.6625	1.9946	94.53	1792.3	0.7574	0.2524	4.1896
mixed_q8k_q4nlv	2	240	0.8153	0.6625	1.6724	93.62	1778.0	0.6175	0.398	4.1903
mixed_q8k_q4nlv	3	240	0.816	0.6708	1.7118	87.10	1760.2	0.5079	0.5241	3.998
mixed_q8k_q4nlv	4	240	0.8065	0.6583	2.084	79.11	1736.0	0.4181	0.6147	3.6929
q8	0	240	0.8153	0.6667	2.2795	102.7	2797.5			4.4896
q8	1	240	0.8117	0.6708	1.8576	108.4	2446.8	0.761	0.2567	4.7209
q8	2	240	0.8098	0.675	1.7007	105.9	2401.9	0.6094	0.4037	4.5087
q8	3	240	0.8145	0.6583	1.5642	98.60	2410.4	0.5026	0.5257	4.5112
q8	4	240	0.8225	0.6875	1.988	89.43	2361.3	0.4179	0.6261	4.2475

Model by quantisation interaction

model	quantization	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s
qwen3.5-0.8b	Q4_K_M	200	0.6898	0.46	0.9853	151.2	3458.0	0.2918	0.6965	5.8313
qwen3.5-0.8b	Q6_K	200	0.7629	0.585	0.9264	148.7	3422.9	0.3183	0.667	6.4542
qwen3.5-0.8b	Q8_0	200	0.7398	0.49	0.9859	134.6	3421.5	0.3256	0.6698	5.9296
qwen3.5-2b	Q4_K_M	200	0.7539	0.585	1.3261	118.4	2822.8	0.5116	0.5317	4.864
qwen3.5-2b	Q6_K	200	0.7749	0.625	1.2094	110.6	2712.7	0.5222	0.5019	4.8281
qwen3.5-2b	Q8_0	200	0.7871	0.635	1.3888	96.08	2680.9	0.5237	0.5076	4.4666
qwen3.5-4b	Q4_K_M	200	0.8769	0.755	1.7047	83.16	1464.2	0.6882	0.3376	4.0289
qwen3.5-4b	Q6_K	200	0.8725	0.77	1.8091	75.04	1361.2	0.7044	0.3146	3.6344
qwen3.5-4b	Q8_0	200	0.8831	0.8	2.1601	64.89	1329.0	0.7156	0.3193	3.2444
qwen3.5-9b	Q4_K_M	200	0.8732	0.755	3.1362	59.85	1101.8	0.7577	0.2948	2.9003
qwen3.5-9b	Q6_K	200	0.8853	0.785	3.8213	51.84	988.2	0.7615	0.2837	2.5975
qwen3.5-9b	Q8_0	200	0.8871	0.79	4.3293	42.56	947.3	0.7671	0.2776	2.1761

Model by KV-cache interaction

model	KV cache config	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s
qwen3.5-0.8b	mixed_q8k_q4nlv	300	0.7333	0.5133	1.0546	131.6	2789.7	0.3099	0.6842	5.6087
qwen3.5-0.8b	q8	300	0.7284	0.51	0.8771	158.1	4078.6	0.314	0.6713	6.5347
qwen3.5-2b	mixed_q8k_q4nlv	300	0.7766	0.62	1.3423	102.4	2353.9	0.5233	0.5094	4.4877
qwen3.5-2b	q8	300	0.7674	0.61	1.2739	114.3	3123.8	0.5151	0.518	4.9515
qwen3.5-4b	mixed_q8k_q4nlv	300	0.8779	0.7733	1.9881	70.26	1166.8	0.7008	0.3222	3.438
qwen3.5-4b	q8	300	0.8771	0.7767	1.7945	78.45	1602.1	0.7046	0.3254	3.8339
qwen3.5-9b	mixed_q8k_q4nlv	300	0.8775	0.7633	3.958	49.70	894.6	0.7669	0.2732	2.4536
qwen3.5-9b	q8	300	0.8862	0.79	3.5665	53.13	1129.9	0.7573	0.2975	2.6624

Draft-token by prompt interaction

MTP draft tokens	prompt name	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s
0	Action Item Extraction	24	0.3102	0	1.3698	94.86	2174.9			2.8945
0	CRM Note Enrichment	24	0.9028	0.9167	1.879	94.23	2577.6			4.1146
0	Contract Clause Extraction	24	0.8833	0.9167	1.2368	94.37	2193.9			4.697
0	Contract Risk Flagging	24	0.7333	0.625	4.2598	93.78	2322.8			1.5727
0	Customer Complaint Triage	24	0.6042	0.2083	1.6923	94.87	2207.6			3.2571
0	Email Classification	24	0.9083	0.875	1.1507	95.50	2005.0			6.1006
0	Employee Handbook QA	24	1	1	0.2306	100.8	2110.3			10.69
0	Expense Claim Policy Check	24	0.5312	0	14.60	96.97	3372.6			1.1
0	HR Leave Policy QA	24	0.7708	0.5417	0.2905	105.4	2219.8			7.415
0	IT Incident Ticket Classification	24	0.875	0.9167	1.8258	94.96	2219.4			4.7803
0	Invoice Extraction	24	0.875	0.875	2.2686	93.84	2406.4			4.9489
0	Meeting Summary	24	0.9688	1	4.8513	93.14	2580.0			2.1184
0	Policy Contradiction Detection	24	0.9792	0.9167	3.3989	94.09	1951.7			1.6476
0	Product Feedback Categorization	24	0.9375	0.75	1.0836	95.43	2393.9			4.7525
0	Purchase Order Extraction	24	0.75	0.75	2.1828	93.86	3412.0			5.0593
0	SQL Generation From Schema	24	0.9833	1	1.1397	96.10	1598.6			6.0556
0	Sales Call Summary	24	0.9722	0.75	2.2096	93.64	2709.0			3.6974
0	Support Knowledge Base Answer	24	0.9062	0.7083	1.2859	96.19	2238.4			4.1727
0	Table To Insight Analysis	24	0.6042	0	4.6326	92.45	2950.9			0.7014
0	Tool Calling Intent Selection	24	0.8889	0.75	0.8678	96.39	1922.2			4.2585
1	Action Item Extraction	24	0.3194	0	1.0368	107.0	1943.9	0.8602	0.2874	3.4429
1	CRM Note Enrichment	24	0.9375	1	1.4067	106.5	2395.0	0.8652	0.2725	4.9343
1	Contract Clause Extraction	24	0.8917	0.9167	0.9671	102.0	1886.8	0.8325	0.2404	5.2675
1	Contract Risk Flagging	24	0.7583	0.5417	3.5611	94.48	2076.1	0.6242	0.274	1.6641
1	Customer Complaint Triage	24	0.6111	0.125	1.3417	103.1	2018.3	0.7524	0.2459	3.5304
1	Email Classification	24	0.925	0.9167	0.9474	102.9	1844.7	0.7595	0.2424	6.6512
1	Employee Handbook QA	24	1	1	0.2036	116.7	1880.4	0.8985	0.2452	11.69
1	Expense Claim Policy Check	24	0.5104	0	7.7287	92.75	3044.1	0.6363	0.2394	1.3273
1	HR Leave Policy QA	24	0.8125	0.6667	0.2692	99.13	1944.9	0.739	0.246	6.9704
1	IT Incident Ticket Classification	24	0.8125	0.8333	1.4065	104.1	2048.0	0.7665	0.2467	4.9197
1	Invoice Extraction	24	0.9167	0.9167	1.7745	107.1	1604.9	0.8923	0.1077	5.4411
1	Meeting Summary	24	0.974	0.9583	4.0859	95.13	2255.6	0.6568	0.2945	2.1877
1	Policy Contradiction Detection	24	0.9479	0.8333	2.6138	95.36	1736.4	0.6553	0.2814	1.7749
1	Product Feedback Categorization	24	0.9271	0.8333	0.8188	103.1	2226.1	0.7842	0.2711	5.3548
1	Purchase Order Extraction	24	0.75	0.75	1.669	104.0	3125.6	0.8384	0.1343	5.1305
1	SQL Generation From Schema	24	0.9833	1	0.9745	104.9	1437.9	0.8293	0.3017	6.04
1	Sales Call Summary	24	0.9653	0.8333	1.7041	102.8	2515.4	0.8087	0.2807	4.1622
1	Support Knowledge Base Answer	24	0.8438	0.5417	1.0222	98.75	2011.0	0.6891	0.2728	4.0718
1	Table To Insight Analysis	24	0.6042	0	4.2509	89.82	2708.7	0.5686	0.305	0.6359
1	Tool Calling Intent Selection	24	0.8333	0.6667	0.7397	99.41	1687.8	0.7272	0.3013	3.9038
2	Action Item Extraction	24	0.3194	0	0.9829	108.1	1923.2	0.7277	0.4331	3.2428
2	CRM Note Enrichment	24	0.9444	1	1.2328	108.6	2350.2	0.729	0.4163	5.2274
2	Contract Clause Extraction	24	0.8583	0.875	0.8708	108.4	1967.6	0.7436	0.3785	5.1508
2	Contract Risk Flagging	24	0.75	0.5417	3.6099	89.99	2089.0	0.4733	0.4204	1.421
2	Customer Complaint Triage	24	0.6736	0.3333	1.3181	100.0	2071.5	0.5812	0.3913	3.6967
2	Email Classification	24	0.9	0.9583	0.8515	101.1	1807.8	0.5869	0.3829	6.6665
2	Employee Handbook QA	24	1	1	0.2114	105.7	1792.3	0.7091	0.3887	10.60
2	Expense Claim Policy Check	24	0.5	0	3.7994	91.83	3036.8	0.5027	0.376	1.5178
2	HR Leave Policy QA	24	0.8333	0.7083	0.2587	98.24	1882.9	0.6805	0.3891	7.0381
2	IT Incident Ticket Classification	24	0.8403	0.6667	1.28	101.6	2022.4	0.62	0.3914	4.9956
2	Invoice Extraction	24	0.8333	0.8333	1.6144	108.4	1588.3	0.724	0.276	5.0234
2	Meeting Summary	24	0.974	1	4.1307	91.42	2247.8	0.5028	0.4411	2.0501
2	Policy Contradiction Detection	24	0.9479	0.8333	2.6274	90.26	1694.2	0.4899	0.4287	1.5954
2	Product Feedback Categorization	24	0.9271	0.7083	0.7606	99.88	2124.3	0.6081	0.4157	5.6265
2	Purchase Order Extraction	24	0.7083	0.7083	1.5054	103.4	3107.8	0.6939	0.2904	4.6436
2	SQL Generation From Schema	24	0.9917	1	0.8555	107.1	1394.7	0.6775	0.4506	6.4383
2	Sales Call Summary	24	0.9514	0.75	1.5052	104.9	2454.5	0.6844	0.4248	4.2736
2	Support Knowledge Base Answer	24	0.875	0.7083	1.0589	94.32	1938.2	0.5263	0.4185	3.7392
2	Table To Insight Analysis	24	0.6042	0.0833	4.5009	84.26	2671.6	0.4131	0.4538	0.5864
2	Tool Calling Intent Selection	24	0.8194	0.6667	0.756	98.19	1634.5	0.5946	0.4497	3.46
3	Action Item Extraction	24	0.3194	0	0.9552	106.2	1878.4	0.6395	0.5585	3.2183
3	CRM Note Enrichment	24	0.9653	1	1.2561	103.6	2327.6	0.6057	0.5406	5.0335
3	Contract Clause Extraction	24	0.9667	1	0.8563	104.1	1955.3	0.6371	0.503	6.0395
3	Contract Risk Flagging	24	0.725	0.5417	3.3792	78.36	2095.5	0.325	0.5507	1.3721
3	Customer Complaint Triage	24	0.6319	0.1667	1.2972	92.00	2018.8	0.4523	0.5166	3.3038
3	Email Classification	24	0.875	0.875	0.8609	97.43	1838.0	0.5144	0.5073	5.9155
3	Employee Handbook QA	24	0.9792	0.9583	0.2114	95.77	1821.9	0.6717	0.5144	10.46
3	Expense Claim Policy Check	24	0.4896	0	2.5895	81.94	2988.0	0.3666	0.5073	1.0607
3	HR Leave Policy QA	24	0.7917	0.625	0.2401	88.86	1893.9	0.5912	0.5159	7.0014
3	IT Incident Ticket Classification	24	0.875	0.875	1.3076	95.29	2008.6	0.4876	0.5183	4.8265
3	Invoice Extraction	24	0.8333	0.8333	1.6175	100.4	1615.4	0.5919	0.4081	4.7948
3	Meeting Summary	24	0.974	1	4.3572	83.18	2266.5	0.3872	0.5679	1.915
3	Policy Contradiction Detection	24	0.9688	0.9167	2.6523	82.66	1679.4	0.3898	0.5485	1.6354
3	Product Feedback Categorization	24	0.9583	0.8333	0.7936	92.04	2135.3	0.4843	0.5398	5.3955
3	Purchase Order Extraction	24	0.7083	0.7083	1.4883	98.66	3088.1	0.5923	0.4078	4.5988
3	SQL Generation From Schema	24	1	1	0.8583	99.76	1384.3	0.5526	0.5669	6.2915
3	Sales Call Summary	24	0.9722	0.7917	1.4887	99.15	2476.7	0.5703	0.5492	4.322
3	Support Knowledge Base Answer	24	0.8333	0.5417	1.0219	91.51	1937.2	0.445	0.5408	3.6824
3	Table To Insight Analysis	24	0.6042	0	4.7739	75.54	2673.0	0.3193	0.57	0.5704
3	Tool Calling Intent Selection	24	0.8333	0.625	0.7546	90.55	1624.1	0.4817	0.5662	3.6523
4	Action Item Extraction	24	0.3148	0	0.9493	96.60	1873.5	0.5428	0.6475	3.1071
4	CRM Note Enrichment	24	0.9514	1	1.2985	95.10	2304.2	0.5116	0.6335	4.6188
4	Contract Clause Extraction	24	0.8333	0.875	0.914	95.20	1960.7	0.5331	0.61	4.5364
4	Contract Risk Flagging	24	0.7083	0.4167	3.5796	72.53	2034.1	0.2843	0.6389	1.1488
4	Customer Complaint Triage	24	0.6458	0.25	1.3694	83.65	1988.2	0.3754	0.6237	3.112
4	Email Classification	24	0.875	0.9583	0.9322	85.67	1782.8	0.3865	0.6176	5.5712
4	Employee Handbook QA	24	1	1	0.2214	86.72	1762.1	0.5813	0.6186	9.9172
4	Expense Claim Policy Check	24	0.4896	0	8.5982	71.42	2881.6	0.2863	0.6138	1.0919
4	HR Leave Policy QA	24	0.7292	0.5	0.2309	86.31	1868.7	0.4944	0.6206	6.7177
4	IT Incident Ticket Classification	24	0.8333	0.8333	1.3911	87.57	1978.5	0.4111	0.622	4.2385
4	Invoice Extraction	24	0.9167	0.9167	1.7028	92.55	1603.9	0.503	0.497	5.1098
4	Meeting Summary	24	0.9844	1	4.8314	74.83	2196.9	0.3112	0.6543	1.8074
4	Policy Contradiction Detection	24	0.9271	0.7083	3.1388	74.12	1642.0	0.3072	0.6415	1.3274
4	Product Feedback Categorization	24	0.9271	0.75	0.8466	82.97	2113.0	0.4043	0.633	4.8433
4	Purchase Order Extraction	24	0.7917	0.7917	1.5298	91.21	3091.6	0.5052	0.4956	5.01
4	SQL Generation From Schema	24	0.9667	0.9583	0.9517	88.43	1354.1	0.4363	0.6549	5.3342
4	Sales Call Summary	24	0.9861	0.9167	1.5577	91.39	2389.6	0.4846	0.6402	4.0587
4	Support Knowledge Base Answer	24	0.9271	0.8333	1.0943	78.33	1875.2	0.3559	0.6339	3.6891
4	Table To Insight Analysis	24	0.5938	0	4.7765	68.40	2624.6	0.2575	0.6568	0.5445
4	Tool Calling Intent Selection	24	0.8889	0.75	0.8058	82.39	1647.3	0.3875	0.6544	3.6199

Model by prompt interaction

model	prompt name	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s	avg. draft accept pct	avg. draft reject pct	avg. semantic points per s
qwen3.5-0.8b	Action Item Extraction	30	0.3037	0	0.6137	146.4	3119.7	0.3241	0.6954	4.5994
qwen3.5-0.8b	CRM Note Enrichment	30	0.9056	0.9667	0.7736	154.4	3984.1	0.401	0.6879	7.2344
qwen3.5-0.8b	Contract Clause Extraction	30	0.62	0.6667	0.5574	144.0	3105.3	0.3345	0.6626	5.5451
qwen3.5-0.8b	Contract Risk Flagging	30	0.5933	0.2667	1.2941	139.4	3336.9	0.273	0.6755	2.5114
qwen3.5-0.8b	Customer Complaint Triage	30	0.6	0.1333	0.7696	150.7	3239.7	0.3573	0.6632	5.0983
qwen3.5-0.8b	Email Classification	30	0.96	0.9667	0.4695	154.0	2948.4	0.3827	0.6665	10.40
qwen3.5-0.8b	Employee Handbook QA	30	0.9833	0.9667	0.1602	156.4	2947.5	0.4554	0.6623	14.44
qwen3.5-0.8b	Expense Claim Policy Check	30	0.3833	0	1.0064	137.7	5001.8	0.2712	0.6744	1.7543
qwen3.5-0.8b	HR Leave Policy QA	30	0.45	0.0333	0.1082	133.7	3127.9	0.088	0.6622	8.5158
qwen3.5-0.8b	IT Incident Ticket Classification	30	0.7722	0.6667	0.7292	150.1	3285.1	0.3577	0.6601	6.6926
qwen3.5-0.8b	Invoice Extraction	30	0.7667	0.7667	1.0178	153.6	2639.4	0.3997	0.6003	6.9854
qwen3.5-0.8b	Meeting Summary	30	0.95	0.9667	2.9958	137.0	3759.4	0.2605	0.6965	2.7178
qwen3.5-0.8b	Policy Contradiction Detection	30	0.8667	0.5333	1.841	134.8	2766.0	0.2369	0.6996	2.1696
qwen3.5-0.8b	Product Feedback Categorization	30	0.7833	0.2667	0.4316	144.8	3498.0	0.2812	0.6887	7.8727
qwen3.5-0.8b	Purchase Order Extraction	30	0.9333	0.9333	0.8408	142.7	5259.4	0.3047	0.6454	10.60
qwen3.5-0.8b	SQL Generation From Schema	30	0.94	0.9667	0.6356	151.1	2226.4	0.3471	0.7099	8.0545
qwen3.5-0.8b	Sales Call Summary	30	0.9	0.4667	1.055	145.2	4122.0	0.3225	0.6938	5.4351
qwen3.5-0.8b	Support Knowledge Base Answer	30	0.725	0.3	0.5445	143.8	3198.2	0.2847	0.6899	5.4493
qwen3.5-0.8b	Table To Insight Analysis	30	0.5583	0.0333	3.0211	132.6	4458.7	0.2334	0.7114	0.8161
qwen3.5-0.8b	Tool Calling Intent Selection	30	0.6222	0.3333	0.4521	144.7	2659.0	0.3228	0.7094	4.5343
qwen3.5-2b	Action Item Extraction	30	0.3	0	0.8265	119.5	2478.5	0.6367	0.5293	3.4266
qwen3.5-2b	CRM Note Enrichment	30	0.8556	0.9667	1.0582	113.7	3022.2	0.588	0.5127	4.9809
qwen3.5-2b	Contract Clause Extraction	30	0.94	1	0.7376	116.2	2574.9	0.6167	0.495	6.524
qwen3.5-2b	Contract Risk Flagging	30	0.6867	0.4333	2.4458	98.07	2710.0	0.3862	0.5257	1.519
qwen3.5-2b	Customer Complaint Triage	30	0.5611	0	0.899	108.3	2629.2	0.4962	0.5149	3.8316
qwen3.5-2b	Email Classification	30	0.8867	0.9	0.6422	112.2	2347.3	0.5331	0.5155	7.0349
qwen3.5-2b	Employee Handbook QA	30	1	1	0.1713	114.8	2401.4	0.7177	0.5049	12.51
qwen3.5-2b	Expense Claim Policy Check	30	0.575	0	2.6459	98.16	3894.7	0.4012	0.5214	1.5794
qwen3.5-2b	HR Leave Policy QA	30	0.7	0.4	0.1636	109.5	2496.1	0.5514	0.5075	8.5432
qwen3.5-2b	IT Incident Ticket Classification	30	0.9111	0.9333	0.8364	113.0	2600.4	0.5386	0.5093	6.6545
qwen3.5-2b	Invoice Extraction	30	0.7333	0.7333	1.3313	110.1	2217.7	0.5509	0.4491	5.1378
qwen3.5-2b	Meeting Summary	30	0.9708	1	3.3944	101.7	2979.3	0.4309	0.5385	2.3699
qwen3.5-2b	Policy Contradiction Detection	30	0.95	0.8333	2.0791	101.3	2229.8	0.4241	0.5195	1.8844
qwen3.5-2b	Product Feedback Categorization	30	0.9667	0.8667	0.7778	108.8	2780.3	0.5239	0.5115	5.2763
qwen3.5-2b	Purchase Order Extraction	30	0.0333	0.0333	1.2949	109.0	4094.9	0.5288	0.4606	0.1744
qwen3.5-2b	SQL Generation From Schema	30	1	1	0.6827	111.7	1848.5	0.5334	0.5423	7.5554
qwen3.5-2b	Sales Call Summary	30	0.9778	0.7667	1.1884	115.7	3219.6	0.6156	0.5189	5.0912
qwen3.5-2b	Support Knowledge Base Answer	30	0.8167	0.5	0.7472	105.4	2524.7	0.4668	0.5127	4.4657
qwen3.5-2b	Table To Insight Analysis	30	0.6083	0	3.6387	93.11	3538.1	0.3373	0.5441	0.7213
qwen3.5-2b	Tool Calling Intent Selection	30	0.9667	0.9333	0.6007	107.3	2188.7	0.506	0.5413	5.1095
qwen3.5-4b	Action Item Extraction	30	0.3296	0	1.1451	85.16	1288.4	0.8729	0.368	2.7123
qwen3.5-4b	CRM Note Enrichment	30	1	1	1.5571	80.58	1498.7	0.8037	0.35	4.0454
qwen3.5-4b	Contract Clause Extraction	30	1	1	1.0416	85.30	1318.5	0.8763	0.2934	5.0218
qwen3.5-4b	Contract Risk Flagging	30	0.7933	0.6667	4.0773	62.64	1414.6	0.4941	0.3529	1.0227
qwen3.5-4b	Customer Complaint Triage	30	0.6389	0.3	1.5396	70.97	1377.9	0.6219	0.3136	2.6568
qwen3.5-4b	Email Classification	30	0.88	0.9	0.9981	68.95	1233.8	0.5889	0.2975	4.5888
qwen3.5-4b	Employee Handbook QA	30	1	1	0.219	85.31	1233.0	1	0.3078	9.2424
qwen3.5-4b	Expense Claim Policy Check	30	0.5667	0	2.7604	65.89	1881.7	0.5501	0.2744	1.1079
qwen3.5-4b	HR Leave Policy QA	30	1	1	0.3289	81.71	1272.2	0.8866	0.3105	6.2318
qwen3.5-4b	IT Incident Ticket Classification	30	0.7889	0.7	1.3891	74.49	1348.7	0.6923	0.3141	3.5284
qwen3.5-4b	Invoice Extraction	30	1	1	1.9831	80.59	1245.3	0.8269	0.1731	4.7187
qwen3.5-4b	Meeting Summary	30	0.9833	1	4.6928	65.72	1443.7	0.5433	0.3797	1.7305
qwen3.5-4b	Policy Contradiction Detection	30	1	1	2.844	67.94	1132.8	0.5837	0.3584	1.4468
qwen3.5-4b	Product Feedback Categorization	30	0.9917	0.9667	0.9368	72.53	1454.9	0.6699	0.3494	4.3347
qwen3.5-4b	Purchase Order Extraction	30	1	1	1.8492	82.79	1915.4	0.8546	0.1598	5.138
qwen3.5-4b	SQL Generation From Schema	30	1	1	1.1157	78.25	957.0	0.7489	0.3863	4.6561
qwen3.5-4b	Sales Call Summary	30	1	1	1.8012	76.88	1572.9	0.7434	0.3596	3.4869
qwen3.5-4b	Support Knowledge Base Answer	30	1	1	1.2016	70.40	1303.0	0.6122	0.3513	3.4267
qwen3.5-4b	Table To Insight Analysis	30	0.6	0	5.4006	60.30	1684.9	0.4526	0.3904	0.4647
qwen3.5-4b	Tool Calling Intent Selection	30	0.9778	0.9667	0.945	70.56	1135.5	0.6318	0.3859	3.1568
qwen3.5-9b	Action Item Extraction	30	0.3333	0	1.6499	59.17	948.5	0.9366	0.3338	1.9863
qwen3.5-9b	CRM Note Enrichment	30	1	1	2.2696	57.62	1058.7	0.9186	0.3123	2.8822
qwen3.5-9b	Contract Clause Extraction	30	0.9867	1	1.5394	57.77	972.9	0.9187	0.2809	3.4621
qwen3.5-9b	Contract Risk Flagging	30	0.8667	0.7667	6.8945	43.16	1032.5	0.5535	0.3299	0.6899
qwen3.5-9b	Customer Complaint Triage	30	0.7333	0.4333	2.4068	48.93	996.8	0.6859	0.2859	1.9333
qwen3.5-9b	Email Classification	30	0.86	0.9	1.6844	50.84	893.2	0.7426	0.2706	2.7001
qwen3.5-9b	Employee Handbook QA	30	1	1	0.3122	48.01	911.8	0.6875	0.2919	6.4946
qwen3.5-9b	Expense Claim Policy Check	30	0.4917	0	23.44	43.37	1363.6	0.5694	0.2663	0.4365
qwen3.5-9b	HR Leave Policy QA	30	1	1	0.4307	57.47	951.9	0.9792	0.2915	4.8234
qwen3.5-9b	IT Incident Ticket Classification	30	0.9167	1	2.8142	49.30	987.3	0.6968	0.2949	2.133
qwen3.5-9b	Invoice Extraction	30	1	1	2.8501	57.50	952.7	0.9338	0.0662	3.4126
qwen3.5-9b	Meeting Summary	30	0.9958	1	6.7222	45.83	1054.9	0.6233	0.3431	1.2447
qwen3.5-9b	Policy Contradiction Detection	30	1	1	4.7809	45.17	834.4	0.5976	0.3227	0.8838
qwen3.5-9b	Product Feedback Categorization	30	1	1	1.2964	52.66	1060.9	0.8058	0.3099	3.2945
qwen3.5-9b	Purchase Order Extraction	30	1	1	2.7153	58.37	1390.4	0.9416	0.0624	3.6365
qwen3.5-9b	SQL Generation From Schema	30	1	1	1.3898	55.91	703.8	0.8663	0.3357	3.8617
qwen3.5-9b	Sales Call Summary	30	1	1	2.7278	55.81	1121.8	0.8666	0.3227	2.3979
qwen3.5-9b	Support Knowledge Base Answer	30	0.9667	0.8667	1.8934	47.72	974.2	0.6527	0.312	2.1425
qwen3.5-9b	Table To Insight Analysis	30	0.6417	0.0333	6.2874	42.38	1221.3	0.5351	0.3397	0.4287
qwen3.5-9b	Tool Calling Intent Selection	30	0.8444	0.5333	1.1414	51.03	829.6	0.7306	0.3349	2.315

Confidence Intervals

Bootstrap confidence intervals from the report, using 500 bootstrap samples over benchmark rows.

Main-effect model confidence intervals

model	metric	mean	ci low	ci high	bootstrap samples
qwen3.5-0.8b	avg_semantic_score	0.7309	0.7079	0.7525	500
qwen3.5-0.8b	overall_pass_rate	0.5117	0.4717	0.5517	500
qwen3.5-0.8b	avg_elapsed_s	0.9659	0.8962	1.0306	500
qwen3.5-0.8b	avg_generation_tok_s	144.8	142.1	147.5	500
qwen3.5-2b	avg_semantic_score	0.772	0.7486	0.7965	500
qwen3.5-2b	overall_pass_rate	0.615	0.5767	0.6533	500
qwen3.5-2b	avg_elapsed_s	1.3081	1.1875	1.4881	500
qwen3.5-2b	avg_generation_tok_s	108.4	107.0	109.9	500
qwen3.5-4b	avg_semantic_score	0.8775	0.8598	0.8943	500
qwen3.5-4b	overall_pass_rate	0.775	0.7417	0.8067	500
qwen3.5-4b	avg_elapsed_s	1.8913	1.7731	2.0201	500
qwen3.5-4b	avg_generation_tok_s	74.36	72.91	75.80	500
qwen3.5-9b	avg_semantic_score	0.8818	0.8653	0.8973	500
qwen3.5-9b	overall_pass_rate	0.7767	0.745	0.8125	500
qwen3.5-9b	avg_elapsed_s	3.7623	3.1691	4.432	500
qwen3.5-9b	avg_generation_tok_s	51.41	50.20	52.52	500

Main-effect draft-token confidence intervals

MTP draft tokens	metric	mean	ci low	ci high	bootstrap samples
0	avg_semantic_score	0.8192	0.7967	0.8416	500
0	overall_pass_rate	0.675	0.6375	0.7136	500
0	avg_elapsed_s	2.6227	2.0422	3.3129	500
0	avg_generation_tok_s	95.54	90.23	100.5	500
1	avg_semantic_score	0.8162	0.7928	0.8384	500
1	overall_pass_rate	0.6667	0.6188	0.7053	500
1	avg_elapsed_s	1.9261	1.6225	2.3464	500
1	avg_generation_tok_s	101.4	97.13	105.7	500
2	avg_semantic_score	0.8126	0.7902	0.8351	500
2	overall_pass_rate	0.6687	0.626	0.7073	500
2	avg_elapsed_s	1.6865	1.4776	1.9739	500
2	avg_generation_tok_s	99.78	96.47	103.0	500
3	avg_semantic_score	0.8152	0.7909	0.8357	500
3	overall_pass_rate	0.6646	0.6208	0.7063	500
3	avg_elapsed_s	1.638	1.5065	1.7752	500
3	avg_generation_tok_s	92.85	90.29	95.39	500
4	avg_semantic_score	0.8145	0.7915	0.8351	500
4	overall_pass_rate	0.6729	0.6312	0.7125	500
4	avg_elapsed_s	2.036	1.6808	2.5478	500
4	avg_generation_tok_s	84.27	82.18	86.27	500

Main-effect quantisation confidence intervals

quantization	metric	mean	ci low	ci high	bootstrap samples
Q4_K_M	avg_semantic_score	0.7985	0.7793	0.8174	500
Q4_K_M	overall_pass_rate	0.6388	0.6038	0.67	500
Q4_K_M	avg_elapsed_s	1.7881	1.555	2.0467	500
Q4_K_M	avg_generation_tok_s	103.2	99.98	106.4	500
Q6_K	avg_semantic_score	0.8239	0.8054	0.8394	500
Q6_K	overall_pass_rate	0.6913	0.6562	0.7238	500
Q6_K	avg_elapsed_s	1.9415	1.638	2.298	500
Q6_K	avg_generation_tok_s	96.64	93.78	99.52	500
Q8_0	avg_semantic_score	0.8243	0.8069	0.8417	500
Q8_0	overall_pass_rate	0.6787	0.6418	0.71	500
Q8_0	avg_elapsed_s	2.216	1.9459	2.5919	500
Q8_0	avg_generation_tok_s	84.53	81.92	87.27	500

Main-effect KV-cache confidence intervals

KV cache config	metric	mean	ci low	ci high	bootstrap samples
mixed_q8k_q4nlv	avg_semantic_score	0.8163	0.8016	0.8305	500
mixed_q8k_q4nlv	overall_pass_rate	0.6675	0.6425	0.6958	500
mixed_q8k_q4nlv	avg_elapsed_s	2.0858	1.8302	2.3797	500
mixed_q8k_q4nlv	avg_generation_tok_s	88.54	86.58	90.47	500
q8	avg_semantic_score	0.8148	0.8005	0.8302	500
q8	overall_pass_rate	0.6717	0.645	0.7013	500
q8	avg_elapsed_s	1.878	1.6701	2.1283	500
q8	avg_generation_tok_s	101.0	98.43	103.5	500

Model by draft-token confidence intervals

model	MTP draft tokens	metric	mean	ci low	ci high	bootstrap samples
qwen3.5-0.8b	0	avg_semantic_score	0.7419	0.6946	0.7878	500
qwen3.5-0.8b	0	overall_pass_rate	0.5167	0.425	0.6	500
qwen3.5-0.8b	0	avg_elapsed_s	0.7636	0.666	0.8717	500
qwen3.5-0.8b	0	avg_generation_tok_s	182.3	178.2	186.0	500
qwen3.5-0.8b	1	avg_semantic_score	0.7134	0.6595	0.7688	500
qwen3.5-0.8b	1	overall_pass_rate	0.5	0.4083	0.5917	500
qwen3.5-0.8b	1	avg_elapsed_s	0.809	0.7048	0.9322	500
qwen3.5-0.8b	1	avg_generation_tok_s	165.8	161.7	170.3	500
qwen3.5-0.8b	2	avg_semantic_score	0.7153	0.6659	0.7687	500
qwen3.5-0.8b	2	overall_pass_rate	0.4917	0.4083	0.571	500
qwen3.5-0.8b	2	avg_elapsed_s	0.9511	0.8201	1.1038	500
qwen3.5-0.8b	2	avg_generation_tok_s	143.1	139.2	146.9	500
qwen3.5-0.8b	3	avg_semantic_score	0.7563	0.7032	0.8049	500
qwen3.5-0.8b	3	overall_pass_rate	0.5417	0.4623	0.6417	500
qwen3.5-0.8b	3	avg_elapsed_s	1.0986	0.9434	1.2807	500
qwen3.5-0.8b	3	avg_generation_tok_s	125.0	121.9	128.2	500
qwen3.5-0.8b	4	avg_semantic_score	0.7275	0.6806	0.78	500
qwen3.5-0.8b	4	overall_pass_rate	0.5083	0.4167	0.6	500
qwen3.5-0.8b	4	avg_elapsed_s	1.207	1.0263	1.4106	500
qwen3.5-0.8b	4	avg_generation_tok_s	108.0	105.6	110.6	500
qwen3.5-2b	0	avg_semantic_score	0.7643	0.7068	0.8183	500
qwen3.5-2b	0	overall_pass_rate	0.6083	0.525	0.6877	500
qwen3.5-2b	0	avg_elapsed_s	1.2039	1.0477	1.3638	500
qwen3.5-2b	0	avg_generation_tok_s	109.0	106.2	111.7	500
qwen3.5-2b	1	avg_semantic_score	0.7793	0.7281	0.8307	500
qwen3.5-2b	1	overall_pass_rate	0.6083	0.525	0.6917	500
qwen3.5-2b	1	avg_elapsed_s	1.4203	1.0337	2.0445	500
qwen3.5-2b	1	avg_generation_tok_s	118.8	116.0	122.0	500
qwen3.5-2b	2	avg_semantic_score	0.7781	0.729	0.8255	500
qwen3.5-2b	2	overall_pass_rate	0.6333	0.55	0.7167	500
qwen3.5-2b	2	avg_elapsed_s	1.2151	1.0659	1.3976	500
qwen3.5-2b	2	avg_generation_tok_s	117.1	114.2	119.7	500
qwen3.5-2b	3	avg_semantic_score	0.7603	0.7044	0.8175	500
qwen3.5-2b	3	overall_pass_rate	0.6083	0.5083	0.6917	500
qwen3.5-2b	3	avg_elapsed_s	1.3079	1.1251	1.5303	500
qwen3.5-2b	3	avg_generation_tok_s	105.0	102.7	107.9	500
qwen3.5-2b	4	avg_semantic_score	0.7779	0.7271	0.8271	500
qwen3.5-2b	4	overall_pass_rate	0.6167	0.5333	0.696	500
qwen3.5-2b	4	avg_elapsed_s	1.3932	1.201	1.603	500
qwen3.5-2b	4	avg_generation_tok_s	92.01	89.77	94.51	500
qwen3.5-4b	0	avg_semantic_score	0.8835	0.8464	0.916	500
qwen3.5-4b	0	overall_pass_rate	0.7833	0.7123	0.8583	500
qwen3.5-4b	0	avg_elapsed_s	2.3499	2.0378	2.6306	500
qwen3.5-4b	0	avg_generation_tok_s	56.11	54.57	57.65	500
qwen3.5-4b	1	avg_semantic_score	0.8894	0.8526	0.9193	500
qwen3.5-4b	1	overall_pass_rate	0.7833	0.7123	0.85	500
qwen3.5-4b	1	avg_elapsed_s	1.8556	1.6062	2.0972	500
qwen3.5-4b	1	avg_generation_tok_s	73.74	71.92	75.88	500
qwen3.5-4b	2	avg_semantic_score	0.8743	0.8406	0.9094	500
qwen3.5-4b	2	overall_pass_rate	0.7583	0.6833	0.8333	500
qwen3.5-4b	2	avg_elapsed_s	1.6964	1.4274	1.9639	500
qwen3.5-4b	2	avg_generation_tok_s	82.38	79.96	84.82	500
qwen3.5-4b	3	avg_semantic_score	0.8702	0.8302	0.9066	500
qwen3.5-4b	3	overall_pass_rate	0.775	0.6917	0.85	500
qwen3.5-4b	3	avg_elapsed_s	1.7952	1.5158	2.1261	500
qwen3.5-4b	3	avg_generation_tok_s	82.42	79.57	85.42	500
qwen3.5-4b	4	avg_semantic_score	0.8702	0.829	0.9093	500
qwen3.5-4b	4	overall_pass_rate	0.775	0.7	0.8417	500
qwen3.5-4b	4	avg_elapsed_s	1.7595	1.5148	2.0002	500
qwen3.5-4b	4	avg_generation_tok_s	77.00	74.02	79.88	500
qwen3.5-9b	0	avg_semantic_score	0.8873	0.8512	0.9205	500
qwen3.5-9b	0	overall_pass_rate	0.7917	0.7123	0.8583	500
qwen3.5-9b	0	avg_elapsed_s	6.1736	4.1172	8.7699	500
qwen3.5-9b	0	avg_generation_tok_s	33.94	32.84	35.09	500
qwen3.5-9b	1	avg_semantic_score	0.8828	0.8456	0.9188	500
qwen3.5-9b	1	overall_pass_rate	0.775	0.6956	0.8417	500
qwen3.5-9b	1	avg_elapsed_s	3.6196	2.5546	5.1254	500
qwen3.5-9b	1	avg_generation_tok_s	47.47	46.07	48.82	500
qwen3.5-9b	2	avg_semantic_score	0.8826	0.8446	0.9226	500
qwen3.5-9b	2	overall_pass_rate	0.7917	0.7167	0.8627	500
qwen3.5-9b	2	avg_elapsed_s	2.8834	2.2325	3.9679	500
qwen3.5-9b	2	avg_generation_tok_s	56.51	54.64	58.25	500
qwen3.5-9b	3	avg_semantic_score	0.874	0.8355	0.9101	500
qwen3.5-9b	3	overall_pass_rate	0.7333	0.65	0.8083	500
qwen3.5-9b	3	avg_elapsed_s	2.3503	2.0175	2.6788	500
qwen3.5-9b	3	avg_generation_tok_s	58.95	56.66	61.09	500
qwen3.5-9b	4	avg_semantic_score	0.8825	0.8456	0.9149	500
qwen3.5-9b	4	overall_pass_rate	0.7917	0.725	0.8583	500
qwen3.5-9b	4	avg_elapsed_s	3.7843	2.3247	5.3045	500
qwen3.5-9b	4	avg_generation_tok_s	60.05	57.57	62.56	500

Enterprise Function Suitability

Function-level scores are shown as exploratory diagnostics only. The function recommendation table was removed because small task slices can overstate weak models, especially qwen3.5-0.8b.

Function suitability matrix

model	quantization	KV cache config	MTP draft tokens	finance	HR	IT	customer support	legal	sales	operations	data analytics	knowledge management	overall
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	0	0.8333	0.75	0.9167	0.1667	0.8	0.9167	0.7708	0.875	1	0.8133
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	1	0.75	0.5	0.75	0.5	0.2	0.8333	0.5729	0.65	0.625	0.6079
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	2	0.5	0.5	0.9167	0.5	0.4	0.6667	0.5764	0.875	0.875	0.6386
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	3	0.5	0.75	0.9167	0.1667	0.6	0.9167	0.6354	0.75	0.875	0.6912
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	4	0.5	0.5	0.8333	0.6667	0.8	0.9167	0.5208	0.525	0.75	0.645
qwen3.5-0.8b	Q4_K_M	q8	0	0.8333	0.75	0.4333	0.6667	0.2	0.8333	0.8021	0.775	0.75	0.6929
qwen3.5-0.8b	Q4_K_M	q8	1	0.5	0.75	0.9167	0.5	0.9	0.9167	0.5347	0.625	0.875	0.7053
qwen3.5-0.8b	Q4_K_M	q8	2	0.0833	0.75	0.8333	0.6667	0.7	0.8333	0.4896	0.625	1	0.6179
qwen3.5-0.8b	Q4_K_M	q8	3	0.8333	0.75	0.7333	0.6667	0.7	1	0.6042	0.75	0.875	0.76
qwen3.5-0.8b	Q4_K_M	q8	4	0.75	0.75	0.8167	0.5	0.4	0.9167	0.6875	0.75	0.875	0.7258
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	0	0.4167	0.75	0.9167	0.5	0.8	0.9167	0.7118	0.625	0.625	0.6932
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	1	0.8333	0.75	0.8333	0.5	0.8	0.9167	0.4792	0.75	0.625	0.7133
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	2	0.8333	0.75	0.9167	0.6667	0.6	0.9167	0.7708	0.65	0.875	0.7833
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	3	0.5	1	0.9167	1	0.6	0.9167	0.691	0.875	0.75	0.769
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	4	0.75	0.75	0.9167	0.6667	0.9	1	0.8056	0.875	0.875	0.8386
qwen3.5-0.8b	Q6_K	q8	0	0.75	0.75	0.9167	0.6667	0.7	1	0.6562	0.65	0.75	0.7538
qwen3.5-0.8b	Q6_K	q8	1	0.75	0.75	0.9167	0.6667	0.9	0.9167	0.6667	0.65	0.75	0.7675
qwen3.5-0.8b	Q6_K	q8	2	0.8333	0.75	1	0.8333	0.4	0.9167	0.6875	1	0.75	0.7858
qwen3.5-0.8b	Q6_K	q8	3	0.75	0.25	0.9167	0.6667	1	1	0.7431	0.75	1	0.7861
qwen3.5-0.8b	Q6_K	q8	4	0.75	0.75	0.9167	0.6667	0.3	0.9167	0.7083	0.875	0.75	0.7383
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	0	0.9167	0.75	0.9167	0.5	0.7	0.6667	0.6875	0.75	0.875	0.7658
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	1	0.9167	0.75	0.9167	0.5	0.8	0.75	0.8333	0.75	0.875	0.8133
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	2	0.75	0.75	0.8333	0.5	0.7	1	0.7708	0.75	0.75	0.77
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	3	0.75	0.75	0.9167	0.6667	0.7	0.9167	0.6875	0.875	1	0.7992
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	4	0.75	0.75	0.75	0.6667	0.2	0.9167	0.7743	0.325	0.625	0.6574
qwen3.5-0.8b	Q8_0	q8	0	0.75	0.75	0.9167	0.8333	0.3	0.9167	0.5729	0.875	0.875	0.7321
qwen3.5-0.8b	Q8_0	q8	1	0.4167	0.75	0.9167	0.5	0.2	0.9167	0.6597	0.875	0.875	0.6728
qwen3.5-0.8b	Q8_0	q8	2	0.75	0.75	0.8333	0.8333	0.5	0.9167	0.5208	0.75	0.625	0.6958
qwen3.5-0.8b	Q8_0	q8	3	0.75	0.75	0.8333	0.6667	0.7	1	0.6042	0.875	0.5	0.7325
qwen3.5-0.8b	Q8_0	q8	4	0.8333	0.75	0.9167	0.5	0.7	0.9167	0.7188	0.75	0.625	0.7596
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	0	0.5	0.75	1	0.6667	0.7	0.6667	0.7743	0.875	0.875	0.7499
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	1	0.5	1	1	0.5	0.7	0.9167	0.8333	0.75	0.875	0.7908
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	2	0.5	1	1	0.6667	0.9	0.9167	0.6285	0.875	0.5	0.7532
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	3	0.25	0.75	0.9	0.6667	0.9	1	0.8333	0.875	0.625	0.7425
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	4	0.5	0.75	0.9167	0.6667	0.6	0.9167	0.8333	0.75	0.875	0.7558
qwen3.5-2b	Q4_K_M	q8	0	0.5	0.75	0.7167	0.1667	0.9	1	0.7431	0.875	1	0.7561
qwen3.5-2b	Q4_K_M	q8	1	0.5	0.75	0.9167	0.6667	0.7	0.9167	0.8333	0.875	0.75	0.7658
qwen3.5-2b	Q4_K_M	q8	2	0.5	1	0.8167	0.5	0.6	0.9167	0.7743	0.75	0.75	0.7382
qwen3.5-2b	Q4_K_M	q8	3	0.1667	0.75	0.8167	0.6667	0.9	0.9167	0.8021	0.75	0.875	0.7196
qwen3.5-2b	Q4_K_M	q8	4	0.5833	0.75	0.7333	0.6667	0.7	0.9167	0.7431	0.875	1	0.7669
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	0	0.5	0.75	0.9	0.5	0.9	0.9167	0.8056	0.75	1	0.7828
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	1	0.5	1	0.9167	0.6667	0.9	0.9167	0.8333	0.875	0.75	0.8108
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	2	0.1667	1	0.9	0.6667	1	0.9167	0.8333	0.75	1	0.7817
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	3	0.5	0.75	0.9167	0.6667	0.8	0.9167	0.8333	0.875	1	0.8008
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	4	0.5	0.75	0.8167	0.5	0.8	1	0.7708	0.875	1	0.7783
qwen3.5-2b	Q6_K	q8	0	0.25	0.75	1	0.6667	0.7	0.9167	0.8333	0.75	0.875	0.7367
qwen3.5-2b	Q6_K	q8	1	0.5	1	0.7167	0.5	0.8	0.9167	0.6667	0.875	0.875	0.7517
qwen3.5-2b	Q6_K	q8	2	0.5833	1	0.9167	0.5	0.9	0.8333	0.8333	0.625	1	0.8067
qwen3.5-2b	Q6_K	q8	3	0.1667	0.75	1	0.5	0.8	0.9167	0.8021	0.875	0.75	0.7196
qwen3.5-2b	Q6_K	q8	4	0.5833	0.75	1	0.1667	0.7	0.9167	0.8021	0.875	1	0.7804
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	0	0.25	0.75	0.9167	0.5	0.8	0.9167	0.8056	0.75	1	0.7369
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	1	0.5	1	0.8333	0.5	0.8	1	0.8056	0.75	0.875	0.7869
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	2	0.5833	1	0.9	0.6667	0.8	1	0.8333	0.75	0.875	0.82
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	3	0.5	1	1	0.5	0.9	0.8333	0.8056	0.625	0.75	0.7719
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	4	0.5	0.75	0.9	0.5	0.8	1	0.8333	0.875	0.875	0.7867
qwen3.5-2b	Q8_0	q8	0	0.5	1	1	0.6667	0.9	0.8333	0.8333	0.75	1	0.8233
qwen3.5-2b	Q8_0	q8	1	0.5	0.75	0.8167	0.6667	0.9	0.9167	0.8021	0.75	0.875	0.7696
qwen3.5-2b	Q8_0	q8	2	0.25	0.75	1	0.6667	0.9	0.9167	0.8333	0.875	0.875	0.7692
qwen3.5-2b	Q8_0	q8	3	0.5	1	0.8	0.3333	0.9	0.9167	0.8333	0.875	1	0.8075
qwen3.5-2b	Q8_0	q8	4	0.5833	0.75	0.9	0.6667	0.8	0.9167	0.8333	0.75	1	0.7992
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	0	0.9167	1	0.9167	0.6667	1	1	0.8333	0.75	1	0.9042
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	1	0.9167	1	0.9167	0.6667	0.8	1	0.8021	0.75	1	0.8779
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	2	0.9167	1	0.8333	0.8333	0.8	1	0.8333	0.875	1	0.8967
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	3	0.8333	1	0.6333	0.6667	0.9	1	0.8333	0.75	1	0.8533
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	4	0.9167	1	0.7333	0.8333	0.8	1	0.8056	0.75	1	0.8686
qwen3.5-4b	Q4_K_M	q8	0	0.8333	1	0.9167	0.6667	1	1	0.6667	0.875	1	0.8708
qwen3.5-4b	Q4_K_M	q8	1	0.9167	1	0.8167	0.6667	0.9	1	0.8333	0.875	1	0.8967
qwen3.5-4b	Q4_K_M	q8	2	0.75	1	0.7333	0.8333	0.8	1	0.8021	0.875	1	0.8554
qwen3.5-4b	Q4_K_M	q8	3	0.8333	1	0.9167	0.6667	0.8	1	0.8333	0.875	1	0.8842
qwen3.5-4b	Q4_K_M	q8	4	0.8333	1	0.8167	0.6667	0.8	1	0.8333	0.75	1	0.8617
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	0	0.9167	1	0.9167	0.3333	0.9	1	0.8333	0.75	1	0.8775
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	1	0.9167	1	0.6333	0.1667	1	1	0.8333	0.875	1	0.8633
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	2	0.9167	1	0.7333	0.6667	0.8	1	0.8021	0.625	1	0.8471
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	3	0.75	1	0.9167	0.3333	0.9	1	0.8333	0.75	1	0.8525
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	4	0.8333	1	0.9167	0.6667	0.8	1	0.8333	0.75	1	0.8717
qwen3.5-4b	Q6_K	q8	0	0.8333	1	0.8167	0.6667	0.9	1	0.8333	0.875	1	0.8842
qwen3.5-4b	Q6_K	q8	1	0.9167	1	0.8167	0.6667	1	1	0.8333	0.75	1	0.8942
qwen3.5-4b	Q6_K	q8	2	0.8333	1	0.9167	0.5	0.9	1	0.8333	0.75	1	0.8733
qwen3.5-4b	Q6_K	q8	3	0.8333	1	0.8167	0.8333	0.8	1	0.8333	0.875	1	0.8825
qwen3.5-4b	Q6_K	q8	4	0.9167	1	0.8167	0.5	0.8	1	0.8333	0.875	1	0.8783
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	0	0.9167	1	0.9167	0.8333	0.9	1	0.8333	0.875	1	0.915
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	1	0.9167	1	0.9167	0.8333	1	1	0.8333	0.75	1	0.9125
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	2	0.9167	1	0.8333	0.3333	0.9	1	0.8333	0.875	1	0.8817
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	3	0.8333	1	0.8167	0.5	1	1	0.8021	0.875	1	0.8796
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	4	0.75	1	0.8167	0.8333	0.9	1	0.8333	0.75	1	0.8675
qwen3.5-4b	Q8_0	q8	0	0.8333	1	0.6333	0.3333	0.9	1	0.8333	0.875	1	0.8492
qwen3.5-4b	Q8_0	q8	1	0.75	1	0.9167	0.6667	1	1	0.8333	0.875	1	0.8917
qwen3.5-4b	Q8_0	q8	2	0.8333	1	0.8333	0.8333	1	1	0.8333	0.75	1	0.8917
qwen3.5-4b	Q8_0	q8	3	0.8333	1	0.8167	0.6667	1	1	0.8333	0.625	1	0.8692
qwen3.5-4b	Q8_0	q8	4	0.75	1	1	0.8333	0.9	1	0.7708	0.75	1	0.8733
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	0	0.8333	1	0.9	0.8333	0.9	1	0.8333	0.875	1	0.9008
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	1	0.8333	1	0.9167	0.6667	1	1	0.75	0.875	0.875	0.875
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	2	0.8333	1	0.6	0.8333	0.8	1	0.75	0.75	1	0.8317
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	3	0.8333	1	0.6	0.6667	0.9	1	0.8333	0.75	0.875	0.8375
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	4	0.8333	1	1	0.6667	1	1	0.75	0.875	1	0.8958
qwen3.5-9b	Q4_K_M	q8	0	0.8333	1	1	0.8333	0.7	1	0.8333	0.75	1	0.8783
qwen3.5-9b	Q4_K_M	q8	1	0.75	1	0.8167	0.6667	0.9	1	0.8333	0.75	1	0.8592
qwen3.5-9b	Q4_K_M	q8	2	0.75	1	0.9	0.6667	1	1	0.75	1	1	0.8858
qwen3.5-9b	Q4_K_M	q8	3	0.9167	1	1	0.6667	0.9	1	0.75	0.75	0.875	0.8733
qwen3.5-9b	Q4_K_M	q8	4	0.8333	1	0.8167	0.6667	1	1	0.8333	0.875	1	0.8942
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	0	0.8333	1	0.9167	0.6667	1	1	0.75	0.75	1	0.875
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	1	0.8333	1	0.9167	0.6667	0.9	1	0.8333	0.875	1	0.8942
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	2	0.75	1	0.9	0.6667	1	1	0.8333	0.875	1	0.89
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	3	0.8333	1	1	0.8333	0.9	1	0.75	0.875	1	0.8942
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	4	0.8333	1	0.9167	0.5	0.9	1	0.75	0.875	1	0.8692
qwen3.5-9b	Q6_K	q8	0	0.8333	1	1	0.8333	1	1	0.8021	0.75	1	0.9021
qwen3.5-9b	Q6_K	q8	1	0.8333	1	0.9	0.6667	0.8	1	0.8333	0.75	1	0.87
qwen3.5-9b	Q6_K	q8	2	0.8333	1	1	0.6667	0.9	1	0.8333	0.875	1	0.9025
qwen3.5-9b	Q6_K	q8	3	0.8333	1	0.9	0.6667	0.9	1	0.75	0.875	1	0.8758
qwen3.5-9b	Q6_K	q8	4	0.8333	1	0.8167	0.8333	0.9	1	0.8333	0.75	1	0.88
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	0	0.8333	1	1	0.6667	0.8	1	0.75	0.75	1	0.8633
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	1	0.8333	1	0.9167	0.8333	0.9	1	0.8333	0.875	1	0.9025
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	2	0.8333	1	0.8167	0.8333	1	1	0.8333	0.75	1	0.89
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	3	0.9167	1	1	0.6667	1	1	0.8333	0.75	0.875	0.9
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	4	0.8333	1	0.5167	0.8333	1	1	0.75	0.75	1	0.8433
qwen3.5-9b	Q8_0	q8	0	0.8333	1	0.9167	0.6667	1	1	0.8333	0.875	1	0.9042
qwen3.5-9b	Q8_0	q8	1	0.8333	1	0.9167	0.8333	1	1	0.75	0.875	1	0.8958
qwen3.5-9b	Q8_0	q8	2	0.8333	1	0.9167	0.8333	1	1	0.75	0.875	1	0.8958
qwen3.5-9b	Q8_0	q8	3	0.8333	1	0.9167	0.8333	0.8	1	0.75	0.75	1	0.8633
qwen3.5-9b	Q8_0	q8	4	0.8333	1	0.9167	0.8333	1	1	0.8333	0.875	1	0.9125

Pareto And Deployment Tables

The full Pareto and deployment-efficiency tables behind the recommendations.

Primary Pareto configurations

model	quantization	KV cache config	MTP draft tokens	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	0	20	0.8133	0.7	0.8791	176.6	3039.9
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	1	20	0.6079	0.35	0.8685	153.6	2777.4
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	2	20	0.6386	0.4	1.0949	137.1	2835.4
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	3	20	0.6912	0.55	1.2848	114.7	2797.7
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	4	20	0.645	0.35	1.3182	101.6	2655.9
qwen3.5-0.8b	Q4_K_M	q8	0	20	0.6929	0.4	0.691	216.0	4405.9
qwen3.5-0.8b	Q4_K_M	q8	1	20	0.7053	0.5	0.7139	192.7	4058.3
qwen3.5-0.8b	Q4_K_M	q8	2	20	0.6179	0.4	0.8494	159.5	4009.3
qwen3.5-0.8b	Q4_K_M	q8	3	20	0.76	0.45	1.0165	137.7	4034.8
qwen3.5-0.8b	Q4_K_M	q8	4	20	0.7258	0.5	1.1367	122.6	3965.4
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	0	20	0.6932	0.5	0.8004	167.0	3064.2
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	1	20	0.7133	0.45	0.9139	152.7	2786.9
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	2	20	0.7833	0.6	0.9773	132.2	2729.4
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	3	20	0.769	0.6	1.0631	118.4	2634.4
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	4	20	0.8386	0.7	1.354	100.6	2619.8
qwen3.5-0.8b	Q6_K	q8	0	20	0.7538	0.55	0.6655	205.5	4717.6
qwen3.5-0.8b	Q6_K	q8	1	20	0.7675	0.65	0.6999	186.3	4020.5
qwen3.5-0.8b	Q6_K	q8	2	20	0.7858	0.6	0.7937	165.7	3953.0
qwen3.5-0.8b	Q6_K	q8	3	20	0.7861	0.65	1.0061	142.5	3995.2
qwen3.5-0.8b	Q6_K	q8	4	20	0.7383	0.55	0.9899	116.3	3707.9
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	0	20	0.7658	0.45	0.8664	149.1	2980.4
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	1	20	0.8133	0.6	0.9454	144.7	2775.1
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	2	20	0.77	0.5	0.9901	122.8	2735.9
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	3	20	0.7992	0.55	1.2497	108.4	2700.9
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	4	20	0.6574	0.4	1.2137	94.38	2711.7
qwen3.5-0.8b	Q8_0	q8	0	20	0.7321	0.5	0.6794	179.6	4643.1
qwen3.5-0.8b	Q8_0	q8	1	20	0.6728	0.45	0.7123	164.9	4010.7
qwen3.5-0.8b	Q8_0	q8	2	20	0.6958	0.45	1.0009	141.4	3863.1
qwen3.5-0.8b	Q8_0	q8	3	20	0.7325	0.45	0.9711	128.0	3910.8
qwen3.5-0.8b	Q8_0	q8	4	20	0.7596	0.55	1.2298	112.5	3883.5
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	0	20	0.7499	0.55	1.0562	115.9	2599.1
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	1	20	0.7908	0.65	1.187	121.3	2393.0
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	2	20	0.7532	0.55	1.2387	116.5	2305.3
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	3	20	0.7425	0.55	1.3647	104.4	2362.9
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	4	20	0.7558	0.6	1.3315	93.27	2349.8
qwen3.5-2b	Q4_K_M	q8	0	20	0.7561	0.6	0.9747	135.0	3747.3
qwen3.5-2b	Q4_K_M	q8	1	20	0.7658	0.55	2.6926	140.8	3222.2
qwen3.5-2b	Q4_K_M	q8	2	20	0.7382	0.65	1.0632	134.8	3122.0
qwen3.5-2b	Q4_K_M	q8	3	20	0.7196	0.6	1.0611	119.3	3085.4
qwen3.5-2b	Q4_K_M	q8	4	20	0.7669	0.55	1.2917	102.9	3041.1
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	0	20	0.7828	0.65	1.3465	101.7	2545.4
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	1	20	0.8108	0.7	1.1638	113.1	2309.9
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	2	20	0.7817	0.7	1.1829	115.7	2296.9
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	3	20	0.8008	0.6	1.3928	103.1	2310.7
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	4	20	0.7783	0.65	1.5348	88.04	2262.7
qwen3.5-2b	Q6_K	q8	0	20	0.7367	0.55	1.0506	115.9	3497.1
qwen3.5-2b	Q6_K	q8	1	20	0.7517	0.5	1.0425	127.8	3038.7
qwen3.5-2b	Q6_K	q8	2	20	0.8067	0.7	1.0857	126.5	2984.6
qwen3.5-2b	Q6_K	q8	3	20	0.7196	0.55	1.1046	113.3	2938.0
qwen3.5-2b	Q6_K	q8	4	20	0.7804	0.65	1.1895	100.9	2943.3
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	0	20	0.7369	0.6	1.4771	87.90	2519.9
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	1	20	0.7869	0.6	1.2796	101.0	2286.5
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	2	20	0.82	0.65	1.3007	100.9	2311.8
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	3	20	0.7719	0.65	1.6397	91.34	2240.9
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	4	20	0.7867	0.6	1.6382	81.88	2213.0
qwen3.5-2b	Q8_0	q8	0	20	0.8233	0.7	1.3184	97.34	3450.1
qwen3.5-2b	Q8_0	q8	1	20	0.7696	0.65	1.1562	108.5	2965.1
qwen3.5-2b	Q8_0	q8	2	20	0.7692	0.55	1.4197	108.2	2958.5
qwen3.5-2b	Q8_0	q8	3	20	0.8075	0.7	1.2846	98.71	2971.7
qwen3.5-2b	Q8_0	q8	4	20	0.7992	0.65	1.3735	85.07	2891.2
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	0	20	0.9042	0.8	2.395	62.31	1306.3
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	1	20	0.8779	0.75	1.692	78.57	1220.1
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	2	20	0.8967	0.75	1.9135	86.40	1212.0
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	3	20	0.8533	0.7	1.5796	84.02	1201.6
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	4	20	0.8686	0.75	1.7142	78.24	1188.3
qwen3.5-4b	Q4_K_M	q8	0	20	0.8708	0.75	1.8838	70.77	1933.7
qwen3.5-4b	Q4_K_M	q8	1	20	0.8967	0.8	1.5285	88.39	1668.1
qwen3.5-4b	Q4_K_M	q8	2	20	0.8554	0.75	1.425	96.34	1617.1
qwen3.5-4b	Q4_K_M	q8	3	20	0.8842	0.75	1.4458	96.69	1653.7
qwen3.5-4b	Q4_K_M	q8	4	20	0.8617	0.75	1.47	89.84	1641.1
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	0	20	0.8775	0.8	2.4657	53.07	1214.9
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	1	20	0.8633	0.7	1.8654	70.72	1165.1
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	2	20	0.8471	0.7	1.6274	78.96	1118.6
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	3	20	0.8525	0.8	1.6489	79.50	1129.7
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	4	20	0.8717	0.75	1.8228	72.90	1125.7
qwen3.5-4b	Q6_K	q8	0	20	0.8842	0.8	2.1702	58.94	1757.9
qwen3.5-4b	Q6_K	q8	1	20	0.8942	0.8	1.8037	78.12	1549.6
qwen3.5-4b	Q6_K	q8	2	20	0.8733	0.8	1.5266	87.44	1514.0
qwen3.5-4b	Q6_K	q8	3	20	0.8825	0.8	1.6206	87.24	1524.5
qwen3.5-4b	Q6_K	q8	4	20	0.8783	0.75	1.5393	82.38	1504.3
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	0	20	0.915	0.85	2.7165	43.70	1183.4
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	1	20	0.9125	0.85	2.1977	60.55	1126.9
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	2	20	0.8817	0.75	2.0305	69.05	1115.2
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	3	20	0.8796	0.8	1.9629	70.25	1108.4
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	4	20	0.8675	0.85	2.1899	64.76	1088.9
qwen3.5-4b	Q8_0	q8	0	20	0.8492	0.7	2.468	47.71	1694.0
qwen3.5-4b	Q8_0	q8	1	20	0.8917	0.8	2.0461	66.07	1515.1
qwen3.5-4b	Q8_0	q8	2	20	0.8917	0.8	1.6557	76.11	1481.0
qwen3.5-4b	Q8_0	q8	3	20	0.8692	0.8	2.5133	76.81	1497.4
qwen3.5-4b	Q8_0	q8	4	20	0.8733	0.8	1.8206	73.90	1479.8
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	0	20	0.9008	0.85	3.8691	39.83	1024.0
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	1	20	0.875	0.7	2.3091	54.27	962.1
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	2	20	0.8317	0.7	2.3634	63.02	965.9
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	3	20	0.8375	0.7	2.0969	64.89	947.8
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	4	20	0.8958	0.75	5.528	65.13	939.6
qwen3.5-9b	Q4_K_M	q8	0	20	0.8783	0.8	5.724	43.09	1370.2
qwen3.5-9b	Q4_K_M	q8	1	20	0.8592	0.8	2.2697	57.84	1203.0
qwen3.5-9b	Q4_K_M	q8	2	20	0.8858	0.8	1.9622	68.57	1199.3
qwen3.5-9b	Q4_K_M	q8	3	20	0.8733	0.65	1.8904	70.55	1206.9
qwen3.5-9b	Q4_K_M	q8	4	20	0.8942	0.8	3.3488	71.28	1199.7
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	0	20	0.875	0.75	8.495	32.27	910.2
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	1	20	0.8942	0.75	6.3432	45.36	859.6
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	2	20	0.89	0.8	2.4604	54.67	875.6
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	3	20	0.8942	0.8	2.3105	57.72	868.8
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	4	20	0.8692	0.75	2.3809	59.07	860.5
qwen3.5-9b	Q6_K	q8	0	20	0.9021	0.85	3.5504	34.58	1198.1
qwen3.5-9b	Q6_K	q8	1	20	0.87	0.75	2.4257	48.94	1072.0
qwen3.5-9b	Q6_K	q8	2	20	0.9025	0.8	2.2967	59.51	1078.7
qwen3.5-9b	Q6_K	q8	3	20	0.8758	0.75	2.2035	61.38	1084.7
qwen3.5-9b	Q6_K	q8	4	20	0.88	0.85	5.747	63.90	1069.6
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	0	20	0.8633	0.7	9.2251	26.07	891.9
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	1	20	0.9025	0.85	3.1701	38.49	845.6
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	2	20	0.89	0.85	2.8886	46.13	834.3
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	3	20	0.9	0.75	2.948	48.31	818.6
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	4	20	0.8433	0.75	2.9817	49.33	815.8
qwen3.5-9b	Q8_0	q8	0	20	0.9042	0.8	6.1778	27.73	1155.0
qwen3.5-9b	Q8_0	q8	1	20	0.8958	0.8	5.1999	39.89	1038.3
qwen3.5-9b	Q8_0	q8	2	20	0.8958	0.8	5.3294	47.16	1042.8
qwen3.5-9b	Q8_0	q8	3	20	0.8633	0.75	2.6524	50.87	1021.7
qwen3.5-9b	Q8_0	q8	4	20	0.9125	0.85	2.7197	51.61	1008.8

Deployment efficiency scores

model	quantization	KV cache config	MTP draft tokens	runs	avg. semantic score	overall pass rate	avg. elapsed s	avg. generation tok s	avg. prompt tok s
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	1	20	0.9125	0.85	2.1977	60.55	1126.9
qwen3.5-9b	Q8_0	q8	4	20	0.9125	0.85	2.7197	51.61	1008.8
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	0	20	0.915	0.85	2.7165	43.70	1183.4
qwen3.5-4b	Q4_K_M	q8	1	20	0.8967	0.8	1.5285	88.39	1668.1
qwen3.5-4b	Q6_K	q8	1	20	0.8942	0.8	1.8037	78.12	1549.6
qwen3.5-4b	Q8_0	q8	2	20	0.8917	0.8	1.6557	76.11	1481.0
qwen3.5-4b	Q6_K	q8	3	20	0.8825	0.8	1.6206	87.24	1524.5
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	1	20	0.9025	0.85	3.1701	38.49	845.6
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	0	20	0.9042	0.8	2.395	62.31	1306.3
qwen3.5-9b	Q6_K	q8	2	20	0.9025	0.8	2.2967	59.51	1078.7
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	0	20	0.9008	0.85	3.8691	39.83	1024.0
qwen3.5-9b	Q6_K	q8	0	20	0.9021	0.85	3.5504	34.58	1198.1
qwen3.5-4b	Q6_K	q8	2	20	0.8733	0.8	1.5266	87.44	1514.0
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	2	20	0.89	0.85	2.8886	46.13	834.3
qwen3.5-4b	Q8_0	q8	1	20	0.8917	0.8	2.0461	66.07	1515.1
qwen3.5-4b	Q4_K_M	q8	3	20	0.8842	0.75	1.4458	96.69	1653.7
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	2	20	0.8967	0.75	1.9135	86.40	1212.0
qwen3.5-9b	Q4_K_M	q8	2	20	0.8858	0.8	1.9622	68.57	1199.3
qwen3.5-9b	Q4_K_M	q8	4	20	0.8942	0.8	3.3488	71.28	1199.7
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	3	20	0.8942	0.8	2.3105	57.72	868.8
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	4	20	0.8675	0.85	2.1899	64.76	1088.9
qwen3.5-9b	Q6_K	q8	4	20	0.88	0.85	5.747	63.90	1069.6
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	3	20	0.8796	0.8	1.9629	70.25	1108.4
qwen3.5-4b	Q8_0	q8	4	20	0.8733	0.8	1.8206	73.90	1479.8
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	2	20	0.89	0.8	2.4604	54.67	875.6
qwen3.5-4b	Q6_K	q8	0	20	0.8842	0.8	2.1702	58.94	1757.9
qwen3.5-4b	Q6_K	q8	4	20	0.8783	0.75	1.5393	82.38	1504.3
qwen3.5-9b	Q8_0	q8	2	20	0.8958	0.8	5.3294	47.16	1042.8
qwen3.5-4b	Q8_0	q8	3	20	0.8692	0.8	2.5133	76.81	1497.4
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	1	20	0.8779	0.75	1.692	78.57	1220.1
qwen3.5-9b	Q8_0	q8	0	20	0.9042	0.8	6.1778	27.73	1155.0
qwen3.5-9b	Q8_0	q8	1	20	0.8958	0.8	5.1999	39.89	1038.3
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	0	20	0.8775	0.8	2.4657	53.07	1214.9
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	3	20	0.8525	0.8	1.6489	79.50	1129.7
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	0	20	0.8133	0.7	0.8791	176.6	3039.9
qwen3.5-4b	Q4_K_M	q8	4	20	0.8617	0.75	1.47	89.84	1641.1
qwen3.5-4b	Q8_0	mixed_q8k_q4nlv	2	20	0.8817	0.75	2.0305	69.05	1115.2
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	3	20	0.9	0.75	2.948	48.31	818.6
qwen3.5-4b	Q4_K_M	q8	2	20	0.8554	0.75	1.425	96.34	1617.1
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	4	20	0.8958	0.75	5.528	65.13	939.6
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	4	20	0.8686	0.75	1.7142	78.24	1188.3
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	4	20	0.8717	0.75	1.8228	72.90	1125.7
qwen3.5-4b	Q4_K_M	q8	0	20	0.8708	0.75	1.8838	70.77	1933.7
qwen3.5-9b	Q4_K_M	q8	1	20	0.8592	0.8	2.2697	57.84	1203.0
qwen3.5-9b	Q6_K	q8	3	20	0.8758	0.75	2.2035	61.38	1084.7
qwen3.5-9b	Q4_K_M	q8	0	20	0.8783	0.8	5.724	43.09	1370.2
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	1	20	0.8942	0.75	6.3432	45.36	859.6
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	4	20	0.8692	0.75	2.3809	59.07	860.5
qwen3.5-9b	Q6_K	q8	1	20	0.87	0.75	2.4257	48.94	1072.0
qwen3.5-9b	Q8_0	q8	3	20	0.8633	0.75	2.6524	50.87	1021.7
qwen3.5-4b	Q4_K_M	mixed_q8k_q4nlv	3	20	0.8533	0.7	1.5796	84.02	1201.6
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	4	20	0.8386	0.7	1.354	100.6	2619.8
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	1	20	0.8633	0.7	1.8654	70.72	1165.1
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	1	20	0.875	0.7	2.3091	54.27	962.1
qwen3.5-2b	Q6_K	q8	2	20	0.8067	0.7	1.0857	126.5	2984.6
qwen3.5-4b	Q6_K	mixed_q8k_q4nlv	2	20	0.8471	0.7	1.6274	78.96	1118.6
qwen3.5-9b	Q6_K	mixed_q8k_q4nlv	0	20	0.875	0.75	8.495	32.27	910.2
qwen3.5-0.8b	Q6_K	q8	1	20	0.7675	0.65	0.6999	186.3	4020.5
qwen3.5-9b	Q4_K_M	q8	3	20	0.8733	0.65	1.8904	70.55	1206.9
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	1	20	0.8108	0.7	1.1638	113.1	2309.9
qwen3.5-2b	Q8_0	q8	0	20	0.8233	0.7	1.3184	97.34	3450.1
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	4	20	0.8433	0.75	2.9817	49.33	815.8
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	1	20	0.8133	0.6	0.9454	144.7	2775.1
qwen3.5-2b	Q8_0	q8	3	20	0.8075	0.7	1.2846	98.71	2971.7
qwen3.5-0.8b	Q6_K	q8	2	20	0.7858	0.6	0.7937	165.7	3953.0
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	3	20	0.8375	0.7	2.0969	64.89	947.8
qwen3.5-4b	Q8_0	q8	0	20	0.8492	0.7	2.468	47.71	1694.0
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	2	20	0.82	0.65	1.3007	100.9	2311.8
qwen3.5-0.8b	Q6_K	q8	3	20	0.7861	0.65	1.0061	142.5	3995.2
qwen3.5-9b	Q4_K_M	mixed_q8k_q4nlv	2	20	0.8317	0.7	2.3634	63.02	965.9
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	2	20	0.7817	0.7	1.1829	115.7	2296.9
qwen3.5-9b	Q8_0	mixed_q8k_q4nlv	0	20	0.8633	0.7	9.2251	26.07	891.9
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	1	20	0.7908	0.65	1.187	121.3	2393.0
qwen3.5-0.8b	Q6_K	q8	0	20	0.7538	0.55	0.6655	205.5	4717.6
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	2	20	0.7833	0.6	0.9773	132.2	2729.4
qwen3.5-2b	Q8_0	q8	4	20	0.7992	0.65	1.3735	85.07	2891.2
qwen3.5-2b	Q6_K	q8	4	20	0.7804	0.65	1.1895	100.9	2943.3
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	0	20	0.7828	0.65	1.3465	101.7	2545.4
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	3	20	0.8008	0.6	1.3928	103.1	2310.7
qwen3.5-2b	Q8_0	q8	1	20	0.7696	0.65	1.1562	108.5	2965.1
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	1	20	0.7869	0.6	1.2796	101.0	2286.5
qwen3.5-2b	Q4_K_M	q8	0	20	0.7561	0.6	0.9747	135.0	3747.3
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	3	20	0.769	0.6	1.0631	118.4	2634.4
qwen3.5-2b	Q6_K	mixed_q8k_q4nlv	4	20	0.7783	0.65	1.5348	88.04	2262.7
qwen3.5-2b	Q4_K_M	q8	2	20	0.7382	0.65	1.0632	134.8	3122.0
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	3	20	0.7992	0.55	1.2497	108.4	2700.9
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	3	20	0.7719	0.65	1.6397	91.34	2240.9
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	4	20	0.7867	0.6	1.6382	81.88	2213.0
qwen3.5-0.8b	Q8_0	q8	0	20	0.7321	0.5	0.6794	179.6	4643.1
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	2	20	0.77	0.5	0.9901	122.8	2735.9
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	4	20	0.7558	0.6	1.3315	93.27	2349.8
qwen3.5-2b	Q8_0	q8	2	20	0.7692	0.55	1.4197	108.2	2958.5
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	0	20	0.7658	0.45	0.8664	149.1	2980.4
qwen3.5-2b	Q4_K_M	q8	4	20	0.7669	0.55	1.2917	102.9	3041.1
qwen3.5-0.8b	Q8_0	q8	4	20	0.7596	0.55	1.2298	112.5	3883.5
qwen3.5-2b	Q4_K_M	q8	1	20	0.7658	0.55	2.6926	140.8	3222.2
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	0	20	0.7499	0.55	1.0562	115.9	2599.1
qwen3.5-0.8b	Q4_K_M	q8	1	20	0.7053	0.5	0.7139	192.7	4058.3
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	2	20	0.7532	0.55	1.2387	116.5	2305.3
qwen3.5-2b	Q4_K_M	q8	3	20	0.7196	0.6	1.0611	119.3	3085.4
qwen3.5-0.8b	Q6_K	q8	4	20	0.7383	0.55	0.9899	116.3	3707.9
qwen3.5-2b	Q6_K	q8	1	20	0.7517	0.5	1.0425	127.8	3038.7
qwen3.5-2b	Q6_K	q8	0	20	0.7367	0.55	1.0506	115.9	3497.1
qwen3.5-0.8b	Q4_K_M	q8	3	20	0.76	0.45	1.0165	137.7	4034.8
qwen3.5-2b	Q8_0	mixed_q8k_q4nlv	0	20	0.7369	0.6	1.4771	87.90	2519.9
qwen3.5-2b	Q4_K_M	mixed_q8k_q4nlv	3	20	0.7425	0.55	1.3647	104.4	2362.9
qwen3.5-2b	Q6_K	q8	3	20	0.7196	0.55	1.1046	113.3	2938.0
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	0	20	0.6932	0.5	0.8004	167.0	3064.2
qwen3.5-0.8b	Q4_K_M	q8	0	20	0.6929	0.4	0.691	216.0	4405.9
qwen3.5-0.8b	Q4_K_M	q8	4	20	0.7258	0.5	1.1367	122.6	3965.4
qwen3.5-0.8b	Q8_0	q8	3	20	0.7325	0.45	0.9711	128.0	3910.8
qwen3.5-0.8b	Q6_K	mixed_q8k_q4nlv	1	20	0.7133	0.45	0.9139	152.7	2786.9
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	3	20	0.6912	0.55	1.2848	114.7	2797.7
qwen3.5-0.8b	Q8_0	q8	1	20	0.6728	0.45	0.7123	164.9	4010.7
qwen3.5-0.8b	Q8_0	q8	2	20	0.6958	0.45	1.0009	141.4	3863.1
qwen3.5-0.8b	Q4_K_M	q8	2	20	0.6179	0.4	0.8494	159.5	4009.3
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	2	20	0.6386	0.4	1.0949	137.1	2835.4
qwen3.5-0.8b	Q8_0	mixed_q8k_q4nlv	4	20	0.6574	0.4	1.2137	94.38	2711.7
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	1	20	0.6079	0.35	0.8685	153.6	2777.4
qwen3.5-0.8b	Q4_K_M	mixed_q8k_q4nlv	4	20	0.645	0.35	1.3182	101.6	2655.9

Charts

All charts referenced by the PDF report.

Semantic score by MTP draft-token setting.

Generation speed by MTP draft-token setting.

Model by draft-token interaction for semantic score.

Model by draft-token interaction for speed.

Pareto view: semantic score versus speed.

Pareto view: semantic score versus latency.

Enterprise function suitability heatmap.

Limitations and next tests

This analysis uses deterministic benchmark scoring rubrics. Confidence intervals are bootstrap estimates over benchmark rows. The ANOVA tables were generated through statsmodels OLS Type-II ANOVA. The dataset was produced by merging the completed MTP 0/1 run with the later MTP 2/3/4 run.

The results are preliminary. The next useful tests are broader context-size sweeps, long-context behaviour, workload-specific prompt families, MTP behaviour under long context, KV-cache quantisation interactions, and prompt-router design for selecting a model based on task type and risk.