Small Qwen3 models on an M4 Pro: a JSON reliability pilot

In this small local pilot, Qwen3-4B-Q4_K_M returned the requested exact JSON on 8 of 12 attempts. Qwen3-0.6B-Q8_0 and Qwen3-1.7B-Q8_0 each passed 3 of 12. These results cover four short synthetic tasks repeated three times on one Apple M4 Pro with 24 GiB unified memory. They are observations about these files and settings, not a general model ranking.

We recorded 54 benchmark task outputs, plus three unscored warmup responses. The task outputs comprise 36 exact-answer JSON responses and 18 summaries. Every request returned an output. The summaries are retained but their factual quality remains unadjudicated and contributes no accuracy score.

Exact JSON results

The tasks tested invoice extraction, corrected owner/date extraction, a deadline boundary, and conditional approval. A pass required JSON-only output with the specified keys, values, types and array order. Markdown fences failed the original requirement even when the enclosed JSON was otherwise correct. We did not change that rule after seeing the results.

Formatting-only means removing one surrounding Markdown code fence reveals JSON that exactly matches the answer key. This is a diagnostic explanation, not a revised score. Content/schema failures can also contain fences; categories are mutually exclusive.

For example, the 1.7B file retained the superseded owner and launch date in all three correction-task attempts. The 4B file approved an item priced at $101 despite the stated $100 ceiling in all three approval-task attempts. The raw outputs preserve those mistakes.

Observed timing and process memory

Timing covers all 18 completed requests per file, including the unscored summaries. Request time starts after the server is ready and excludes downloads, model loading and file hashing. Generation speed is the generation-duration-weighted mean of the runtime’s reported throughput, retaining its first-token timing convention. It excludes prompt processing and loading.

RSS was sampled from the server process with macOS ps roughly every 100 ms. It is not total unified-memory or GPU physical footprint, and sampling can miss brief peaks. We did not use model file size as a substitute for measured memory. No electricity or total ownership cost was measured.

How this run was controlled

We used llama.cpp 0.4.0-dev, build 10809, commit 5266f24da. One model and one server slot ran at a time, with four CPU inference and batch threads, requested all-layer GPU offload (-ngl 99), a 4,096-token context, prompt caching disabled, temperature 0.6, top-p 0.95, a 256-token output cap, and seeds 4101–4103. The retained logs do not independently confirm actual device placement. Other runtime defaults are retained in the metadata. Each model received one unscored warmup capped at eight output tokens. Pinned source files were SHA-256 verified before inference.

The complete cohort interval was 79.98 seconds, including startup, warmups, hashing and requests. Paid provider charges were $0. Hardware and electricity are excluded from that figure.

What this can and cannot establish

There are four distinct JSON tasks per model, not 12 independent task designs. Three repetitions describe this narrow sample; they do not establish broad capability or population-level reliability. The 4B file uses Q4_K_M while the smaller files use Q8_0, so this is not a controlled model-size comparison. Summary quality remains pending, and these timings should not be generalized to other machines, prompts, contexts or runtime versions.

Primary sources and evidence

• Qwen3-0.6B-Q8_0.gguf at the tested source revision
• Qwen3-1.7B-Q8_0.gguf at the tested source revision
• Qwen3-4B-Q4_K_M.gguf at the tested source revision
• Official llama.cpp build 10809 release

The source links identify the files and runtime; they do not substantiate Kingy’s measured scores. Our saved requests, outputs, answer key, checksums and timing/RSS records provide that evidence.