Can Inkling-Small Run on a 128GB Mac Studio? The 163GB Q4 Reality

No. The current Q4 does not fit as a fully resident model. UD-Q4_K_XL is 163,273,894,112 bytes: 163.273894GB, or 152.060663GiB. A 128GB Mac Studio is short before macOS, the inference graph, KV cache or multimodal projector gets a byte. It may start through memory mapping and SSD paging, but that is not the same as fitting; neither proves practical sustained use. On 128GB, Q3 is the boundary. The strongest Apple evidence so far is a 91.205GiB IQ3_XXS community run at 25–31 output tokens per second on experimental llama.cpp code, with a serious long-context corruption bug still open.

Our earlier report, “Inkling-Small Is 276 Billion Parameters—and Still the Efficient One”, assessed the native release, its sparse architecture and its official hardware guidance. Consumer-oriented GGUF quantizations and one unusually useful Apple Silicon test are what changed after publication. Those files expand the local options without making the headline Q4 fit in 128GB.

Four different meanings of “runs”

Hardware-feasibility claims become slippery when these states are treated as synonyms:

1. Fits fully in unified memory. The selected weights plus macOS and the working set stay resident without sustained SSD pressure.
2. Fits only with restricted context or headroom. The weights fit on paper, but normal desktop use, longer context or concurrent requests erase the margin.
3. Runs through SSD-backed memory mapping. The runtime maps the checkpoint and macOS faults pages in from storage. A process starting is not proof that all weights are resident.
4. Is practical for sustained interactive use. Load time, prompt processing, token generation, stability, thermal behaviour and repeated turns all remain acceptable for the intended workload.

The first three describe memory residency. The fourth is an observed usability result. A model can be fully resident and still be slow or unstable; it can also produce a few tokens through paging without being remotely practical.

Verdict by Mac memory tier

The table below uses a deliberately transparent engineering screen. Apple memory tiers are treated as their nominal GiB-scale capacities, while actual free memory is lower. R means the model leaves at least 20GiB before runtime allocations; T leaves 12GiB to under 20GiB and is context/headroom constrained; S leaves under 12GiB, exceeds physical memory or fails the model maker’s stated minimum, so SSD-backed mapping or an unsupported configuration should be expected. The 20GiB line approximates a 64K, one-sequence working set explained later. It is not a vendor guarantee.

Direct verdict: if the purchase decision is specifically “a 128GB Mac Studio for Inkling-Small Q4,” do not buy on the assumption that it fits. A 192GB-class machine is the minimum memory-plausible Q4 tier; 256GB is the more defensible tier for context, desktop headroom and experimentation. Neither is a speed guarantee.

Exact checkpoint and quantization sizes

Official checkpoints

The official repositories include an optional multi-token-prediction head, mtp.safetensors. It is a speculative-drafting component, not a vision or audio projector.

The complete repository trees, including configuration, tokenizer and metadata files, total 531.944669GB for BF16 and 170.764923GB for NVFP4. Hugging Face’s scanner reports 265,956,439,090 stored BF16 tensor elements, while Thinking Machines describes the architecture as 276B parameters. This article does not silently force those two counting conventions to agree.

Every GGUF variant currently published

These totals include every shard in each quantization folder. GB is decimal; GiB is binary. The current “163GB Q4” headline maps to UD-Q4_K_XL.

For GGUF multimodal use, add one combined projector: BF16 or F16 is 138,684,128 bytes (0.138684GB; 0.129160GiB), while F32 is 277,311,968 bytes (0.277312GB; 0.258267GiB). Q4_K_XL plus the F16 projector is 163,412,578,240 bytes: 163.412578GB, or 152.189823GiB.

Full memory-state matrix

This matrix applies the same R/T/S headroom rule to every current checkpoint. It is a screening tool, not a benchmark.

† GGUF BF16 crosses the article’s 20GiB weight-file screen by less than 1GiB, but that arithmetic conflicts with the official model card’s 600GB minimum memory for BF16. The official package is classified below minimum. Neither 512GB entry should be read as demonstrated residency or as a recommendation.
* A community tester demonstrated interactive IQ3_XXS decoding on 128GB Apple Silicon. That is not proof of sustained long-context stability or quantized task quality.

Why Q4 does not automatically mean 138GB

The tempting calculation is simple:

It is also the wrong storage model for this file.

Verified fact: “276B” is the architectural headline, not a promise that every stored tensor will consume exactly four payload bits. Hugging Face’s own scan reports 265.956B stored tensor elements for the BF16 repository. K-quants add block scales, minima and other metadata. Unsloth’s Dynamic 2.0 conversions assign different precisions to different tensors instead of flattening the entire network to one bit width. Sensitive layers and non-expert components can stay at higher precision; the S, M and XL recipes make different trade-offs.

Container and tokenizer metadata contribute a little. Mixed precision contributes much more. Q4_K_XL works out to roughly 4.73 storage bits per headline 276B parameter, or 4.91 bits per Hugging Face-scanned tensor element. The result is 163.274GB, not 138GB.

The official NVFP4 repository makes the same point from another direction. Its live tensor composition includes substantial BF16 and FP8 regions alongside packed U8 data. “Four-bit” names are family labels; the exact file bytes are the number that must be compared with a Mac’s memory.

Why “12B active” is not a 12B storage problem

Inkling-Small has 42 layers, 256 routed experts, six routed experts selected per token and two shared experts that are always active. Its 12B active parameters describe the compute path for one token. They do not describe the complete checkpoint.

Different prompts and tokens can select different experts, so the runtime must keep every expert addressable. Sparse routing reduces arithmetic. It does not turn 276B worth of possible routes into a 12B download.

Engineering inference: if the full expert set is not resident, memory mapping may turn routing into irregular storage traffic. Some pages can remain cached and some experts may be used more often, but there is no honest way to convert “12B active” into a universal 12B RAM claim. The exact working-set behaviour needs workload-level measurement.

128GB headroom and context calculations

The first subtraction is brutally simple:

Even IQ4_XS is not a healthy fit. Its 9.344GiB remainder is smaller than a realistic macOS desktop footprint before a model graph or KV cache exists.

For a reproducible planning scenario, reserve:

• 12GiB for macOS and an otherwise quiet desktop;
• 5GiB of runtime and safety margin;
• 0.861GiB for the 882MiB compute buffer observed in the current Apple test;
• 0.129GiB for the F16 vision/audio projector; and
• the context-dependent KV cache.

Those are engineering allowances, not fixed runtime constants. The observed graph buffer belongs to one IQ3_XXS build, and other builds can differ.

Inkling-Small’s configuration has eight KV heads, 128-dimensional heads, 42 layers, 35 local sliding-window layers with a 512-token window, and seven global-attention layers. With an F16 KV cache, a useful approximation beyond 512 tokens is:

That is roughly 28KiB per additional token plus a 70MiB fixed local-window component:

* macOS + runtime/safety + observed graph + F16 projector + KV.

At 64K, Q3_K_XL’s 16.656GiB raw margin becomes a 3.152GiB deficit in this scenario. IQ3_S retains about 8.112GiB; IQ3_XXS retains about 16.987GiB. A second independent sequence adds roughly another KV cache of the same order, plus runtime-specific buffers. Concurrency is not free.

The model advertises a one-million-token context window. The KV math says the cache can be smaller than many readers expect because most layers use a local window. It does not say processing a million tokens is fast, stable or useful on today’s Mac path.

Runtime support: llama.cpp, Unsloth, MLX, Ollama and LM Studio

Runtime support is moving faster than app release notes. This is the state at the evidence cutoff:

Verified fact: llama.cpp PR #25731 remains open. Unsloth’s own Inkling guide tells manual users to check out that PR. A generic ollama run snippet on a model page is not proof that Ollama’s current runner understands the architecture.

Verified fact: MLX support belongs to mlx-vlm. Its Inkling-Small and native NVFP4 loading landed in PR #1756. Text and image paths are supported. Audio currently fails through the bundled processor on main; PR #1767 is the pending repair. The separate mlx-lm project does not currently implement Inkling.

Requires testing: LM Studio hot-swaps inference engines, so its status can change without a major desktop-app release. Kingy found no current Inkling support announcement, and its open-source MLX engine pin predates Inkling support. Importing the files or seeing them in a library does not prove they load.

Multimodal support versus text-only support

The official BF16 and NVFP4 safetensors contain model.visual.* and model.audio.* tensors. Native MLX use therefore does not need a separate projector. The 4.464GB MTP file is optional speculative-drafting machinery, not a multimodal encoder.

The GGUF layout is different:

• Text only: the selected model shards are sufficient; tokenizer and chat-template metadata are embedded.
• Image or audio: add one compatible mmproj file and use an Inkling-capable multimodal runtime.
• One projector, two modalities: direct metadata inspection shows both the vision and audio encoders in the same file.
• Audio input: the official model card specifies 16kHz WAV, ideally below two minutes.

The current patched llama.cpp path reports image and audio support, and a community Apple test demonstrated image input. Small-specific audio, automatic projector discovery and Unsloth Studio’s projector-fetch behaviour still need independent reproduction.

Preserving the encoder files does not prove that a Q2 or Q3 language quant preserves multimodal task quality. The projector can work mechanically while the quantized language model becomes less reliable at grounding, OCR, spatial reasoning or audio interpretation.

Q2/Q3 quality risk

Q3 is where 128GB becomes physically credible. That does not make it the quality-equivalent of native NVFP4 or Q4.

Unsloth Dynamic 2.0 uses model-specific, mixed-precision decisions intended to protect sensitive tensors. That is more sophisticated than treating every tensor identically, but the live repository does not include Inkling-Small task-level evaluations for every Q4, Q3, IQ3, Q2, IQ2 and IQ1 variant.

The responsible hierarchy is:

• Verified: the files exist at the sizes reported above.
• Community result: IQ3_XXS generated coherent reasoning, completed 20/20 structured tool calls and passed 8/8 small executed coding prompts in one report.
• Not established: equivalence to native benchmark scores, long-form factual reliability, multimodal retention, difficult coding quality or low-frequency expert behaviour.
• High risk: Q2, IQ2 and IQ1 should be treated as access quants until they pass the buyer’s actual task suite.

Eight small programs are useful smoke tests, not a benchmark. This article does not transfer the native model’s published scores to any aggressive quant.

Expected performance: what has been reproduced

The strongest Apple evidence is a detailed llama.cpp PR test report, not a Kingy run. The tester used:

• a 128GB M5 Max Apple Silicon system on macOS 26.5.2;
• llama.cpp PR commit d69b7e606, build 10207, with Metal;
• the 97.931GB / 91.205GiB UD-IQ3_XXS checkpoint;
• all 42 layers on Metal, F16 KV, one parallel sequence and a 65,536-token context setting.

Reported results were approximately five seconds to load, 25–31 output tokens per second, 336 instantaneous prefill tokens per second at about 2K preceding context, and an 882MiB compute buffer. Vision input worked. That is genuinely interactive decode performance.

Long-context prompt processing deteriorated sharply in the same environment:

A roughly 42K-token request took about 2,375 seconds to reach generation, then triggered persistent garbage output until the server restarted. The trigger threshold was not bisected: the tester remained clean through 15,849 tokens and observed poisoning after roughly 42K, so the boundary lies somewhere between those observations. A Q8 KV-cache attempt also crashed on Metal; F16 KV worked.

Community result: IQ3_XXS on a 128GB Apple system is interactive at short to moderate context.
Requires testing: sustained sessions, the exact corruption threshold, Q3 quality, Q4 speed on 192GB/256GB Macs, Studio-versus-Max performance and multimodal audio.
No evidence-led estimate: Kingy will not project Q4 speed from memory bandwidth or “12B active.” No reproducible Q4 Mac measurement was found.

SSD-backed mmap is a launch mechanism, not free memory

llama.cpp normally memory-maps model files. That lets the operating system fault file-backed pages into memory instead of eagerly copying the whole checkpoint. It is useful for startup and can make an oversized model appear to run.

It does not create RAM. If the active pages plus the rest of the system exceed physical memory, macOS must reclaim, refault, compress or swap pages. MoE expert selection can make that access pattern less friendly than a sequential read.

For a clean residency test, compare the normal mmap run with a build using --mlock where supported, while recording memory pressure, page-ins, swap growth, prompt processing and decode over repeated turns. If the model only survives with file-backed eviction and storage traffic, classify it as SSD-backed, not as a full fit.

API versus local ownership economics

Inkling-Small’s API prices are low enough that local ownership is rarely justified by token savings alone.

At the snapshot:

• Thinking Machines’ Tinker serverless beta lists Inkling-Small NVFP4 at $0.30 per million input tokens, $0.06 cached input and $1.20 output. Tinker says the beta is not recommended for intensive production use.
• OpenRouter lists it at $0.50 per million input tokens and $1.20 output.

At a 3:1 input-to-output mix, that is about $0.525 per million combined tokens through Tinker or $0.675 through OpenRouter. A 100K-input, 10K-output session costs roughly $0.042 or $0.062, respectively.

Apple’s current 96GB M3 Ultra Mac Studio starts at $5,299 in the United States, and that base machine cannot hold Q4. Even using that artificially low hardware floor, purchase-price break-even is about 10.09 billion combined Tinker tokens or 7.85 billion OpenRouter tokens. A Q4-suitable higher-memory Mac costs more; storage, power, depreciation and setup time push the crossover further away.

Local ownership can still be rational for privacy, offline work, weight-level control, predictable availability, research access or avoiding per-request data transfer. Those are ownership benefits. For ordinary interactive or research use where the service terms and beta status are acceptable, the API is the cheaper default.

Reproducible test protocol

Anyone claiming “Inkling-Small runs on this Mac” should publish enough detail to identify which of the four states they measured.

1. Pin the evidence. Record Mac model and chip, unified memory, macOS version, free SSD space, repository commit, exact quant folder, shard byte total, runtime commit and build flags.
2. Start text-only. Use one sequence, a 4K context, F16 KV, full Metal offload and no desktop workload. Record cold-load time, peak memory, memory pressure, page-ins and swap.
3. Separate residency from mmap. Run the default mapped configuration, then attempt --mlock with the same settings. A launch that depends on continued file faults is state 3, not state 1.
4. Scale context deliberately. Repeat at 4K, 16K, 32K and 64K. Measure prompt processing, time to first token, decode rate and output integrity. Restart between known corruption tests.
5. Test sustained interaction. Run at least 30–60 minutes of repeated turns and report thermals, power, swap growth, failures and restart requirements. Then test concurrency separately.
6. Add multimodality explicitly. For GGUF, name mmproj-F16.gguf; run a fixed image and a 16kHz WAV. Report whether the encoder loads and whether the answer passes a task-specific check.
7. Measure quality against a reference. Use identical coding, tool-use, retrieval, factual, image and audio tasks on native NVFP4 where feasible and on each candidate quant. Publish pass/fail criteria and outputs; do not borrow native benchmark scores.

The community Apple command was broadly equivalent to:

llama-server \
  -m Inkling-Small-UD-IQ3_XXS-00001-of-00003.gguf \
  --ctx-size 65536 \
  --n-gpu-layers 999 \
  --flash-attn on \
  --parallel 1 \
  --batch-size 2048 \
  --ubatch-size 1024 \
  --jinja

Reproduction currently requires the Inkling PR branch, not a standard llama.cpp release. For image/audio, use the patched multimodal executable and pass the projector explicitly.

FAQ

Can Inkling-Small Q4 run on a 128GB Mac Studio?

It can potentially start through memory mapping and SSD paging, but the current Q4_K_XL is 152.061GiB before runtime allocations, so it cannot fit fully in 128GB unified memory. Even the smaller Q4_K_S is 141.875GiB. That is not a resident or practical Q4 configuration.

Which Inkling-Small quant is most credible on 128GB?

IQ3_XXS has the best current Apple evidence: 91.205GiB and 25–31 reported output tokens per second on a 128GB M5 Max system. Its task quality is not comprehensively measured, and the experimental runtime showed long-context corruption. IQ3_S leaves less headroom and lacks an equally detailed Mac result.

Does 12B active mean Inkling-Small needs about 12GB of RAM?

No. Twelve billion describes the parameters used in one token’s sparse compute path. The model still needs all experts available because later tokens can route differently. Storage follows the full checkpoint, not the active subset for one token.

Is 192GB enough for Q4?

It is memory-plausible. Q4_K_XL leaves about 39.94GiB before macOS and runtime allocations; the article’s 64K planning reserve leaves roughly 20.13GiB. No reproducible Q4 result on a 192GB Mac was found, so speed and stability remain open questions.

Does the one-million-token context fit?

The simplified F16 KV estimate is about 28.068GiB per sequence because seven layers use global attention and 35 use a 512-token sliding window. That is only one allocation. Processing a million-token prompt on the current Mac branch has not been shown practical, and much shorter contexts have already exposed severe prompt-speed and corruption problems.

Do image and audio still work locally?

With official safetensors, the image and audio towers are inside the checkpoint; current mlx-vlm supports text and image, while its audio path awaits a fix. With GGUF, one combined mmproj file is required. The patched llama.cpp path reports both image and audio support; upstream llama.cpp, Ollama and LM Studio support were not verified at the cutoff.

Do I need the 4.464GB MTP file?

Not for ordinary generation and not for multimodal input. It is an optional multi-token-prediction head for speculative drafting. The GGUF repository does not currently publish a separate MTP GGUF.

Are Q2 and IQ2 good enough?

Their storage sizes make 96GB and 128GB machines more accessible, but Inkling-Small-specific task evaluations are not published. Treat them as experimental access quants, especially for difficult coding, factual work and multimodal grounding.

Will Ollama or LM Studio load the GGUF?

Not reliably at this evidence cutoff. Inkling support is still outside upstream llama.cpp, Ollama has no verified architecture handler and LM Studio’s current engines were not shown to include the required changes. Recheck runtime versions before downloading more than 100GB of files.

The bottom line

The new GGUFs materially improve Inkling-Small’s local accessibility, but they do not repeal arithmetic. Q4 is a 192GB-class memory proposition and a 256GB-class comfort proposition; it is not a 128GB fit. A 128GB Mac can reach IQ3, and one IQ3_XXS result is impressively interactive, but quality coverage and long-context stability are not yet good enough to call it a settled daily-driver configuration.

Buy local hardware for ownership, privacy and control. Do not buy a 128GB Mac Studio because “12B active” or “Q4” made a 163GB checkpoint sound smaller than it is.

Dated source list

Checked August 1, 2026 (PDT) unless otherwise noted.

• Official / verified: Thinking Machines Inkling-Small model card and BF16 repository, commit 8cc5877b44d343f88b92086aa1fb72897950f06a.
• Official / verified: Thinking Machines Inkling-Small-NVFP4 repository, commit b6a99534467840620d411e4cd4ad5819b2610d9c.
• Official conversion repository / verified file sizes: Unsloth Inkling-Small GGUF repository, commit 1a19ef82883cb7b9c581b93c30ea252dabbf658d.
• Official runtime guidance: Unsloth Inkling guide and Unsloth Dynamic 2.0 GGUF methodology.
• Primary code evidence: llama.cpp Inkling support PR #25731, open at cutoff; llama.cpp mmap discussion.
• Community performance evidence: 128GB M5 Max IQ3_XXS report and prompt-processing follow-up. These are community results, not Kingy measurements.
• Primary MLX code evidence: mlx-vlm Inkling-Small/NVFP4 PR #1756 and pending audio PR #1767.
• Primary Ollama code evidence: current llama.cpp version pin and compatibility dispatch table.
• Primary LM Studio evidence: MLX engine dependency pin and runtime update model.
• Official pricing: Thinking Machines Tinker model pricing, OpenRouter Inkling-Small pricing and Apple 96GB M3 Ultra Mac Studio configuration.
• Kingy context: “Inkling-Small Is 276 Billion Parameters—and Still the Efficient One”.