Artifact and allocation first
Buy local-LLM hardware for an exact model artifact, runtime, context, concurrency, placement plan, and acceptance target—not for a parameter count in isolation.
For a wider route from requirements to tested rigs, compare this guide with the AI Hardware hub.
Identify what will actually load
Record the model repository and revision, exact files and hashes, architecture, parameter and expert structure where relevant, model format, weight precision or quantization, tokenizer, chat template, adapters, and any vision, audio, or other projector. Then record the runtime build and backend that claim to support that combination.
A short quantization label is not a capacity guarantee. File contents, tensor types, metadata, auxiliary components, runtime overhead, context, KV cache representation, buffers, and device placement all affect the working set. Quantization can also alter model quality; test the exact artifact against the task.
Build a complete allocation model
Weights and runtime
Use the actual artifact size and observed load, then include runtime state, graphs or kernels, allocator reservations, and any duplicated or converted tensors.
Context and KV cache
Record configured context, prompt and generated tokens, parallel sequences, KV cache type, model architecture, and observed allocation. Larger context increases memory demand; advertised maximum context does not prove practical capacity.
Concurrent service
Record loaded models, simultaneous requests, parallel sequences, batch behavior, queueing, speculative or auxiliary models, embeddings or reranking, and per-user state.
Other memory
Add the operating system, integrated display use, server and client applications, retrieval index, input documents, temporary buffers, and safety or post-processing components.
Use the local AI RAM guide and local AI VRAM guide for the detailed accounting method; this page applies it to LLM procurement.
Distinguish placement architectures
| Path | What to verify | What not to assume |
|---|---|---|
| Dedicated GPU | Exact GPU and VRAM, driver, runtime backend, supported kernels, layer or tensor placement, context and KV cache placement, transfers, and display use. | System RAM does not become dedicated VRAM; a supported GPU family does not prove the exact artifact fits. |
| Unified memory | Installed capacity, memory available after the OS and applications, runtime behavior, CPU/GPU access, pressure limits, context, and concurrency. | The full advertised capacity is not available to one process, and unified capacity alone does not predict throughput. |
| CPU plus GPU offload | Which layers or tensors reside on each device, system RAM, VRAM, transfer path, fallback, observed latency and throughput, and sustained behavior. | Partial offload has the same behavior as full device residency or removes host-memory needs. |
| CPU only | Instruction support, memory channels and capacity, thread settings, model artifact, context, latency, throughput, power, and session duration. | A successful load meets the interactive or service target. |
| Multiple GPUs | Runtime split mode, device selection, tensor distribution, topology, power, thermals, per-device memory, synchronization, and measured scaling. | VRAM transparently pools or every runtime supports the topology. |
| NPU | Model format, supported architecture and operators, precision, execution provider, accessible memory, tooling, logs, and fallback. | An NPU badge or TOPS number establishes general LLM support. |
Make runtime behavior observable
llama.cpp documents several hardware backends, quantized GGUF files, and CPU-plus-GPU hybrid inference. Its quantization documentation warns that reduced precision may introduce accuracy loss and treats multimodal encoders or projectors as separate artifacts. Ollama documents that increasing context increases memory demand and provides a process view for checking context allocation and model offloading. Those are runtime-specific controls, not universal hardware rules.
MLX documents a unified-memory model in which CPU and GPU access the same arrays on Apple silicon. NVIDIA’s CUDA programming model distinguishes host and device memory on conventional CUDA systems. Select the article and acceptance method that matches the actual architecture; do not translate one memory label into another.
Local LLM acceptance packet
Load and identity
Verify artifact hashes, tokenizer and template, runtime and backend, device identities, loaded context, placement, and absence of unexpected fallback.
Task quality
Run a versioned prompt and evaluation set that represents the use case. Compare the exact quantized or converted artifact with the approved reference and record tolerances.
Memory and service
Measure every relevant pool at the configured context, prompt length, generation length, parallelism, loaded-model mix, and concurrent application state.
Sustained behavior
Run the required session length while recording time to first token, generation throughput where useful, queue delay, errors, thermals, power mode, memory growth, cancellation, and restart behavior.
Choose a chassis only after the packet passes
For unified-memory systems, use Can a Mac Run Local LLMs?. For portable systems, use Can an AI Laptop Run Local LLMs?. For compact stationary nodes, use Can an AI Mini PC Run Local LLMs?. Those pages test form-factor constraints; none replaces the exact artifact and allocation packet.