AI Chip and Accelerator Selection Guide

Workload-to-platform selection guide

Do not buy an “AI chip.” Freeze the workload artifact, prove the software path, size every memory pool, and validate the complete system under sustained conditions.

Official documentation reviewed July 18, 2026. This is a selection method, not a performance ranking, laboratory review, or claim that similarly named hardware and software combinations are interchangeable.

Start with the job, not the badge

Training and fine-tuning

Record framework, exact model and revision, precision, optimizer, batch and sequence shapes, distributed strategy, checkpoint path, required kernels, target time, dataset movement, and failure recovery. Capacity and interconnect requirements can dominate the accelerator label.

Batch inference

Record artifact format, precision, batch distribution, queueing, latency ceiling, throughput floor, concurrency, pre/post-processing, host work, transfer path, and utilization. A high-throughput configuration may be a poor interactive configuration.

Interactive inference

For language and multimodal models, freeze quantization, context policy, KV-cache behavior, prompt length, generation length, concurrency, time to first output, sustained output rate, and cancellation behavior. Measure the whole request path.

Vision, media and embedded pipelines

Freeze sensors, codecs, frame sizes and rates, colorspace, pre/post-processing, supported operators, control deadlines, power envelope, environmental limits, recovery behavior, and the boundary between perception and actuation.

Different compute blocks have different jobs

Compute block	Useful role	What must be proved
CPU	Control flow, tokenization, data loading, sparse or irregular operators, orchestration, fallback, small models, and latency-sensitive work that does not map efficiently elsewhere.	Instruction set, library kernels, thread and NUMA behavior, memory bandwidth, preprocessing share, and whether “accelerated” execution silently returns work to the CPU.
Discrete GPU	Parallel training, fine-tuning, high-throughput inference, broad research frameworks, and workloads that benefit from dedicated high-bandwidth memory.	Exact GPU, VRAM, driver, toolkit, framework/runtime build, precision, kernels, host link, power supply, chassis clearance, cooling, and sustained behavior.
Integrated GPU	Client inference, media and vision pipelines, and shared-memory workloads where low transfer overhead and compact form factor matter.	Reserved versus dynamically shared memory, bandwidth contention with the CPU, driver/runtime support, thermals, and actual device placement.
NPU or DSP	Efficient supported inference graphs, often inside a client or embedded SoC, including always-on vision, audio, sensor and selected generative workloads.	Compiler, model format, supported precisions and operators, shape limits, graph partitioning, fallback, firmware, OS build, and application access—not the NPU badge alone.
FPGA	Custom dataflow, deterministic I/O, sensor pipelines, low-latency transforms, and specialized inference when hardware design effort is justified.	Board image, toolchain, IP and runtime versions, compiled design, host interface, resource use, timing closure, license, maintenance skills, and production device path.
ASIC	A fixed accelerator optimized for a bounded model family, precision, power envelope or deployment volume.	Compiler/runtime, supported graph and operators, host integration, memory, firmware, model-conversion path, supply/lifecycle, and what happens to unsupported work.
SoC	A complete package combining CPUs with one or more GPUs, NPUs, DSPs, media blocks, memory controllers and I/O.	Which block runs each stage, how memory is shared, synchronization and copy costs, SDK/BSP support, power modes, thermal design, security updates and system availability.

Prove the entire software chain

The minimum chain is: model artifact → framework or application → runtime → execution provider or backend → driver and compiler → supported operators and precisions → device placement → complete system. ONNX Runtime, for example, delegates supported graph regions to ordered execution providers; an unsupported node may run on another provider. That makes a successful launch different from complete accelerator execution.

NVIDIA’s TensorRT support matrix ties inference engines to software versions, platforms, GPU capabilities and portability conditions. AMD publishes ROCm compatibility matrices for exact operating-system and hardware combinations. OpenVINO publishes device and feature-support boundaries across CPU, GPU and NPU targets. Apple’s MLX documentation explains its CPU/GPU use of Apple silicon unified memory. Each is evidence for a specific stack—not proof that an arbitrary application supports every device in the family.

Placement gate: log registered providers, graph partitions, fallback, compiled artifacts, driver and runtime versions, device utilization, and measured host/device time. “The application opened” is not placement evidence.

Treat memory as a budget and a path

Account for every resident object

Model weights are only the starting point. Add runtime and driver allocations, activations or workspaces, framework caches, input/output buffers, application state, and—for autoregressive models—the context-dependent KV cache. Concurrency can multiply per-request state.

Name the physical pool

System RAM serves the CPU and application. Dedicated VRAM is local to a discrete GPU. Unified or shared memory lets processors address a common pool but does not create infinite capacity or eliminate bandwidth contention. Capacity, bandwidth and access rules all matter.

Price transfers and offload

Moving tensors across PCIe or another fabric consumes time and bandwidth; partial GPU offload can help capacity while adding transfers and synchronization. Measure the selected layer split and request shape rather than assuming more offload is always faster.

Leave operating margin

A model that barely allocates may fail under longer context, another user, a display workload, driver growth or a larger workspace. Define a headroom policy and test the worst approved prompt, batch and concurrency. See the system-RAM guide and VRAM guide for the component-level worksheet.

Headline metrics are specifications, not rankings

TOPS or FLOPS usually describe a bounded operation type, precision and counting convention. Core counts describe an architecture, not utilization by your graph. Peak memory bandwidth does not include every access pattern, contention or transfer. Vendor benchmarks can document a vendor’s exact configuration, but they become comparable only when model, quality, precision, shapes, batch, software, power, cooling, host and measurement method are aligned.

Use these figures to reject impossible candidates and to form a test hypothesis. Then measure application-level latency, throughput, quality, energy, memory peak, fallback, startup, thermals and failure behavior on the exact system. The GPU vs NPU guide explains execution-path verification; the Apple silicon vs NVIDIA guide treats unified memory and dedicated VRAM as workload-specific system choices.

Select the system around the accelerator

Packet field	Record exactly	Acceptance evidence
Hardware identity	System vendor/model, processor and accelerator SKU, board revision, firmware, memory capacity/topology, storage, expansion link and power mode.	Delivered inventory and firmware export match the approved packet.
Software identity	OS image, kernel, driver, compiler/toolkit, framework, runtime/provider, model revision, conversion settings, precision and licenses.	A reproducible environment manifest and preserved deployable artifact.
Mechanical and electrical	Form factor, slot and link, host requirements, supply capacity, cable, cooling, acoustic limit, ambient range and enclosure airflow.	Worst approved workload remains within power, temperature, clock and noise limits without errors or unplanned throttling.
Service and lifecycle	Warranty, repair route, driver/security policy, API deprecations, hardware availability, change notices, replacement plan and data handling.	Named owner, review cadence, tested upgrade/rollback and an acceptable replacement path.
Application result	Dataset or prompts, quality gate, latency/throughput, concurrency, memory peak, energy method, logs, fallback and failure tests.	Independent rerun on the delivered system passes the written threshold.

A practical acceptance sequence

1. Compatibility proof

Build the exact environment, enumerate the device and provider, compile or load the exact artifact, and capture every unsupported operator, fallback and warning.

2. Capacity and correctness

Run the largest approved batch, context and concurrency; compare outputs against the written quality tolerance; record peak memory for every pool and preserve the test inputs.

3. Sustained system test

Measure warm and cold behavior, thermals, clocks, power, host load, transfers and errors for a duration representative of deployment. Test recovery from process, device and power interruption.

4. Operations and change control

Prove installation, update, rollback, observability, remote support, replacement and artifact retention. Assign an owner to revalidate after any model, runtime, driver, OS or hardware change.