How to Choose an AI Inference Provider

A plain-English guide to models, price, performance, privacy, and reliability

Publication date: August 18, 2026
Research cutoff and last verified: August 17, 2026
Geography: Global, with product- and region-level caveats
Currency: US dollars; taxes, marketplace adjustments, and currency conversion are excluded unless stated

Standfirst. The cheapest token is not necessarily the cheapest useful answer. Choose an inference stack by eliminating anything that fails your non-negotiable requirements, then measure the risk-adjusted cost of completed, acceptable work. That means testing the exact model version, provider, deployment mode, service tier, region, and application configuration you intend to run.

Disclosure and evidence labels. This is an independent desk-researched buyer’s guide. There are no affiliate links, sponsored rankings, or undisclosed commercial relationships. We did not run a universal cross-provider speed benchmark: doing so without identical workloads, endpoints, regions, quotas, and timestamps would manufacture precision. Current prices, catalogs, privacy controls, and service claims are taken from direct official sources and were checked on August 17, 2026. A contractual fact is enforceable only when it appears in your agreement. A published product fact is documented but may change. A vendor claim is the vendor’s characterization and is not treated as independent measurement. An independently measured observation comes from a disclosed third-party methodology. An author inference is analysis based on cited evidence and explicit assumptions.

The answer in one sentence. The best inference provider is the option that passes your non-negotiable constraints and delivers the lowest risk-adjusted cost per accepted job at the required quality, latency, capacity, privacy, and reliability—while preserving a credible exit path.

1. The 60-second answer

You are not choosing a logo. You are choosing a deployable stack:

model → model version → access route/provider → deployment mode → service tier → region → application configuration

Change any link and you may change quality, latency, price, privacy, feature behavior, or availability. A model alias can move. A partner may host a different revision or quantization. A global endpoint may process outside the region where data is stored. A router may send the same model to several upstream companies with different retention policies. An “OpenAI-compatible” endpoint may accept familiar JSON while differing on tools, errors, usage accounting, or streaming.

Use four stages:

1. Define the job. State what a successful business outcome is, the traffic shape, data sensitivity, required features, and latency target.
2. Apply hard gates. Eliminate any option that fails a non-negotiable requirement. Do this before scoring preferences.
3. Score the survivors. Compare outcome quality, cost per accepted job, tail latency, capacity, privacy, reliability, operations, and exit cost. Score evidence confidence separately.
4. Pilot three to five finalists. Freeze model IDs and settings, run representative traffic, publish failures, and canary the winner.

If you only remember one economic idea, remember this:

Cost per accepted job = all costs of attempts, retries, tools, capacity, operations, and human correction ÷ jobs that actually pass both the quality and latency bar.

The provider with the lowest input-token price can lose if it produces longer answers, needs more retries, misses schemas, triggers more human review, or fails at peak traffic.

2. Who this guide is—and is not—for

This guide is for developers, product and engineering leaders, finance and procurement teams, security reviewers, and enterprises buying generative-AI inference through an API or managed deployment. Language and multimodal models are the main focus; embeddings, reranking, speech, image, and video inference appear where they change the decision.

It is not a guide to choosing a consumer chatbot subscription. ChatGPT, Claude, Gemini, Grok, and similar end-user products bundle a user interface, product features, and subscription terms. An API is a programmable service with separate models, billing, limits, retention controls, and contracts. Giving a third-party app your API key also creates a second trust boundary: the app can see data, consume credits, and sometimes retain results even if the model provider does not.

It is also not legal, security, tax, or medical advice. Regulated and high-risk workloads require review of the exact contract, endpoint, feature set, subprocessors, region, and controls—not a certification logo or this guide.

3. What inference means in plain English

Training is how a model learns general patterns. Inference is what happens when you send a trained model an input and ask it to produce an output. For a language model, the input may include instructions, documents, images, tool definitions, and conversation history. The output may be text, JSON, tool calls, embeddings, audio, an image, or a video.

One business job can involve many inferences. A support answer may embed a query, retrieve documents, rerank passages, call a language model, check citations, retry malformed JSON, and fall back to another model. Billing one call tells you little about the cost of that whole outcome.

A token is a model-specific chunk of text or other content used for metering and context. The same prose can produce different token counts across tokenizers. Images, audio, video, and documents may be converted to model-specific units. Preserve each provider’s native billing units before normalizing them.

4. Model versus provider versus router versus host

A model developer creates or trains a model. A provider or host runs the compute that processes your request. They can be the same company, but not always. A router, gateway, or reseller accepts your request and forwards it to one or more upstream inference hosts. A deployment mode describes shared serverless capacity, batch, reserved throughput, a managed dedicated endpoint, self-hosting, or another operating arrangement.

The word “provider” can therefore mean seven different things:

1. A first-party model-maker API, such as a model developer serving its own proprietary models.
2. A hyperscaler model platform that offers several model families through cloud identity, networking, billing, and contracts.
3. A specialist inference cloud optimized for open-weight or selected partner models.
4. A marketplace or general serverless host carrying many community and commercial models.
5. A gateway, broker, or router that normalizes APIs and chooses upstream hosts.
6. A managed dedicated endpoint with customer-specific replicas or reserved capacity.
7. Self-hosted, private-cloud, on-premises, or edge infrastructure.

Keep these distinctions straight:

• Product versus API. A $20 assistant plan does not grant production API usage.
• Model developer versus processor. A partner-hosted model can place the cloud or specialist host in the processor chain.
• Router versus upstream. The router’s privacy policy is only one layer; the chosen upstream’s policy still matters.
• Family name versus deployable version. “Claude,” “Gemini,” “GPT,” “Grok,” “Llama,” or “Qwen” is not a complete SKU.
• Open-weight versus open source. Downloadable weights may still have license, field-of-use, attribution, redistribution, or scale restrictions. Call a model open source only when its license and complete source materials justify that stronger term.
• Compatible versus equivalent. vLLM documents an OpenAI-compatible server while also listing unsupported or ignored parameters; compatibility describes an interface surface, not identical behavior (published product fact, verified 2026-08-17).
• Quota versus capacity. A published rate limit is an account ceiling, not a reservation that the platform will always serve at low latency.
• No training versus no retention. A provider can promise not to train on customer data while retaining abuse logs or application state.
• Capacity labels. “Serverless,” “batch,” “flex,” “fast,” “priority,” “provisioned,” “reserved,” and “dedicated” have provider-specific definitions.

What this means for you

Write the exact SKU on every evaluation row: model ID or snapshot, provider, API, deployment, tier, region, context/output settings, quantization if applicable, and verification date. If your spreadsheet says only “GPT” or “Llama,” it cannot support a production decision.

5. Provider archetypes

Scroll or swipe to compare →

6. Stage 1: define the workload

Before discussing vendors, complete the one-page brief in Appendix B. The minimum questions are:

• What is one job or business outcome? A support answer, extracted invoice, resolved coding issue, translated call, or generated product image are different jobs.
• What makes it accepted: factual correctness, a valid schema, tool completion, citations, latency, safety, or human approval?
• What are p50, p95, and maximum input and output sizes? Averages hide expensive and slow tails.
• How many model, tool, search, retrieval, rerank, media, retry, and fallback calls occur per job?
• Is traffic interactive, streaming, asynchronous, batchable, or mixed? What share can use discount tiers?
• How much prompt content repeats, and what cache hit rate is plausible in production?
• Which modalities and API features are mandatory?
• What are p50, p95, and p99 latency requirements?
• What are monthly volume, peak RPM, input/output/total TPM, concurrency, and burst size?
• Where are users and data? What classification, residency, retention, training-use, and deletion rules apply?
• What is the monthly budget and likely growth curve?
• What support, invoicing, marketplace, DPA, BAA, SLA, and procurement terms are required?

Do not write “low latency” or “enterprise privacy.” Write measurable requirements: “95% of accepted jobs return first text within 800 ms and finish within 4 seconds in Canada” or “no prompt/output content stored at rest; processing and storage in the EU; signed DPA; exact stateful features excluded.”

7. Stage 2: apply hard gates

Hard gates are pass/fail. They come before weighted scoring because a cheap, fast service that cannot legally process the data is not a candidate.

Typical gates include:

• Exact required model or an approved family substitute.
• Account, country, payment, and contractual availability. OpenAI, for example, publishes a supported-country list and warns that access outside it may lead to suspension (published product fact, verified 2026-08-17). xAI documents that model access can vary by geography and account, and its billing FAQ says Indian payment cards were unsupported when checked (published product fact, verified 2026-08-17).
• Model and feature availability in the required processing region.
• Context and maximum output length for the exact endpoint.
• Structured output, tools, streaming, logprobs, embeddings, rerank, batch, caching, fine-tuning, files, speech, image, or video.
• Generally available production status.
• Sustainable peak capacity and acceptable cold-start behavior.
• Retention, training use, residency, private networking, licensing, compliance, and acceptable-use compatibility.
• Required SLA and support response.
• Version pinning and minimum deprecation/migration window.

Use “not publicly documented—confirm contractually” instead of treating silence as “no.” Appendix C provides the copyable checklist.

What this means for you

Expect the shortlist to shrink sharply. That is useful. Weighted scorecards are most dangerous when they allow a high score in convenience to compensate for a failed legal, regional, capacity, or lifecycle requirement.

8. Models and frontier access

“Does it have frontier models?” is not a yes/no question. Ask which exact model, on what date, through which API, in what status and region, with what quota and features.

At the research cutoff, OpenAI’s catalog identified gpt-5.6-sol as a frontier model with a 1.05-million-token context window and 128,000-token maximum output; the same catalog listed functions, web search, file search, and computer use (published product fact, verified 2026-08-17). Anthropic’s live pricing catalog listed Claude Sonnet 5, Opus 4.8, and other newer variants, with differences in price, tokenizer, batch, fast mode, and data-residency modifiers (published product fact, verified 2026-08-17). Google’s current Gemini docs marked gemini-3.6-flash GA, with text, image, video, audio, and PDF inputs, a 1,048,576-token input limit, and several tool capabilities—including computer use in preview (published product fact, verified 2026-08-17). xAI’s July 2026 pricing page listed grok-4.5, grok-4.3, and dated Grok 4.20 variants, plus separate batch, priority, tool, storage, and download charges (published product fact, verified 2026-08-17). These are dated findings, not permanent labels.

Check all of the following:

• Exact snapshot versus alias. Google’s model guide says a “latest” alias can be hot-swapped, while stable IDs usually do not change; previews may have short notice (published product fact, verified 2026-08-17). Pin when reproducibility matters.
• First-party versus partner-hosted. Anthropic explicitly distinguishes its API, Claude Platform on AWS, Amazon Bedrock, Google Cloud, and Microsoft Foundry by operator, processor, API surface, and release schedule (published product fact, verified 2026-08-17).
• Release timing and status. A preview can be billable and still unsuitable for a gated production workload.
• Region, account, quota, and approval. Catalog presence does not guarantee your account can deploy it where needed.
• Native feature parity. Tools, files, caching, reasoning controls, batch, fine-tuning, and multimodal behavior can differ across routes.
• Context and output limits. Confirm them on the endpoint, not in a launch post.
• Revision, tokenizer, quantization, prompt template, and safety layer. These can change behavior and cost even under a similar model name.
• Weights and license. A specialist can host weights without having created them. Download permission is not automatically permission for every commercial use.
• Deprecation. Google maintains dated deprecation and shutdown tables; at the cutoff it showed several stable and preview models with different replacement windows (published product fact, verified 2026-08-17).

Open-weight control trades model-maker dependence for deployment and license responsibility. Mistral’s model selector, for example, displayed exact IDs, prices, licenses, weight availability, parameter counts, and GPU requirements for selected models (published product fact, verified 2026-08-17). That is the correct level of specificity.

What this means for you

Treat “frontier access” as a dated matrix field, not a provider personality trait. If release-day access matters, make it a scored preference or contractual requirement and accept that native first-party APIs may lead while cloud controls or residency may favor a later partner route.

9. Quality: evaluate the work, not the leaderboard

Public leaderboards are discovery tools. They tell you which candidates deserve a test; they do not tell you which configuration will resolve your tickets, extract your documents, call your tools, obey your policies, or fit your latency budget.

Chatbot Arena uses pairwise human preference votes, which is valuable for broad conversational preference but not a substitute for your acceptance rubric (independent methodology, verified 2026-08-17). Stanford’s HELM emphasizes reproducible scenarios and metrics beyond accuracy, and its enterprise benchmark documentation explicitly notes that domain performance can differ from general benchmarks (independent methodology, verified 2026-08-17). LongBench and related work exist precisely because advertised context length is not the same as long-context understanding (independent methodology, verified 2026-08-17). Benchmark sets can also age or contain broken tasks; a 2026 audit estimated that about 30% of one coding benchmark’s tasks were broken, reinforcing the need to inspect examples and failures rather than repeat a headline score (published research observation, verified 2026-08-17).

Measure what acceptance requires:

• Correctness and task completion.
• Unsupported claims and hallucinations.
• Schema and structured-output validity.
• Tool selection, argument validity, execution, and multi-step completion.
• Citation support and source fidelity.
• Long-context retrieval and reasoning.
• Domain, language, tone, instruction following, refusal, and safety behavior.
• Human correction, escalation, or review time.
• Whole-workflow success, not isolated eloquence.

Run two separate experiments:

1. Same-model infrastructure comparison. Freeze the exact weights/version, prompt template, tokenizer, quantization, settings, tools, region, and retry policy as far as the services allow. This isolates serving, price, latency, errors, and capacity.
2. Complete-solution comparison. Compare different model-provider combinations on accepted business outcomes. This answers which stack should run the job.

Do not merge the results. A specialist may serve the same open-weight model faster, while another model-provider pair may still achieve a higher acceptance rate.

What this means for you

Quality is a denominator in the economic decision. A model that costs twice as much per call can be cheaper per accepted job if it completes in fewer turns, produces shorter valid outputs, and avoids human correction.

10. Stage 3: score preferences, with confidence

After hard gates, set weights before looking at provider results. Use the anchored 0–5 scorecard in Appendix D. Keep two numbers for every criterion:

• Score: how well the candidate meets the requirement.
• Evidence confidence: how trustworthy and current the evidence is.

A provider marketing claim may merit a strong provisional performance score but low confidence until you reproduce it. A contractual SLA may have high confidence but still score poorly if its remedy and measurement window do not fit the workload.

Normalize weighted scores to 100, but do not let the total hide weak evidence. Report both a point estimate and an evidence-adjusted view. One simple method is:

Evidence-adjusted contribution = weight × score/5 × confidence/5

This is not probability. It is a discipline that prevents an undocumented feature from receiving the same decisional weight as a reproduced result.

Run sensitivity analysis by varying:

• traffic volume and peak concurrency;
• input/output lengths and reasoning effort;
• cache hit rate;
• retry/fallback rates;
• task acceptance and latency-SLO pass rates;
• human review time;
• weights by ±20% or to plausible stakeholder alternatives.

If modest changes switch the leader, report a conditional tie: “Alpha for low-volume interactive traffic; Beta above 4 million jobs/month; Gamma when EU-only processing is a hard gate.” That is more honest and more useful than declaring one winner.

11. Pricing: read the whole meter

Price pages are specifications, not decoration. Capture every billed component in its native unit:

• Uncached input tokens: new prompt content processed at the standard input rate.
• Cache writes, reads, and storage: some providers bill writing a reusable prefix, reading it later, and retaining it for time. A high theoretical discount is worthless if real prompts do not hit the cache.
• Output tokens: usually more expensive because generation is compute-intensive. Long or meandering answers can erase a cheap input rate.
• Reasoning or thinking: providers differ. Some expose separate reasoning tokens but bill them at output rates; others state that output prices include thinking; others meter an execution mode. Verify the exact usage fields and price page.
• Long context: thresholds can raise input and output rates, reduce priority eligibility, or alter capacity behavior.
• Server-side tools: web search, grounding, code execution, file retrieval, computer use, and model-managed agents may add per-call, per-query, storage, or runtime charges.
• Media: images may be billed by size/quality, audio by minute or token, and video by generated second, resolution, or compute.
• Batch and flex: these trade completion time or availability for a discount. Google’s Gemini docs said both batch and flex could be 50% of standard for supported models at the cutoff (published product fact, verified 2026-08-17). Anthropic published a 50% Batch API discount on input and output tokens (published product fact, verified 2026-08-17). xAI’s discount varied by model and applied to listed token types, illustrating why the word “batch” alone is not enough (published product fact, verified 2026-08-17).
• Fast, priority, or premium: OpenAI’s Fast mode published premium per-token prices plus enterprise-only uptime and latency SLAs for supported models; xAI documented a 2× token-price multiplier for Priority Processing (published product facts, verified 2026-08-17, xAI pricing).
• Provisioned or reserved throughput: fixed capacity can be billed by token unit, PTU, model unit, tokens-per-minute bundle, replica, or hour. It creates idle cost when underused.
• Gateway and credit fees: OpenRouter documented pass-through inference pricing but a 5.5% credit-purchase fee with a minimum, plus BYOK rules; Hugging Face Inference Providers said it adds no markup to provider cost; Vercel documented zero markup and credits; Cloudflare said its core gateway features were free and inference passed through at provider rates at the cutoff (published product facts, verified 2026-08-17, Hugging Face, Vercel, Cloudflare). Payment processing, tax, and future premium features can still apply.
• Minimums, rounding, failures, and cancellations: establish whether a failed request, partial stream, tool call, or retry is charged and how small units round.
• Prepaid credits: Replicate’s policy said purchased credit was valid for one year and non-refundable; unused balances are a real economic cost (published product fact, verified 2026-08-17).
• Dedicated compute: bill initialization, active and idle replicas, scale-down delay, storage, network egress, logs, and observability. Replicate states that most private models bill setup, idle, and active time; Baseten bills per minute while replicas deploy, scale, or serve (published product facts, verified 2026-08-17, Baseten).
• Fine-tuning: include training, hosting, inference, adapter/checkpoint storage, evaluation, and future migration.
• People and operations: engineering, security review, migration, on-call, human correction, vendor management, and audit support often exceed model charges.

Taxes, VAT/GST, currency conversion, cloud marketplace credits, committed-spend offsets, support plans, negotiated discounts, and invoice timing belong in the procurement model, not in a footnote after selection.

Official-price conflict example

During this research, current OpenAI model pages returned inconsistent live rates for some GPT-5.6 variants across official catalog and comparison surfaces. The stable takeaway is not which cached rendering was right; it is that buyers must capture the exact official URL, model ID, timestamp, and invoice meter, then resolve material conflicts with the provider before commitment. The source ledger records this as an unresolved price conflict rather than manufacturing a clean number.

What this means for you

Build a rate card per deployable SKU, not per company. Add a “not publicly documented / confirm” value when necessary. Never silently treat unlike units—tokens, searches, images, seconds, GPUs, PTUs, or CCUs—as comparable.

12. Cost per accepted job

An accepted job is an outcome that is sufficiently correct, obeys formatting or tool rules, finishes inside the latency target, and avoids a disqualifying safety, compliance, or operational failure.

The formulas are:

Token cost per attempt
= uncached input × uncached input rate

• cache writes × cache-write rate
• cache reads × cache-read rate
• visible output × applicable output rate
• separately billed reasoning/thinking × applicable rate

Variable workflow cost
= sum of all model calls

• tool, search, retrieval, media, and routing charges
• expected retry and fallback cost

Effective cost per accepted job
= (variable workflow costs

• allocated provisioned/dedicated/idle capacity
• storage, networking, logging, and observability
• allocated engineering and operational expense
• human review and correction)
  ÷ accepted jobs

For a steady stream of attempts, a useful estimate is:

Accepted jobs = attempted jobs × quality acceptance rate × latency-SLO pass rate × compliance/operational pass rate

Do not assume those rates are independent when evidence shows correlation. A long output can simultaneously cost more, miss latency, and increase schema failure.

Worked example 1: small support classification

Illustrative rates, not a provider quote: $0.20/M uncached input, $0.02/M cached input, $1.20/M output. One call uses 600 uncached input tokens, 1,400 cached tokens, and 120 output tokens. Retry probability is 3%. Quality acceptance is 94%; latency pass is 98%; monthly volume is 1,000,000 attempted jobs.

• Base call: (600×0.20 + 1,400×0.02 + 120×1.20) / 1,000,000 = $0.000292.
• Expected API cost with retries: $0.000292 × 1.03 = $0.000301 per attempted job.
• Accepted jobs: 1,000,000 × 0.94 × 0.98 = 921,200.
• Monthly API cost: about $301.
• API cost per accepted job: about $0.000327.

If 5% of attempts require a 30-second review at $30/hour, human review adds $0.0125 per attempt—more than 40 times the API cost. Total cost per accepted job rises to roughly $0.0139. The economic lever is reducing review, not shaving a fraction from input price.

Sensitivity: 50% longer output raises API cost per accepted job by about 25%; losing all cache hits and billing the repeated 1,400 tokens at the uncached rate raises it about 86%; a 10% retry rate raises it about 7%; reducing quality acceptance from 94% to 85% raises cost per accepted job about 11% before human review.

Worked example 2: retrieval-augmented document answer

One job embeds a 2,500-token query/document slice at $0.15/M tokens, performs retrieval ($0.001) and reranking ($0.002), then calls a model with 1,500 uncached input, 6,000 cached input, and 500 output tokens. Model rates are $2.50/M uncached input, $0.25/M cached input, and $15/M output. Seven percent of workflows retry. Quality acceptance is 90%; latency pass is 95%; volume is 100,000 attempts/month.

• Embedding: $0.000375; retrieval and rerank: $0.003.
• Generation: $0.00375 + $0.00150 + $0.00750 = $0.01275.
• Base workflow: $0.016125; with 7% retry: $0.017254 per attempt.
• Accepted jobs: 100,000 × 0.90 × 0.95 = 85,500.
• Monthly variable cost: about $1,725.
• Cost per accepted job: about $0.0202.

Sensitivity: doubling output to 1,000 tokens raises accepted-job cost to about $0.0296. Cutting cache hit content from 6,000 to 3,000 tokens and shifting the other 3,000 to uncached adds about $0.0080 per accepted job. Improving citation/answer acceptance from 90% to 95% cuts cost per accepted job about 5.3% and may reduce review cost further.

Worked example 3: high-volume batch extraction

One asynchronous call uses 2,500 input and 300 output tokens. Standard rates are $0.30/M input and $1.00/M output; a verified batch tier discounts both 50%. Two percent retry. Quality acceptance is 97%; completion-window pass is 99%; volume is 10,000,000 attempts/month.

• Batch call: (2,500×0.15 + 300×0.50)/1,000,000 = $0.000525.
• With retries: $0.0005355 per attempted job.
• Accepted jobs: 9,603,000.
• Monthly variable cost: $5,355.
• Cost per accepted job: about $0.000558.

Sensitivity: using real-time standard pricing doubles the token component. A drop to 90% quality acceptance raises cost per accepted job about 7.8%. Output growing from 300 to 600 tokens raises the batch call 29%. Batch expiry must count as a latency failure, not disappear from the denominator.

Worked example 4: multi-call coding or research agent

An eight-call workflow averages, per call, 3,000 uncached input tokens, 9,000 cached tokens, 1,200 visible output tokens, and 800 separately billed reasoning tokens. Rates are $5/M uncached input, $0.50/M cached input, and $25/M output/reasoning. Twelve server tools cost $0.005 each. Retries and fallbacks add 20%. Quality acceptance is 72%; latency pass is 85%; volume is 50,000 attempts/month.

• Each model call: $0.015 + $0.0045 + $0.030 + $0.020 = $0.0695.
• Eight calls: $0.556; tools: $0.060; base workflow: $0.616.
• With retry/fallback overhead: $0.7392 per attempt.
• Accepted jobs: 50,000 × 0.72 × 0.85 = 30,600.
• Monthly variable cost: $36,960.
• Cost per accepted job: about $1.21.

If 15% of attempts need ten minutes of correction at $30/hour, correction adds $0.75 per attempt and lifts total cost per accepted job to about $2.43. If the 15% applies only to failed jobs and average correction is longer, the result can be much higher—model it from measured operations.

Sensitivity: 25% fewer calls cuts variable API/tool cost about 25% if tool use scales with calls. Halving cached-token hit rate adds roughly $0.053 per attempt. Raising acceptance from 72% to 82% cuts variable cost per accepted job about 12%. A faster but less accurate model can lose if it creates more loops.

Worked example 5: latency-sensitive voice interaction

Assume 90 seconds of combined billable audio per session at $0.05/minute, three billed text events at $0.004 each, and one tool call at $0.005. Five percent retry/reconnect overhead. Quality acceptance is 93%; latency pass is 97%; volume is 2,000,000 attempts/month. The audio and event rates mirror one documented commercial pattern but are illustrative for the calculation; confirm how a chosen service counts sent versus received audio (xAI Voice Agent published product fact, verified 2026-08-17).

• Base session: 1.5×$0.05 + 3×$0.004 + $0.005 = $0.092.
• With retry overhead: $0.0966 per attempt.
• Accepted sessions: 2,000,000 × 0.93 × 0.97 = 1,804,200.
• Monthly variable cost: $193,200.
• Cost per accepted session: about $0.1071.

Sensitivity: adding 30 seconds increases base cost by $0.025. Moving quality acceptance from 93% to 88% raises accepted-session cost about 5.7%. A reconnect bug that lifts retry overhead from 5% to 15% adds about 9.5% to cost. In voice, network jitter, turn detection, and tool latency can matter more than headline output tokens per second.

Serverless-versus-dedicated break-even

Do not convert a GPU hour into a token price without workload assumptions. Suppose:

• serverless costs $0.006 per attempted job at the measured call shape;
• a two-GPU dedicated deployment plus storage/network costs $9,000/month;
• allocated SRE, security, and platform work is $5,000/month;
• dedicated variable cost is $0.001 per job;
• benchmarked utilization is 65%, average context is 4,000 input/500 output tokens, effective concurrency is 48, and the deployment sustains 70 accepted jobs/second at the quality bar during load tests;
• autoscaling keeps two warm replicas, so idle time is included in fixed cost.

Break-even attempted volume is:

($9,000 + $5,000) ÷ ($0.006 - $0.001) = 2,800,000 jobs/month.

Below that, serverless is cheaper under these assumptions. Above it, dedicated may be cheaper if measured throughput and acceptance persist. Change context length, concurrency, quantization, failure rate, replica floor, utilization, or staffing and the answer moves. Capacity, model license, region, and disaster recovery can justify dedicated service even when nominal cost is higher.

13. Performance and capacity

Latency is not one number:

• Time to first token (TTFT): request arrival to first streamed output. It includes network, queue, prompt processing, and model prefill.
• Inter-token latency / output tokens per second: generation pace after the first token.
• End-to-end time: request arrival to a complete usable result, including tools, retries, validation, and post-processing.
• Throughput: work completed per time, measured as tokens, requests, or preferably accepted jobs.
• Concurrency: simultaneous in-flight work.
• Queue time: waiting before compute begins.
• Cold start: loading weights and initializing a scaled-to-zero or new replica.
• p50/p95/p99: the median and tail of the distribution. Interactive products are often defined by p95/p99, not the average.

Measure RPM, input/output/total TPM, burst duration, concurrent streams, batch completion/expiry, 429s, timeouts, and all non-success outcomes. Model input and output separately: long inputs stress prefill and memory; long outputs stress generation. OpenAI’s Fast mode, for example, publishes latency SLA details for supported enterprise customers using a stated percentile and measurement window; that is far more specific than “fast” (contractually scoped published term, verified 2026-08-17). Groq publishes model-specific speed, context, price, and developer-plan limits, but those speeds remain vendor claims until reproduced with your workload (vendor claim, verified 2026-08-17). Never compare its published number directly with another vendor’s differently measured claim.

Shared serverless capacity is easy to start and efficient for bursts, but achievable throughput can vary. Dedicated endpoints expose sizing and idle-cost decisions. Hugging Face documents scale-to-zero cold starts and proxy behavior; Baseten documents minute billing during deploy/scale/serve and configurable replica floors; Fireworks distinguishes shared serverless from GPU-billed on-demand deployments (published product facts, verified 2026-08-17, Baseten, Fireworks).

Quality-adjusted goodput is the most useful combined measure:

accepted jobs completed within the latency target ÷ cost or unit time

It penalizes fast wrong answers and cheap timeouts.

What this means for you

Load-test the exact production route with realistic prompt-size distributions and concurrency. Run long enough to reach steady state and include ramps, bursts, cache behavior, and cold starts. A free tier tests syntax; it does not predict production capacity.

14. Availability and reliability

“Available” has five meanings:

1. Account access: can you create, fund, and contract for an account?
2. Catalog availability: does the exact model and feature exist on the route?
3. Regional availability: can it process data where required?
4. Capacity availability: can you obtain sustained throughput at acceptable tails?
5. Lifecycle continuity: will the endpoint remain long enough for your product?

A quota is not a capacity reservation. Azure’s provisioned-throughput documentation explicitly says dedicated capacity is held whether requests are made or not, while standard is shared and batch is asynchronous; PTUs are region-specific and throughput varies by model and call shape (published product fact, verified 2026-08-17). AWS similarly distinguishes on-demand tiers, reserved capacity, batch, and cross-region inference, with model-specific details and routing implications (published product fact, verified 2026-08-17).

Evaluate:

• observed uptime and incident distributions versus the contractual SLA;
• SLA scope, exclusions, measurement windows, required tier, and service-credit remedy;
• 429s, timeouts, 5xx, capacity errors, malformed outputs, and stream disconnects;
• regional and provider failover;
• whether fallback changes the model, behavior, latency, price, or data handling;
• correlated failure domains: same model developer, cloud, region, control plane, identity system, network, or upstream host;
• public status and incident-history transparency;
• support escalation and response commitments;
• version rollback and deprecation notice.

A router is not automatic independent resilience. OpenRouter lets buyers constrain provider order, fallbacks, quantization, data collection, ZDR, price, and performance preferences; that control is valuable, but the buyer must still choose routes that do not share the same upstream dependency (published product fact, verified 2026-08-17). Vercel similarly exposes provider ordering and fallback behavior. Neither fact proves your chosen routes are independent (published product fact, verified 2026-08-17).

What this means for you

Design two fallbacks: a model fallback for capability continuity and a provider/region fallback for infrastructure continuity. Decide in advance which quality, privacy, or feature degradation is acceptable. Test failover under load; do not count a route you have never exercised.

15. Privacy, security, compliance, and legal fit

Start by drawing the data flow: client → your application → gateway/router → inference host → model/tool/search/file service → logs and storage. Label the controller, processor, subprocessors, storage locations, processing locations, and retention at every hop.

Ask separately:

• Are prompts/outputs used for training or product improvement by default, by opt-in, or under contract?
• What is default abuse-monitoring retention?
• What application state is stored by Responses, Assistants, files, batch, caching, fine-tuning, or managed agents?
• Is zero data retention available, to whom, and for which exact features?
• Where is data stored at rest? Where does inference processing occur? Where do metadata and the control plane run?
• What does a gateway add to the chain, and which upstream policy applies?
• Which subprocessors, DPAs, BAAs, transfer mechanisms, encryption, customer-managed keys, private networking, tenant isolation, RBAC, SSO, SCIM, project boundaries, and audit logs apply?
• What is the certification scope? Does it cover the endpoint, region, and feature?
• Who owns input and output? What indemnity exists? Which model license and acceptable-use restrictions bind the workload?
• What deletion, audit, and incident-notification terms are required?

“No training” does not mean “no retention.” OpenAI says API data is not used to train by default, while separately documenting abuse logs, application-state retention, ZDR eligibility, and feature exceptions; for example, the Responses API can store application state by default and some caching/background features are not ZDR-compatible (published product fact, verified 2026-08-17). Anthropic’s current API retention matrix says ZDR is enabled by arrangement, applies per organization, and excludes or qualifies several stateful features; it also documents model-specific 30-day retention for designated covered models and retention for flagged content (published product fact, verified 2026-08-17).

Storage location is not necessarily processing location. Anthropic exposes separate inference-geo and workspace-geo controls (published product fact, verified 2026-08-17). Microsoft says regional, Data Zone, and Global deployment types route differently; its data page also distinguishes stateless inference from stored state and abuse review (published product fact, verified 2026-08-17). Google Cloud says customer data is not used to train managed models without permission, but documents retention exceptions for abuse monitoring, Search/Maps grounding, live-session resumption, and in-memory caching, along with steps required for zero-retention goals (published product fact, verified 2026-08-17).

Self-hosting is not automatically private. It removes a third-party inference path only if weights, telemetry, container images, updates, logs, support, monitoring, and tool calls are controlled. A default request dump, cloud bucket, or crash trace can recreate the same privacy problem inside your estate.

What this means for you

Turn privacy language into a feature-level matrix signed off by legal and security. Obtain the DPA/BAA and exact terms. Treat certifications as evidence about a scoped control environment, not proof that your configuration complies.

16. Developer and business operations

A production provider is also an operational system. Evaluate:

• API/SDK maturity, streaming, async, structured output, tool behavior, idempotency, request IDs, error schemas, and documented retry rules;
• usage export, billing latency, budgets, caps, alerts, and allocation by project, key, user, model, and environment;
• logs, traces, token counts, TTFT, output speed, total latency, queue/cold-start metrics, and debugging controls;
• key and environment management, RBAC, SSO/SCIM, audit logs, private networking, and customer-managed keys;
• changelogs, pinning, deprecation notices, rollback, fine-tune export, and data portability;
• self-serve versus sales-led access, credit card versus invoice, cloud marketplace, minimum commitments, support, and response guarantees;
• vendor viability, concentration risk, migration effort, and exit assistance.

OpenAI-compatible interfaces reduce client rewrite, but real differences remain. vLLM’s own compatibility page notes unsupported/ignored parameters. Hugging Face’s Inference Providers adds a unified route to many named providers without markup, while the upstream still defines model behavior and processing (published product fact, verified 2026-08-17). Cloudflare provides gateway analytics, caching, and rate limiting, but persistent logs have plan-specific limits and must be configured to match retention policy (published product fact, verified 2026-08-17).

Dedicated and self-hosted stacks require deeper operations. vLLM supports tensor, pipeline, and data parallelism, but the buyer owns environment consistency, high-speed networking, capacity, upgrades, monitoring, and recovery (published product fact, verified 2026-08-17). SGLang exposes Prometheus metrics and does not log request contents by default, yet its optional request/crash dumps illustrate why runtime configuration belongs in the privacy review (published product fact, verified 2026-08-17).

What this means for you

Count the hours between an incident and an explainable answer. A nominally cheap endpoint with poor request IDs, delayed billing, or missing tail metrics can create expensive engineering and finance work.

17. Current provider landscape

Inclusion and exclusion criteria

The dated market map includes representative services that, as of August 17, 2026, had an official public product or documentation page for generative inference and were material to at least one archetype: proprietary frontier access, broad cloud catalog, specialist/open-weight serving, gateway routing, managed dedicated endpoints, or self-hosting. Inclusion is not endorsement. We excluded consumer-only assistants, unverified or inactive products, providers with no discoverable official inference documentation, and minor resellers without a distinct routing, deployment, regional, or contractual proposition. The market is larger than this map.

Representative market map, verified August 17, 2026

First-party model APIs (alphabetical): Anthropic, Cohere, Google Gemini Developer API, Mistral AI, OpenAI, xAI. Strength: native model access and features. Limitation: each is a separate commercial and operational dependency, with route-specific geography and policy.

Hyperscaler platforms (alphabetical): Amazon Bedrock, Google Vertex AI, Microsoft Foundry/Azure Direct Models. Strength: cloud identity, network, contract, marketplace, region, and provisioned-capacity integration. Limitation: exact model and feature parity varies by route, region, and release.

Specialist inference platforms (alphabetical): Baseten, Cerebras, DeepInfra, Fireworks AI, Groq, Modal, Nebius, Replicate, RunPod, Together AI. Strength: serving economics, hardware choice, rapid open-weight catalogs, or developer-friendly deployment. Limitation: catalogs, quantization, enterprise controls, and notice/SLA terms vary substantially.

Gateways and routers (alphabetical): Cloudflare AI Gateway, Hugging Face Inference Providers, OpenRouter, Portkey, Vercel AI Gateway. Strength: one integration, policy routing, experiments, budgets, and fallbacks. Limitation: an additional processor/failure domain and possible native-feature gaps; upstream dependencies remain.

Managed dedicated and self-hosted options (alphabetical by product/company): Amazon Bedrock Provisioned/Reserved options, Azure Provisioned Throughput, Baseten deployments, Cohere Model Vault/private deployment, Fireworks on-demand deployments, Hugging Face Inference Endpoints, Replicate deployments, self-managed cloud GPUs, and vLLM/SGLang on rented, private-cloud, on-premises, or edge compute. Strength: version/capacity/isolation control. Limitation: commitments, idle cost, sizing, and operations.

Representative provider matrix

Scroll or swipe to compare →

No row is a winner. Each is a starting hypothesis for a workload-specific pilot.

18. Decision paths by buyer scenario

These are conditional archetype recommendations, not universal provider rankings.

A nontechnical person who wants an AI assistant

Buy an end-user assistant subscription, not raw API inference. Compare the product’s features, privacy settings, file handling, account security, and regional availability. An API shifts security, billing, prompting, reliability, and interface work onto you.

A nontechnical person supplying an API key to a third-party app

Prefer an app with transparent data handling, scoped keys, hard spend limits, and a provider you understand. Create a separate project/key, cap it, never reuse a production credential, and assume the app can read submitted data and consume the balance. A gateway can centralize budgets, but adds another processor.

A solo builder creating an MVP

Start with first-party serverless if one model family is central, or a reputable gateway/specialist if rapid multi-model comparison matters. Optimize for documentation, spend caps, schema/tool support, and low operational burden. Move only after measured cost, capacity, privacy, or portability justifies migration.

A startup with intermittent traffic

Use shared serverless with per-token pricing, optionally batch/flex for tolerant work. Avoid warm dedicated replicas until cold-start risk or a measured break-even warrants them. Together, Fireworks, Groq, Replicate, or hyperscaler on-demand routes may be candidates for open-weight or multimodal workloads; availability, catalog, and enterprise terms decide the shortlist.

A latency-sensitive consumer application

Shortlist a first-party premium/fast tier, a latency-optimized specialist, and provisioned capacity. Measure TTFT and p95/p99 end-to-end time from user regions. Groq’s and Cerebras’s speed positioning makes them discovery candidates for supported open-weight models; OpenAI Fast, xAI Priority, Fireworks Fast, Azure priority/provisioned, and AWS priority/reserved are candidates only where the exact model/tier is supported. Published speed is not the pilot result.

High-volume asynchronous classification or extraction

Start with batch/flex on the smallest model that meets the acceptance rubric. Compare a specialist open-weight batch route, a first-party batch API, and dedicated capacity above the measured break-even. Output control, schema validity, retries, and human review usually matter more than broad conversational ranking.

A multi-call agentic workflow

Favor native tool semantics, reproducible usage metadata, cache effectiveness, state/retention clarity, and loop controls. Compare whole-workflow acceptance and cost; per-call price is especially misleading. A router helps experiments, but pin routes during evaluation so provider changes do not contaminate results.

A team requiring a proprietary frontier model

Begin with the first-party API for native, timely access, then test a hyperscaler route if cloud contract, region, private networking, or provisioned capacity outweighs release lag or feature differences. Record exact model snapshot and API parity.

A team requiring open weights or custom fine-tunes

Start with a specialist managed endpoint or hyperscaler managed compute. Shortlist self-hosting only if license, utilization, data placement, custom optimization, and staffing justify it. Together, Fireworks, Baseten, Hugging Face Endpoints, Replicate, and cloud GPU stacks offer different balances; vLLM/SGLang provide serving control but not managed SRE.

A regulated enterprise handling sensitive data

Apply legal/privacy gates before model quality. Hyperscaler regional or data-zone deployment, a first-party enterprise arrangement with validated ZDR/residency, a private managed endpoint, or on-premises may fit. Obtain DPA/BAA, subprocessor list, feature-level retention matrix, private networking, audit logs, encryption/key details, and incident terms. Disable or exclude stateful tools that break the policy.

A steady high-utilization workload considering dedicated capacity

Benchmark serverless and a dedicated deployment at the same exact model revision and workload. Include warm replica floor, idle capacity, peak headroom, DR, SRE, storage, network, and commitments. Dedicated wins only when the operational and capacity benefits exceed the all-in fixed cost.

An organization requiring multi-region or multi-provider resilience

Use explicit routes across independent clouds/regions/control planes where policy permits. Separate provider and model fallback. Test degraded behavior and data handling. A router can implement policy, but independence comes from route design, not the router label.

19. Stage 4: benchmark and pilot

Shortlist three to five candidates that passed every gate. Then:

1. Build a representative, versioned evaluation set from real workload distributions. Remove or protect personal data as policy requires.
2. Define acceptance rubrics and adjudication before running models.
3. Include normal, difficult, long-context, malformed, safety-sensitive, multilingual, and edge cases.
4. Freeze exact model IDs, prompts, tool definitions, parameters, regions, clients, retry/fallback rules, and concurrency.
5. Record permitted outputs plus request ID, timestamps, token/media/tool usage, route, tier, cache fields, errors, and cost.
6. Use blinded human review for subjective or high-impact outcomes; measure agreement and adjudicate disputes.
7. Run multiple samples and realistic concurrency at different times. Disclose sample size and uncertainty.
8. Capture TTFT, output speed, completion time, p50/p95/p99, queue/cold start, errors, retries, timeouts, cost, and acceptance.
9. Keep same-model infrastructure results separate from complete-solution results.
10. Publish failure categories and examples, not just averages and winners.
11. Calculate cost per accepted job and quality-adjusted goodput.
12. Shadow or canary the finalists before migration or commitment.

For proportions such as acceptance rate, publish a confidence interval. For heavy-tailed latency and cost, bootstrap the median and tail or report repeated-run ranges. Ten prompts are a demo, not capacity evidence. The two-to-four-week plan is in Appendix G.

20. Weighted scorecard and sensitivity analysis

Use the 0–5 anchors in Appendix D. A reasonable starting weight set—not a universal truth—is:

Scroll or swipe to compare →

Calculate Σ(weight × score/5). Then show evidence confidence beside it. Re-run with finance, engineering, security, and product weight sets; vary volumes, output lengths, cache, retry, acceptance, and latency. A provider that wins only under one fragile assumption is not the winner—it is one branch of a decision.

21. Resilience and exit planning

An exit path is credible only if you have exercised it. At selection time:

• Put model/provider routing behind your own thin application interface; do not pretend this erases behavioral differences.
• Keep prompts, tool schemas, evals, and acceptance rubrics versioned outside a vendor console.
• Export usage, traces, fine-tune data, adapters/checkpoints where allowed, and configuration regularly.
• Store exact model IDs, tokenizer/revision/quantization, region, and tier with test results.
• Avoid state that exists only inside an API unless you have an export/rebuild plan.
• Maintain a warm or periodically tested secondary route for critical workloads.
• Pre-negotiate data return/deletion, assistance, deprecation notice, and transition terms where material.
• Estimate migration engineering, quality revalidation, regulatory review, and dual-run cost.

More providers do not always increase resilience. Two routes can share a model developer, region, cloud, identity system, or router. They can also multiply monitoring, security review, and semantic variance. Add a provider only when it removes a measured concentration risk or improves an economic/operational outcome more than the complexity it creates.

What this means for you

Price the exit before signing the commitment. A slightly more expensive stack with pinned versions, portable data, reproducible evals, and a tested alternate route may have lower risk-adjusted cost.

22. Common myths and expensive mistakes

Myth: Lowest token price means lowest cost. It ignores output length, tools, retries, failures, capacity, operations, and human correction. Use cost per accepted job.

Myth: The most intelligent model is always best. A smaller model can be faster, cheaper, more consistent, and sufficient for a narrow rubric.

Myth: The largest context window gives the best long-document result. Capacity to accept tokens is not evidence of retrieval, reasoning, citation, or tail latency at that length.

Myth: The same model name is the same product everywhere. Revision, quantization, tokenizer, chat template, safety layer, tools, output limits, and serving settings can differ.

Myth: OpenAI-compatible means interchangeable. It usually means some familiar endpoints and payload shapes. Errors, usage, tools, parameters, streaming, and behavior can differ.

Myth: A free tier predicts production. Free capacity is designed for evaluation and often has different limits or data terms.

Myth: A rate limit guarantees capacity. It is normally a ceiling. Reserved/provisioned capacity is a different commercial product.

Myth: An SLA prevents outages. An SLA defines measurement and remedies after a failure; it does not remove failure.

Myth: “No training” means no retention. Abuse logs and stateful features can retain content.

Myth: Data stored in a region is processed there. Storage and inference routing are separate controls on several platforms.

Myth: A router automatically creates independent redundancy. Upstreams may share dependencies, and fallback may change behavior or policy.

Myth: Self-hosting is always cheaper. Low utilization, idle replicas, GPU fragmentation, staffing, DR, and upgrades can dominate.

Myth: Self-hosting is automatically private. Telemetry, logs, dumps, tools, storage, model downloads, and support paths still matter.

Myth: More providers always means greater resilience. Complexity and correlated dependencies can increase failure risk.

Myth: Frontier access matters more than task-specific quality. The accepted outcome is the objective, not a model’s prestige.

Myth: Published tokens per second predicts user latency. It omits network, queue, prompt prefill, tool time, and tail behavior.

23. Final one-page checklist

• 
  Define one job and its acceptance rubric.
• 
  Record p50/p95/max input, output, reasoning, media, tools, and calls.
• 
  Record monthly and peak RPM/TPM/concurrency/burst.
• 
  Set p50/p95/p99 TTFT and completion SLOs.
• 
  Classify data and state retention, training, processing-region, DPA/BAA, network, and audit requirements.
• 
  List mandatory model, modality, tool, schema, streaming, logprob, caching, batch, fine-tune, and file features.
• 
  Specify support, SLA, invoice, marketplace, and deprecation requirements.
• 
  Eliminate candidates that fail a hard gate.
• 
  Capture exact model ID / version / provider / deployment / tier / region / quantization.
• 
  Build a complete native-unit rate card, including tools, storage, failures, credits, taxes, and people.
• 
  Freeze a representative evaluation set and rubric.
• 
  Run same-model infrastructure and complete-solution tests separately.
• 
  Measure p50/p95/p99, errors, retries, cache, cost, acceptance, and goodput.
• 
  Calculate effective cost per accepted job.
• 
  Score preferences and evidence confidence separately.
• 
  Run sensitivity analysis; report conditional ties.
• 
  Review contracts, privacy, license, AUP, and certification scope.
• 
  Test capacity, cold starts, failover, rollback, and support escalation.
• 
  Canary the finalist and monitor drift.
• 
  Preserve a tested exit path.

24. Glossary

Accepted job: A completed business outcome that passes defined quality, formatting/tool, latency, safety, compliance, and operational rules.
Alias: A model name that may point to a changing version.
Batch: Asynchronous processing, often discounted and completed within a window.
Cache hit: Reused prompt content billed or scheduled differently from new input.
Cold start: Delay while an idle or new replica initializes and loads weights.
Concurrency: Requests or streams processed simultaneously.
Context window: Maximum tokens the endpoint can consider, not a guarantee of retrieval quality.
Dedicated endpoint: Customer-specific model replicas managed by a platform.
Data residency: Contract/product controls for where data is stored and/or processed; these are distinct.
DPA/BAA: Data processing agreement / business associate agreement.
Embedding: Numeric representation used for semantic retrieval or classification.
Flex/economy: Best-effort or delay-tolerant capacity sold at a discount; definition varies.
Gateway/router: Service that proxies and may select among upstream inference providers.
Goodput: Accepted work completed within the latency target, rather than raw tokens or requests.
Inference: Running a trained model on input to produce output.
Inter-token latency: Time between generated output tokens.
Open-weight: Model weights are available under a license; not automatically open source.
OpenAI-compatible: Implements some OpenAI-like API shapes; not full feature or behavioral equivalence.
p50/p95/p99: Percentiles describing typical and tail behavior.
Priority/fast: Premium shared or reserved scheduling for lower/more consistent latency; exact terms vary.
Provisioned/reserved throughput: Capacity purchased in advance for defined duration or units.
Quantization: Reduced-precision model representation that changes memory, speed, and potentially quality.
Rate limit/quota: Maximum allowed usage, not necessarily guaranteed service capacity.
Reranker: Model that reorders retrieved items by relevance to a query.
Serverless: Shared managed inference billed by use, with no customer-managed replicas.
Service tier: Scheduling/capacity class such as standard, batch, flex, fast, or provisioned.
SLA/SLO: Contractual service-level agreement / internal target objective.
Snapshot: A fixed dated model version.
TTFT: Time to first token.
TPM/RPM: Tokens per minute / requests per minute.
ZDR: Zero data retention arrangement; scope and exceptions are product- and feature-specific.

25. Frequently asked questions

Which AI inference provider is best?

There is no universal winner. The best option passes every hard gate and has the lowest risk-adjusted cost per accepted job for your exact workload, while meeting latency, capacity, privacy, reliability, and exit requirements.

Should I use the model maker or a cloud platform?

Use first-party when native features and release timing matter most. Use a hyperscaler when cloud contracts, identity, networking, regions, or provisioned capacity outweigh possible catalog or feature differences. Pilot both if they pass the gates.

Is a router safer than calling providers directly?

It can improve policy enforcement, observability, and failover, but adds a processor and failure domain. Safety and resilience depend on the configured upstream routes, terms, and independence.

When is self-hosting cheaper?

Usually when workload is sustained enough to use expensive capacity efficiently and your team can operate it well. Calculate a break-even with measured throughput, utilization, context, concurrency, idle capacity, SRE, storage, network, and DR.

How many prompts do I need for a provider evaluation?

Enough to represent important job types and estimate uncertainty at the decision boundary. Start with hundreds for narrow automated rubrics and more for diverse/rare failures; use repeated runs for stochastic outputs and load tests for tails. Report the sample and interval instead of treating a fixed number as universal.

Does “no training” protect confidential data?

Not by itself. Verify abuse-monitoring retention, application state, files, batch, caching, tools, processing regions, subprocessors, deletion, and contract scope.

What should I monitor after launch?

Exact model/route, acceptance, human correction, TTFT and total p50/p95/p99, error/retry/fallback, cache hit, input/output/reasoning/tool usage, cost per accepted job, capacity saturation, and version or price changes.

Methodology, source policy, limitations, and changelog

We searched current official model catalogs, pricing pages, API and product documentation, data-handling pages, cloud deployment documentation, and deprecation pages across the representative market. Volatile facts were retained only when a direct official source was available. Independent benchmark sources were used to explain methodology and limitations, not to rank current providers. Vendor speed or quality claims remain labeled vendor claims.

Prices are public list prices, excluding negotiation, taxes, credits, currency conversion, and most regional/marketplace adjustments. Catalogs and terms can change between verification and publication. Some official pages conflict or are dynamically generated; these are flagged. “Not publicly documented” is not converted to “no.” No universal hands-on cross-provider benchmark was performed; the guide provides the protocol readers should run.

Changelog

• 2026-08-17: Initial publication. Current market map, official rate examples, privacy-feature exceptions, service-tier distinctions, calculation examples, tools, templates, and maintenance schedule added.

Appendix B: one-page workload-requirements worksheet

Workload name / owner / date: ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​

One job: ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​_​

Acceptance rule: Quality ​​​​​​ Schema/tool ​​​​​​ Latency ​​​​​​ Safety/compliance ​​​​​​ Human review ​​​​​​

Scroll or swipe to compare →

Traffic: Interactive ____ Streaming ____ Async ____ Batchable share % Mixed ____
Modalities: Text ____ Image ____ Audio ____ Video ____ Embedding ____ Rerank ____
Mandatory API features: ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​
Users/data locations: ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​
Data classifications: ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​
Training-use rule: ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​
Maximum retention / state: ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​
Required processing and storage regions: ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​
Network/security/compliance: DPA ____ BAA ____ Private endpoint ____ CMK ____ SSO/SCIM ____ Audit logs ____ Other ____
Monthly volume / budget: ​​​​​​​​​​ 12-month growth: ​​​​​​​​​​
Procurement / support / SLA: ​​​​​​​​ Version/deprecation: ​​​​​​​​ Exit: ​​​​​​​​

Appendix C: yes/no hard-gate checklist

Answer Yes or Not applicable for every row. “Unknown” is a blocker until resolved.

Scroll or swipe to compare →

Appendix D: copyable weighted scorecard

Anchored 0–5 definitions

Scroll or swipe to compare →

Evidence confidence: 0 = contradicted/absent; 1 = unverified marketing or stale; 2 = current vendor claim; 3 = current published product documentation; 4 = reproduced limited test or contract; 5 = representative repeated measurement or directly applicable contractual fact. Evidence confidence does not turn a preference into a hard gate.

Scroll or swipe to compare →

Sensitivity runs: base weights; finance +20% cost; product +20% quality/latency; security +20% privacy/reliability; low/base/high volume; p95/max output; cache −20 points; retries +5 points; acceptance −5 points. Report the winner, margin, and any rank reversal.

Appendix E: price-per-accepted-job calculator

Input schema

Scroll or swipe to compare →

Spreadsheet formulas

TokenCostPerCall = UncachedInput*UncachedRate + CacheWrite*CacheWriteRate + CacheRead*CacheReadRate + VisibleOutput*OutputRate + Reasoning*ReasoningRate

BaseWorkflowCost = SUM(TokenCostPerCall*Calls) + SUM(ToolOrMediaUnits*UnitRate)

ExpectedRetryFallbackCost = SUM(FailureProbability*ReplacementWorkflowCost)

VariableCostPerAttempt = BaseWorkflowCost + ExpectedRetryFallbackCost

AcceptedJobs = AttemptedJobs * QualityAcceptance * LatencyPass * ComplianceOperationalPass

HumanCost = AttemptedJobs * ReviewIncidence * ReviewMinutes/60 * LoadedHourlyRate

MonthlyAllInCost = AttemptedJobs*VariableCostPerAttempt + ProvisionedDedicated + StorageNetworkLogs + EngineeringOperations + HumanCost + TaxCurrencySupport

EffectiveCostPerAcceptedJob = MonthlyAllInCost / AcceptedJobs

QualityAdjustedGoodputPerDollar = AcceptedJobs / MonthlyAllInCost

Keep native units in separate columns. Never add token counts to audio minutes or GPU hours. Normalize only costs after each native meter is priced.

Appendix F: provider-comparison matrix

Create one row per deployable SKU, not per company.

Scroll or swipe to compare →

Appendix G: two-to-four-week proof-of-concept plan

Week 1: freeze the decision

Owners: Product defines jobs and acceptance; engineering freezes harness/configuration; security/legal resolves hard gates; finance builds rate cards. Select three to five viable SKUs. Version the eval set, prompts, tool schemas, route, region, client, parameters, retries, and logging policy. Run a small smoke test to catch integration errors—not to rank winners.

Exit criteria: Every candidate passes hard gates; data handling is approved for the test; each rate card is traceable; the harness records exact SKU, route, usage, timings, outputs, and errors.

Week 2: quality and same-model infrastructure tests

Run blinded/reproducible task evaluation. Where the same open-weight model is available across hosts, freeze revision, quantization, prompt template, and settings as far as possible and compare serving separately. Publish acceptance by task slice, schema/tool/citation failures, token/output length, retries, and uncertainty.

Exit criteria: Failure taxonomy is stable; no candidate wins only because of an integration bug; same-model and complete-solution results are separated.

Week 3: performance, capacity, reliability, and economics

Replay realistic p50/p95/max prompt shapes at normal, peak, and burst concurrency from user regions. Test cache hit/miss, cold start, batch completion, 429/5xx/timeout handling, fallback, and a provider/region impairment. Measure p50/p95/p99 TTFT and completion, goodput, costs, and acceptance. Allocate fixed and human costs.

Exit criteria: Sustainable capacity and tail behavior are evidenced; effective cost per accepted job is computed; failover and privacy consequences are known.

Week 4: canary and decision (optional but recommended for meaningful commitments)

Run shadow traffic or a small canary with rollback thresholds. Review invoices/usage lag and support escalation. Run stakeholder weight and workload sensitivity sets. Select primary and fallback, or report a conditional tie. Document migration and exit.

Go/no-go thresholds: no hard-gate regression; minimum acceptance ​​​​​​%; p95 TTFT ​​​​​​; p99 total ​​​​​​; peak goodput ​​​​​​; max cost/accepted job $​​​​​​; error/retry ​​​​​​%; rollback tested; security/legal/procurement signed.

Appendix H: one-page executive recommendation memo

Decision: Approve / conditionally approve / reject ​​​​​​​​​​​​​​​​​​ as primary inference stack for ​​​​​​​​​​​​​​​​​​.

Why now: ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​

Workload: One job is ​​​​​​​​​​​​​​​​​​. Volume p50/peak ​​​​​​​​​​​​​​​​​​. Acceptance requires ​​​​​​​​​​​​​​​​​​. Data/region ​​​​​​​​​​​​​​​​​​.

Hard gates: Passed ​​​​​​ of ​​​​​​. Open items: ​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​

Shortlist: Primary ​​​​​​​​​​​​​​​​​​; fallback ​​​​​​​​​​​​​​​​​​; rejected ​​​​​​​​​​​​​​​​​​ because ​​​​​​​​​​​​​​​​​​.

Measured result: Acceptance ​​​​​​%; latency p50/p95/p99 ​​​​​​​​​​​​​​​​​​; sustainable peak ​​​​​​​​​​​​​​​​​​; error/retry ​​​​​​​​​​​​​​​​​​; effective cost per accepted job $​​​​​​ at ​​​​​​ jobs/month.

Economic sensitivity: Primary remains preferred from ​​​​​​ to ​​​​​​ volume. Rank reverses when ​​​​​​​​​​​​​​​​​​. Dedicated break-even ​​​​​​​​​​​​​​​​​​.

Risk and controls: Privacy/retention ​​​​​​​​​​​​​​​​​​; capacity ​​​​​​​​​​​​​​​​​​; lifecycle ​​​​​​​​​​​​​​​​​​; concentration ​​​​​​​​​​​​​​​​​​; contract/SLA ​​​​​​​​​​​​​​​​​​.

Exit plan: Alternate route ​​​​​​​​​​​​​​​​​​; migration estimate ​​​​​​ engineer-weeks; data/config export ​​​​​​​​​​​​​​​​​​; last failover test ​​​​​​​​​​​​​​​​​​.

Approval requested: Spend/commitment ​​​​​​​​​​​​​​​​​​; term ​​​​​​​​​​​​​​​​​​; conditions ​​​​​​​​​​​​​​​​​​; owner/date ​​​​​​​​​​​​​​​​​​.

Appendix I: source ledger

All entries verified August 17, 2026. Next review dates are intentionally short for catalogs/prices and longer for stable methodology.

Scroll or swipe to compare →

Maintenance and re-verification schedule

Scroll or swipe to compare →

Maintain a diffable source ledger. Recompute worked examples whenever a quoted rate changes. Do not silently update the “last verified” date unless every volatile row was actually checked.

Continue exploring AI infrastructure

Use our current AI inference provider comparison for a market-level matrix of models, prices, measured speed, limits, privacy, and availability. You can also browse the Kingy.ai AI model database or read more independent analysis in the Kingy.ai Blog.

Featured image: original AI-generated Kingy.ai editorial illustration. It depicts an inference decision pipeline and is not a measured benchmark or product UI.