A provider can charge almost nothing for one model call while the finished business outcome costs dollars. Retries, tool calls, rejected outputs, human review and idle GPUs all sit between the rate card and the result.
AI pricing pages usually lead with dollars per million tokens. That number is useful for checking an invoice. It is a poor unit for choosing a model, designing an agent or forecasting the cost of a product.
A customer does not buy 10,000 input tokens. They buy a routed support ticket, an answered question, an extracted invoice, a completed coding task or a resolved voice call. The workflow may make several model calls, search the web, retry after a timeout, fall back to another model and send the result to a human. It may still fail.
The useful denominator is the number of jobs that pass the required quality, format, tool, latency, reliability, privacy and safety gates.
Kingy’s rule: Optimize fully loaded cost per accepted outcome. A cheaper token rate only matters when the resulting workflow still passes the same acceptance contract.
Prices and product rules were checked against current official provider pages on August 18, 2026. Workload sizes, traffic, acceptance rates, staffing and capacity assumptions in the six examples are illustrative planning inputs. Kingy did not run an independent provider benchmark for this article. All amounts are USD and use public list prices before tax, negotiated discounts or regional uplifts unless stated otherwise.
Define the job before calculating its cost
A job is one complete user or business outcome. It may contain one API request or fifty. Count the whole workflow.
A job is accepted only when it clears every mandatory gate:
- Quality: The label is correct, the answer is grounded, the extraction meets its accuracy threshold or the code passes its tests.
- Validity: The result parses and contains every required field.
- Tool completion: Retrieval, search, code execution, database work or another required action finishes correctly.
- Latency: The complete workflow, including queues and retries, finishes inside its service-level objective.
- Reliability: The provider, network and orchestration path complete without a disqualifying error.
- Policy: Privacy, safety, jurisdiction and operational rules pass.
Measure the joint pass rate from production-like traces when possible. Multiplying separate quality, validity, latency and reliability rates is only an approximation. Failures are often correlated: a long task may time out, miss a tool step and receive a poor quality score at the same time.
If one attempt costs $0.40 and 80% of attempts are accepted, the accepted-job cost is already $0.50:
$0.40 / 0.80 = $0.50 per accepted jobThat calculation still excludes any fixed infrastructure, engineering or human-review cost omitted from the $0.40 numerator.
The accepted-job cost formula
Start with the model calls. For one attempt:
Token cost per attempt =
uncached input × input rate
+ cache writes × cache-write rate
+ cache reads × cache-read rate
+ visible output × output rate
+ separately billed reasoning or thinking × applicable rateThen add the rest of the workflow:
Variable workflow cost =
all model-call costs
+ search, retrieval, code, media and gateway charges
+ retry costs
+ fallback costsFinally, calculate the business unit:
Effective cost per accepted job =
(variable workflow costs
+ provisioned and idle capacity
+ storage, networking and observability
+ engineering and operations
+ human review and correction)
/ accepted jobsUse one accounting period, normally a month. Reconcile provider bills, infrastructure, labor and shared services into that period before dividing by attempted and accepted jobs.
Three cost boundaries answer different questions:
| Boundary | What it includes | Best use |
|---|---|---|
| Provider-variable | Tokens, tools, media, retries, fallbacks and gateway fees | API tuning and invoice reconciliation |
| Service total cost | Provider-variable plus review, capacity, logs, egress, engineering and support | Architecture and build-versus-buy decisions |
| Business outcome | Service total cost plus downstream rework, escalations, errors and value impact | Portfolio and ROI decisions |
Name the boundary on every chart. A low provider-variable cost and a high service total cost can both be correct.
Why token prices miss the bill
Input and output rates differ. The same text can produce different token counts under different tokenizers. A model that writes longer answers can cost more even when its published output rate is lower.
Caching adds at least three separate quantities: eligible prefix tokens, cache writes and cache reads. Anthropic documents a five-minute cache write at 1.25 times base input and a cache hit at 0.1 times input. OpenAI’s GPT-5.4 nano lists cached input at one-tenth of ordinary input. Google adds a cache-storage price as well as a cached-token rate. The savings depend on actual hit rate, cache lifetime and routing affinity, not the amount of text that looks theoretically reusable.
Reasoning and thinking tokens need model-specific treatment. Google’s current Gemini pricing includes thinking tokens in output. Other services expose or bill reasoning through different rules. Do not move hidden work into a generic output estimate without checking the provider documentation.
Long-context requests can cross a threshold that reprices the entire request. Tools and media add their own native units: search queries, vector requests, code sessions, image units, audio tokens, minutes and GPU-hours. Keep those units visible until the final cost calculation.
Operational behavior can overwhelm all of them. Retries repeat paid work. A fallback may use a larger model and a fresh context. Partial responses can be billed even when the job is rejected. A multi-call agent can keep spending after the useful path has already failed.
Contracts add another layer: batch discounts, priority premiums, minimum spend, prepaid-credit expiry, router fees, taxes, currency conversion, marketplace commitments and support. Open weights remove a vendor token bill but add hardware, serving, redundancy, observability, security and on-call work. Kingy’s analysis of open-weight AI and capital spending covers that transfer of cost in more detail.
Current rate facts used in the examples
These are observed public prices. They are not normalized model rankings.
| Provider and model | Current public rate used | Scope |
|---|---|---|
| OpenAI GPT-5.4 nano | $0.20 input, $0.02 cached input, $1.25 output per million tokens | Standard API; snapshot available as gpt-5.4-nano-2026-03-17 |
| Anthropic Claude Sonnet 5 | $2 input and $10 output per million tokens through August 31, 2026 | Introductory price; five-minute writes use 1.25× input and hits use 0.1× input |
| Google Gemini 3.5 Flash-Lite | Batch/Flex: $0.15 input, $0.02 cached input, $1.25 output per million tokens | Thinking is included in output; cache storage is separate |
| Google Gemini 3.6 Flash | Standard promo: $0.75 input, $0.075 cached input, $3.75 output per million tokens through December 31, 2026 | Search: 5,000 shared free requests, then $14 per 1,000 queries |
| OpenAI GPT-Realtime-2.1 mini | Text $0.60/$0.06/$2.40; audio $10/$0.30/$20 per million input/cached/output tokens | Text and audio stay in separate billing units |
| Together AI Llama 3.3 70B | Serverless $1.04 input and output per million tokens | The current dedicated-inference H100 price is $6.49 per GPU-hour |
Anthropic’s introductory Sonnet 5 price expires soon. Google’s Gemini 3.6 Flash promotional rate changes on January 1, 2027. A forecast that spans those dates needs two rate periods. Kingy’s guide to Anthropic usage-based billing and prompt caching explains the cache mechanics.
Six worked examples
Each example uses an 80% p50 plus 20% p95 planning blend. That shortcut is transparent, but it is not a statistical identity. Replace it with the measured mean or replay the full job distribution when trace data is available.
1. Short classification or support routing
Observed rate: OpenAI GPT-5.4 nano, Standard, checked August 18, 2026.
Illustrative workload: 5 million jobs a month, a peak of 6,000 jobs per minute and one call per job. Input is 550 tokens at p50 and 900 at p95. Output is 12 tokens at p50 and 30 at p95. The workflow has a 350-token cacheable prefix and an 80% hit rate. It assumes a 0.8% retry probability, a 3% fallback probability and a 0.5% human-review rate at 45 seconds per review and $30 an hour. The quality, validity, latency and operations gates produce a 95.19% planning acceptance rate.
| Result | Expected case |
|---|---|
| Monthly spend | $32,446.22 |
| Cost per attempted job | $0.006489 |
| Cost per accepted job | $0.006817 |
| Best / expected / stress | $0.0077 / $0.0068 / $0.0071 |
The model bill is small enough that fixed operating layers and review matter more. Doubling the review rate adds about $9,375 a month. Shortening a 12-token output does very little.
2. Retrieval-augmented document answering
Observed rate: Anthropic Claude Sonnet 5 at its introductory Standard price with a five-minute prompt cache, checked August 18, 2026.
Illustrative workload: 200,000 jobs a month, 120 jobs per minute at peak and two calls per job. Input is 9,000 tokens at p50 and 24,000 at p95. Visible output is 500 and 1,200 tokens. Each call uses 300 reasoning tokens in the planning model. A 6,000-token prefix has a 75% assumed hit rate. The workflow makes two vector queries per job at an illustrative $0.0004 each, retries 3% of jobs, falls back on 6% and sends 8% to a three-minute human review at $45 an hour. The planning acceptance rate is 80.18%.
| Result | Expected case |
|---|---|
| Monthly spend | $77,537.40 |
| Cost per attempted job | $0.387687 |
| Cost per accepted job | $0.483495 |
| Best / expected / stress | $0.4957 / $0.4835 / $0.5525 |
A cache miss reprices 6,000 tokens per call from $0.20 per million to $2 for ordinary input or $2.50 for a five-minute write. Human review still exceeds the model bill in this scenario.
3. Batch invoice or document extraction
Observed rate: Google Gemini 3.5 Flash-Lite, Batch, checked August 18, 2026.
Illustrative workload: 1 million jobs a month, a 5,000-job-per-minute peak and one call per job. Input is 1,600 tokens at p50 and 5,000 at p95; output is 220 and 450. A 500-token prefix has a 70% assumed hit rate. PDF and document units remain in Google’s native modality accounting. The workflow assumes a 2% retry rate, 4% fallback rate, 2% human review rate and a 91.52% planning acceptance rate.
| Result | Expected case |
|---|---|
| Monthly spend | $44,241.58 |
| Cost per attempted job | $0.044242 |
| Cost per accepted job | $0.048342 |
| Best / expected / stress | $0.0542 / $0.0483 / $0.0506 |
Batch token pricing makes provider spend small. Fixed engineering and the validity gate drive more of the accepted-job cost. A cheap extraction that fails the schema is still a rejected job.
4. Multi-step coding or research agent
Observed rate: Google Gemini 3.6 Flash at its Standard promotional price, checked August 18, 2026.
Illustrative workload: 30,000 jobs a month, 10 jobs per minute at peak and six calls per job. Input is 7,000 tokens at p50 and 20,000 at p95. Output is 900 and 2,500 tokens, with 1,200 thinking tokens per call in the planning model. A 4,000-token prefix has a 55% hit rate. The agent makes three Google Search requests per job. It retries 8%, falls back 12% and sends 25% of jobs to a ten-minute review at $70 an hour. The combined quality, validity, tool, latency and operations gates yield a 62.03% planning acceptance rate.
| Result | Expected case |
|---|---|
| Monthly spend | $137,720.38 |
| Cost per attempted job | $4.590679 |
| Cost per accepted job | $7.400202 |
| Best / expected / stress | $7.6306 / $7.4002 / $8.2789 |
Human review contributes $87,500 a month. Raising joint acceptance from 62% to 75% would reduce accepted-job cost by about 17% before any other change. The six-call loop and repeated context also make a one-call token comparison meaningless.
5. Real-time voice interaction
Observed rate: OpenAI GPT-Realtime-2.1 mini, Standard, checked August 18, 2026.
Illustrative workload: 100,000 voice jobs a month, 250 per minute at peak and eight model turns per job. Text input is 1,200 tokens at p50 and 3,000 at p95; text output is 100 and 300. Audio input is 9,000 and 18,000 tokens; audio output is 7,000 and 14,000. The model assumes an 85% hit rate on an 800-token text prefix and keeps cached audio at its separate rate. Telephony adds five minutes per job at an illustrative $0.0085 a minute. Retries are 4%, fallbacks 5%, human review 3% and planning acceptance 80.07%.
| Result | Expected case |
|---|---|
| Monthly spend | $87,476.63 |
| Cost per attempted job | $0.874766 |
| Cost per accepted job | $1.092452 |
| Best / expected / stress | $1.1258 / $1.0925 / $1.2308 |
Audio output, telephony, fallback and latency dominate. A text-only cost estimate would miss most of the bill.
6. Open-weight serverless versus dedicated capacity
Observed rate: Together AI Llama 3.3 70B at $1.04 per million input and output tokens for serverless. Together’s current Dedicated Inference table lists one H100 80GB at $6.49 an hour. Both were checked August 18, 2026.
Illustrative workload: 4 million jobs a month, 100 per minute at peak and one call per job. Input is 1,800 tokens at p50 and 3,500 at p95. Output is 350 and 800. Both paths assume 2% retries, 3% fallbacks, 1% one-minute human review at $35 an hour and the same 90% quality pass rate. Serverless has a 96% latency pass rate; the two-H100 dedicated path assumes 98%. The dedicated path provides 120 jobs per minute, leaving 20% peak margin.
| Result | Serverless | Two H100 dedicated |
|---|---|---|
| Monthly spend | $53,480.79 | $59,508.73 |
| Cost per attempted job | $0.013370 | $0.014877 |
| Planning acceptance | 85.36% | 87.14% |
| Cost per accepted job | $0.015662 | $0.017072 |
The current two-GPU list price produces $9,475.40 in raw monthly H100 capacity at 730 hours. Raw inference reaches break-even near 3.98 million jobs a month, or about 75.6% of the modeled maximum throughput. That is not service-TCO break-even. Dedicated infrastructure still needs matched quantization and quality, failover, serving software, monitoring, security and operators.
Do not compare the two paths until the same-model deployment passes the same acceptance contract. A cheaper quantized build that loses tool reliability has changed the solution, not merely the infrastructure.
What the examples show
| Workload | Accepted-job cost | Main driver |
|---|---|---|
| Short classification | $0.0068 | Fixed operations and review |
| RAG answering | $0.4835 | Review, cache effectiveness and acceptance |
| Batch extraction | $0.0483 | Fixed engineering and validity |
| Coding/research agent | $7.4002 | Human review, six calls and low joint acceptance |
| Real-time voice | $1.0925 | Audio, telephony, fallback and latency |
| Open-weight serverless / dedicated | $0.0157 / $0.0171 | Utilization, engineering and matched quality |
These are different jobs with different acceptance contracts. Their unit costs are teaching examples, not a provider ranking.
How to collect numbers that survive scrutiny
Use representative production-like tasks. Freeze the exact model version, region, tier, prompt, tools, retrieval corpus, cache policy, timeout, retry, fallback and decoding settings.
Record one row per attempted job with:
- job ID, workload stratum, timestamp, candidate, model snapshot, region and tier;
- calls, input, output, reasoning and cached tokens, plus audio, image and tool units;
- queue time, time to first token, tool time, generation time and total latency;
- errors, timeouts, retries, fallback path and termination reason;
- quality score, schema validity, tool completion, policy result, latency result and joint acceptance; and
- provider-variable cost, review cost, capacity allocation and total attempted-job cost.
Measure p50, p95 and p99. The mean still matters for cost because invoices sum all usage. Test representative concurrency and peak bursts; serial calls cannot establish queueing or peak latency.
Randomize candidate order and time blocks. Run cold, warm and steady-state phases. Blind-score quality where practical. Use the same jobs for paired comparisons.
Acceptance is binomial, while cost and latency are usually skewed. Report intervals:
- use a Wilson 95% interval for acceptance;
- bootstrap complete job rows for mean cost and accepted-job cost;
- bootstrap p95 latency and run enough peak-load jobs to populate the tail; and
- use the rule of three for rare failures: zero observed events in n trials still gives an approximate 95% upper bound of 3/n.
An approximate acceptance sample size is:
n = z² × p × (1 - p) / e²At p = 0.5, 95% confidence and a margin of three percentage points, the approximation gives about 1,068 independent jobs.
Recompute provider cost from raw usage, then reconcile it with an invoice or billing-export sample. Investigate rounding, cache treatment, minimums, free allowances and tier labels. Finish with shadow traffic and a bounded canary with rollback limits for acceptance, p95 latency, error rate and spend velocity.
Practical decision rules
- Improve the denominator first. A small acceptance gain can beat a large token discount.
- Route by measured job type. Easy and hard work may need different models and fallback paths.
- Cache stable prefixes. Measure realized hits, lifetime and routing affinity.
- Cap runaway work. Set per-job limits for calls, tokens, tools, retries, elapsed time and dollars.
- Use batch only when the latency contract allows it. A discounted job rejected for being late is expensive.
- Pay for priority when it improves acceptance. The premium should buy a measured latency or capacity gain.
- Demand sustained utilization before choosing dedicated capacity. Idle headroom, replicas and operators erase raw GPU savings.
- Separate audit review from rescue review. Both cost money; only rescue review can convert a failed machine result into an accepted job.
- Rebaseline after pricing or model changes. Promotional rates, tokenizers and product rules move.
- Set a decision margin. Do not migrate for a modeled 2% saving inside a 10% uncertainty interval.
Kingy’s AI Stack Builder can help map the broader workflow before provider selection. Keep its categories separate from the measured economics here.
Copyable accepted-job worksheet
| Field | Your measurement |
|---|---|
| Job definition and acceptance contract | |
| Model/version, tier, region and date | |
| Monthly attempts, peak jobs/minute and calls/job | |
| p50/p95/p99 input, output and reasoning tokens | |
| Cacheable tokens, write rate, hit rate and storage | |
| Audio, image, document and tool units | |
| Error, timeout, retry and fallback rates | |
| Quality, validity, tool, latency and policy gates | |
| Observed joint acceptance | |
| Human review rate, minutes and hourly cost | |
| Provider-variable monthly cost | |
| Productive, idle and redundant capacity | |
| Logs, storage, egress, engineering and support | |
| Discounts, minimums, gateway fees, tax and FX | |
| Fully loaded cost per attempted job | |
| Fully loaded cost per accepted job | |
| Peak capacity margin and p95 latency pass | |
| Confidence interval and best/stress range |
Adopt a candidate only when the lower bound of its measured advantage clears migration cost and risk while quality, latency, privacy, safety and peak capacity remain inside the contract. That is the calculation a token rate cannot do for you.