AI Accuracy Index · first edition
How often do AI models give a wrong number?
We asked 9 AI models 125 everyday questions that each have one official answer: a tax limit, a benefit rate, a passport fee, a minimum wage, a recall. Every answer was marked by fixed rules against the official page.
Short answer: From memory, the best models gave a wrong figure about 1% of the time and the worst about 20%. GPT-6 Astra (0.8%) and GPT-6.1 Sol (1.2%) were right as often as Claude but wrong far less: Claude Opus 5.5 8.4%, Claude Sonnet 5.5 10.4%. With a working web search, nearly every answer was right.
Scorecard
| Model | Right | Wrong figure | 95% range | Said not sure | No answer | Warned when wrong |
|---|---|---|---|---|---|---|
| Gemini 4 Argon | 12% | 0.0% | 0–0% | 88% | 0% | 0 of 0 |
| Grok 4.7 | 19% | 0.4% | 0–1.2% | 78% | 3% | 0 of 1 |
| GPT-6 Astra | 58% | 0.8% | 0–2.4% | 41% | 0% | 0 of 2 |
| GPT-6.1 Sol | 59% | 1.2% | 0–3.2% | 40% | 0% | 0 of 3 |
| GPT-6.1 Terra | 59% | 6.0% | 2.4–9.6% | 35% | 0% | 0 of 15 |
| Claude Opus 5.5 | 60% | 8.4% | 4–13.2% | 32% | 0% | 19 of 21 |
| Claude Sonnet 5.5 | 62% | 10.4% | 5.6–15.2% | 28% | 0% | 25 of 26 |
| Gemini 3.8 Flash | 22% | 10.8% | 5.6–16.4% | 67% | 0% | 2 of 27 |
| Qwen 3.8 27B | 22% | 19.6% | 13.6–26% | 18% | 41% | 0 of 49 |
Share of 250 answers per model. "Wrong figure" means the model gave a value and it did not match the official source, although the question told it that it could answer "not sure". The range comes from resampling questions, because a model usually answers the same question the same way twice. "Warned when wrong" counts wrong answers whose text told the reader to check the figure.
What the results show
- Two styles of good. GPT-6 Astra and GPT-6.1 Sol answered about 58% right and were almost never wrong. Gemini 4 Argon and Grok 4.7 were almost never wrong either, but mostly by saying "not sure": they answered only 12% and 19% right.
- Claude answers as often as GPT, but is wrong more. Claude Sonnet 5.5 gave a wrong figure on 14 questions where GPT-6 Astra did not, and Astra on 0 where Sonnet did not. Claude Opus 5.5 against Astra: 10 to 0.
- Claude warns you; most others don't. 44 of Claude's 47 wrong answers told the reader to check the official source. Only 2 of the other models' 97 wrong answers did.
- Last year's number is the usual mistake. 73 of the 144 wrong answers were exactly a previous official figure.
- New figures are mostly unknown. On the 53 questions about figures first published in the last 12 months, models were right 22% of the time and said "not sure" 66%.
- Travel fees are the riskiest area: 22.2% of answers on passport, visa and travel-permit fees gave a wrong figure.
By topic (all models)
| Topic | Questions | Right | Wrong figure | Said not sure |
|---|---|---|---|---|
| Travel documents | 12 | 40% | 22.2% | 34% |
| Travel rules | 12 | 78% | 7.4% | 14% |
| Work and consumer | 31 | 46% | 6.3% | 42% |
| Taxes | 33 | 44% | 4.7% | 45% |
| Benefits | 26 | 34% | 3.2% | 57% |
| Recalls | 11 | 2% | 1.0% | 94% |
The questions models got wrong most often
Public questions only, with the official answer and what each model said.
As of November 2026, what is the total fee for an approved ESTA travel authorization to visit the United States under the Visa Waiver Program?
Official answer: $40.62 (Certain DHS Immigration-Related Fees Required by HR–1: Fiscal Year 2027 Adjustments for Inflation, checked November 9, 2026). Earlier figures: $40.27 (January 1 to October 15, 2026, fiscal year 2026 amount); $40 (September 30 to December 31, 2025); $21 (before September 30, 2025).
14 wrong answers (each model was asked twice). Claude Opus 5.5: $40; Claude Sonnet 5.5: $40; GPT-6 Astra: $40.27; GPT-6.1 Sol: $40.27; GPT-6.1 Terra: $40; Gemini 3.8 Flash: $21; Qwen 3.8 27B: $21
As of November 2026, what is the total fee for an adult (18 and over) to apply for Canadian citizenship, including both the processing fee and the right of citizenship fee?
Official answer: CA$653 (Citizenship and immigration application fees: Fee list, checked November 9, 2026).
8 wrong answers (each model was asked twice). Claude Opus 5.5: CA$649.75; Claude Sonnet 5.5: CA$630 and CA$730; GPT-6.1 Terra: CA$649.75; Gemini 3.8 Flash: CA$630; Qwen 3.8 27B: CA$731
For the 2026–27 income year, what income tax rate applies to the part of an Australian resident's taxable income between $18,201 and $45,000, not counting the Medicare levy?
Official answer: 15% (Tax rates – Australian resident | Australian Taxation Office, checked November 9, 2026). Earlier figure: 16% (2025–26 income year).
5 wrong answers (each model was asked twice). Gemini 3.8 Flash: 16%; Grok 4.7: 16%; Qwen 3.8 27B: 16%
As of November 2026, how much does a standard 34-page adult UK passport (age 16 and over) cost when you apply online from inside the UK?
Official answer: £102 (Passport fees, checked November 9, 2026). Earlier figure: £94.50 (before 8 April 2026).
5 wrong answers (each model was asked twice). Claude Opus 5.5: £94.50; Claude Sonnet 5.5: £94.50; Qwen 3.8 27B: £93
For business miles driven in a car between July 1 and December 31, 2026, what is the IRS standard mileage rate, in cents per mile?
Official answer: 76 (Standard mileage rates, checked November 9, 2026). Earlier figures: 72.5 (January 1 to June 30, 2026); 70 (2025).
5 wrong answers (each model was asked twice). Claude Sonnet 5.5: 72.5; GPT-6.1 Sol: 72.5; GPT-6.1 Terra: 72.5
According to the EU's official ETIAS website as of November 2026, how much does it cost to apply for an ETIAS travel authorisation, for a traveller aged 18 to 70 who has no fee exemption?
Official answer: €20 (What is ETIAS, checked November 9, 2026). Earlier figure: €7 (previous fee, before the increase announced in July 2025).
4 wrong answers (each model was asked twice). Gemini 3.8 Flash: €7; Qwen 3.8 27B: €7
For the UK tax year from 6 April 2026 to 5 April 2027, what rate of tax does a basic rate taxpayer pay on dividend income above the dividend allowance?
Official answer: 10.75% (Tax on dividends: Check if you have to pay tax on dividends - GOV.UK, checked November 9, 2026). Earlier figure: 8.75% (tax year 2025 to 2026).
4 wrong answers (each model was asked twice). Gemini 3.8 Flash: 8.75%; Qwen 3.8 27B: 8.75%
As of November 2026, how much does one US First-Class Mail Forever stamp cost at the Post Office?
Official answer: $0.82 (First-Class Mail & Postage | USPS, checked November 9, 2026). Earlier figures: $0.78 (July 13, 2025 to July 11, 2026); $0.73 (before July 13, 2025).
4 wrong answers (each model was asked twice). Claude Opus 5.5: $0.78; Claude Sonnet 5.5: $0.78
As of November 2026, how much does it cost to mail a standard-sized rectangular postcard within the United States with a USPS postcard stamp?
Official answer: $0.65 (Postage Rates & Prices, checked November 9, 2026). Earlier figure: $0.61 (July 13, 2025 to July 11, 2026).
4 wrong answers (each model was asked twice). Claude Opus 5.5: $0.61; Claude Sonnet 5.5: $0.61
As of November 2026, how much is the TSA ConfirmID fee for an adult who arrives at a US airport security checkpoint without a REAL ID or any other acceptable ID?
Official answer: $45 (TSA ConfirmID | Transportation Security Administration, checked November 9, 2026).
3 wrong answers (each model was asked twice). GPT-6.1 Terra: $75; Qwen 3.8 27B: $15 and $25
With web search
On the first 43 questions we also gave every model the same web search tool (OpenRouter's web search with the Parallel engine; the model writes its own search). 379 of 380 answers were right, 0 were wrong and 1 said not sure. An earlier setup that searched on the whole prompt returned poor sites and was dropped; its results are not used here.
How we tested
- Questions. 125 questions from the US (72), UK (18), Canada (16), Australia (11), the EU, New Zealand and Ireland. Each names the country, the year or period and the exact case. 73 have an earlier official figure on record.
- Answer keys. Each key was written from an official government or agency page, then found again by a second checker who never saw the key. Only questions where both agreed are in the test. Some official sites block automated readers; there the checker read the same page in an ordinary browser.
- Asking. Each model got the same prompt with today's date, the question, and a request to end with "ANSWER: ..." or "ANSWER: NOT SURE". Default settings, no tools, each question asked twice.
- Marking. Fixed rules read the last ANSWER line. No AI judge. A figure with a warning still counts as an answer. We read every distinct wrong answer on the newest 82 questions and a random sample of the rest, and found no mis-marks. That check was done with Claude; a check by a person is still to come.
- Private questions. 39 of the 125 questions are kept private and count in the scores, so the test cannot simply be learned. Wrong figures: 5.8% of answers on public questions, 7.7% on private ones.
- Limits. These are the models through their APIs, not the apps. A different set of questions would give somewhat different numbers; the ranges above show how much.
Conflict of interest. Kingy AI's research and writing are done with Claude, made by Anthropic, and Claude models are in this test. To guard against bias, marking is done by fixed rules, not by any AI, and every public question, key, source, reply and mark is below to check.
Download the data
- questions.json: the 86 public questions with keys, sources and earlier figures.
- answers.json: every model's reply to those questions, the value read and the mark.
- results.json: the scorecard figures for all 125 questions.
Questions
Which AI model is most accurate on everyday official figures?
From memory, GPT-6 Astra and GPT-6.1 Sol: each answered about 58% of questions right and gave a wrong figure on 0.8% and 1.2% of answers. Gemini 4 Argon and Grok 4.7 were wrong even less often, but they answered only 12% and 19% of questions right; the rest of the time they said they were not sure.
Is Claude less accurate than ChatGPT's models?
On this test, yes. Claude Sonnet 5.5 gave a wrong figure on 14 questions where GPT-6 Astra did not, and Astra was wrong on 0 where Sonnet was not. For Claude Opus 5.5 it was 10 to 0. Claude usually added a warning: 44 of its 47 wrong answers told the reader to check the official source. Kingy's own work is done with Claude, and we say so below.
Why do AI models get these numbers wrong?
Mostly because the figure changed after the model was trained. 73 of the 144 wrong answers were exactly a previous year's figure. On the 53 questions about figures first published in the last 12 months, models answered right 22% of the time, said not sure 66% and gave a wrong figure 7%.
Does web search fix it?
In our test, yes. With the same web search tool given to every model, 379 of 380 answers on the first 43 questions were right and 0 were wrong. Turn search on for anything with a date or a price.
Is this the same as using the ChatGPT, Claude or Gemini apps?
No. We called each model through its developer API with default settings and no tools. The apps add their own instructions, memory and web search, so they can answer differently.
Tested November 9 to 10, 2026. Updated November 10, 2026. Next edition: a third of the questions replaced and every key re-checked.