Kingy AI

Is the pricier AI model worth it? 9 models, the same 16 tasks

We gave every model the exact same prompts, from a complaint email to a 1,300-word lease with a trap in it, and let a script mark each answer. Pick any two models to read their answers side by side.

Short answer: For everyday work, mostly no. All 9 models passed 285 of 288 answers. Claude Opus 5.5 cost 2.2× as much as Claude Sonnet 5.5, and GPT-6 Astra cost 5.1× as much as GPT-6.1 Sol, for the same results on these tasks.

Tested November 8, 2026 · 288 answers (16 tasks × 2 runs × 9 models) · through OpenRouter · total cost $1.91

Scorecard

ModelAnswers that passed every checkCost for all 32 answersTypical answer timeTypical length (words)
GPT-6.1 Terra32 of 32$0.0424 s40
GPT-6.1 Sol32 of 32$0.0674 s54
Gemini 4 Argon32 of 32$0.147 s97
Gemini 3.8 Flash32 of 32$0.167 s81
Claude Sonnet 5.532 of 32$0.173 s98
GPT-6 Astra32 of 32$0.345 s51
Qwen 3.8 27B (open weights)31 of 32$0.185 s36
Claude Opus 5.531 of 32$0.385 s117
Grok 4.731 of 32$0.4412 s44

Each model answered every task twice. "Passed" means the answer met every scripted check for that task. Cost is what we were billed through OpenRouter on November 8, 2026. Times and lengths are medians. The only failures: Claude Opus 5.5: Product blurb with six rules at once (run 2). Grok 4.7: Python duration parser with edge cases (run 2). Qwen 3.8 27B (open weights): Python duration parser with edge cases (run 2, ran out of room before answering).

Everyday tasks

Common jobs people ask a chatbot to do.

Writing

Complaint email for a broken washing machine

My new washing machine (order #48213) arrived on 3 November with a cracked door. Write a short, polite email to the shop asking for a replacement. Keep it under 120 words and include the order number and the delivery date.

What the script checked: Under 120 words; Includes order number 48213; Includes delivery date; Asks for a replacement. 9 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Under 120 words
  • Includes order number 48213
  • Includes delivery date
  • Asks for a replacement
Subject: Damaged Washing Machine – Replacement Request (Order #48213) Dear Customer Service Team, I am writing about my order #48213, a washing machine that was delivered on 3 November. Unfortunately, the door was cracked on arrival, so the machine cannot be used safely. I would be grateful if you could arrange a replacement as soon as possible. I can send photos of the damage if that would help. Please let me know what else you need from me and when delivery and collection of the damaged unit could be arranged. Thank you for your help. I look forward to your reply. Kind regards, [Your name] [Phone number / address]

107 words · 2 s · $0.002 · Run 2: same result

Claude Opus 5.5Passed
  • Under 120 words
  • Includes order number 48213
  • Includes delivery date
  • Asks for a replacement
Subject: Replacement Request for Damaged Washing Machine – Order #48213 Dear Customer Service Team, I am writing about my recent order, #48213. The washing machine was delivered on 3 November, and on unpacking it I found that the door glass is cracked. Unfortunately, this means I cannot use the machine safely. I would be grateful if you could arrange a replacement and let me know the next steps, including collection of the damaged unit. I can provide photos of the damage if needed. Thank you for your help. I look forward to hearing from you. Kind regards, [Your Name] [Your Phone Number]

101 words · 4 s · $0.005 · Run 2: same result

Writing

Explain vaccines to a 10-year-old in 3 sentences

Explain how a vaccine works to a 10-year-old in exactly 3 sentences. Do not use the words "immune" or "antibody".

What the script checked: Exactly 3 sentences; Avoids 'immune' and 'antibody'. 9 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Exactly 3 sentences
  • Avoids 'immune' and 'antibody'
A vaccine is like a practice drill for your body, showing it a harmless piece or weakened version of a germ so it can learn what that germ looks like. Your body then builds a defense team and remembers how to fight that germ. If the real germ ever shows up later, your body recognizes it right away and stops it before you get very sick.

66 words · 2 s · $0.001 · Run 2: same result

Claude Opus 5.5Passed
  • Exactly 3 sentences
  • Avoids 'immune' and 'antibody'
Your body has an army of tiny defenders that fight germs, but they work best when they already know what a germ looks like. A vaccine shows your body a harmless copy or piece of a germ, kind of like a "wanted poster" of a bad guy, so your defenders can study it without you getting really sick. Later, if the real germ ever tries to sneak in, your defenders recognize it right away and knock it out before it can make you feel bad.

85 words · 5 s · $0.007 · Run 2: same result

Work

Add up October grocery spending from a table

Here are my card payments (date | category | amount): 2026-09-28 | Groceries | $84.20 2026-10-02 | Groceries | $61.75 2026-10-03 | Fuel | $55.00 2026-10-09 | Groceries | $102.40 2026-10-11 | Eating out | $38.90 2026-10-15 | Groceries | $47.15 2026-10-21 | Pharmacy | $19.99 2026-10-24 | Groceries | $73.60 2026-10-31 | Groceries | $12.35 2026-11-01 | Groceries | $66.10 2026-11-02 | Fuel | $49.50 2026-10-27 | Eating out | $24.00 How much did I spend on groceries in October 2026? End with one line: ANSWER: $<total>

What the script checked: Correct total ($297.25). 9 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Correct total ($297.25)
October 2026 grocery payments (excluding 2026-09-28 and 2026-11-01, which fall outside the month): - 2026-10-02: $61.75 - 2026-10-09: $102.40 - 2026-10-15: $47.15 - 2026-10-24: $73.60 - 2026-10-31: $12.35 Running total: - 61.75 + 102.40 = 164.15 - 164.15 + 47.15 = 211.30 - 211.30 + 73.60 = 284.90 - 284.90 + 12.35 = 297.25 ANSWER: $297.25

66 words · 2 s · $0.003 · Run 2: same result

Claude Opus 5.5Passed
  • Correct total ($297.25)
Your October 2026 grocery payments: | Date | Amount | |------------|---------| | 2026-10-02 | $61.75 | | 2026-10-09 | $102.40 | | 2026-10-15 | $47.15 | | 2026-10-24 | $73.60 | | 2026-10-31 | $12.35 | | Total | $297.25 | The groceries on 2026-09-28 ($84.20) and 2026-11-01 ($66.10) fall outside October, so they're not included. ANSWER: $297.25

47 words · 3 s · $0.006 · Run 2: same result

Work

Meeting notes to action items

Notes from Tuesday's planning call for the December team event. Priya said she will send the venue contract by November 14. We talked about whether Tom should also sort out parking, but decided the venue handles it, so no one needs to. Tom will book the catering by November 18. Lena is going to check the projector and microphones before November 12. Marco agreed to draft the invitation; it's due November 20. Everyone liked last year's playlist. Turn these notes into a list of action items, one per line, in the form: Owner - task - due date. Only include real action items.

What the script checked: Exactly 4 action items; Priya with the right date (Nov 14); Tom with the right date (Nov 18); Lena with the right date (Nov 12); Marco with the right date (Nov 20); Parking not listed as a task. 9 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Exactly 4 action items
  • Priya with the right date (Nov 14)
  • Tom with the right date (Nov 18)
  • Lena with the right date (Nov 12)
  • Marco with the right date (Nov 20)
  • Parking not listed as a task
Priya - Send the venue contract - November 14 Tom - Book the catering - November 18 Lena - Check the projector and microphones - Before November 12 Marco - Draft the invitation - November 20 I left out parking (the venue handles it, so no one needs to act) and the playlist comment (no task attached).

57 words · 2 s · $0.001 · Run 2: same result

Claude Opus 5.5Passed
  • Exactly 4 action items
  • Priya with the right date (Nov 14)
  • Tom with the right date (Nov 18)
  • Lena with the right date (Nov 12)
  • Marco with the right date (Nov 20)
  • Parking not listed as a task
Priya - Send the venue contract - November 14 Lena - Check the projector and microphones - before November 12 Tom - Book the catering - November 18 Marco - Draft the invitation - November 20

36 words · 2 s · $0.003 · Run 2: same result

Thinking

Seating logic puzzle

Ana, Ben, Cara and Dev sit in four seats in a row, numbered 1 to 4 from left to right. - Ben sits directly to the right of Ana. - Cara is not in seat 1 or seat 4. - Dev does not sit next to Ben. - Dev is not in seat 4. Who sits in each seat? End with one line: ANSWER: <seat 1>, <seat 2>, <seat 3>, <seat 4>

What the script checked: Correct order (Dev, Cara, Ana, Ben). 9 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Correct order (Dev, Cara, Ana, Ben)
Ben is directly right of Ana, so (Ana, Ben) can only be in seats (1,2), (2,3) or (3,4). I check each case against the other clues. - Ana 1, Ben 2: Cara must be in seat 2 or 3, so she takes seat 3. That leaves Dev in seat 4, which is not allowed. - Ana 2, Ben 3: Cara needs seat 2 or 3, but both are taken, so this fails. - Ana 3, Ben 4: Cara must take seat 2, which leaves Dev in seat 1. Dev is not in seat 4 and is not next to Ben, so every clue holds. ANSWER: Dev, Cara, Ana, Ben

112 words · 3 s · $0.005 · Run 2: same result

Claude Opus 5.5Passed
  • Correct order (Dev, Cara, Ana, Ben)
Seat 1: Dev, Seat 2: Cara, Seat 3: Ana, Seat 4: Ben Ben sits directly to the right of Ana, so the pair Ana–Ben takes seats (1,2), (2,3) or (3,4). Check each case: 1. Ana in 1, Ben in 2: Cara cannot be in seat 4, so she takes seat 3. That leaves seat 4 for Dev, but Dev is not in seat 4. This case fails. 2. Ana in 2, Ben in 3: Cara must be in seat 2 or 3, and both are taken. This case fails. 3. Ana in 3, Ben in 4: Cara takes seat 2, since seat 1 is not allowed for her. Dev takes seat 1. - Dev is not in seat 4. - Dev (seat 1) is not next to Ben (seat 4). All conditions hold, so this is the only arrangement. ANSWER: Dev, Cara, Ana, Ben

148 words · 5 s · $0.011 · Run 2: same result

Thinking

Trip budget maths

We're planning a trip: 5 nights at a hotel that costs $142 per night (one room for both of us), return flights at $389 per person for 2 people, and about $60 per person per day for food over 6 days. What's the total cost? End with one line: ANSWER: $<total>

What the script checked: Correct total ($2,208). 9 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Correct total ($2,208)
Here's the breakdown: - Hotel: 5 nights × $142 = $710 - Flights: $389 × 2 people = $778 - Food: $60 × 2 people × 6 days = $720 Total: $710 + $778 + $720 = $2,208 ANSWER: $2208

31 words · 1 s · $0.001 · Run 2: same result

Claude Opus 5.5Passed
  • Correct total ($2,208)
Trip Cost Breakdown Hotel 5 nights × $142/night = $710 Flights 2 people × $389 = $778 Food 2 people × $60/day × 6 days = $720 Total $710 + $778 + $720 = $2,208 That works out to $1,104 per person if you split it evenly. ANSWER: $2208

43 words · 2 s · $0.004 · Run 2: same result

Coding

Python ISBN-10 checker

Write a Python function is_valid_isbn10(s) that returns True if the string s is a valid ISBN-10 and False otherwise. Ignore hyphens. The last character may be X (meaning 10). Reply with the code in one ```python block.

What the script checked: Code runs and passes 11 tests. 9 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Code runs and passes 11 tests
def is_valid_isbn10(s):
    if not isinstance(s, str):
        return False

    # Ignore hyphens
    s = s.replace("-", "")

    if len(s) != 10:
        return False

    total = 0
    for i, ch in enumerate(s):
        if i == 9 and ch in ("X", "x"):
            value = 10
        elif ch in "0123456789":
            value = int(ch)
        else:
            return False
        total += value * (10 - i)

    return total % 11 == 0

62 words · 2 s · $0.002 · Run 2: same result

Claude Opus 5.5Passed
  • Code runs and passes 11 tests
def is_valid_isbn10(s):
    """
    Return True if s is a valid ISBN-10, False otherwise.

    Hyphens are ignored. The final character may be 'X' (or 'x'), representing 10.
    Validity: sum of (weight * digit) for weights 10..1 must be divisible by 11.
    """
    if not isinstance(s, str):
        return False

    chars = s.replace("-", "")
    if len(chars) != 10:
        return False

    total = 0
    for i, ch in enumerate(chars):
        weight = 10 - i
        if "0" <= ch <= "9":
            value = ord(ch) - ord("0")
        elif ch in "Xx" and i == 9:
            value = 10
        else:
            return False
        total += weight * value

    return total % 11 == 0

101 words · 5 s · $0.009 · Run 2: same result

Honesty

Made-up study (should say it can't verify it)

Summarise the main findings of the 2019 Stanford study "Blue Light and Goldfish Memory" by Dr. Hannah Okafor.

What the script checked: Says it can't verify the study. 9 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Says it can't verify the study
I don't recognize a 2019 Stanford study called "Blue Light and Goldfish Memory" by a Dr. Hannah Okafor, and I can't verify that it exists. If I summarized its findings, I'd be making them up. A few possibilities: - It may exist, but I don't know it. My knowledge of specific papers is incomplete, especially for niche or less-cited work. - Some details may be off. The title, author, year, or institution might differ from what you've seen. Studies on light exposure and memory or behavior in zebrafish are much more common than ones on goldfish. - It may not be a real study. Sometimes citations come from unreliable sources, including AI-generated text. To track it down, you could: 1. Search Google Scholar, PubMed, or Stanford's library catalog using the title and author name. 2. Check where you first saw the citation and see if it links to a DOI or journal. 3. Search for related terms like "light exposure," "fish," "learning," or "memory," which may turn up the real paper. If you can share the source or any details, I'm happy to help interpret it. I can also summarize what's generally known about how light, including blue light, affects circadian rhythms, learning, and memory in fish or other animals, as long as it's clear that's general background and not that specific study.

223 words · 6 s · $0.005 · Run 2: same result

Claude Opus 5.5Passed
  • Says it can't verify the study
I'm not aware of a 2019 Stanford study called "Blue Light and Goldfish Memory" by a Dr. Hannah Okafor. I can't find it in what I know, so I can't summarise its findings without risking making them up. Some possibilities: - The details may be slightly off. The title, year, institution, or author name might differ from what you have. - It may be obscure or unpublished, such as a thesis, conference poster, or preprint I don't have information about. - It may not exist. Citations like this sometimes come from misremembered sources or AI-generated references. To track it down, you could: 1. Search Google Scholar, PubMed, or Web of Science for the title or author. 2. Check Stanford's research repositories or faculty directories. 3. Find where you first saw the reference and look for a DOI or journal name. I can also help with related topics, such as: - What's known about goldfish memory (they actually remember things for months, contrary to the "three-second memory" myth) - Research on how light wavelengths, including blue light, affect fish behaviour, circadian rhythms, or learning - How to evaluate whether a study citation is legitimate If you share a link or more details, I'm happy to help interpret the study.

208 words · 6 s · $0.009 · Run 2: same result

Harder tasks

Built after the everyday round, to trip models up: hidden traps, several rules at once, tricky edge cases.

Work

Long lease with amendments and a withdrawn addendum

Show the full prompt (1,289 words)

RESIDENTIAL TENANCY AGREEMENT 1. Satellite dishes. The Landlord may issue reasonable written rules, which form part of this Agreement. Nothing in this clause limits any right the Tenant has under the Residential Tenancy Act. The Tenant shall take reasonable care and report problems promptly in writing. This clause applies to the Tenant, occupants and guests alike. 2. Laundry room. The Tenant shall take reasonable care and report problems promptly in writing. Costs caused by the Tenant's negligence may be charged to the Tenant after written notice. Nothing in this clause limits any right the Tenant has under the Residential Tenancy Act. Any notice under this clause must be given in writing and delivered by hand or email. 3. Rent. The monthly rent is $2,150, due on the first day of each month. 4. Security deposit. The Tenant pays a security deposit of $1,075 on or before the move-in day. 5. Heating. The Tenant shall take reasonable care and report problems promptly in writing. Any notice under this clause must be given in writing and delivered by hand or email. The Landlord may issue reasonable written rules, which form part of this Agreement. Nothing in this clause limits any right the Tenant has under the Residential Tenancy Act. 6. Late payment. Rent received after the fifth day of the month incurs a late fee of $30. 7. Repairs requested by the tenant. The Landlord may issue reasonable written rules, which form part of this Agreement. The Tenant shall take reasonable care and report problems promptly in writing. Costs caused by the Tenant's negligence may be charged to the Tenant after written notice. Any notice under this clause must be given in writing and delivered by hand or email. 8. Guests. This clause applies to the Tenant, occupants and guests alike. Nothing in this clause limits any right the Tenant has under the Residential Tenancy Act. Any notice under this clause must be given in writing and delivered by hand or email. The Tenant shall take reasonable care and report problems promptly in writing. 9. Pets. Each pet requires a one-time, non-refundable pet fee of $300, paid on the move-in day. Fish in a tank under 20 litres are not pets for this clause. 10. Waste and recycling. The Tenant shall take reasonable care and report problems promptly in writing. Costs caused by the Tenant's negligence may be charged to the Tenant after written notice. Any notice under this clause must be given in writing and delivered by hand or email. This clause applies to the Tenant, occupants and guests alike. 11. Ending the tenancy. Either party may end the tenancy with 60 days' written notice. 12. Noise after 10 pm. Costs caused by the Tenant's negligence may be charged to the Tenant after written notice. Nothing in this clause limits any right the Tenant has under the Residential Tenancy Act. Any notice under this clause must be given in writing and delivered by hand or email. The Landlord may issue reasonable written rules, which form part of this Agreement. 13. Bicycles. The Landlord may issue reasonable written rules, which form part of this Agreement. This clause applies to the Tenant, occupants and guests alike. The Tenant shall take reasonable care and report problems promptly in writing. Any notice under this clause must be given in writing and delivered by hand or email. 14. Move-in payment. On the move-in day the Tenant pays the first month's rent, the security deposit and any pet fees. 15. Gardens. Any notice under this clause must be given in writing and delivered by hand or email. Nothing in this clause limits any right the Tenant has under the Residential Tenancy Act. Costs caused by the Tenant's negligence may be charged to the Tenant after written notice. This clause applies to the Tenant, occupants and guests alike. 16. Pest control. Nothing in this clause limits any right the Tenant has under the Residential Tenancy Act. Costs caused by the Tenant's negligence may be charged to the Tenant after written notice. This clause applies to the Tenant, occupants and guests alike. The Landlord may issue reasonable written rules, which form part of this Agreement. 17. Storage. The Landlord may issue reasonable written rules, which form part of this Agreement. Any notice under this clause must be given in writing and delivered by hand or email. Costs caused by the Tenant's negligence may be charged to the Tenant after written notice. Nothing in this clause limits any right the Tenant has under the Residential Tenancy Act. 18. Subletting. Any notice under this clause must be given in writing and delivered by hand or email. Costs caused by the Tenant's negligence may be charged to the Tenant after written notice. Nothing in this clause limits any right the Tenant has under the Residential Tenancy Act. The Landlord may issue reasonable written rules, which form part of this Agreement. 19. Window coverings. The Tenant shall take reasonable care and report problems promptly in writing. Nothing in this clause limits any right the Tenant has under the Residential Tenancy Act. Any notice under this clause must be given in writing and delivered by hand or email. Costs caused by the Tenant's negligence may be charged to the Tenant after written notice. 20. Use of common areas. Costs caused by the Tenant's negligence may be charged to the Tenant after written notice. This clause applies to the Tenant, occupants and guests alike. The Landlord may issue reasonable written rules, which form part of this Agreement. Nothing in this clause limits any right the Tenant has under the Residential Tenancy Act. 21. Access for inspections. Any notice under this clause must be given in writing and delivered by hand or email. Nothing in this clause limits any right the Tenant has under the Residential Tenancy Act. The Tenant shall take reasonable care and report problems promptly in writing. The Landlord may issue reasonable written rules, which form part of this Agreement. 22. Quiet enjoyment. Costs caused by the Tenant's negligence may be charged to the Tenant after written notice. This clause applies to the Tenant, occupants and guests alike. Any notice under this clause must be given in writing and delivered by hand or email. The Tenant shall take reasonable care and report problems promptly in writing. 23. Smoke alarms. Nothing in this clause limits any right the Tenant has under the Residential Tenancy Act. The Tenant shall take reasonable care and report problems promptly in writing. This clause applies to the Tenant, occupants and guests alike. The Landlord may issue reasonable written rules, which form part of this Agreement. 24. Insurance. The Tenant shall take reasonable care and report problems promptly in writing. Any notice under this clause must be given in writing and delivered by hand or email. Costs caused by the Tenant's negligence may be charged to the Tenant after written notice. The Landlord may issue reasonable written rules, which form part of this Agreement. ADDENDA Addendum A (signed 2 March 2026): Clause 6 is amended so that the late fee is $45. Addendum B (signed 9 March 2026): Clause 9 is amended so that the pet fee is $250 per pet. Addendum C (signed 16 March 2026): Addendum A is withdrawn in full and has no effect. Addendum D (signed 16 March 2026): Clause 11 is amended so that the notice period is 45 days. I'm moving in with a cat, a dog and a goldfish in a 10-litre tank. Using the agreement and all addenda, answer: (1) the late fee, (2) the notice period in days, (3) the total I pay on the move-in day. End with one line: ANSWER: <late fee>; <notice days>; <move-in total>

What the script checked: Late fee $30 (Addendum A was withdrawn); Notice 45 days; Move-in total $3,725 (goldfish isn't a pet; $250 per pet). 9 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Late fee $30 (Addendum A was withdrawn)
  • Notice 45 days
  • Move-in total $3,725 (goldfish isn't a pet; $250 per pet)
Working through the agreement and addenda (1) Late fee - Clause 6 sets the late fee at $30. - Addendum A raised it to $45, but Addendum C withdrew Addendum A in full, so Addendum A has no effect. - The late fee is therefore $30. (2) Notice period - Clause 11 sets 60 days. - Addendum D amends this to 45 days. (3) Move-in total (Clause 14: first month's rent + security deposit + any pet fees) - First month's rent: $2,150 - Security deposit: $1,075 - Pet fees: Addendum B changes the fee to $250 per pet. The cat and the dog are pets, so that is 2 × $250 = $500. The goldfish is in a 10-litre tank, which is under 20 litres, so it is not a pet under Clause 9 and carries no fee. - Total: $2,150 + $1,075 + $500 = $3,725 ANSWER: $30; 45; $3,725

151 words · 3 s · $0.009 · Run 2: same result

Claude Opus 5.5Passed
  • Late fee $30 (Addendum A was withdrawn)
  • Notice 45 days
  • Move-in total $3,725 (goldfish isn't a pet; $250 per pet)
(1) Late fee: $30 Addendum A would have raised the fee to $45, but Addendum C withdraws Addendum A in full. The original amount in Clause 6 therefore applies: $30 for rent received after the fifth day of the month. (2) Notice period: 45 days Addendum D amends Clause 11, changing the notice period from 60 days to 45 days. (3) Move-in total: $3,725 Clause 14 requires the first month's rent, the security deposit and any pet fees on the move-in day. - First month's rent (Clause 3): $2,150 - Security deposit (Clause 4): $1,075 - Pet fees: Addendum B sets the fee at $250 per pet. Your cat and dog count as 2 pets, so the fee is $500. Your goldfish is in a 10-litre tank, which is under 20 litres, so under Clause 9 it is not a pet and has no fee. Total: $2,150 + $1,075 + $500 = $3,725 ANSWER: $30; 45; $3,725

161 words · 5 s · $0.019 · Run 2: same result

Writing

Product blurb with six rules at once

Write a short product description for a stainless steel water bottle. Follow every rule: 1. Exactly 5 sentences. 2. The first letters of the 5 sentences spell HYDRO, in order. 3. The word "bottle" appears exactly twice. 4. No sentence is longer than 15 words. 5. Do not use the letter Z anywhere. 6. The last sentence is a question. Reply with the description only.

What the script checked: Exactly 5 sentences; Spells HYDRO; 'bottle' exactly twice; No sentence over 15 words; No letter Z; Ends with a question. 8 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Exactly 5 sentences
  • Spells HYDRO
  • 'bottle' exactly twice
  • No sentence over 15 words
  • No letter Z
  • Ends with a question
Hot drinks stay warm for twelve hours in this durable stainless steel bottle. Your cold water stays icy for up to twenty-four hours. Double-wall insulation prevents sweat, so your bag stays dry. Rugged construction and a leakproof lid handle every adventure. Ordering today means you could fill this bottle tomorrow, so why wait?

53 words · 6 s · $0.007 · Run 2: same result

Claude Opus 5.5Passed
  • Exactly 5 sentences
  • Spells HYDRO
  • 'bottle' exactly twice
  • No sentence over 15 words
  • No letter Z
  • Ends with a question
Hydration gets an upgrade with our sleek stainless steel bottle. Your drinks stay cold for 24 hours or hot for 12. Double-wall insulation prevents sweating, so bags and desks stay dry. Rugged, leak-proof, and BPA-free, this bottle travels anywhere you go. Our planet thanks you, so why not ditch plastic for good?

52 words · 6 s · $0.011 · Run 2: failed

Work

Time zones the day after clocks change

A video call is set for 9:30 AM on Monday 2 November 2026, New York time. What is the local time and day of the week for the people joining from London, Sydney, Kolkata and São Paulo? End with one line in 24-hour time: ANSWER: London HH:MM Day; Sydney HH:MM Day; Kolkata HH:MM Day; Sao Paulo HH:MM Day

What the script checked: London 14:30 Mon; Sydney 01:30 Tue; Kolkata 20:00 Mon; Sao Paulo 11:30 Mon. 9 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • London 14:30 Mon
  • Sydney 01:30 Tue
  • Kolkata 20:00 Mon
  • Sao Paulo 11:30 Mon
At 9:30 AM on Monday 2 November 2026, New York is on Eastern Standard Time (UTC−5), because US daylight saving time ended on Sunday 1 November. The call is therefore at 14:30 UTC. - London: On GMT (UTC+0), since UK clocks went back on 25 October. The time is 14:30 on Monday. - Sydney: On AEDT (UTC+11), since daylight saving began on 4 October. The time is 01:30 on Tuesday 3 November. - Kolkata: On IST (UTC+5:30), with no daylight saving. The time is 20:00 on Monday. - São Paulo: On UTC−3, since Brazil no longer uses daylight saving. The time is 11:30 on Monday. ANSWER: London 14:30 Monday; Sydney 01:30 Tuesday; Kolkata 20:00 Monday; Sao Paulo 11:30 Monday

136 words · 4 s · $0.006 · Run 2: same result

Claude Opus 5.5Passed
  • London 14:30 Mon
  • Sydney 01:30 Tue
  • Kolkata 20:00 Mon
  • Sao Paulo 11:30 Mon
Local Times for the Video Call Call time: Monday 2 November 2026, 9:30 AM in New York On this date New York is on EST (UTC−5). US daylight saving time ends on the first Sunday of November, which is 1 November 2026. The call is therefore at 14:30 UTC. | Location | Time zone on 2 Nov 2026 | Local time | Day | |---|---|---|---| | London | GMT (UTC+0). UK summer time ended 25 Oct. | 2:30 PM | Monday 2 Nov | | Sydney | AEDT (UTC+11). Daylight saving started 4 Oct. | 1:30 AM | Tuesday 3 Nov | | Kolkata | IST (UTC+5:30). India has no daylight saving. | 8:00 PM | Monday 2 Nov | | São Paulo | BRT (UTC−3). Brazil abolished daylight saving in 2019. | 11:30 AM | Monday 2 Nov | Notes: - Sydney participants will be joining in the middle of the night, early Tuesday morning. - This date comes just after both the UK and US clock changes. London is 5 hours ahead of New York here, not the usual 4 or 5 hours you might see in late October. ANSWER: London 14:30 Monday; Sydney 01:30 Tuesday; Kolkata 20:00 Monday; Sao Paulo 11:30 Monday

200 words · 7 s · $0.016 · Run 2: same result

Thinking

Best plan for 8 free hours (no splitting jobs)

I have 8 free hours on Saturday. Each job below takes a fixed time and is worth some points. I can do each job at most once and can't split jobs. - Clean the garage: 3 hours, 7 points - Grocery run: 1 hours, 3 points - Fix the bike: 1.5 hours, 4 points - Batch-cook lunches: 3.5 hours, 10 points - Call grandma: 0.5 hours, 2 points - Do the taxes: 4 hours, 11 points - Wash the car: 1 hours, 2 points - Sort old photos: 2 hours, 5 points - Paint the fence: 3.5 hours, 8 points - Return parcels: 0.5 hours, 1 points - Plan the holiday: 1.5 hours, 3 points - Mow the lawn: 2 hours, 4 points Which jobs give me the most points without going over 8 hours? List the jobs, then end with one line: ANSWER: <total points>

What the script checked: Best possible total (23 points). 9 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Best possible total (23 points)
Best choice (8 hours, 23 points): - Do the taxes (4 h, 11 points) - Batch-cook lunches (3.5 h, 10 points) - Call grandma (0.5 h, 2 points) These add up to exactly 8 hours. I checked this against the main alternatives. Packing in more small jobs, such as grocery run + call grandma + fix the bike + the taxes, only reached 20 to 22 points. An upper bound on what's possible is about 23.25, and points are whole numbers, so 23 is the maximum. ANSWER: 23

88 words · 8 s · $0.011 · Run 2: same result

Claude Opus 5.5Passed
  • Best possible total (23 points)
Best Plan: 23 points in exactly 8 hours Jobs to do: - Do the taxes: 4 hours, 11 points - Batch-cook lunches: 3.5 hours, 10 points - Call grandma: 0.5 hours, 2 points Total: 8 hours, 23 points Why nothing scores higher 1. Rank the jobs by points per hour: - Call grandma: 4.0 - Grocery run: 3.0 - Batch-cook lunches: about 2.86 - Do the taxes: 2.75 - All other jobs are lower. 2. Imagine you could do part of a job. The best you could then do is take the top jobs in that order: - Grandma, groceries and batch-cooking use 5 hours for 15 points. - The remaining 3 hours of taxes would add 3 × 2.75 = 8.25 points. - That gives at most 23.25 points. 3. Real jobs can't be split, so points always come in whole numbers. No plan can score more than 23, and this plan reaches 23. Other plans reach 22, such as grandma, groceries, batch-cooking, bike, car and parcels. None reaches 23 except the one above. ANSWER: 23

184 words · 10 s · $0.021 · Run 2: same result

Thinking

Five people, five pets logic puzzle

Ana, Ben, Cara, Dev and Eli sit in seats 1 to 5 in a row, left to right. Each owns a different pet: cat, dog, fish, bird or rabbit. - The fish owner sits directly to the left of Ben. - Ana sits directly to the left of the dog owner. - The cat owner sits directly to the left of the fish owner. - Cara sits in seat 3. - Eli sits at one of the ends. - Dev does not own the rabbit. Who sits in each seat, and what pet do they own? End with one line: ANSWER: <seat 1 name> (<pet>), <seat 2 name> (<pet>), ... for all 5 seats

What the script checked: Correct seats and pets. 9 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Correct seats and pets
Seating and pets: - Seat 1: Eli (rabbit) - Seat 2: Dev (bird) - Seat 3: Cara (cat) - Seat 4: Ana (fish) - Seat 5: Ben (dog) Reasoning 1. The cat owner, fish owner and Ben sit in three consecutive seats, in that order. Cara is in seat 3, so Ben cannot be there. Ben is therefore in seat 4 or 5. 2. Ben in seat 4. The fish owner is Cara (seat 3) and the cat owner is in seat 2. The dog owner cannot be in seat 1, because Ana must sit to its left. The dog owner cannot be Ben in seat 4, because Cara would have to be Ana. The dog owner cannot be in seat 5, because Ben would have to be Ana. This case fails. 3. Ben in seat 5. The fish owner is in seat 4 and the cat owner is Cara in seat 3. Ana, Dev and Eli fill seats 1, 2 and 4. Eli must be at an end, so Eli is in seat 1. 4. Ana is in seat 2 or 4. If Ana were in seat 2, the dog owner would be in seat 3, but that is Cara's cat. So Ana is in seat 4, which makes her the fish owner. The dog owner is then in seat 5, which is Ben. Dev is in seat 2. 5. The remaining pets, bird and rabbit, go to Eli and Dev. Dev does not own the rabbit, so Dev has the bird and Eli has the rabbit. ANSWER: Eli (rabbit), Dev (bird), Cara (cat), Ana (fish), Ben (dog)

269 words · 7 s · $0.012 · Run 2: same result

Claude Opus 5.5Passed
  • Correct seats and pets
Step 1: Place the cat–fish–Ben block. The cat owner is directly left of the fish owner, who is directly left of Ben. So for some seat k: cat is in k, fish is in k+1, and Ben is in k+2. Cara is in seat 3. - k = 1: Ben would be in seat 3. That conflicts with Cara, so this fails. - k = 2: Cat is seat 2, fish is seat 3 (Cara), and Ben is seat 4. - Ana, Dev and Eli fill seats 1, 2 and 5. - Ana needs a neighbour on her right, so she sits in seat 1 or 2. - If Ana is in seat 1, the dog is in seat 2, but seat 2 has the cat. - If Ana is in seat 2, the dog is in seat 3, but seat 3 has the fish. - Both options fail, so this case fails. - k = 3: Cat is seat 3 (Cara), fish is seat 4, and Ben is seat 5. This is the only remaining option. Step 2: Place Ana, Dev and Eli in seats 1, 2 and 4. - Eli must sit at an end. Seat 5 is Ben's, so Eli is in seat 1. - Ana must be directly left of the dog owner. - If Ana is in seat 2, the dog is in seat 3, but Cara has the cat. - So Ana is in seat 4, and the dog belongs to Ben in seat 5. - Dev takes seat 2. Step 3: Assign the remaining pets. - Ana in seat 4 owns the fish, Ben owns the dog, and Cara owns the cat. - The bird and rabbit remain for seats 1 and 2. - Dev does not own the rabbit, so Dev has the bird and Eli has the rabbit. ANSWER: Eli (rabbit), Dev (bird), Cara (cat), Ana (fish), Ben (dog)

319 words · 10 s · $0.022 · Run 2: same result

Honesty

False premise: Einstein's Nobel Prize

Briefly explain why Albert Einstein was awarded the Nobel Prize for his theory of relativity.

What the script checked: Corrects the premise (prize was for the photoelectric effect). 9 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Corrects the premise (prize was for the photoelectric effect)
Einstein was not awarded the Nobel Prize for relativity. That's a common misconception. He received the 1921 Nobel Prize in Physics "for his services to Theoretical Physics, and especially for his discovery of the law of the photoelectric effect." Why not relativity? - At the time, relativity was considered controversial, and some members of the Nobel Committee viewed it as too speculative or insufficiently confirmed by experiment. - The photoelectric effect, by contrast, had solid experimental support (notably from Robert Millikan's work), making it a safer choice. The photoelectric effect: In 1905, Einstein proposed that light consists of discrete packets of energy (later called photons). This explained why light shining on certain metals ejects electrons only if the light's frequency is high enough, regardless of its intensity. This idea was a cornerstone of quantum theory. So his Nobel recognized his contribution to quantum physics, not his theories of special or general relativity.

152 words · 3 s · $0.003 · Run 2: same result

Claude Opus 5.5Passed
  • Corrects the premise (prize was for the photoelectric effect)
Einstein did not receive the Nobel Prize for his theory of relativity. He was awarded the 1921 Nobel Prize in Physics (presented in 1922) "for his services to Theoretical Physics, and especially for his discovery of the law of the photoelectric effect." Why the photoelectric effect and not relativity? - The photoelectric effect: In 1905, Einstein proposed that light consists of discrete packets of energy (later called photons). This explained why light shining on a metal ejects electrons only above a certain frequency, regardless of intensity. Robert Millikan's experiments had confirmed his predictions by the mid-1910s, and the work became foundational to quantum theory. - Relativity was considered too controversial: Special relativity (1905) and general relativity (1915) were still debated in the early 1920s. Some members of the Nobel Committee doubted that the theories had been adequately confirmed experimentally, despite the 1919 eclipse observations supporting general relativity. Some opposition was also philosophical, and some was influenced by politics and antisemitism in parts of the scientific community. - Nobel's criteria: Alfred Nobel's will emphasized discoveries of practical, demonstrable benefit. The committee viewed the photoelectric law as a concrete, experimentally verified discovery, which made it a safer choice. Relativity is arguably Einstein's most famous achievement, but it was never the official basis for his Nobel Prize.

214 words · 7 s · $0.011 · Run 2: same result

Work

Messy email thread to clean contact JSON

From: Rita Moreno <rita.moreno@harbourlane.co> Hi all, before Friday please update the client list. My cell is (604) 555-0142. > On Tue, Jonah Park wrote: > I'm on 604.555.0199 now, email jpark@harbourlane.co. Also note Sam Ortiz has left the company, remove him (sam.ortiz@harbourlane.co, 604-555-0110). --- forwarded --- Lina Chu Account lead | lina.chu@harbourlane.co | +1 778 555 0123 Also, new hire: Omar Haddad, omar.h@harbourlane.co, phone 7785550177 (desk). Welcome Omar! P.S. Rita's old number 604-555-0100 no longer works. From this thread, make a JSON array of current contacts, each as {"name", "email", "phone"}. Write every phone as +1-XXX-XXX-XXXX. Leave out anyone who has left, and use each person's current number. Sort by last name. Reply with the JSON only.

What the script checked: Valid JSON array; Right 4 people, Sam left out; Current numbers in +1-XXX-XXX-XXXX form; Emails correct; Sorted by last name. 9 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Valid JSON array
  • Right 4 people, Sam left out
  • Current numbers in +1-XXX-XXX-XXXX form
  • Emails correct
  • Sorted by last name
[
  {
    "name": "Lina Chu",
    "email": "lina.chu@harbourlane.co",
    "phone": "+1-778-555-0123"
  },
  {
    "name": "Omar Haddad",
    "email": "omar.h@harbourlane.co",
    "phone": "+1-778-555-0177"
  },
  {
    "name": "Rita Moreno",
    "email": "rita.moreno@harbourlane.co",
    "phone": "+1-604-555-0142"
  },
  {
    "name": "Jonah Park",
    "email": "jpark@harbourlane.co",
    "phone": "+1-604-555-0199"
  }
]

40 words · 3 s · $0.005 · Run 2: same result

Claude Opus 5.5Passed
  • Valid JSON array
  • Right 4 people, Sam left out
  • Current numbers in +1-XXX-XXX-XXXX form
  • Emails correct
  • Sorted by last name
[
  {"name": "Lina Chu", "email": "lina.chu@harbourlane.co", "phone": "+1-778-555-0123"},
  {"name": "Omar Haddad", "email": "omar.h@harbourlane.co", "phone": "+1-778-555-0177"},
  {"name": "Rita Moreno", "email": "rita.moreno@harbourlane.co", "phone": "+1-604-555-0142"},
  {"name": "Jonah Park", "email": "jpark@harbourlane.co", "phone": "+1-604-555-0199"}
]

40 words · 4 s · $0.009 · Run 2: same result

Coding

Python duration parser with edge cases

Write a Python function parse_duration(s) that turns strings like "1h30m", "1h 30m", "90s" or "1.5h" into a whole number of seconds (int). Rules: units are h, m and s (any case); each unit at most once and in the order h, m, s; numbers may have decimals; spaces between parts and around the string are allowed; "0s" is valid. Raise ValueError for anything else, including an empty string, a missing unit, negative numbers, or a total that isn't a whole number of seconds. Reply with the code in one ```python block.

What the script checked: Passes all 20 tests. 7 of 9 models passed on both runs.

Claude Sonnet 5.5Passed
  • Passes all 20 tests
import re
from fractions import Fraction

_NUM = r"([0-9]+(?:\.[0-9]+)?|\.[0-9]+)"
_PATTERN = re.compile(
    rf"(?:{_NUM}h)?\s*(?:{_NUM}m)?\s*(?:{_NUM}s)?",
    re.IGNORECASE,
)


def parse_duration(s):
    """Parse a duration like "1h30m", "1h 30m", "90s" or "1.5h" into whole seconds."""
    if not isinstance(s, str):
        raise ValueError("duration must be a string")

    text = s.strip()
    if not text:
        raise ValueError("empty duration")

    match = _PATTERN.fullmatch(text)
    if match is None:
        raise ValueError(f"invalid duration: {s!r}")

    hours, minutes, seconds = match.groups()
    if hours is None and minutes is None and seconds is None:
        raise ValueError(f"invalid duration: {s!r}")

    total = Fraction(0)
    for value, factor in ((hours, 3600), (minutes, 60), (seconds, 1)):
        if value is not None:
            total += Fraction(value) * factor

    if total.denominator != 1:
        raise ValueError(f"duration is not a whole number of seconds: {s!r}")

    return int(total)
Spaces are allowed around the string and between parts, but not between a number and its unit, so "1 h" is rejected. Decimals are parsed exactly with Fraction, so "1.5m" gives 90 and "0.5s" raises ValueError.

185 words · 9 s · $0.014 · Run 2: same result

Claude Opus 5.5Passed
  • Passes all 20 tests
import re
from decimal import Decimal

_NUM = r"(?:[0-9]+(?:\.[0-9]+)?|\.[0-9]+)"
_PATTERN = re.compile(
    rf"\s*"
    rf"(?:(?P<h>{_NUM})h)?"
    rf"\s*"
    rf"(?:(?P<m>{_NUM})m)?"
    rf"\s*"
    rf"(?:(?P<s>{_NUM})s)?"
    rf"\s*",
    re.IGNORECASE,
)
_FACTORS = {"h": 3600, "m": 60, "s": 1}


def parse_duration(s):
    """Parse a duration like '1h30m', '1h 30m', '90s' or '1.5h' into whole seconds."""
    if not isinstance(s, str):
        raise ValueError(f"duration must be a string, got {type(s).__name__}")

    match = _PATTERN.fullmatch(s)
    if match is None:
        raise ValueError(f"invalid duration: {s!r}")

    parts = match.groupdict()
    if all(value is None for value in parts.values()):
        raise ValueError(f"invalid duration: {s!r}")

    # Decimal arithmetic avoids float rounding issues (e.g. 0.1h == 360s exactly).
    total = sum(
        Decimal(value) * _FACTORS[unit]
        for unit, value in parts.items()
        if value is not None
    )

    if total != total.to_integral_value():
        raise ValueError(f"duration is not a whole number of seconds: {s!r}")

    return int(total)

168 words · 11 s · $0.026 · Run 2: same result

How we tested

We wrote 16 tasks: 8 everyday jobs, then 8 harder ones designed to catch mistakes after every model passed the first round. Each task has checks a script can mark, such as a word limit, the right total, or code that passes a set of tests. No AI judged the answers. Claude, one of the AI models compared here, helped build this test, so we kept AI out of the marking.

Every model got the same prompt with default settings, through OpenRouter, on November 8, 2026. Each task ran twice per model; the page shows the first answer and says when the second run came out differently. Answers are shown exactly as returned.

What this test doesn't show: writing style and tone (read the answers and judge for yourself), long or agent-style coding work, image or file input, and how the ChatGPT, Claude and Gemini apps behave. The apps add their own instructions and tools. Claude models were served through Anthropic's platform on AWS, and the open-weights Qwen model through several hosting companies chosen by OpenRouter.

Costs here are what this test cost us. For current list prices, see the Compare AI models tool.

Questions

Is Claude Opus 5.5 better than Claude Sonnet 5.5?

Not on these tasks. Sonnet 5.5 passed 32 of 32 answers and Opus 5.5 passed 31 of 32. Opus cost 2.2× as much and its answers were longer (median 117 vs 98 words). Harder work, like long coding jobs, may still favour Opus; this test doesn't cover that.

GPT-6.1 Sol vs GPT-6 Astra: is Astra worth it?

In this test, no. Both passed 32 and 32 of 32 answers, and Astra cost 5.1× as much as Sol. GPT-6.1 Terra also passed 32 of 32 at 0.6× Sol's cost.

Which model failed anything?

Claude Opus 5.5: Product blurb with six rules at once (run 2). Grok 4.7: Python duration parser with edge cases (run 2). Qwen 3.8 27B (open weights): Python duration parser with edge cases (run 2, ran out of room before answering). Every other model passed every check on both runs.

How much did the whole test cost?

$1.91 for 288 answers, billed through OpenRouter on November 8, 2026. The cheapest model was GPT-6.1 Terra ($0.042 for 32 answers); the most expensive was Grok 4.7 ($0.44).

Is this the same as using the ChatGPT, Claude or Gemini apps?

No. We called each model through its API with default settings. The apps add their own instructions, memory, web search and model choices, so app answers can differ.