AI News

GPT-6.2 Sol passed all 15 coding trials after a prompt rewrite

Editing an earlier requirement caused OpenAI’s API to drop a reasoning item in every one of our 15 coding trials. Appending the same requirement to the conversation caused zero removals. Both workflows produced 15 patches that passed every test.

The distinction matters if you build software around GPT-6.2 Sol. OpenAI gives developers a way to see when a history edit discards earlier work. On these three small tasks, we could see exactly when that happened. The test suite recorded a tie.

For each of three tasks, all five rewrites and zero appends removed an earlier reasoning item. All five patches per task passed in both workflows.

Each circle represents one scored trial. Source: Kingy.ai’s November 18, 2026 Responses API experiment. Run-level data.

Two half-cent invoice lines

Take two invoice lines worth half a cent each. Round each line to the nearest cent using half-up rounding, and the invoice comes to $0.02. Add the lines first and round the total, and it comes to $0.01. Both answers make sense. Only one follows the latest instruction.

We gave GPT-6.2 Sol a small Python invoice module and asked it to plan for line-by-line rounding. After it had explained its approach, we changed the rule to rounding the total. In one copy of the conversation, we added the new rule at the end. In the other, we edited the earlier rule in place. Both copies kept the model’s original written plan. In both, we ended with the same request to implement the current requirement.

This is a choice developers make when their software sends a conversation back to a model. An application might keep every instruction in order, or update a saved specification whenever a user changes their mind. OpenAI says its Responses API checks the history preceding a reasoning item. Editing that history can make the item ineligible for the next response. The API can report the removal through a diagnostic called prefix_mismatch. OpenAI’s integration guide

How we tested the two workflows

We used three fictional modules. The invoice changed its rounding rule. A CSV importer switched from keeping the first row for each ID to keeping the newest timestamp, with the earlier row winning a tie. A booking scheduler switched from assigning a booking to its UTC date to using the calendar date at the venue.

For each module we generated five fresh plans. We forked each plan, including its visible text and any opaque reasoning item returned by the API, into the two workflows. We used gpt-6.2-sol at medium reasoning on the default service tier throughout and randomized which branch ran first. The model saw the same starting file and returned a complete replacement module in one call. We ran the code locally against four tests for the changed rule and four for behavior that should survive the change. The model never saw the grader or its results.

The first three-pair pilot exposed an easy way to misread this experiment. All three short planning replies used zero reasoning tokens and returned no reasoning item. The API had nothing to remove, and all six patches passed. For the second pilot, we asked Sol to work through a concrete edge case before giving its plan. All three revised plans produced a reasoning item. Each rewrite triggered a removal and each appended update preserved eligibility. We used those planning prompts for all 15 scored pairs and kept both pilots out of the score.

What changed in the code?

In the first scored CSV pair, the original plan called for a set of IDs the importer had already seen. That would satisfy the first-row rule. It would also discard a newer record for the same ID. Once we changed the rule, both branches returned the same comparison.

if row_id not in selected or row["stamp"] > selected[row_id]["stamp"]:

The strict greater-than comparison matters. A newer timestamp replaces the saved row. An equal timestamp leaves the earlier row in place, as requested. The two branches independently returned the same complete module on that trial. Across all 15 pairs, only two pairs had identical visible submissions. The other 13 produced different source text that passed the same checks. Saved plans and submissions

Scored result Append the update Rewrite the earlier rule
Modules passing the full suite 15/15 15/15
Revised-rule checks passed 60/60 60/60
Preserved-behavior checks passed 60/60 60/60
Branches with an earlier reasoning item removed 0/15 15/15
Median implementation response time 2.30 seconds 1.90 seconds
Cost of the 15 implementation responses, from API usage US$0.053030 US$0.045498

We matched each reported removal to the ID of a reasoning item in that pair’s planning response. The original written plan remained visible even when the API removed the opaque item. Sol could still read the module, the revised rule and its earlier explanation. The diagnostic tells us which item the API removed. It cannot tell us how much the model relied on an eligible item when writing code. OpenAI’s reasoning documentation

The difference in cost was less than a cent

Across the 15 implementation calls, the rewrite branches consumed 9,564 input tokens and 2,637 output tokens. The append branches consumed 11,485 input and 3,006 output. At Sol’s Standard rates of US$2 per million input tokens and US$10 per million output tokens, the difference came to US$0.007532. OpenAI reported zero cache reads and zero cache writes on the scored requests. Reusing a long cached prompt can change the economics. OpenAI pricing · Prompt caching

The 15 shared planning calls added US$0.073820, bringing the scored study to US$0.172348. Both pilots added US$0.053144. OpenAI’s cost export for the experiment key came to US$0.225492 for all 63 model responses, matching our token accounting exactly. Billing receipt

Rewriting finished sooner in nine of the 15 pairs. The median within-pair advantage was 0.20 seconds. We timed the implementation request from the client and excluded planning, token counting and local tests. The randomized order put rewrite first in 11 pairs. A fraction of a second on short network requests is too little evidence for a production speed claim.

What this means for a coding agent

These were three distinct problems, each repeated five times. All were small enough for Sol to read the entire module and return a replacement in one response. Both workflows hit the ceiling of our tests. A change that hurts work across several files or a long sequence of tool calls could easily escape this study. Only the appended update came after the model’s old plan. The measured outcomes belong to those two complete workflows.

If you maintain a coding agent, pick a task from your own regression suite where a requirement changed after several tool calls. Look for prefix_mismatch in the next response’s input_transformations, then check the resulting patch against that task’s acceptance tests. In our sample, the first check changed in every pair and the second changed in none. Experiment method and fixtures (ZIP)

Receipt2026-11-18
Tested
Three fictional Python coding tasks, five paired repeats per task, 30 scored submissions. Two three-pair pilots preceded scoring.
Runner
Direct OpenAI Responses API. Each branch returned a complete replacement Python module. Tests ran locally under Python 3.14.
Cloud side
gpt-6.2-sol, medium reasoning, default service tier.
Settings
Same starting code and tests within each task. Full initial response replayed into both branches. Seeded random order. No model-callable tools.
Grading
Four changed-rule and four preserved-behavior checks per task. Passing required all eight. Every saved submission was regraded.
Cost
OpenAI billed US$0.225492 for all 63 responses, including both pilots and shared planning. The billing export exactly matched the token records.
Dates
run 2026-11-18 · checked 2026-11-18

Sources and downloadable evidence

Download the experiment evidence package (ZIP). It includes the prompts, starting code, tests, all 63 visible model responses, trial timings, chart and billing receipt.