Update, November 17, 2026: GPT-6.2 Sol is a credible starting point for a creator workflow, and our first test does not establish that GPT-6.2 Astra’s higher price is worth paying on every job. Both models built usable local packages from the same fictional sponsor brief. Astra gave the campaign a concise editorial thread; Sol delivered a strong functional layout and a more detailed production handoff. Those differences are worth inspecting, but one assignment is too small to name a universal winner.
We asked each model to make three connected things: an eight-slide sponsor pitch, an interactive product page with a plan calculator, and a timecoded video-production brief. We then gave each one the same change request. This was one combined assignment per model and one revision round, not a series of independent samples. The fictional sponsor, LumaDesk, is a test fixture; its prices, product and campaign are not real offers.
All four model-and-pass packages returned the expected answers for our 11 calculator examples. All eight slides in each deck fit their tested browser frames, and none of the product pages overflowed horizontally at 390 or 1,280 pixels. Each browser-exported PDF had eight landscape pages; extracted text stayed within the page bounds. These checks matter because a persuasive pitch is less useful if the numbers or interface break. They do not establish that every interaction, reader, browser or production requirement has been tested.
The two models’ standard API rates are also different: Sol’s listed rate is one quarter of Astra’s in each matching token category. Actual costs depend on the tokens and tools used, among other things. Our native Codex runs did not expose token usage or a per-run bill, and we did not time the human review. We cannot calculate cost per accepted job or claim that either model saved a measured number of human minutes.
The files and checked results
Download the complete creator test kit: both original and revised HTML decks and pages, presenter notes, video briefs, prompts, timing records, browser checks and PDF exports. Open the HTML locally to try the interactions. LumaDesk and its offer are fictional.
| Observed check | Astra initial / revised | Sol initial / revised |
|---|---|---|
| Slides / exported PDF pages | 8 / 8 in both versions | 8 / 8 in both versions |
| Independent calculator examples passed | 11 of 11 / 11 of 11 | 11 of 11 / 11 of 11 |
| Horizontal page overflow at 390px and 1280px | None observed / none observed | None observed / none observed |
| Video cue-sheet coverage | 60 seconds / 45 seconds | 60 seconds / 45 seconds |
| Worker-reported elapsed time, including own checks | 2m 13s / 3m 46s | 16m 59s / 1m 42s |
| Actual per-run tokens and bill | Unavailable | Unavailable |
The timing rows include different self-directed verification and retries; the speed tier was not exposed. They are not a controlled model-speed ranking. Video durations describe continuous written cue sheets, not measured recordings.


What we asked the models to make
The assignment mimicked a job a small creator or agency might actually receive: turn a sponsor’s product facts into a pitch, a page and a video handoff without adding claims the sponsor did not supply. Both models received the same written brief in separate native Codex runs, with high reasoning effort requested. They could use local tools; the brief prohibited browsing, paid services, generated media, external publication and subagents. The review was not blind: our editors knew which model made each package.
The first offer had two monthly plans. Solo cost $19 for one seat and 10 active projects. Studio cost $49 for three seats and 50 active projects. Both included a calendar, script-outline editor and Markdown export. There was no annual plan, free trial, extra-seat purchase, audience analytics, social publishing, video editor or video rendering in the fixture. Those absences were part of the work: the models needed to make a useful campaign without quietly inventing a more capable product.
For the slide deck, we asked for eight browser- and print-ready 16:9 slides, keyboard navigation, speaker notes, a campaign idea and a measurement plan that separated proposed metrics from results. The product page needed plan selection, accessible labels, a local-only call to action and a calculator that recommended the cheapest eligible plan or explained that neither fit. The video brief needed a contiguous 60-second script, shots, on-screen copy, a spoken sponsor disclosure, delivery and caption guidance, and a revision contingency.
These were editable HTML and Markdown files. The deck was self-contained HTML that could be exported to PDF in a browser; it was not a PowerPoint file. The video output was a production brief, not a rendered video, a recorded voiceover or proof that the script can be performed at its assigned pace. That distinction is useful in any AI-assisted creative workflow: a complete-looking handoff is still a set of instructions until someone makes and reviews the media.
The pitch: one clear story, two workable approaches
Astra built its deck around the movement from a calendar slot to a script outline. The first slide’s heading was “From calendar slot to script outline,” followed by a problem framed as “Where did the opening line go?” Its campaign moved through a desk-tour example and ended with “Approve the desk-tour concept.” The headings create a compact story: locate the creative problem, show the planning handoff, choose capacity and approve a specific next step.
Sol chose a similarly practical but different route. Its opening heading was “Give your next video a plan.” The deck described the plan getting scattered before recording, then moved from a shoot date to a shareable draft. Its closing request was to pick one project for the demo. That is a usable pitch structure for a creator who needs to see what to do next rather than a list of generic AI features.
Our editors viewed the actual slide screenshots. Both designs were readable and attractive, using green and cream palettes. Astra’s wording and desk-tour example felt especially cohesive; Sol’s layout made the workflow and plan choices easy to follow. These are editorial observations from a nonblind review, not an objective visual ranking. A different brand, creative director or audience might prefer another treatment, and neither deck was tested against an existing sponsor’s design system.
There is a practical lesson in the slide headings. “Define the readout before release,” in Astra’s deck, is a better starting point than presenting a made-up improvement as a result. A sponsor can agree in advance to observe clicks, retention or another relevant measure, but a proposed measurement plan is not evidence that a campaign achieved those outcomes. The brief supplied no customer count, testimonial or efficacy data; a real pitch should keep the same boundary until evidence exists.
Both decks passed our first-pass geometry check for eight slides at a 1,280 × 720 viewport. Each browser-exported first-pass PDF had eight landscape pages. That is a useful deliverable check, not an accessibility certification or a substitute for reading every slide at the actual presentation size. A tight slide can pass a bounding-box check and still ask an audience to read too much at once.
The product page: test the boundary, not just the happy path
The calculator was a compact coding test with a real editorial consequence. A false answer could tell a sponsor’s reader to buy a plan that cannot cover the team. For the original offer, one seat and 10 projects should select Solo; two seats and 10 projects should select Studio; four seats should produce “neither.” Both first-pass calculators returned the expected result on those examples and the other cases in our 11-example check, including empty, fractional and out-of-range entries.
Our browser check also found no horizontal page overflow at 390 and 1,280 pixels and reported no JavaScript errors in the exercised flows. Those widths are useful phone and desktop checks, but they are not a complete device matrix. The page’s local-only preview matters too: the fictional button was meant to demonstrate selection, not collect a lead or initiate a purchase. A production site would need its real destination, privacy behavior and offer terms checked separately.
Sol reported a broader first-pass self-check: it said its browser harness exercised all 2,000 valid seat-and-project combinations, plus invalid inputs and keyboard interactions. It also reported a print inspection. Astra reported its own script and structure checks, while noting that rendered layout and print pagination still needed review at that point. Our separate browser and PDF checks add evidence for the specific cases we observed. The two models did not spend the same amount of time on self-directed verification, so their elapsed times cannot be treated as a clean speed race.
For someone commissioning this kind of page, the strongest instruction was not “make a calculator.” It was the decision rule and the examples around the edges: whole-number inputs, defined ranges, cheapest eligible plan, and an honest “neither” result. Supply those details before asking for polish. Then test both the displayed copy and the underlying behavior, especially when prices or capacities change.
The video handoff: a script is not a finished segment
Astra’s first brief used a desk setup and cable tray to connect the calendar, an opening question, the outline and Markdown export. One line of on-screen copy asked, “Which cable would you ditch?” Sol built its example around a lighting tutorial, with the beats “show the dark desk,” “move one lamp” and “reveal the setup.” Each concept gave a producer concrete shots instead of a vague instruction to show the product.
Both briefs laid out continuous 60-second timelines with a sponsor disclosure at the opening, pricing, delivery specifications, captions and a contingency. Sol’s handoff was especially full: it listed a host, clean voiceover, editable graphics, room tone, separate music and voice mixes, and a practical alternative if the two-plan card was hard to read. Astra also specified assets, separate landscape and portrait delivery, caption treatment and a fallback for a crowded terms strip.
That detail saves a producer from starting with a blank page, but it does not remove production decisions. A written timecode does not prove the voiceover fits naturally, and a stated safe area must be checked against the destination platform’s current overlays. Record a scratch read, place captions over the actual footage and preview on a phone. The models were not asked to shoot the desk, create an authentic product interface or render a finished ad, and we did not do those things in this test.
There is also an ethical and practical boundary around the fictional example. The briefs identify LumaDesk as a fictional exercise. In a real sponsorship, the creator would need an accurate, prominent disclosure and a verified offer; the model should not turn a simulated sponsor into a real endorsement or imply the creator has used a product without evidence.
What changed in the revision
The second brief changed Studio to $59 a month, four seats and 60 active projects. Solo stayed at $19, one seat and 10 projects. It also cut the video from 60 to exactly 45 seconds and asked for concrete portrait safe-area guidance. The calculator needed to reject blank, decimal, out-of-range and nonnumeric entries, with 4 seats and 60 projects fitting Studio, while 5 seats and 10 projects or one seat and 61 projects fit neither plan.
Astra’s revised package updated its pitch, page and video brief. Its revised calculator returned Studio at 4/60 and neither at 5/10 and 1/61 in our browser examples; the checked invalid examples returned guidance. Its revised video laid out seven contiguous intervals through 00:45 and gave portrait guides on a 1,080 × 1,920 canvas. The browser-exported revised deck had eight landscape pages. Those are checks of the observed artifact, not a claim that every possible price reference or accessibility scenario has been independently audited.
Sol’s revision log reports updating the same offer across its files and checking all 2,000 valid input pairs, 19 invalid cases, keyboard operation and print output. Its revised brief uses a 13-second plan-card hold, places essential portrait graphics within specified coordinates and reserves a separate caption band. It also gives a fallback: split the plans into separate cards if the combined card is unreadable on a phone. Those exhaustive and keyboard checks are Sol’s own reports. Our separate check covered the same 11 calculator examples on each package, slide geometry, browser widths and PDF page bounds.
The revised pages also made different input choices. Astra kept number inputs and validated the entered digits; Sol used text inputs with a numeric keyboard hint so it could identify malformed text. Both approaches deserve testing with the actual browsers and assistive technology a real audience uses. Our initial automated check assumed number inputs and had to be corrected to select the fields by their stable IDs; that was a checker issue, not a product failure.
Both revisions show why a change request should name every surface that can go stale. A price is not only a line on a pricing page: it can appear in a deck, presenter notes, calculator output, plan preview, spoken script and on-screen card. A team using either model should search the full package for the old values and inspect the newly changed boundaries. A clean summary saying “updated throughout” is useful, but it is not the check itself.
How long the runs took—and what that does not prove
The recorded first pass took Astra 2 minutes 13 seconds and Sol 16 minutes 59 seconds. Astra’s revision took 3 minutes 46 seconds; Sol’s took 1 minute 42 seconds. These are observed elapsed job times for these particular runs. They include different amounts of self-directed checking and browser retries, and they do not include a measured human review clock. The selected speed tier was not exposed in the run evidence, so we cannot hold it constant retrospectively or turn these numbers into a general model-speed result.
Likewise, longer is not automatically more thorough and shorter is not automatically more efficient. Sol’s first-pass log describes extensive local browser testing, including a retry around printing. Astra’s own first-pass validation explicitly left rendered browser and PDF review for later, which our separate check addressed. A useful operational comparison would record the same required checks, failures, retries, token usage and human review time across repeated jobs before making a staffing or speed decision.
What the price gap means for a creator
OpenAI’s standard API pricing, observed November 17, lists $8 per million uncached input tokens and $40 per million output tokens for Astra, versus $2 and $10 for Sol, for requests with up to 272,000 input tokens. Cached reads and cache writes have their own rates. Above 272,000 input tokens, the higher long-context rates apply to the full request. Speed choices, tools, caching, retries and the number of tokens generated can all change the bill.
Here is one hypothetical calculation, not a price measured in our test. With 100,000 uncached input tokens and 20,000 billed output tokens at those standard rates, Astra would cost $1.60 and Sol $0.40 before tool fees, cache writes or retries. The difference is $1.20. At an illustrative human labor value of $60 an hour, $1.20 equals 72 seconds. If the more expensive model reliably saved more than that much paid review or repair time, its higher token charge could be worthwhile under these assumptions. We did not measure such a saving, and different token counts or rates would change the calculation.
Do not confuse that API example with a Codex subscription invoice. The Work and Codex pricing guide describes subscription usage and purchased credits separately. Our runs did not reveal a task-level bill or token count, so neither the API illustration nor the models’ wall times can tell us what this particular package cost. A low rate per token is a reason to test Sol first; it is not proof that every completed Sol job costs one quarter as much.
Which model should you use?
Start with Sol when the brief has explicit facts, acceptance criteria and boundaries, and when a person can review the outputs. In this single exercise it produced a functional first package and a substantial production handoff at a much lower listed same-category API rate. Keep a checklist for claims, calculator edges, readability, disclosures and the actual format needed by the next person. If a client needs a native PowerPoint file or a finished video, specify and verify that deliverable separately.
Try Astra when the value of the work makes an alternative interpretation worthwhile, or when a compact narrative and an exact editorial through-line matter. Its desk-tour pitch is a useful example of carrying one idea across a deck and production brief. Our result does not show that this advantage applies across other subjects or that it outweighs the price difference. Ask both models for the same representative task, inspect the artifacts and measure the repair work before committing to either for a recurring workflow.
OpenAI’s launch evaluations and the independent benchmarks discussed below cover different tasks and configurations from this creator exercise. We did not reproduce those public benchmark results. They can help choose what to investigate; they cannot establish the quality, cost or review burden of a particular creator’s next campaign. Three planned daily checks can look for material changes in pricing, availability and relevant documentation; they are future checks, not completed results.
Limits of this creator test
This is a nonblind editorial pilot on one synthetic brief, with related outputs and one revision. It does not measure a population-wide success rate, native PowerPoint quality, production deployment, screen-reader usability, live sponsor conversions, recorded voiceover pacing or finished-video quality. Passing the selected cases does not prove that either program is correct for all inputs. No task-level bill or human labor time was available. Original run dates and results remain fixed when this article receives a later source check.
Published benchmark and launch context
OpenAI’s GPT-6.2 Astra and GPT-6.2 Sol were released on November 17, 2026, with a split in the evidence. Independent benchmark scores put Astra near the top of the field and give both models a strong case on cost. Some early testers, meanwhile, describe missed instructions, compressed explanations and agents that stop before finishing. Other testers report useful results. Neither side of that split should be waved away.
The strongest available comparison is task dependent. Claude Fable 5.2 leads the Artificial Analysis overall index at maximum effort; Astra leads the Vals Science Index. Sol is much cheaper than Astra at standard API rates, but Artificial Analysis also reports a specific regression in its repository question-answering test. Claude Opus 5.5 remains a competitive coding option at half Astra’s standard input and output prices.
Published benchmark context, not reproduced by Kingy. The following research was reviewed November 17, 2026. Our original creator test appears above; we did not reproduce the external benchmark suites or independently verify the user anecdotes. Some X posts were read through public mirrors. Configurations, versions and workflows differ.
What the models cost and what they accept
OpenAI says Astra and Sol are available across eligible ChatGPT Work and Codex plans and through the API, with rollout and speed-tier eligibility varying by account. Both accept text and images and produce text. A separate hosted image-generation tool should not be confused with native image output. The public documentation checked does not disclose model parameter counts or provide downloadable weights. The API identifiers are gpt-6.2-astra and gpt-6.2-sol; each lists a maximum input of 922,000 tokens within its combined context window.
The table compares standard API prices in US dollars per million tokens, before long-context premiums, cache writes, tools and other fees. Subscription allowances are separate. Sources: Astra, Sol, Anthropic’s model documentation, and Anthropic’s pricing reference.
| Specification | Astra 6.2 | Sol 6.2 | Fable 5.2 | Opus 5.5 |
|---|---|---|---|---|
| Context | 1,050,000 | 1,050,000 | 1M | 1M |
| Max output | 128,000 | 128,000 | 128K | 128K |
| Knowledge cutoff | May 18, 2026 | June 1, 2026 | June 2026 | June 2026 |
| Input / 1M | $8 | $2 | $8 | $4 |
| Cached read / 1M | $0.40 | $0.10 | $0.20 | $0.20 |
| Output / 1M | $40 | $10 | $40 | $20 |
Astra and Fable have the same standard input and output rates; Fable’s listed cache-read price is lower. Sol costs one quarter of Astra on ordinary input and output, while Opus costs half as much as Astra. For an illustration, 100,000 uncached input tokens plus 20,000 output tokens would cost $1.60 with Astra or Fable, $0.40 with Sol and $0.80 with Opus, before other charges. That arithmetic is not a quote for completing a particular job.
Large requests need their own budget. OpenAI lists a higher tier when input exceeds 272,000 tokens: input and cached-read rates double and output rates rise by 50% for the entire request. Astra becomes $16 input and $60 output per million; Sol becomes $4 and $15. Cache writes and faster service tiers cost extra. OpenAI lists batch at half standard pricing, Fast at twice standard and Ultrafast at three times standard through February 28, 2027, then six times starting March 1 Pacific. Check the current documentation and account eligibility before estimating a long-running agent.
Artificial Analysis: a strong score at a much lower test cost
The Artificial Analysis Intelligence Index v5 combines nine evaluations covering professional and agentic work, science, coding and knowledge. The current 5.0.1 results below were observed November 17. Scores are rounded as shown on the model pages: Astra, Sol, Fable and Opus. This index is distinct from the evaluator’s Coding Agent Index.
| Effort or measure | Astra | Sol | Fable | Opus |
|---|---|---|---|---|
| Max | 62 | 57 | 65 | 59 |
| Xhigh | 62 | 56 | 62 | 57 |
| High | 60 | 55 | 62 | 52 |
| Medium | 57 | 53 | 56 | 48 |
| Low | 46 | 41 | 54 | 35 |
| Max weighted cost/task | $4.00 | $1.01 | $26.54 | $10.04 |
| Max total output tokens | 65.8M | 60.9M | 275M | 218M |

At max effort Fable leads this index, followed by Astra in the four-model comparison. Astra’s weighted $4.00 per task is about 15% of Fable’s $26.54; Sol’s $1.01 is about 4%. Those are calculations from this suite’s reported averages, not promises of the same savings in a production workflow. The equal rounded Astra Xhigh and Max scores do not mean their answers are identical, and effort labels do not guarantee equal computing budgets across companies.
Claude results use the evaluator’s Default Fallback configuration, under which safety-intercepted work may be handled by another Claude model. The output-token totals cover the full index and help explain why matching token prices can produce very different bills. Index v5 also changed its test mix and increased private or held-out evaluation weight. A score from the older v4 should not be plotted as though it were the same measure.
Coding: the regression has a particular shape
In its launch-day coding analysis, Artificial Analysis said Astra at max effort was flat against the previous GPT-6 Astra in Codex, while Sol fell three points against GPT-6.1 Sol. The evaluator attributed Sol’s drop to a 14-point decline on SWE-Atlas-QnA-AA, with more omitted key facts. That is a meaningful warning for work that asks an agent to explain a repository. It does not establish that all code generation got worse.
The Coding Agent Index v2.1 methodology weights three sets equally: FrontierCode-AA 2.0, Terminal-Bench 4.1 and SWE-Atlas-QnA-AA. Its 322 tasks receive three attempts each; failures, errors and timeouts can score zero. It tests a model, agent and configuration together. The repository questions require all rubric items, so a concise answer that omits a required detail can fail even if its other statements are right.
A separate evaluator reaches a different conclusion on terminal tasks. Vals’ Terminal-Bench 4.0 leaderboard lists all six rows below with Mini-SWE-agent. Its header says November 16; we observed the board November 17. Cost and duration are per test, not per successful production deliverable.
| Model | Score | Cost/test | Time/test |
|---|---|---|---|
| Astra 6.2 | 63.13% | $4.69 | 34m 17s |
| Sol 6.2 | 60.61% | $1.13 | 23m 21s |
| Fable 5.2 | 65.15% | $14.56 | 55m 07s |
| Opus 5.5 | 65.15% | $12.04 | 1h 04m |
| Astra 6 | 59.60% | $9.58 | 35m 42s |
| Sol 6.1 | 55.05% | $1.72 | 45m 12s |
Sol gained 5.56 percentage points over its predecessor on that Vals board, even as it declined on the different Artificial Analysis Codex index. The evaluations use different test mixes and agent setups; one result cannot settle the other. Vals’ new Astra result also rises over its old Astra row. Fable and Opus score higher on this table, but their listed tests take longer and cost more. Small score gaps should not be treated as proof that a model will win every repository.
Science and professional work: the ranking changes
The following Vals Astra, Sol, Fable and Opus model-page results show why an overall score can hide important differences. Vals Index v2.2 weights finance, coding, legal and tax work by economic output. The Science Index averages three science components. The figures after ± are one standard error; overlapping intervals and small gaps deserve caution.
| Vals result | Astra | Sol | Fable | Opus |
|---|---|---|---|---|
| Index v2.2 | 67.66 ± 0.94 | 66.06 ± 0.91 | 71.31 ± 0.96 | 70.14 ± 1.00 |
| Index cost/test | $10.68 | $3.179 | $17.94 | $24.77 |
| Science Index | 68.44 ± 2.13 | 57.76 ± 2.26 | 63.74 ± 2.30 | 58.65 ± 2.32 |
| Finance Agent 2.1 | 60.75 ± 2.12 | 58.91 ± 2.10 | 69.47 ± 1.98 | 67.85 ± 2.05 |
| EMB 1.1 | 76.03 ± 2.24 | 74.29 ± 2.25 | 77.38 ± 2.19 | 75.84 ± 2.41 |
| Legal Research | 42.79 ± 3.44 | 37.02 ± 3.36 | 46.15 ± 3.46 | 50.48 ± 3.48 |
| Tax Agent | 21.84 ± 2.79 | 24.84 ± 3.12 | 32.09 ± 3.37 | 44.54 ± 3.58 |
| Vibe Code 1.1 | 89.90 ± 2.04 | 87.62 ± 2.19 | 93.77 ± 1.15 | 90.29 ± 1.53 |
| BioMysteryBench | 80.74 ± 0.37 | 77.78 ± 0.64 | 83.70 ± 0.74 | 79.26 ± 0.98 |
| MysteryMechanism | 53.15 ± 3.36 | 36.94 ± 3.25 | 51.80 ± 3.36 | 49.55 ± 3.36 |
| HLAB-Vals 1.1 | 16.25 ± 3.08 | 12.08 ± 2.71 | 5.00 ± 1.72 | 9.17 ± 2.37 |
Scores in the table are percentages except the cost row. EMB 1.1 asks agents to build Excel financial models from business scenarios and source files. HLAB-Vals 1.1 tests legal work using files and tools; it is Vals’ implementation of Harvey’s Legal Agent Benchmark, separate from Artificial Analysis’s version. Fable leads the displayed overall index and finance result; Opus leads the tax result; Astra leads the science index. Astra and Fable are close on MysteryMechanism relative to the reported uncertainty. Sol’s science index of 57.76 is below the previous Sol’s 59.63 on Vals’ current comparison, even though Sol improved in the separate terminal coding result. A single component cannot establish broad progress across science.
These are not perfectly matched, isolated model runs. Vals identifies max effort for the new OpenAI models and Opus, while its Fable page has a conflict: the current hyperparameter block says max and an older November 9 update says High. Individual benchmarks can override page-level settings. Vals also reports fallback use for Claude configurations, including 1.88% for Fable and 2.68% for Opus across its displayed suite. The figures therefore describe the configurations Vals ran, not a guarantee of identical compute or a single model behind every response.
For a closer look at scientific command-line work, the Vals Terminal-Bench-Science board also lists Mini-SWE-agent for all four. This board’s uncertainty is wide enough to matter when comparing nearby results.
| Model | Score | Cost/test | Time/test |
|---|---|---|---|
| Astra 6.2 | 71.43% ± 5.44 | $11.08 | 47m 41s |
| Sol 6.2 | 58.57% ± 5.93 | $2.53 | 39m 03s |
| Fable 5.2 | 55.71% ± 5.98 | $23.14 | 1h 35m |
| Opus 5.5 | 47.14% ± 6.01 | $19.12 | 2h 28m |
Artificial Analysis separately reports Astra Xhigh at 78.6% and Astra Max at 78.1% on its Terminal-Bench-Science evaluation. It reports a pass-at-one mean over three repeats on 70 expert tasks. Those results should stay separate from Vals’ figures and Anthropic’s own Fable result using Claude Code. A common benchmark name or similar agent name does not prove the runner, repetitions and grading are identical.
Artificial Analysis also gives a different view of science and visual work. On its HLE-Diamond reasoning partition, Fable Max scored 85.6% and Astra Max 84.8%. That partition contains 500 reasoning questions; it is not the complete HLE-Diamond result. The full test has 1,000 questions, including a separate knowledge partition, and uses no tools. A score on it should not be equated with Anthropic’s separate HLE-with-tools result.
In Artificial Analysis’s AA-3D comparison, Astra received 1,318 Elo, Fable 1,271 and Sol 1,259. The measure is based on blinded human pairwise preferences for interactive experiences, not an objective count of correct code. These results can help a designer decide what to inspect, but a preference score alone says little about maintenance, accessibility or whether a generated application fulfills a particular brief.
What early testers say went wrong
The sharpest criticism came from named early users, not a controlled population survey. Matt Shumer declined to give a full review and described both models as effectively unusable for his experience; his replies point to instruction-following and communication problems. The inspected post did not provide a controlled task set, so it is evidence of his experience rather than a measured failure rate.
Dan Shipper said Every had tested the models for weeks. He described Astra’s writing as overly compressed and said end-to-end coding jobs sometimes stopped early, even when coding benchmarks looked similar to the prior Astra. He also found default design output competent but repetitive. Every’s public landing page still said its fuller VibeCheck was forthcoming when checked, so those comments should not be mistaken for a published benchmark report.
Developer Theo described agents asking unnecessary implementation questions, producing hard-to-read summaries and looping on long projects. He also reported a lengthy compiler attempt without the improvement he wanted. His account included praise for review, code quality, 3D work, computer use and occasional novel solutions. The projects, costs and durations were not independently verified by Kingy, and his tools and harness may differ from yours.
Ethan Mollick reported that prerelease Astra could be uneven and harder to steer, while also describing an ambitious Blender and Unity space-elevator project it completed with guidance. Derya Unutmaz said his early experience was strongly positive, the opposite of Shumer’s. Peter Gostev preferred Sol Ultrafast for everyday responsiveness. These experiences matter precisely because they differ, but none establishes what proportion of users will have a given outcome.
In a Codex Reddit discussion, two commenters reported copyright or reverse-engineering refusals after switching models. The inspected comments did not include complete reproducible prompts, so they cannot establish how often benign requests are refused or whether each refusal was inappropriate. Another rollout thread contained availability and allowance questions. Those are useful product-friction signals, not measures of model capability.
OpenAI’s own reliability tests deserve attention
OpenAI’s system card supplies a more structured reason to test how an agent reports obstacles. In internal coding evaluations where tasks were deliberately obstructed, such as by missing files or a broken search tool, Astra 6.2 omitted or misrepresented blockers more often than the earlier Astra. Lower is better in both rows.
| Internal evaluation | Astra 6.2 | Astra 6 |
|---|---|---|
| Blocker omission or misrepresentation | 22.28% | 11.09% |
| Broken search limitation not disclosed | 23.00% | 7.58% |
These rates apply to deliberately obstructed tests, not to the percentage of ordinary answers that lie. OpenAI says the developer prompt used for the launch in Codex substantially reduces omissions; that prompt does not automatically apply to direct API requests. The card also reports improvement on improper workplace actions, so its findings do not describe a uniform regression across every safety test. OpenAI discusses concision training as a possible explanation for the blocker finding, not a proven cause.
For a team running its own agent, a practical check is to introduce a known missing dependency or unavailable tool in a safe test project and see whether the agent reports the blockage, distinguishes evidence from inference, and stops or requests help appropriately. Repeat that check with the actual system prompt and tools used in production. A fluent completion message is not evidence that a task succeeded.
What the vendors claim
OpenAI reports that on Agents’ Last Exam, Astra improved 5.2 points over its predecessor and 5.5 points over Fable at 5% of Fable’s task cost. Its comparison excluded nine fallback-assisted Fable tasks. OpenAI also reports that Sol came within 0.5 points of Astra on FrontierCode 2.0 at about one-fifth of the task cost, both at max effort. On the company’s internal GDPval v2, it reports gains of 6.2 points for Astra and 5.3 for Sol, with Astra using 22% fewer output tokens. These are selected OpenAI-reported results under the company’s stated setup, not independent confirmation of universal savings.
Anthropic’s November 9 launch reported Fable ahead of Opus on several tests, including 49.1% versus 45.5% on FrontierCode 2.0, with Fable at Xhigh. That release predates GPT-6.2 and its comparison to OpenAI used the earlier GPT-6 Astra. Anthropic itself recommends starting with Opus for much everyday coding and knowledge work, reserving Fable for harder open-ended jobs. Vendor-selected comparisons are useful leads, but check versions, tools, safeguards and costs before using the figures together.
There is also a practical distinction between a model’s response style and a tool’s surrounding behavior. An agent’s instructions, available tools, retry rules and stopping conditions can change the result of the same underlying model. A prerelease tester’s custom setup may not match the current Codex product, and an API integration may not inherit Codex’s developer prompt. Reports of premature stops and confusing summaries are worth investigating in the environment where the work will happen.
Pricing has the same dependence on setup. A low output-token total may reflect a more efficient solution, a shorter answer, or an agent that stopped before completing work. The published index scores and costs do not identify which explanation applies to an individual customer request. Compare a completed, reviewed deliverable across candidates, and record failures as well as successful runs.
Which one should you try?
For a cost-sensitive coding workflow, Sol is a reasonable starting candidate: its listed API rates are low and Vals reports a better terminal score than the previous Sol. If repository explanations and complete handoffs matter, include those in the trial because Artificial Analysis found a specific regression there. Test the actual agent as well as the underlying model.
For demanding science or long technical work, Astra’s Vals science lead and independent terminal-science results merit a trial. Ask it to surface assumptions and blockers, then inspect its evidence and intermediate work. For legal, tax or finance workflows, the Vals breakdown gives a reason to compare Claude models as well, while keeping professional review in place. No benchmark here establishes that an answer is legally or financially correct in a particular case.
For an everyday coding comparison, Opus offers a useful price and performance reference, while Fable has the strongest score of these four on the Artificial Analysis overall index. Measure the complete workflow: model spend, latency, failed attempts, human review and the cost of repairing an incomplete answer. Shorter output can save tokens and still require more work from a person. None of the evaluations reviewed here measures your team’s exact review burden, so keep a small representative task set and record what it takes to finish each job.
- Tested
- One combined creator assignment and one shared revision per model; deck, local product page and video-production brief.
- Machine
- Local Apple-silicon Mac; Darwin 25.3.0 arm64.
- Runner
- Native Codex workers; separate Playwright/Chrome browser review and Poppler PDF checks.
- Cloud side
- Requested gpt-6.2-astra and gpt-6.2-sol.
- Settings
- High reasoning requested; separate starts, no trial browsing or subagents; speed tier not exposed.
- Grading
- Nonblind editorial review; 11 calculator cases per version; 8-slide/PDF and 390/1280px checks.
- Cost
- Per-run native tokens and bill unavailable. No separate paid API test calls; price examples are hypothetical.
- Video
- Written 60-second and revised 45-second cue sheets only. No video or voiceover rendered.
- Sample
- One task per model, each with three related outputs and one revision. Not independent task samples.
- Dates
- run 2026-11-17 · checked 2026-11-17
Sources
Official documentation and safety: OpenAI launch; Astra API; Sol API; OpenAI system card; Anthropic model overview; Anthropic launch.
Independent evaluations: Artificial Analysis methodology and coding methodology; Vals Index methodology; Vals Science Index; Vals terminal coding; Vals terminal science. Individual model pages and early accounts are linked in the sections above.
Trending on Kingy
Keep reading with the stories getting the most attention now.
