First-party API pilot · November 21, 2026 Pacific
Four customer emails met the brief. One booking app and one video edit passed. Two Blender scenes rendered after relocation, and both attempts fell short of the full delivery requirements.
We gave Gemini 4 Neon, GPT‑6.2 Astra, GPT‑6.2 Sol, Claude Fable 5.2 and Claude Opus 5.5 one attempt each on four fictional jobs. Acceptance came from a single editorial pass with independent local checks where applicable. Six more jobs have prepared fixtures and offline checks. The full ten-job study remains on hold. Trial setup and evidence

Accepted submissions. Email 4/5, booking app 1/5, video edit 1/5, Blender 0/5. “Partial files” marks exported work from an attempt that missed acceptance. “No exported files” covers empty submissions.
Runner limitations. Tool-call errors and token caps stopped some attempts. The runner marked one empty video attempt complete, and the Blender geometry check needed correction. Open the full-size graphic · Read the result table
What the four pilot jobs showed
Reply to a difficult customer email 4/5 accepted
The model read a support thread and current policy, then wrote a 110–180-word reply with valid options, deadlines and refund terms. Astra, Fable and Opus included the original parcel status and all three options. Sol passed with the parcel status missing. Neon returned an incomplete response without usable output, and a targeted retest ended the same way at 5,000 billed output tokens. The accepted replies took roughly 14–41 seconds and cost US$0.026–$0.186 per attempt in the pilot. Opus resumed the same attempt after a runner pause. Brief · Astra’s accepted reply
Build a workshop booking app 1/5 accepted
Sol’s booking app was the only accepted submission. The trial took 188.743 seconds and cost US$0.2575815. The saved app then passed independent Chrome checks for prices, validation, duplicate booking prevention, capacity, reload persistence, cancellation and keyboard operation. The fixture also required email to stay in memory and all data to stay in the browser. Brief · Sol’s app screenshot
Astra saved illustrations and session data before reaching the input limit. Fable and Opus reached output caps before issuing visible actions. Neon returned a malformed tool call. Fable also exhausted a larger output allowance in the targeted retest, and Neon’s app retest was held under its email gate. The fixture used a local, single-browser app. Payments, deployment and multi-user capacity fall outside its scope.
Edit a vertical workshop promo 1/5 accepted
The brief called for a 35–45-second H.264/AAC export at 720 × 1280 and 24 fps, with correct campaign facts, captions and an editable timeline. Sol delivered a 35.531-second edit using seven takes with a timeline and verification note. The run took 146.824 seconds and cost US$0.3564645. Fable and Opus delivered playable 37.665-second and 41.540-second edits. Both rerendered after the media moved and both missed the required verification note. Fable also added words to a caption that the narrator had not said. Brief · Sol’s 35.531-second video
Neon returned a malformed tool call. Astra signaled completion with an empty submission. The Astra and Opus v2 retests also ended without exports. A later offline replay ran Opus’s saved timeline and render actions and exposed a frame-rate mismatch. The fixture used synthetic source footage, transcripts, a contact sheet and local rendering tools.
Build and light a Blender product scene 0/5 accepted
The task required an editable desk speaker with metric dimensions, an 800 × 800 still and a three-second turntable. Astra and Sol saved editable scenes that rendered after relocation. Sol also delivered the required-size still. Both attempts reached the cumulative input limit before exporting the turntable and verification note. Fable and Opus hit output caps, and Neon returned a malformed tool call. Those three attempts exported no scene. Brief · Sol’s still · Astra’s relocated-scene render
The original diagnostic treated object origins as mesh centers. A corrected check verified evaluated geometry and reopened both scenes. Sol delivered a relocatable scene again in the v2 retest, which ended with the still, turntable and verification note outstanding. Billed input dropped from 142,051 to 21,709 over 15 calls in each attempt. Other retest settings also changed, so the two runs leave the effect of context compaction alone unresolved.
What the runner checks established
On November 22, each of the five models used the same fictional card in a five-turn check at low reasoning effort. It tested a prose-wrapped JSON action, a 1,800-character page limit, a second read using the returned cursor, a saved ID and a missing-file completion gate. All five delivered both exact files. One passed all eight original checks. The prompt had conflicting instructions about prose, and the checkpoint test counted an updated record as a miss even when it carried the correct ID.
All five slots passed the revised checkpoint check in offline replay. Three fresh requests then tested the corrected wording on Astra, Sol and Opus. All three returned prose followed by one labeled JSON block and executed the exact read. Live evidence for the corrected wording covers one first turn per model. Compatibility scores and costs
Six fixture previews
These jobs have prepared fixtures and offline checks from November 21. Model performance is unmeasured for all six.
Meeting recap
Write a 300-word recap and action CSV with owners, deadlines and source timestamps. Preserve final decisions and unresolved ownership after proposals change during the meeting.
Vendor proposals
Compare 12 months of pre-tax CAD costs and signed requirements across three PDFs. Cite pages for prices and eligibility. The offline checks extracted all 18 proposal pages across the variants.
Purchase research
Answer a plan restriction question from a fictional, dated HTML corpus. Cite the exact sections and distinguish documented availability from something a buyer still needs to verify.
Spreadsheet analysis
Clean a sales export, preserve source rows and deliver a workbook with formulas and a native chart. Authored controls recalculated after separate edits to gross amounts and refunds.
Board presentation
Build a six-slide, editable PowerPoint with a recommendation, budget headroom, capacity conditions and a native bookings chart. The authored deck rendered in LibreOffice with editable chart data.
Draft-store bug fix
Repair saved values lost on reload while preserving legacy files and the public API. In each variant, the authored regression failed on the starter and passed on its repaired control.
Where the evidence stops
Each pilot cell is one result on one frozen fixture. The first-party routes and local tools differ from consumer chat interfaces and native editing products. The v2 retest used different turn and output limits. The full study’s three variants per job and 150 planned attempts remain unrun. Open the evidence package
Sources and evidence
Pilot evidence, task briefs and methods · Per-attempt results CSV · Download the article and evidence ZIP
