First-party API pilot · November 21, 2026 Pacific

Results and evidence for the four-job AI pilot

Twenty attempts, six accepted submissions, five partial exports and nine attempts with no exported files. The full study remains on hold.

Per-attempt results

Partial files came from attempts that exported work and missed acceptance. Times cover the runner’s recorded attempt, with independent diagnostics conducted on saved files afterward. Opus’s email resumed from its saved transcript in the same attempt. Costs use the metered API records before tax. Download the 20-row CSV

One attempt per model on each job
JobModelExport outcomeRunner stopSecondsUS dollars
Customer emailGemini 4 NeonNo exported filesIncomplete provider response24.799$0.0481440
Customer emailGPT-6.2 AstraAcceptedFinish requested14.235$0.0670680
Customer emailGPT-6.2 SolAcceptedFinish requested18.415$0.0257455
Customer emailClaude Fable 5.2AcceptedFinish requested32.256$0.1858880
Customer emailClaude Opus 5.5AcceptedFinish requested41.229$0.1226760
Booking appGemini 4 NeonNo exported filesMalformed tool call83.735$0.0033810
Booking appGPT-6.2 AstraPartial filesCumulative input limit393.104$2.2731880
Booking appGPT-6.2 SolAcceptedFinish requested188.743$0.2575815
Booking appClaude Fable 5.2No exported filesOutput limit107.278$0.3926800
Booking appClaude Opus 5.5No exported filesOutput limit91.427$0.1963400
Video editGemini 4 NeonNo exported filesMalformed tool call10.752$0.0079770
Video editGPT-6.2 AstraNo exported filesFinish requested with an empty export119.725$0.5639600
Video editGPT-6.2 SolAcceptedFinish requested146.824$0.3564645
Video editClaude Fable 5.2Partial filesOutput limit317.826$1.9309280
Video editClaude Opus 5.5Partial filesCumulative input limit170.579$0.8824000
BlenderGemini 4 NeonNo exported filesMalformed tool call91.005$0.0060810
BlenderGPT-6.2 AstraPartial filesCumulative input limit653.854$2.5610460
BlenderGPT-6.2 SolPartial filesCumulative input limit806.294$0.5084605
BlenderClaude Fable 5.2No exported filesOutput limit109.354$0.3960000
BlenderClaude Opus 5.5No exported filesOutput limit106.139$0.2298320

The attempts cost US$11.015841. Five access checks added US$0.001978, for US$11.017819 total. The five dispatched v2 retest slots separately cost US$1.351440.

Open or save the full-size comparison graphic · Download the article and evidence ZIP · View hosted file checksums

Four pilot briefs

These are the instructions supplied with the fictional pilot fixtures.

Selected pilot outputs

How the pilot ran

The original pilot used the Gemini Interactions, OpenAI Responses and Anthropic Messages APIs. Models received the same input bundle within a job. The shared local workbench replayed visible actions and observations. The email job used the providers’ medium reasoning settings and the three production jobs used high. Those labels refer to each provider’s own settings.

Pilot acceptance came from one editorial pass with task keys and independent local checks where applicable. Checks included Chrome behavior, video decoding and relinking, and reopening Blender scenes. The later v2 retest ran five of six planned slots with different limits and reached no submissions.

Compatibility checks

These synthetic text-fixture results measure the workbench protocol. The offline replay applied the revised checkpoint criterion to the saved replies. The corrected wording had one fresh first-turn request for each of Astra, Sol and Opus.

Original scores, revised offline scores and fresh first-turn scores
ModelFive turns, liveFive turns, replayCorrected first turn, live
Gemini 4 Neon8/88/8Outside follow-up scope
GPT-6.2 Astra7/87/84/4
GPT-6.2 Sol7/87/84/4
Claude Fable 5.27/88/8Outside follow-up scope
Claude Opus 5.56/87/84/4

The five-slot live check made 25 generation calls for US$0.32064750. The three corrected first turns cost US$0.02584550 in total. Offline replay used the archived responses. Both live compatibility checks were metered separately from the pilot.