First-party API pilot · November 21, 2026 Pacific
Results and evidence for the four-job AI pilot
Twenty attempts, six accepted submissions, five partial exports and nine attempts with no exported files. The full study remains on hold.
Per-attempt results
Partial files came from attempts that exported work and missed acceptance. Times cover the runner’s recorded attempt, with independent diagnostics conducted on saved files afterward. Opus’s email resumed from its saved transcript in the same attempt. Costs use the metered API records before tax. Download the 20-row CSV
| Job | Model | Export outcome | Runner stop | Seconds | US dollars |
|---|---|---|---|---|---|
| Customer email | Gemini 4 Neon | No exported files | Incomplete provider response | 24.799 | $0.0481440 |
| Customer email | GPT-6.2 Astra | Accepted | Finish requested | 14.235 | $0.0670680 |
| Customer email | GPT-6.2 Sol | Accepted | Finish requested | 18.415 | $0.0257455 |
| Customer email | Claude Fable 5.2 | Accepted | Finish requested | 32.256 | $0.1858880 |
| Customer email | Claude Opus 5.5 | Accepted | Finish requested | 41.229 | $0.1226760 |
| Booking app | Gemini 4 Neon | No exported files | Malformed tool call | 83.735 | $0.0033810 |
| Booking app | GPT-6.2 Astra | Partial files | Cumulative input limit | 393.104 | $2.2731880 |
| Booking app | GPT-6.2 Sol | Accepted | Finish requested | 188.743 | $0.2575815 |
| Booking app | Claude Fable 5.2 | No exported files | Output limit | 107.278 | $0.3926800 |
| Booking app | Claude Opus 5.5 | No exported files | Output limit | 91.427 | $0.1963400 |
| Video edit | Gemini 4 Neon | No exported files | Malformed tool call | 10.752 | $0.0079770 |
| Video edit | GPT-6.2 Astra | No exported files | Finish requested with an empty export | 119.725 | $0.5639600 |
| Video edit | GPT-6.2 Sol | Accepted | Finish requested | 146.824 | $0.3564645 |
| Video edit | Claude Fable 5.2 | Partial files | Output limit | 317.826 | $1.9309280 |
| Video edit | Claude Opus 5.5 | Partial files | Cumulative input limit | 170.579 | $0.8824000 |
| Blender | Gemini 4 Neon | No exported files | Malformed tool call | 91.005 | $0.0060810 |
| Blender | GPT-6.2 Astra | Partial files | Cumulative input limit | 653.854 | $2.5610460 |
| Blender | GPT-6.2 Sol | Partial files | Cumulative input limit | 806.294 | $0.5084605 |
| Blender | Claude Fable 5.2 | No exported files | Output limit | 109.354 | $0.3960000 |
| Blender | Claude Opus 5.5 | No exported files | Output limit | 106.139 | $0.2298320 |
The attempts cost US$11.015841. Five access checks added US$0.001978, for US$11.017819 total. The five dispatched v2 retest slots separately cost US$1.351440.
Four pilot briefs
These are the instructions supplied with the fictional pilot fixtures.
Selected pilot outputs
- Astra’s accepted email from the fictional customer support fixture.
- Sol’s accepted app as captured by the Chrome evaluator.
- Sol’s accepted 35.531-second video from the shared synthetic footage.
- Sol’s Blender still from a partial submission that lacked a turntable and verification note.
- Astra’s relocated scene render produced by the geometry check from a partial submission.
How the pilot ran
The original pilot used the Gemini Interactions, OpenAI Responses and Anthropic Messages APIs. Models received the same input bundle within a job. The shared local workbench replayed visible actions and observations. The email job used the providers’ medium reasoning settings and the three production jobs used high. Those labels refer to each provider’s own settings.
Pilot acceptance came from one editorial pass with task keys and independent local checks where applicable. Checks included Chrome behavior, video decoding and relinking, and reopening Blender scenes. The later v2 retest ran five of six planned slots with different limits and reached no submissions.
Compatibility checks
These synthetic text-fixture results measure the workbench protocol. The offline replay applied the revised checkpoint criterion to the saved replies. The corrected wording had one fresh first-turn request for each of Astra, Sol and Opus.
| Model | Five turns, live | Five turns, replay | Corrected first turn, live |
|---|---|---|---|
| Gemini 4 Neon | 8/8 | 8/8 | Outside follow-up scope |
| GPT-6.2 Astra | 7/8 | 7/8 | 4/4 |
| GPT-6.2 Sol | 7/8 | 7/8 | 4/4 |
| Claude Fable 5.2 | 7/8 | 8/8 | Outside follow-up scope |
| Claude Opus 5.5 | 6/8 | 7/8 | 4/4 |
The five-slot live check made 25 generation calls for US$0.32064750. The three corrected first turns cost US$0.02584550 in total. Offline replay used the archived responses. Both live compatibility checks were metered separately from the pilot.