{ "schema": "kingy-astra-sol-observations-v1", "tested_date": "2026-10-22", "protocol": "Three predeclared synthetic tasks, one response per model/task, same CLI, Medium, fresh sessions, no tools; order alternated by task. Not a production workflow or statistical model ranking.", "billing": "Included ChatGPT subscription access; no paid API calls. CLI token metadata is not a dollar invoice.", "timing": "End-to-end CLI process wall time includes client startup and service variance; one observation per task, no general speed conclusion.", "context": "Identical user prompt per task; model-specific harness instructions and input/cache token counts can differ. No independent backend snapshot attestation.", "runs": [ { "model_requested": "gpt-6-astra", "model_identity_limit": "Requested exact model slug; backend snapshot not independently attested by CLI JSON.", "task": "quote-normalization", "attempt": 1, "effort": "medium", "cli_version": "0.163.0-alpha.10.2", "prompt": "Do not use tools, files, browsing, or external information. Return the requested JSON only.\n\nYou are checking synthetic supplier quotes for a four-seat team buying 12 months of service. Use only these rows. Q1: A costs USD18 per seat per month, paid monthly, plus a one-time USD40 setup fee. Q2: B costs USD16 per seat per month, billed for all 12 months upfront, plus a one-time USD100 setup fee. Q3: C costs EUR15 per seat per month, no setup fee; no EUR/USD exchange rate is supplied. Ignore tax for every quote. Total first-year budget is USD900; upfront cash is not separately constrained. Return only a JSON object with eligible_totals_usd (object keyed A and B), recommendation (supplier letter), excluded (object keyed by supplier with the missing information), and source_rows (object mapping A/B/C to the source row IDs). Calculate first-year totals, recommend the cheapest qualifying USD quote, and do not invent currency conversion.", "prompt_sha256": "8a52a09ec68125a66938e99b450931eaf0d65c38914c428bcf0e16ee99ab716f", "response": "{\n \"eligible_totals_usd\": {\n \"A\": 904,\n \"B\": 868\n },\n \"recommendation\": \"B\",\n \"excluded\": {\n \"C\": \"EUR/USD exchange rate not supplied.\"\n },\n \"source_rows\": {\n \"A\": \"Q1\",\n \"B\": \"Q2\",\n \"C\": \"Q3\"\n }\n}", "response_sha256": "466c6890bfa2388d286032c10c6a6384af4ec7c4efb6aacd9f4b1d7db9232164", "started_utc": "2026-10-22T07:35:10.526852+00:00", "ended_utc": "2026-10-22T07:35:19.031105+00:00", "wall_seconds": 8.504, "usage_reported_by_cli": { "input_tokens": 18458, "cached_input_tokens": 12160, "cache_write_input_tokens": 0, "output_tokens": 128, "reasoning_output_tokens": 36 }, "tool_calls": 0, "validation": { "raw_json_pass": true, "reference_arithmetic_and_fields_pass": true, "field_checks": [ true, true, true, true ] }, "repair_calls": 0, "human_correction_seconds": null, "actual_api_charge_usd": null }, { "model_requested": "gpt-6-astra", "model_identity_limit": "Requested exact model slug; backend snapshot not independently attested by CLI JSON.", "task": "seat-optimization", "attempt": 1, "effort": "medium", "cli_version": "0.163.0-alpha.10.2", "prompt": "Do not use tools, files, browsing, or external information. Return the requested JSON only.\n\nSolve this synthetic seat-purchase decision using only the stated rules. Monthly demand for 12 months is [2,2,2,8,8,8,3,3,3,6,6,6]. Buy an integer k annual seats from 0 through 8 on January 1. Each annual seat costs USD144 upfront for the whole year. If k>=5, discount only the annual-seat charge by 10%. Add a one-time USD60 setup fee if k>0. Every month buy max(0, demand-k) extra seats for USD20 each; those overage charges are paid monthly and are not part of January 1 upfront cash. Unused annual seats have no refund or carryover. First-year total must be <=USD1000 and January 1 upfront cash <=USD500. Return only JSON with evaluations (nine objects, each with k, upfront_usd, monthly_overage_total_usd, first_year_total_usd, feasible), recommendation_k, recommended_total_usd, recommended_upfront_usd. Enumerate all k values and choose the cheapest feasible option.", "prompt_sha256": "7e8a0ff4a5a83d6a1eb4fe9bbd5c4c1dc66f6d6d0663c43be3e1347268045004", "response": "{\n \"evaluations\": [\n {\"k\": 0, \"upfront_usd\": 0, \"monthly_overage_total_usd\": 1140, \"first_year_total_usd\": 1140, \"feasible\": false},\n {\"k\": 1, \"upfront_usd\": 204, \"monthly_overage_total_usd\": 900, \"first_year_total_usd\": 1104, \"feasible\": false},\n {\"k\": 2, \"upfront_usd\": 348, \"monthly_overage_total_usd\": 660, \"first_year_total_usd\": 1008, \"feasible\": false},\n {\"k\": 3, \"upfront_usd\": 492, \"monthly_overage_total_usd\": 480, \"first_year_total_usd\": 972, \"feasible\": true},\n {\"k\": 4, \"upfront_usd\": 636, \"monthly_overage_total_usd\": 360, \"first_year_total_usd\": 996, \"feasible\": false},\n {\"k\": 5, \"upfront_usd\": 708, \"monthly_overage_total_usd\": 240, \"first_year_total_usd\": 948, \"feasible\": false},\n {\"k\": 6, \"upfront_usd\": 837.6, \"monthly_overage_total_usd\": 120, \"first_year_total_usd\": 957.6, \"feasible\": false},\n {\"k\": 7, \"upfront_usd\": 967.2, \"monthly_overage_total_usd\": 60, \"first_year_total_usd\": 1027.2, \"feasible\": false},\n {\"k\": 8, \"upfront_usd\": 1096.8, \"monthly_overage_total_usd\": 0, \"first_year_total_usd\": 1096.8, \"feasible\": false}\n ],\n \"recommendation_k\": 3,\n \"recommended_total_usd\": 972,\n \"recommended_upfront_usd\": 492\n}", "response_sha256": "d93e29b39cf93bce1610731ba3e592afe05b0ff38b0725365ce0d81bde96fd4a", "started_utc": "2026-10-22T07:36:02.566975+00:00", "ended_utc": "2026-10-22T07:36:26.150939+00:00", "wall_seconds": 23.584, "usage_reported_by_cli": { "input_tokens": 21579, "cached_input_tokens": 12160, "cache_write_input_tokens": 0, "output_tokens": 668, "reasoning_output_tokens": 216 }, "tool_calls": 0, "validation": { "raw_json_pass": true, "reference_arithmetic_and_fields_pass": true, "field_checks": [ true, true, true, true ] }, "repair_calls": 0, "human_correction_seconds": null, "actual_api_charge_usd": null }, { "model_requested": "gpt-6-astra", "model_identity_limit": "Requested exact model slug; backend snapshot not independently attested by CLI JSON.", "task": "upgrade-evidence-boundary", "attempt": 1, "effort": "medium", "cli_version": "0.163.0-alpha.10.2", "prompt": "Do not use tools, files, browsing, or external information. Return the requested JSON only.\n\nA synthetic buyer asks whether an upgrade is guaranteed to pay for itself. Current plan is USD200/month; proposed plan is USD500/month. Their time value is USD60/hour. Vendor marketing says the proposed speed mode is 'up to 2.5x faster'. There are no matched timings, no count of latency-bound jobs, and no measured allowance data. Current plan quotas are described qualitatively, not as fixed tasks. Return only JSON with verdict, monthly_increment_usd, break_even_hours_saved_per_month, break_even_minutes_saved_per_month, guaranteed_savings_usd (null if not established), required_measurements (array), and reasoning (one sentence). Decide whether guaranteed ROI or exact task capacity is established and calculate only the time-savings threshold supported by the inputs. Do not treat a marketed maximum as a measured result.", "prompt_sha256": "0b75c8cc1212028c8f999c7999a9bf678047908b45dd1b0584fe8b37433fc2b3", "response": "{\n \"verdict\": \"Neither guaranteed ROI nor exact task capacity is established.\",\n \"monthly_increment_usd\": 300,\n \"break_even_hours_saved_per_month\": 5,\n \"break_even_minutes_saved_per_month\": 300,\n \"guaranteed_savings_usd\": null,\n \"required_measurements\": [\n \"Matched timings for representative jobs on both plans\",\n \"Monthly count of latency-bound jobs\",\n \"Actual usable time saved per job\",\n \"Measured allowances and consumption per representative task on both plans\"\n ],\n \"reasoning\": \"The USD300 monthly increase requires 5 hours (300 minutes) of usable time savings valued at USD60/hour to break even, but an 'up to 2.5x faster' marketing claim and qualitative quotas establish neither guaranteed savings nor exact task capacity.\"\n}", "response_sha256": "0c68e4118c136c50d2c548bae3ccd3f76f7e48ae1480b84e970bc981815b79d4", "started_utc": "2026-10-22T07:37:14.596867+00:00", "ended_utc": "2026-10-22T07:37:24.218103+00:00", "wall_seconds": 9.621, "usage_reported_by_cli": { "input_tokens": 18437, "cached_input_tokens": 12160, "cache_write_input_tokens": 0, "output_tokens": 177, "reasoning_output_tokens": 0 }, "tool_calls": 0, "validation": { "raw_json_pass": true, "reference_arithmetic_and_fields_pass": true, "field_checks": [ true, true, true, true ] }, "repair_calls": 0, "human_correction_seconds": null, "actual_api_charge_usd": null }, { "model_requested": "gpt-6.1-sol", "model_identity_limit": "Requested exact model slug; backend snapshot not independently attested by CLI JSON.", "task": "quote-normalization", "attempt": 1, "effort": "medium", "cli_version": "0.163.0-alpha.10.2", "prompt": "Do not use tools, files, browsing, or external information. Return the requested JSON only.\n\nYou are checking synthetic supplier quotes for a four-seat team buying 12 months of service. Use only these rows. Q1: A costs USD18 per seat per month, paid monthly, plus a one-time USD40 setup fee. Q2: B costs USD16 per seat per month, billed for all 12 months upfront, plus a one-time USD100 setup fee. Q3: C costs EUR15 per seat per month, no setup fee; no EUR/USD exchange rate is supplied. Ignore tax for every quote. Total first-year budget is USD900; upfront cash is not separately constrained. Return only a JSON object with eligible_totals_usd (object keyed A and B), recommendation (supplier letter), excluded (object keyed by supplier with the missing information), and source_rows (object mapping A/B/C to the source row IDs). Calculate first-year totals, recommend the cheapest qualifying USD quote, and do not invent currency conversion.", "prompt_sha256": "8a52a09ec68125a66938e99b450931eaf0d65c38914c428bcf0e16ee99ab716f", "response": "{\n \"eligible_totals_usd\": {\n \"A\": 904,\n \"B\": 868\n },\n \"recommendation\": \"B\",\n \"excluded\": {\n \"C\": \"Missing EUR/USD exchange rate.\"\n },\n \"source_rows\": {\n \"A\": \"Q1\",\n \"B\": \"Q2\",\n \"C\": \"Q3\"\n }\n}", "response_sha256": "fe4dcad55f0ee398e983beae44cb9d4330472709d40fe138a3018a9722316a8d", "started_utc": "2026-10-22T07:34:33.549965+00:00", "ended_utc": "2026-10-22T07:34:42.008869+00:00", "wall_seconds": 8.459, "usage_reported_by_cli": { "input_tokens": 18522, "cached_input_tokens": 12288, "cache_write_input_tokens": 0, "output_tokens": 134, "reasoning_output_tokens": 43 }, "tool_calls": 0, "validation": { "raw_json_pass": true, "reference_arithmetic_and_fields_pass": true, "field_checks": [ true, true, true, true ] }, "repair_calls": 0, "human_correction_seconds": null, "actual_api_charge_usd": null }, { "model_requested": "gpt-6.1-sol", "model_identity_limit": "Requested exact model slug; backend snapshot not independently attested by CLI JSON.", "task": "seat-optimization", "attempt": 1, "effort": "medium", "cli_version": "0.163.0-alpha.10.2", "prompt": "Do not use tools, files, browsing, or external information. Return the requested JSON only.\n\nSolve this synthetic seat-purchase decision using only the stated rules. Monthly demand for 12 months is [2,2,2,8,8,8,3,3,3,6,6,6]. Buy an integer k annual seats from 0 through 8 on January 1. Each annual seat costs USD144 upfront for the whole year. If k>=5, discount only the annual-seat charge by 10%. Add a one-time USD60 setup fee if k>0. Every month buy max(0, demand-k) extra seats for USD20 each; those overage charges are paid monthly and are not part of January 1 upfront cash. Unused annual seats have no refund or carryover. First-year total must be <=USD1000 and January 1 upfront cash <=USD500. Return only JSON with evaluations (nine objects, each with k, upfront_usd, monthly_overage_total_usd, first_year_total_usd, feasible), recommendation_k, recommended_total_usd, recommended_upfront_usd. Enumerate all k values and choose the cheapest feasible option.", "prompt_sha256": "7e8a0ff4a5a83d6a1eb4fe9bbd5c4c1dc66f6d6d0663c43be3e1347268045004", "response": "{\n \"evaluations\": [\n {\"k\": 0, \"upfront_usd\": 0, \"monthly_overage_total_usd\": 1140, \"first_year_total_usd\": 1140, \"feasible\": false},\n {\"k\": 1, \"upfront_usd\": 204, \"monthly_overage_total_usd\": 900, \"first_year_total_usd\": 1104, \"feasible\": false},\n {\"k\": 2, \"upfront_usd\": 348, \"monthly_overage_total_usd\": 660, \"first_year_total_usd\": 1008, \"feasible\": false},\n {\"k\": 3, \"upfront_usd\": 492, \"monthly_overage_total_usd\": 480, \"first_year_total_usd\": 972, \"feasible\": true},\n {\"k\": 4, \"upfront_usd\": 636, \"monthly_overage_total_usd\": 360, \"first_year_total_usd\": 996, \"feasible\": false},\n {\"k\": 5, \"upfront_usd\": 708, \"monthly_overage_total_usd\": 240, \"first_year_total_usd\": 948, \"feasible\": false},\n {\"k\": 6, \"upfront_usd\": 837.6, \"monthly_overage_total_usd\": 120, \"first_year_total_usd\": 957.6, \"feasible\": false},\n {\"k\": 7, \"upfront_usd\": 967.2, \"monthly_overage_total_usd\": 60, \"first_year_total_usd\": 1027.2, \"feasible\": false},\n {\"k\": 8, \"upfront_usd\": 1096.8, \"monthly_overage_total_usd\": 0, \"first_year_total_usd\": 1096.8, \"feasible\": false}\n ],\n \"recommendation_k\": 3,\n \"recommended_total_usd\": 972,\n \"recommended_upfront_usd\": 492\n}", "response_sha256": "d93e29b39cf93bce1610731ba3e592afe05b0ff38b0725365ce0d81bde96fd4a", "started_utc": "2026-10-22T07:36:26.152879+00:00", "ended_utc": "2026-10-22T07:37:01.998983+00:00", "wall_seconds": 35.846, "usage_reported_by_cli": { "input_tokens": 18558, "cached_input_tokens": 12288, "cache_write_input_tokens": 0, "output_tokens": 648, "reasoning_output_tokens": 196 }, "tool_calls": 0, "validation": { "raw_json_pass": true, "reference_arithmetic_and_fields_pass": true, "field_checks": [ true, true, true, true ] }, "repair_calls": 0, "human_correction_seconds": null, "actual_api_charge_usd": null }, { "model_requested": "gpt-6.1-sol", "model_identity_limit": "Requested exact model slug; backend snapshot not independently attested by CLI JSON.", "task": "upgrade-evidence-boundary", "attempt": 1, "effort": "medium", "cli_version": "0.163.0-alpha.10.2", "prompt": "Do not use tools, files, browsing, or external information. Return the requested JSON only.\n\nA synthetic buyer asks whether an upgrade is guaranteed to pay for itself. Current plan is USD200/month; proposed plan is USD500/month. Their time value is USD60/hour. Vendor marketing says the proposed speed mode is 'up to 2.5x faster'. There are no matched timings, no count of latency-bound jobs, and no measured allowance data. Current plan quotas are described qualitatively, not as fixed tasks. Return only JSON with verdict, monthly_increment_usd, break_even_hours_saved_per_month, break_even_minutes_saved_per_month, guaranteed_savings_usd (null if not established), required_measurements (array), and reasoning (one sentence). Decide whether guaranteed ROI or exact task capacity is established and calculate only the time-savings threshold supported by the inputs. Do not treat a marketed maximum as a measured result.", "prompt_sha256": "0b75c8cc1212028c8f999c7999a9bf678047908b45dd1b0584fe8b37433fc2b3", "response": "{\n \"verdict\": \"Neither guaranteed ROI nor exact task capacity is established.\",\n \"monthly_increment_usd\": 300,\n \"break_even_hours_saved_per_month\": 5,\n \"break_even_minutes_saved_per_month\": 300,\n \"guaranteed_savings_usd\": null,\n \"required_measurements\": [\n \"Matched timings for representative jobs on both plans using the proposed speed mode.\",\n \"Monthly count of latency-bound jobs and actual usable time saved.\",\n \"Measured allowances and consumption per representative task on each plan.\"\n ],\n \"reasoning\": \"The USD300 monthly increase requires 5 hours of usable time savings at USD60/hour to break even, but the marketed maximum of up to 2.5x faster and qualitative quotas establish neither guaranteed savings nor exact task capacity.\"\n}", "response_sha256": "f690e4fef851201f11f664c2832cb82bdf908591c9bd01900d191543d018b164", "started_utc": "2026-10-22T07:37:02.000120+00:00", "ended_utc": "2026-10-22T07:37:14.593787+00:00", "wall_seconds": 12.594, "usage_reported_by_cli": { "input_tokens": 25380, "cached_input_tokens": 12288, "cache_write_input_tokens": 0, "output_tokens": 222, "reasoning_output_tokens": 48 }, "tool_calls": 0, "validation": { "raw_json_pass": true, "reference_arithmetic_and_fields_pass": true, "field_checks": [ true, true, true, true ] }, "repair_calls": 0, "human_correction_seconds": null, "actual_api_charge_usd": null } ] }