# Operator protocol — freeze before first run
## Setup
The kit itself makes no calls; execution is now authorized up to US$100 total. See EXECUTION-SETTINGS.md and the two execution ledgers. Check access and obtain any
required spending authorization before paid usage. Do not infer API access from an
app subscription. Record actual billed usage separately from allocated subscription
cost; unknown is blank, not $0. Currency is USD in the logs; record conversion/source
in notes if the provider bills elsewhere. No model family is a pinned endpoint.

Use one common local runner with filesystem, Python/JS, shell and browser tools for
all candidates where possible. All six controlled tasks have web access OFF. Same
machine, same allowed dependencies, same files. Disable cross-chat memory and external
connectors. Copy only the active task and COMMON into an isolated workspace. Restrict
the agent's filesystem so it cannot read operator keys or other submissions. If that
is impossible, record leakage risk and do not claim a blind controlled test.

Record provider/model identifier, visible version, app or runner version, settings,
reasoning effort, tool capabilities, output budget, cache handling and run timestamp.
Choose vendor-default reasoning and do not change it between tasks or repeats. Vendor
defaults differ: this is a practical default-configuration comparison, not equal
compute. Set maximum output budget high enough for required files; log actual caps.
If a default or hidden routing is unknown, write unknown. Do not pool preview and
final versions. Changing endpoint/settings requires a separately labelled cohort.

If using native apps instead, use the same attachments/prompts but label the results
as product+tools comparisons. Unsupported capabilities are unavailable, not a score
of zero. Do not silently give one model better tools. Never mix runner and app cohorts.

## Execution and timing
Use run-order.csv for a rotated model order. Each repetition uses identical fixtures
and a fresh context/workspace. Clear the working site between runs. Two repetitions
measure only limited variability, not population significance. No per-model prompt
tuning. No operator hints. The revision prompt is identical for everyone, not tailored
feedback. Preserve first submission before sending revision.

Initial wall-clock budget30min; revision15min. Record elapsed time separately for each
stage, including model/tool waiting and automatic retries. Record startup/auth delays
separately. At timeout keep partial outputs and mark timeout; score what was delivered.
A provider outage/no access is unavailable: one reschedule is allowed, recorded; do
not score it as reasoning failure. A model's own faulty tool use consumes its budget.
Repeated infrastructure failures end as unavailable. Record token/truncation errors.

## Evidence and grading
Save prompt+input hashes, raw transcript, tool trace, output files and screenshots
under runs/<run_id>/initial and revision. Give blinded copies to the human grader;
keep model mapping separately. Do not use another competing model as the sole judge.
Use the rubric and exact keys; each 0–4 criterion needs an evidence note. Record the
critical failure check explicitly even for a20. Availability/status is separate from
quality. Never fill unrun scores with zero. Model self-reported testing is unverified
until the operator checks it. Time the human's necessary output verification/repair
separately from optional editorial polish, including failed repair attempts; distinguish
checking minutes from repair minutes. Native app task3 needs exported runnable files.

grading.xlsx has one row for each initial/revision stage (200 rows); run-log.csv stores
100 run identities/settings. Fill actual values only; no sample results are prefilled.
Score columns C1–C5 follow that task's Rubric tab in order. Evidence field should point
to a per-run grading note if too long. Enter critical Yes/No and exact run status.
Statuses: not run, complete, timeout, unavailable, invalid. Invalid = protocol violation
(e.g. leaked key), excluded pending rerun; complete/timeout are gradable.

Summarize initial and revised results separately. Report all task/repetition outcomes,
medians/ranges of actual elapsed/checking/repair times and costs only where observed.
Show unavailable runs and failures. Do not extrapolate general intelligence, declare
statistical significance from two runs, or hide failures by showing only revised work.
No overall winner is needed: report which tasks worked and what correction remained.

## Optional live purchase-research extension (not in the60-run main cohort)
Task2 main packet fixes sources for fairness. It tests source synthesis, not browsing.
For a separate live-web test, keep a dated real buyer brief and shortlist, same region,
budget and browsing tools. Replace fictional offers with independently checked live
sources before any run; save snapshots and operator answer key. Capture timestamps,
stock, SKU, tax, shipping and delivery commitment. Score relative to each captured
source time; record changes during the run. Do not reuse Task2's fictional key or pool
these results with the controlled task. No live browsing has been run in this kit.
