{ "schemaVersion": "kingy-document-three-pair-report/1.0.0", "metadata": { "schemaVersion": "kingy-document-three-pair-report/1.0.0", "studyId": "document-comparison-v19-20261015.1", "scopeDescription": "Ten synthetic PDFs and 40 held-out questions were frozen before inference, then attempted in three original runs per configured workflow. Model bytes, generation seed, candidate and scoring were frozen. All 240 original attempts are preserved. Candidate 1.5.4 is held: all three Kingy runs missed the automatic quality targets, the independent source audit found concise-answer false failures in the scorer, and one native completion remains unverified after a host restart. Frozen automatic tallies are not a semantic accuracy ranking. The report documents source-verified differences in both directions. The active download remains workflow 1.4.0.", "limitations": [ "The final unblinded AI-assisted source audit covers 82 distinct response/grade/citation bundles across all 240 originals. Native Q12 and Q15 are definite contextual concise-answer false failures under the frozen scorer. Many unresolved native paraphrases are semantically complete. Original grades remain unchanged; no corrected numeric leaderboard or overall quality winner is issued.", "Kingy declined six answerable questions in every repetition; one further returned answer has a conservative unresolved citation. The independent audit documents each case, including the Q19 focused-question context loss. These failures are preserved rather than repaired or retried on this consumed corpus.", "Native B3 original22 (V19-Q38) has one uniquely matched final-answer event at 92.536 seconds, exceeding the frozen 90-second deadline. This is separate from the CUA timeout observation. The timing record does not establish its cause or general product reliability.", "One shared Apple M4 Pro Mac with 24 GiB, one synthetic English corpus, and Qwen3-4B Q4_K_M. No result establishes behavior on other hardware, private real-world documents, scans, OCR, long libraries or other models.", "Separate AI agents authored and reviewed the fresh questions, obligations, aliases and source spans before inference. These are AI-assisted reviews, not human adjudication or separate-machine reproduction.", "Forty unique questions repeated three times are not 120 independent tasks per workflow. The published corpus is consumed and cannot serve as a future unseen test.", "The shared scorer remains version 1.4.0. It requires all declared atomic obligations and support, and has finite aliases and composition rules. Unresolved judgments remain in conservative bounds.", "Candidate 1.5.4 isolates retrieval, selection and sufficiency by field, preserves source-page diversity, and uses finite lexical role witnesses. Unfamiliar synonyms can cause false abstention; lexical presence alone does not prove entailment. Document-scope enforcement still includes inferred title matches.", "Both configured workflows use identical model bytes but different engines, retrieval and output pipelines. This is a whole-workflow comparison, not a model-only leaderboard.", "Native GUI control provenance and every retained original are available. Control interruptions must not be attributed to product reliability.", "Native B3 scheduled original23 (V19-Q09) spans an unobserved host-restart gap and has no saved final answer or matching final-generation event. The unchanged original is retained as incomplete in the primary denominator. Actual generation and answer timing are unverified; no comparative product advantage or reliability conclusion may use this interrupted row.", "Native B1 and B3 contain preparation pauses and additional cold loads and neutral warmups, fully enumerated in the final inventory. Logical repetition labels do not mean uninterrupted native sessions. Missing log evidence does not prove that an unobserved model request never occurred.", "The native arm uses a retained ten-PDF conversation state. During replacement preparation, a sixth sequential upload was refused by the installed default five-file add handler. This study does not establish a fresh sequential ten-PDF installation path or a universal native file limit; no limit, policy or vendor code was changed.", "Complete-query timing is descriptive on a shared Mac. OS caches were not flushed and wall power was not measured. No general speed, setup-time or cost advantage follows.", "Kingy emits document, page and exact-quote records. The native arm uses its frozen factual prompt and stock expandable citations. Other prompts may change native output.", "Kingy uses a local deny-egress profile. Native app-wide egress was not independently enforced or verified. This study does not establish comparative privacy.", "Earlier studies, their original scores, raw evidence and downloads remain preserved. The new comparison does not retroactively replace their outcomes." ], "history": { "studyId": "document-comparison-v14-20261014.1", "href": "https://kingy.ai/wp-content/plugins/kingy-local-ai/app/documents/comparisons/document-comparison-v14-20261014.1/report.html" }, "instrumentation": { "amended": true, "phase": "B1-after-20-before-21", "beforeFirstAffectedInference": true, "retainedNativeOriginalsBefore": 20, "nativeB1ExtraColdStartAfterOriginal": 35, "nativeB1ColdLoadCount": 3, "nativeB1NeutralWarmupCount": 3, "nativeB1ExtraColdLoadsAfterOriginal35": 2, "nativeB1ExtraNeutralWarmupsAfterOriginal35": 2, "nativeInterruptedOriginals": [ { "session": "B3", "scheduledPosition": 23, "questionId": "V19-Q09", "status": "incomplete-unverified", "actualModelGeneration": "unverified", "questionRetried": false, "nativeRecordSha256": "d98481153cec34f759ee58a1dc1a429f8238d08857a0c2811ae13e2ce90b6a6c" } ], "candidateCorpusPromptsScorerUnchanged": true, "freshReferencesOrQualityGradesReviewed": false, "description": "Native LM Studio used the separately reviewed direct System Events template-menu control. After two blank-copy persistence delays, an independently reviewed wrapper was added before B1 original21. It waits up to ten seconds for a single acknowledged Duplicate action to persist a blank ten-attachment record, then returns to unchanged frozen checks. It adds no UI retry, model request, question submission or change to Send/answer timing. After original35, two unscheduled preparation generations occurred outside every measured answer window. The first followed unexpected short input; the user later reported concurrent typing, but exact event attribution was not established. The second contained file parts without text during menu preparation; its cause remains unresolved. Both affected native chats and their original bytes were preserved privately. Coordinate-based measured-trial template-menu preparation was retired before original36. An independently reviewed PID-scoped AXPress control verifies the exact owned row, outer AXGroup popup trigger, enabled popup state and labelled inner More options child, then presses the outer trigger once. It requires a focused More menu, and the binding rejects any second menu, including unnamed native overlays. An initial inner-button target mismatch and stale dual-menu guard defect were corrected before original36, with prior sources and failed probes preserved. The installed vendor UI source was inspected read-only; vendor code was not changed. The blank ten-attachment template is checked before and after preparation. The neutral native preflight created zero-file unscored copies once each and proved canonical reselection and repeat menu access without inference. Foreground indicators and Cancel acknowledgments were not treated as proof of selection or dismissal. A fresh owned blank chat was renamed and five PDFs were sequentially attached with exact PDF/configuration checks. The sixth sequential upload was visibly refused by the installed default five-file add handler. No policy, limit or vendor code was changed, and no over-limit upload was forced. Independent review allowed conditional reuse of a pre-existing, never-measured blank already containing all ten frozen PDFs, through ordinary native identity, name-only and duplicate controls. The native arm therefore depends on an existing ten-file conversation state; this does not establish that a fresh sequential ten-file setup works or that the UI has a universal five-file limit. Native Reveal in Finder proved the exact unused ten-PDF record before it was renamed. The five-file preparation was preserved under a quarantine name, and only the verified ten-file blank name changed to the canonical template name. One ordinary native Duplicate then produced a blank ten-file copy with exact frozen hashes and ten visible previews; the required screenshot API passed. The final model selection and eleven configuration keys are bound by the later AX freeze and verification receipts. Later draft-only characters were preserved privately; the user reported concurrent Mac use, but exact attribution remains unresolved. After the user confirmed an idle Mac, a clean unloaded GUI restart restored fresh observation. New inference was held while a separate finite local-model job occupied the shared host. The study model was kept unloaded during UI recovery. B1 used three recorded model loads and three neutral warmups in total, including two extra cold loads and two neutral READY warmups after original35. Each warmup was one model request without documents or a bank question. All receipts are retained. During attachment-only preparation, another job loaded a separate model and selected runtime2.60 while frozen2.56 remained installed. All34 frozen app, model, RAG and runtime artifacts were reverified unchanged. That observation does not identify who changed the selection or prove a cause for UI interruptions. A mistaken sheet-count diagnostic was corrected against an explicit native sheet lookup. Normal window and sidebar layout changes restored full-width controls; a fresh native menu-item point click opened the actual picker. These preparation actions did not change candidate inference, grading or retained original attempts. Extra conditioning and preparation pauses can affect cache state. Native runs include preparation pauses; B1 also includes extra cold loads and is not an uninterrupted-session timing result. Four unrelated input/output log events are omitted from the public derivative with exact hashes disclosed; all other event bytes and original private logs are preserved. The public engine-process proof omits a local credential-bearing command and binds the unchanged private original by hash. Independent export checks verify the derivative and raw-receipt exclusion; the final completed public artifacts still require the actual privacy scan. Before B3 original1, the native control connection closed before a response. Reconciliation proved zero retained or pending questions, an unconsumed menu token and only the two neutral warmup log events. A documented control reset recovered the actual singleton menu; the old token and unchanged log prefix were preserved. No model reload, extra warmup, control-source edit or question replay was needed. Before B3 original2, the attachment guard rejected one newly duplicated blank: it had ten entries but only nine distinct backing-file identifiers and hashes. The canonical source still had all ten correct frozen PDFs. The rejected blank was preserved unchanged and its native identity verified. A fresh ordinary copy passed ten distinct frozen PDF hashes before any input; original2 was then submitted once. The same guard also rejected a second blank before original22, before prompt, pending record or Send. Its separate interruption and recovery receipts preserve that preparation event. These storage-identifier observations do not establish which filenames the UI or model displayed, their cause, or general product reliability. B3 scheduled original23 (V19-Q09) was submitted before an unobserved gap spanning a host restart. The saved record has no documented final answer, final-generation block or matching final model-log event. Old owned processes were unavailable; a new main app instance existed and remained uncontrolled during reconciliation. The exact native record, pending clock and46-event log prefix were preserved. An initial capture-helper preflight required an explicit null timestamp while the original JSON omitted that field; it stopped before a marker, journal write or coordinator call. A separately reviewed correction accepts only absence or explicit null and rejects every present completion value. The approved capture-only helper then invoked the unchanged frozen coordinator once, using a truthful observation scoped to the unavailable old owned instance and forbidding all UI/model actions. It retained original23 as incomplete without inventing UI completion, answer timing or an answer, and cleared the pending slot. No question was resubmitted. The missing final evidence does not prove that no model request occurred, a wrong model answer, or product unreliability. A fresh neutral ten-PDF duplicate, screenshot-API, canonical reselection and single focused menu preflight then passed on the restarted main instance. A separate reviewer bound the exact frozen2.56 engine, all eleven configuration keys, effective SDK settings, unchanged ten PDF hashes,48 log lines and original46-event prefix before allowing original24 onward. One additional cold model load and one neutral warmup followed the host restart. Their actual continuation and final restoration are disclosed in the linked final load/warmup inventory. No comparative advantage may be claimed from this interrupted row. After that preflight, a focus-only helper observed native focus, but CUA omitted its focused-element metadata. That pilot did not pass and was stopped; unavailable metadata does not prove native focus was absent. A distinct transport amendment moved the current native focus check to the same bounded call as exactly one Duplicate AXPress. Its source, failed neutral attempts and actual verification are preserved separately. This changes control transport only; frozen candidate, scorer, corpus, later prompt and Send behavior remain unchanged. Kingy A2 initially stopped before any server load, warmup or question submission because a model link pointed to a disconnected external volume. The zero-attempt start was preserved. The link was repaired to the existing local native model after exact frozen-byte verification and a separate metadata-only review. Candidate, corpus, scorer and model bytes did not change, and no measured question was replayed. All planned original submissions are retained. Preparation interruptions and unscheduled activity are not product reliability evidence.", "evidenceIds": [ "focus-pilot-review", "focus-pilot-discrepancy", "duplicate-boundary-freeze", "duplicate-boundary-review", "duplicate-neutral-proof", "duplicate-neutral-review", "resumed-original-settings", "transport-freeze", "transport-review", "readiness-addon-freeze", "readiness-addon-review", "template-recovery-review", "replacement-template-review", "replacement-template-verification", "ax-menu-freeze", "ax-menu-review", "ax-menu-preflight", "final-native-preflight", "final-native-template", "final-native-runtime", "final-native-runtime-selection", "native-load-warmup-final", "b3-control-interruption", "b3-control-recovery", "b3-attachment-guard", "b3-attachment-recovery", "b3-attachment-guard-22", "b3-attachment-recovery-22", "host-gap-observation", "host-gap-boot", "host-gap-processes", "host-gap-artifacts", "host-gap-recovery-review", "host-gap-capture-first-review", "host-gap-capture-preflight", "host-gap-capture-review", "host-gap-capture-execution", "host-gap-capture-verification", "host-gap-neutral-preflight", "host-gap-neutral-review", "outer-trigger-review", "outer-trigger-proof", "native-target-diagnosis", "native-attachment-diagnosis", "native-five-file-refusal", "native-file-limit-source", "existing-native-state-review", "final-native-identity", "final-native-rename", "final-native-duplicate", "native-layout-reconciliation", "native-runtime-observation", "frozen-native-artifacts-reverified", "model-log-redaction", "concurrent-use-clarification", "second-concurrent-use-clarification", "native-host-preflight", "native-process-credential-review", "private-screenshot-export-review", "local-model-start-recovery", "local-model-recovery-source", "local-model-recovery-review", "local-model-failed-start", "measurement-integrity" ] }, "repairedScorer": false, "engineeringRepair": true, "finalAudit": true, "artifacts": [ { "id": "paired-results", "label": "Paired calculations and unresolved bounds", "sha256": "d0962641cb099e85c964fc7cbebd4e04a56555d0c2445bf9b7bc141203be5f94", "role": "paired-results", "href": "evidence/paired-results.json", "bytes": 98582 }, { "id": "summaries", "label": "All six original-session summaries", "sha256": "8c44369c679f56858da1c0bb2ef490ac29700626b31b82e894fb6abd65e62d06", "role": "summaries", "href": "evidence/summaries.json", "bytes": 54096 }, { "id": "analysis-completion", "label": "Analysis completion and output hashes", "sha256": "4500ac6c8072a209f29f75a26f60473354bb9407e88e5eed8bfa37493a4af2a6", "role": "analysis-completion", "href": "evidence/analysis-completion.json", "bytes": 1131 }, { "id": "session-provenance", "label": "All six original-session integrity checks", "sha256": "8a54a1d98a4e787eb926a0feac3a0a67b0e8534075fd038224724167af2f917e", "role": "session-provenance", "href": "evidence/session-provenance.json", "bytes": 42541 }, { "id": "kingy-raw", "label": "All120 Kingy originals with frozen grades", "sha256": "821f2bc055e6462eb0d303e6e37b2d28f14e5502455fd9ecde9426dcb2ce5b54", "role": "kingy-raw", "href": "evidence/kingy-raw.jsonl", "bytes": 1803758 }, { "id": "native-raw", "label": "All120 native LM Studio originals with frozen grades", "sha256": "9de8b48cc63896a1d91b0d029818b1874920326dccd06809bfa31e8b01c6c3f2", "role": "native-raw", "href": "evidence/native-raw.jsonl", "bytes": 21383746 }, { "id": "protocol", "label": "Before-output comparison protocol", "sha256": "a0638194c552ef306b218d1c43a3c6adb3798507e02406c792496ef7072e3069", "role": "protocol", "href": "evidence/protocol.md", "bytes": 8726 }, { "id": "scoring", "label": "Atomic task-completeness rubric1.4.0", "sha256": "ebc5601d23f5bfd3a35f3c1e9a5a3b601a249733adb2fd2d86dd285c6bbb2e82", "role": "scoring", "href": "evidence/scoring.md", "bytes": 3883 }, { "id": "freeze", "label": "Complete method, candidate and corpus freeze", "sha256": "d1abac923f15a5b6c4a10e51b3f17938789cf99747a218e91ca5d622d22a8f6f", "role": "freeze", "href": "evidence/freeze.json", "bytes": 9669 }, { "id": "method-freeze", "label": "Method frozen before corpus authoring", "sha256": "eb135d1b71a98b762302ea4d66e86dee55514ff453eb419174baddf4a9b22182", "href": "evidence/method-freeze.json", "bytes": 8307 }, { "id": "scorer-validation", "label": "Separate-agent engineering and preserved scorer validation", "sha256": "dbdb9bc52018d8ab6bdb8379bf11e5444d7679d2f7de2cc6757604d9dcd05d6d", "href": "evidence/scorer-validation.json", "bytes": 2896 }, { "id": "scorer-review", "label": "Independent review scope and retained failures", "sha256": "ac854f2c9026dc73dc4fed178831366df2e0f7a5f2c2a0fb73add1f03ebfcf29", "href": "evidence/scorer-review.md", "bytes": 1616 }, { "id": "corpus-review", "label": "Independent every-item corpus review", "sha256": "155d2edd134a87b535c71e602e88c166fae821345c409bb52413deaef51bf073", "href": "evidence/corpus-review.json", "bytes": 208382 }, { "id": "corpus-freeze", "label": "All frozen corpus file hashes", "sha256": "6c3e12958be51f064f01e868ee51836a427479fc2400d71807553beb7774f3a5", "href": "evidence/corpus-freeze.json", "bytes": 7663 }, { "id": "qualification-decision", "label": "Actual launch decision and failed gates", "sha256": "41c7af304d4b83c290186e9b84522a2767ffbc8a421e2bf5d0738eff77b9752a", "role": "qualification-decision", "href": "evidence/qualification-decision.json", "bytes": 3870 }, { "id": "fresh-failures", "label": "Every Kingy attempt failed or unresolved under the frozen scorer", "sha256": "6081a96142de0904f690ae01fdd3618589f61c005ebe4d4d2a75d67d9458902c", "href": "evidence/fresh-failures.json", "bytes": 67908 }, { "id": "native-settings", "label": "Effective native settings checked against originals", "sha256": "d28305b7df7214eb6e4797591b90991ab6add825980bbc38a91ae10958a7668d", "role": "native-settings", "href": "evidence/native-settings.json", "bytes": 11338 }, { "id": "native-runtime", "label": "Engine and embedding artifact pins", "sha256": "d2f3a8e469022f4a1dd2b78a7ac685d21baa53afa8b6c334e9b190455a354d76", "role": "native-runtime", "href": "evidence/native-runtime.json", "bytes": 3121 }, { "id": "native-rag", "label": "Unmodified official native RAG artifact pins", "sha256": "c46211b91224f40b122b8140db0d2dd1b12f164ffbbe6047050d4da9f850a0dc", "role": "native-rag", "href": "evidence/native-rag.json", "bytes": 1268 }, { "id": "runtime-recheck", "label": "After-measurement runtime recheck", "sha256": "0cc68282fd550ac7ae7ebaf5d4e4ac714958004beeb53a416d72eeb493510566", "href": "evidence/runtime-recheck.json", "bytes": 11646 }, { "id": "history-manifest", "label": "Original immutable1.4 manifest", "sha256": "fc28d7f64e75a37768586ded40807195f55aac7d1337d27ecf7a700fbd63a4e6", "role": "history-manifest", "href": "evidence/history-manifest.json", "bytes": 3822 }, { "id": "v15-history-manifest", "label": "Original1.5.0 manifest", "sha256": "f62295ede008d7312b9dc967d7c7394fd552a3736558b04286f69b075827ed5e", "href": "evidence/v15-history-manifest.json", "bytes": 4793 }, { "id": "v17-history-manifest", "label": "Preserved1.5.2 clarified-study manifest", "sha256": "bf95982a0d533c456fe045a38f6f2e75c25f828c71c098cd88ff01b3c0143407", "href": "evidence/v17-history-manifest.json", "bytes": 6305 }, { "id": "history-preservation", "label": "Originalv18 file preservation", "sha256": "b0dcd639b88aa373b2311594389edba8294f7238eb68db548733f33e31835714", "href": "evidence/history-preservation.json", "bytes": 128583 }, { "id": "measurement-integrity", "label": "Originals sealed before reference access", "sha256": "e82d77530e170318798472bfd3485eb6e528486950608de56111adf7b5fbdd59", "href": "evidence/measurement-integrity.json", "bytes": 228385 }, { "id": "local-model-start-recovery", "label": "Zero-question Kingy startup failure and byte-identical model-link recovery", "sha256": "bdeb764fd8ab826bc03c02817ce2a78c9c051a24def8c716ffaae6a7e2278621", "href": "evidence/local-model-start-recovery.json", "bytes": 1175 }, { "id": "local-model-recovery-source", "label": "Existing local model matches the frozen GGUF bytes", "sha256": "6902fa6a07f1a6a4f45a51d9dd73be81657ba809ad5e0803a6d0cd2c54ca1cfb", "href": "evidence/local-model-recovery-source.json", "bytes": 331 }, { "id": "local-model-recovery-review", "label": "Independent zero-inference and unchanged-input recovery check", "sha256": "321fd56a80cf1848b64c0fa301054c1621481ba916be6518ac8fd0c82513a445", "href": "evidence/local-model-recovery-review.json", "bytes": 3377 }, { "id": "local-model-failed-start", "label": "Preserved zero-attempt Kingy startup failure", "sha256": "071ee06fb2657502e4b3fcd4b887d1c16efe870fffa7ad751b6e09d2c5a552ac", "href": "evidence/local-model-failed-start.json", "bytes": 4379 }, { "id": "corpus-authorship", "label": "Separate-agent authorship and counts", "sha256": "6352bb66e316bafbb708579d269711e2cd4614e788c9fc8371c25a0679ab2d8f", "href": "evidence/corpus-authorship.json", "bytes": 1086 }, { "id": "input-provenance", "label": "Exact author-pack to execution-pack copy", "sha256": "f19a99ad8d1c4548707bf7be4c6b0a9c5afd29affcff696731a9da864d2bd917", "href": "evidence/input-provenance.json", "bytes": 7401 }, { "id": "candidate-changes", "label": "Field-isolated inference repair; shared scorer unchanged", "sha256": "9dad582976341395e13965bb86bdcd6e694c7a7262f3e31d7afba26b0db4c805", "href": "evidence/candidate-changes.json", "bytes": 1313 }, { "id": "final-audit", "label": "Final independent source and scoring audit", "sha256": "eb6314a911c50064383d5a7a721c00aca5453fe893c186ceba7154c09e2f1634", "role": "final-audit", "href": "evidence/final-audit.json", "bytes": 220353 }, { "id": "final-audit-notes", "label": "Every independently reviewed response bundle", "sha256": "88314f9cc8016a977056299271e2b3100f8b4cf12589f974cd22d5784bfb0e5b", "href": "evidence/final-audit-notes.md", "bytes": 80520 }, { "id": "launch-gates", "label": "All gates declared before fresh inference", "sha256": "106c5dd5aeb8224c8b713a62d7018d77932c0fb4430ca493c9032df9808def4b", "href": "evidence/launch-gates.json", "bytes": 995 }, { "id": "measurement-completion-review", "label": "Independent frozen completion-gate interpretation: original23 unverified, candidate held", "sha256": "1543fe3733a6683228cdb1d1edbbfc554a47277e168be344c4db5cef6a49b04a", "href": "evidence/measurement-completion-review.json", "bytes": 5626 }, { "id": "completion-gate-controls", "label": "Forty publication controls, including refusal to launch with an interrupted original", "sha256": "dcc640c64327c0d78632fd2c14267129b25994f47028b75d8a892144bff73f1d", "href": "evidence/completion-gate-controls.json", "bytes": 895 }, { "id": "publication-guard-review", "label": "Independent publication guard review and closed findings", "sha256": "f2e47865a8c7b979bf4b6ecc0d49fb4738027de3d30a91d82bed9b7ff89a4542", "href": "evidence/publication-guard-review.json", "bytes": 3052 }, { "id": "d3-publication-source-review", "label": "Independent sealing and report-link checks for the distinct native transport amendment", "sha256": "21eb13ffa52cf98d34fc2503f7c6f3997ea2118f09cdc11fe2c7afcb34fdd626", "href": "evidence/d3-publication-source-review.json", "bytes": 1456 }, { "id": "native-process-credential-review", "label": "Independent local-credential export fix and thirteen focused checks", "sha256": "761a20747232f1b283a89106a11b050b10ef769262ac53290c6d1f5f8580b795", "href": "evidence/native-process-credential-review.json", "bytes": 2388 }, { "id": "private-screenshot-export-review", "label": "Independent private neutral screenshot exclusion and five scanner cases", "sha256": "a5683f2782fdddcb40c7a70af3300aa7834690864ca66026e5f4a728e4652de9", "href": "evidence/private-screenshot-export-review.json", "bytes": 1817 }, { "id": "private-diagnostic-export-review", "label": "Independent private native diagnostic exclusion and focused scanner cases", "sha256": "ac8cb59f8aac1bcd3e661b12b789b57d9127f77a146b360c7bcb67b15dfd5fa1", "href": "evidence/private-diagnostic-export-review.json", "bytes": 1022 }, { "id": "competitor-claim-boundaries", "label": "Official competitor scope and bounded claims", "sha256": "9f1e028937f0c1f06467a344b385a47c1be6798db65126ce50e9e227d342366f", "href": "evidence/competitor-claim-boundaries.json", "bytes": 1361 }, { "id": "transport-freeze", "label": "Native transport frozen before fresh corpus", "sha256": "2c187429ee5c112a14386af85d3161419af6e9c43a06639f8aa1f53f45820247", "href": "evidence/transport-freeze.json", "bytes": 1060 }, { "id": "transport-review", "label": "Independent native transport review", "sha256": "c13801f5e50bc1d36ce9b7a58be124f6ee1b29482ec4b38b54d6c5082a27c7f7", "href": "evidence/transport-review.json", "bytes": 3002 }, { "id": "readiness-addon-freeze", "label": "Bounded preparation wait added after B1 original20", "sha256": "b14f1df8baec70f274dc9288175f597a695341b3c38792c83037a8260b11f3d8", "href": "evidence/readiness-addon-freeze.json", "bytes": 625 }, { "id": "readiness-addon-review", "label": "Independent bounded-wait review and controls", "sha256": "936533a0ff58c5b5d96b90a9749d07928389d17637f00c6c41527213f6bde70c", "href": "evidence/readiness-addon-review.json", "bytes": 18061 }, { "id": "template-recovery-review", "label": "Independent original-attempt integrity and template recovery review", "sha256": "ef1bb181eb7b10edccb4767bd2c6ddd333b37c66bc08238409d17e757fc59e7a", "href": "evidence/template-recovery-review.json", "bytes": 4310 }, { "id": "replacement-template-freeze", "label": "Pinned replacement blank-template identity and configuration", "sha256": "5966dbad9c04b02bcb60e461f232cfb49c8a12c8f29445c5bdb5d2662c0b7339", "href": "evidence/replacement-template-freeze.json", "bytes": 8525 }, { "id": "replacement-template-review", "label": "Independent replacement-template validation", "sha256": "bb470c413699ca4023f9b0624d38bca635ecd4befe3267a0e466f954233981c1", "href": "evidence/replacement-template-review.json", "bytes": 3838 }, { "id": "replacement-template-verification", "label": "Native UI preservation, PDF and configuration checks", "sha256": "fdb74f24878ea3b848fc33015376fddabfd148a7f7e31675faba9235d09f47b0", "href": "evidence/replacement-template-verification.json", "bytes": 16519 }, { "id": "ax-menu-freeze", "label": "AXPress-only native menu and replacement-template pins", "sha256": "5c33516fcb42a21ce3aea7a66104a63933871f19bcdbcf2fe267c0e1e33a16c9", "href": "evidence/ax-menu-freeze.json", "bytes": 10962 }, { "id": "ax-menu-review", "label": "Independent AXPress guard and native preflight review", "sha256": "a2841253cb3c6b3b921cf4753b5c8bcee375556e9782f5fb16d1ee2d96bb3b3e", "href": "evidence/ax-menu-review.json", "bytes": 5115 }, { "id": "ax-menu-preflight", "label": "Historical zero-file native menu preflight without inference", "sha256": "397d2d130d670e3cae856b1a71223c7a3accfd8629829ba5399d7f6956d64b90", "href": "evidence/ax-menu-preflight.json", "bytes": 13056 }, { "id": "final-native-preflight", "label": "Actual ten-PDF duplicate, canonical reselection and single focused menu", "sha256": "a4c13e94dec2433f50ebfc71242823f6f7d32396d5cf0cd46b0fadcc67564eef", "href": "evidence/final-native-preflight.json", "bytes": 2157 }, { "id": "final-native-template", "label": "Actual selected model, ten PDFs and eleven frozen configuration fields", "sha256": "21585baccad70d85502cab5cd7e5747bccde1fa3e86395a977e525594b61a3f5", "href": "evidence/final-native-template.json", "bytes": 11567 }, { "id": "final-native-runtime", "label": "Actual frozen runtime2.56; credential-bearing command omitted", "sha256": "00fedc0d00fee9cfc8e07b2795bbd80bb0ae55a42502c28ce4a7ecb1157d246b", "href": "evidence/final-native-runtime.json", "bytes": 911 }, { "id": "final-native-runtime-selection", "label": "Previous and frozen runtime selection receipts", "sha256": "33eeeabf7a4d9a6f287ec1978b5ff243f8b5b3085647532b2e5bda761001d599", "href": "evidence/final-native-runtime-selection.json", "bytes": 1390 }, { "id": "native-load-warmup-final", "label": "Actual native loads, neutral warmups and final runtime-selection restoration", "sha256": "3bbc527517d7ec98ce49f9618b8ac8beb98edfafc8503699cc388143ed48f48d", "href": "evidence/native-load-warmup-final.json", "bytes": 9558 }, { "id": "b3-control-interruption", "label": "Zero-question native control-connection interruption", "sha256": "e0ddb0dba754b53b78defe3de466977a68f06a84d55f501750ef48f31e90129e", "href": "evidence/b3-control-interruption.json", "bytes": 451 }, { "id": "b3-control-recovery", "label": "Unconsumed token and zero-inference native connection recovery", "sha256": "229385f04b2d7a301774715e37a0b301d8473d4e1a033952d82b672d5ae012ef", "href": "evidence/b3-control-recovery.json", "bytes": 1018 }, { "id": "b3-attachment-guard", "label": "Rejected blank copy with duplicate backing-file identifiers before input", "sha256": "7e918272239bdced029bd788f862349f537a07fcfcb15a1d2074751baaee29ba", "href": "evidence/b3-attachment-guard.json", "bytes": 5558 }, { "id": "b3-attachment-recovery", "label": "Failed blank preserved; ten distinct frozen PDF hashes verified before original2", "sha256": "7ad0b67c659c8db05eae8282af6fc9b860a5cd4701a72497eac508918042d675", "href": "evidence/b3-attachment-recovery.json", "bytes": 2260 }, { "id": "b3-attachment-guard-22", "label": "Second rejected blank copy before original22", "sha256": "3601084edb858def503327e4fc62da50af0da4de6e493fc84be691dedf5645eb", "href": "evidence/b3-attachment-guard-22.json", "bytes": 2817 }, { "id": "b3-attachment-recovery-22", "label": "Original22 preparation recovery with failed blank preserved", "sha256": "256c34258ce741d61296bd26bdeade25d1f4332c2f21668bdd16d793baa83b99", "href": "evidence/b3-attachment-recovery-22.json", "bytes": 1022 }, { "id": "host-gap-observation", "label": "Interrupted pending native23: preserved record and log observations", "sha256": "9c03ed6dd45b0882574847fc903f3ace60784b1e8437c1bbf1fce4e37e99bafe", "href": "evidence/host-gap-observation.json", "bytes": 1325 }, { "id": "host-gap-boot", "label": "Current boot occurred after the interrupted native record", "sha256": "49fe85301e1601d4d0811b5ceacbb5dbbb4ff30534d9fe35c44f362ff35b8e1c", "href": "evidence/host-gap-boot.json", "bytes": 561 }, { "id": "host-gap-processes", "label": "Old owned processes unavailable; new main instance observed without control", "sha256": "2644ff01b7d7cc9a5e21109354aa7ae8e50cfcd4e98c397b98bf53981e8114d6", "href": "evidence/host-gap-processes.json", "bytes": 371 }, { "id": "host-gap-artifacts", "label": "All34 frozen runtime, model, app and RAG artifacts still match after restart", "sha256": "204bfc7a40bde768f389ce61ae043097079ac310dedb4e2a19275a326b07f491", "href": "evidence/host-gap-artifacts.json", "bytes": 11646 }, { "id": "host-gap-recovery-review", "label": "Independent preserved-original and capture-only continuation review", "sha256": "ed18bb70a8cc96b540d01ea717ea67e910b81088b98d2f428f1d9a06820e4c37", "href": "evidence/host-gap-recovery-review.json", "bytes": 7387 }, { "id": "host-gap-capture-first-review", "label": "Initial capture-helper review; preflight later refused an absent timestamp field", "sha256": "ff4f29c4f4f600e618775d2d7fdf8715e04e587f1bf7d9f52d810b1a011e36df", "href": "evidence/host-gap-capture-first-review.json", "bytes": 4196 }, { "id": "host-gap-capture-preflight", "label": "Initial capture preflight stopped before any action or journal write", "sha256": "4b14a92b3ee41d51e67d8bb75feb698daeb3bc9d9692f36c361bd6c1c6d8e559", "href": "evidence/host-gap-capture-preflight.json", "bytes": 842 }, { "id": "host-gap-capture-review", "label": "Exact revised capture helper and absent-or-null guard independently approved", "sha256": "fc631d7570e7a12cd7c318e4bb0bb25c455a0220a90ff4d9f9063f95cff08682", "href": "evidence/host-gap-capture-review.json", "bytes": 2833 }, { "id": "host-gap-capture-execution", "label": "Interrupted native23 retained once with zero UI/model actions and no completion claim", "sha256": "baff3cbecabae5ca557e870b215b740d53285dabeef2145734cb3f0bdd0720f3", "href": "evidence/host-gap-capture-execution.json", "bytes": 2045 }, { "id": "host-gap-capture-verification", "label": "Root verifies exact interrupted original, cleared pending slot and zero resubmissions", "sha256": "b340fff9b174c67f32b57bdbe1eef5827f15bf53334e14a25508318411ffbe5e", "href": "evidence/host-gap-capture-verification.json", "bytes": 657 }, { "id": "host-gap-neutral-preflight", "label": "Fresh post-restart neutral ten-PDF duplicate and actual frozen runtime/settings", "sha256": "139add61a0ffb12671cdb8919bbbaff12370b3e4da5b92d16f906d418eb9f6eb", "href": "evidence/host-gap-neutral-preflight.json", "bytes": 4566 }, { "id": "host-gap-neutral-review", "label": "Independent binding of fresh preflight and conditional continuation without retries", "sha256": "2af2814fee61d1cbe6b68613dffa534bb04a8d581777b269b549cc4621d7faf2", "href": "evidence/host-gap-neutral-review.json", "bytes": 4395 }, { "id": "focus-pilot-review", "label": "Focus-only pilot stopped: CUA metadata did not establish the required boundary", "sha256": "56da125cf2c043c4a045ef91a93d72bccf3bd2c5bfce38f01ada8d485572638b", "href": "evidence/focus-pilot-review.json", "bytes": 2455 }, { "id": "focus-pilot-discrepancy", "label": "Native focus observation and unavailable CUA focus metadata", "sha256": "61d4ea0a1a66c952bc75a79eef5e01967f2a6062d20e4e3b4e46269c1344d967", "href": "evidence/focus-pilot-discrepancy.json", "bytes": 1192 }, { "id": "duplicate-boundary-freeze", "label": "Distinct source pins for actual native Duplicate boundary", "sha256": "8a5f05ec2c5b18111fe3a0e8083593c6c788ad825c656561feaec9b892feb6ba", "href": "evidence/duplicate-boundary-freeze.json", "bytes": 4939 }, { "id": "duplicate-boundary-review", "label": "Separate-agent source review for the finite native Duplicate boundary", "sha256": "7afc7124484d33b298eb18b007d87de1cdff421c49334c3a0aa566a622e5802b", "href": "evidence/duplicate-boundary-review.json", "bytes": 17004 }, { "id": "duplicate-neutral-proof", "label": "Actual same single blank copy, ten PDF hashes, settings and visible previews", "sha256": "35bab062e5d5bae65f59f4f1e6216822f879fbdb55f9e1576a8a7c934ecc93fa", "href": "evidence/duplicate-neutral-proof.json", "bytes": 1737 }, { "id": "duplicate-neutral-review", "label": "Independent actual neutral-copy binding with late acknowledgment disclosed", "sha256": "c3479a27ec20fe37d8a7048d7445c63b86f94e9ccec93cb3542a88af8f82959f", "href": "evidence/duplicate-neutral-review.json", "bytes": 5353 }, { "id": "resumed-original-settings", "label": "Actual resumed original24 model and five generation/configuration groups match the frozen baseline", "sha256": "44d4cbebe895e314f7d59d7c7405279a581669eaa679d7c8557370345c3c29a2", "href": "evidence/resumed-original-settings.json", "bytes": 509 }, { "id": "outer-trigger-review", "label": "Independent outer popup trigger and overlay guard review", "sha256": "de1f5e0b8cbb0bfba08905d4a352c04095531cb78d0e58d29253dc874b6ffb0e", "href": "evidence/outer-trigger-review.json", "bytes": 3521 }, { "id": "outer-trigger-proof", "label": "Actual single focused menu from one neutral outer AXPress", "sha256": "6a1201d26454e1d574d12f68be8dd5f42e0594e55c52455ebff6871003d3cd36", "href": "evidence/outer-trigger-proof.json", "bytes": 1007 }, { "id": "native-target-diagnosis", "label": "Installed native UI target diagnosis; vendor code unchanged", "sha256": "7d6a49f76f3057117b9e5f500b5cbfa028b8ca761e40427720574dce41177c52", "href": "evidence/native-target-diagnosis.json", "bytes": 1786 }, { "id": "native-attachment-diagnosis", "label": "Read-only installed attachment callback diagnosis", "sha256": "81c8ab712b3740e1ad67a78592e7ce41cc57c0d3a87d8807217ebe805f7a6f64", "href": "evidence/native-attachment-diagnosis.json", "bytes": 1051 }, { "id": "native-five-file-refusal", "label": "Observed refusal of a sixth sequential native PDF upload", "sha256": "39b6e43d7ce11f6cbe8c4f89093f9c26dfcf18aacc1cccda1f0776916fba72fb", "href": "evidence/native-five-file-refusal.json", "bytes": 1101 }, { "id": "native-file-limit-source", "label": "Read-only installed default file-count guard diagnosis", "sha256": "dbba175bf16a9da71e8677b666a84111d9d85fdc74493fde8da151d8b890298e", "href": "evidence/native-file-limit-source.json", "bytes": 1411 }, { "id": "existing-native-state-review", "label": "Independent conditional reuse review of an unused10-PDF native blank", "sha256": "9bb6df63636a848f14ab14c72e5c5d0d798e840a7d34fbe12001d69ff81acebf", "href": "evidence/existing-native-state-review.json", "bytes": 6283 }, { "id": "final-native-identity", "label": "Native Reveal proves the exact unused ten-PDF blank identity", "sha256": "7b544f98ca6d2187205ee403f0edcbe290d6c059708e71e297c14df49c8acb25", "href": "evidence/final-native-identity.json", "bytes": 2929 }, { "id": "final-native-rename", "label": "Only the verified ten-PDF blank name changed", "sha256": "309957ed021bd6be9a834622ca871d5afc33bae88063eb02cca173b8517133d0", "href": "evidence/final-native-rename.json", "bytes": 728 }, { "id": "final-native-duplicate", "label": "Actual ten-PDF blank duplicate and screenshot API preflight", "sha256": "c6dfcfd57d74a19a7a994956555e80401cfba8d80d473173f23f53f966e118f5", "href": "evidence/final-native-duplicate.json", "bytes": 2476 }, { "id": "native-layout-reconciliation", "label": "Actual layout recovery and corrected sheet-count diagnostic", "sha256": "87233400a0da04a7d84511cd2c66c9fc01c43a0cb93ef057ed3267d693580b80", "href": "evidence/native-layout-reconciliation.json", "bytes": 1838 }, { "id": "native-runtime-observation", "label": "Unrelated model and changed runtime selection observed during preparation", "sha256": "4b569b63b45c83edb7d3c326dad4f7c7fc098f1deb1e1d67703ec81b897f94ea", "href": "evidence/native-runtime-observation.json", "bytes": 1304 }, { "id": "frozen-native-artifacts-reverified", "label": "All34 frozen app, model, RAG and runtime artifacts unchanged", "sha256": "563a05b0e997b75bbb98c47af527cb49e1794f7e197234424c7b5c17bd4c85fc", "href": "evidence/frozen-native-artifacts-reverified.json", "bytes": 11646 }, { "id": "model-log-redaction", "label": "Public log derivative: four unplanned events omitted, originals preserved", "sha256": "6e794c8f6fdbdd7eb4a48d56898ec32c031372e8dc2cb3ce021dcdfc081900f7", "href": "evidence/model-log-redaction.json", "bytes": 1999 }, { "id": "concurrent-use-clarification", "label": "User-reported concurrent use during preparation", "sha256": "a2cf25b595ddf7801ef52774b0b0aca5426f1af5f08b6fb89b7727d3c6ab2d9e", "href": "evidence/concurrent-use-clarification.json", "bytes": 361 }, { "id": "second-concurrent-use-clarification", "label": "Later draft-only activity: user-reported concurrent Mac use", "sha256": "2e8668b0af62ad92e07744d9b200f7848ab6d95937caf68ae5d0e6cbd3b175e7", "href": "evidence/second-concurrent-use-clarification.json", "bytes": 653 }, { "id": "native-host-preflight", "label": "Separate local job observed; new measurements held", "sha256": "26a8bb2f2b962575c2614f2b9fc574fd2ce2ec64ca19955c3cad51677fa3b5ea", "href": "evidence/native-host-preflight.json", "bytes": 1002 }, { "id": "descriptive-latency", "label": "Descriptive paired timing", "sha256": "d674601703f8db2c29afa17a893cfedd9ab655112fb86bbc657bb612faa6c889", "href": "evidence/descriptive-latency.json", "bytes": 571 }, { "id": "installation", "label": "Clean package installation, authentication and denied-egress checks", "sha256": "72bed774e85968acc494fe58558068540b2522c58d64369fa483f94eb0c467b1", "role": "installation", "href": "evidence/installation.json", "bytes": 1654 }, { "id": "package-security-precheck", "label": "Release archive path and private-key checks", "sha256": "1952349d5dab7266f486e71bb5ca40ef31fc108856f0a3a7a9ce2581a90b9742", "href": "evidence/package-security-precheck.json", "bytes": 274 }, { "id": "package-selftest-raw", "label": "Four public package self-test originals; not held-out qualification", "sha256": "79d9bec1adcc48dd6dc5b0bee9fbff7aea6071250ba0af9a46dba07241c74616", "href": "evidence/package-selftest-raw.jsonl", "bytes": 47043 }, { "id": "package-selftest-summary", "label": "Public package self-test summary", "sha256": "1db58db5d2bf63213df2a527406c3182c227bec6b6ee242ae326fab4d8971259", "href": "evidence/package-selftest-summary.json", "bytes": 1376 }, { "id": "package-network", "label": "Actual local authentication and denied-egress probes", "sha256": "77f7a86eb014d334417ba7d0525d0b676a5a0af3359faf3c2fcb31f7b8714560", "href": "evidence/package-network.json", "bytes": 639 }, { "id": "recipe-package", "label": "Candidate archive file hashes", "sha256": "80dae80ad5db00781c22bc1683fb56c02c15bb063cbe17b8e5cf2515632ef705", "href": "evidence/recipe-package.json", "bytes": 16344 }, { "id": "kingy-settings", "label": "Actual Kingy inference configuration", "sha256": "70725499b0b69ba1a5eea0b9ec59ea76bfe800c8df9b12e62152972c89242233", "role": "kingy-settings", "href": "evidence/kingy-settings.json", "bytes": 1688 }, { "id": "control-provenance", "label": "Frozen control and original-attempt provenance", "sha256": "356a00a3a026daa3e76e8246360f97eb6c19cff7475a8f468aa303bde31c7fdf", "role": "control-provenance", "href": "evidence/control-provenance.json", "bytes": 12208 }, { "id": "conditions", "label": "Shared-Mac conditions and neutral warmups", "sha256": "a96f037a03c5045d7b37fff15d93a0e8f9acc9b86d30e0505580f4f955e34c74", "href": "evidence/conditions.json", "bytes": 102947 }, { "id": "study-materials", "label": "Audit bundle: exact frozen inputs and all originals", "sha256": "f5b93cd59a8d41d47c40536d16318edef0eb3c6801971c68d597d773a130eab6", "href": "evidence/study-materials.zip", "bytes": 11513384 } ] }, "analysisCompletedAt": "2026-10-17T23:17:24.253519+00:00", "uniqueQuestionClusters": 40, "repetitions": 3, "attemptsPerWorkflow": 120, "summaries": { "A1": { "attempts": 40, "answer_correctness": { "passed": 24, "total": 30, "unresolved": 0, "lower": 0.8, "upper": 0.8 }, "supported_answer_success": { "passed": 23, "total": 30, "unresolved": 1, "lower": 0.7666666666666667, "upper": 0.8 }, "abstention_accuracy": { "passed": 10, "total": 10, "unresolved": 0, "lower": 1, "upper": 1 }, "citation_precision": { "passed": 27, "total": 28, "unresolved": 1, "lower": 0.9642857142857143, "upper": 1 }, "unresolved_questions": 1, "explicit_page_citations": 27, "uniquely_quote_mapped_citations": 0, "all_semantic_judgments_resolved": false, "qualified": false, "complete_planned_ids": true, "thresholds": { "supported_answer_success": 0.9, "citation_precision": 0.95, "abstention_accuracy": 0.9 }, "thresholds_passed_on_lower_bounds": { "supported_answer_success": false, "citation_precision": true, "abstention_accuracy": true }, "latency": { "n": 40, "unavailable": 0, "p50": 5.948283728997922, "p95": 9.807647916022688, "max": 13.89305316703394, "errors_retained": 0, "timeouts_retained": 0, "boundary": null }, "answer_model_request_count": 90, "original_session_receipt": { "original_summary": { "executor_version": "kingy-comparison-session/1.0.0", "system": "A", "repetition": 1, "status": "complete", "qualificationGranted": false, "complete_query_deadline_seconds": 90, "generation_seed": 271828, "client_offline_profile": "offline.sb", "client_offline_profile_sha256": "e5462ef1daae3785012cb8e4fa05b89105bf4ac4d98be9580a8c357365a4f54f", "timing_boundary": "runner.ask entry through its complete returned answer, source resolution and sufficiency check; excludes prior PDF import/index, cold start, warmup and later grading.", "cache_policy": "cache_prompt false; no claim that OS file caches were flushed.", "freeze_sha256": "d1abac923f15a5b6c4a10e51b3f17938789cf99747a218e91ca5d622d22a8f6f", "schedule_sha256": "a568a9bbf9a68dcb23a8bc0191cc151be64ab27e932a62873ba616eed2a1215c", "questions_sha256": "677e88becba7b22d057e5f172421686c61cecf79d170eb0b56d1c9232d51be90", "executor_sha256": "9e8e702e3967ab53f501de1145f26c98ff9582ae7867ab8853cd17f91babf1e2", "runner_sha256": "911298ba21797fe0704874ed85c8f306193de2a69d130ea7ebb32b18bb9064f7", "model_sha256": "7485fe6f11af29433bc51cab58009521f205840f5b4ae3a32fa7f92e8534fdf5", "corpus_pdf_sha256": { "V19-D01.pdf": "c6e49b6cd7067bfac30a956e09d00aaee438b2d021aae474d33a3a23a05694c7", "V19-D02.pdf": "e7bf34f61d9835bcac343fa16fd1bd5754ef97a071fdbea6562171bfaa569cdf", "V19-D03.pdf": "6a04c61ec3b02ef5a08dd3d648cc369f08da86366becc786d44c0abb1a5afe72", "V19-D04.pdf": "7c2dd8a8f3696639941362cdfab9bdac0f35735296168059f95ff4a869a87bc9", "V19-D05.pdf": "009256748075d06133d973fb17dcb28479e81f0325d73978903c160fd4feadb6", "V19-D06.pdf": "f9a23da40c1781721167177c61ebce0129f0fabca7a5bc346ae8af388db800cf", "V19-D07.pdf": "170213e1018f0e0593aec90f515c7563a45f7ae786c4832ca325c682e4507c8b", "V19-D08.pdf": "139ff8485db0f1ae53bf676522fe83656c305d1312ea4657ac119742a143a768", "V19-D09.pdf": "91d65f821a4edf1fb8c9689db251b82edfa32c08cb7445e8b23f15466001fbd0", "V19-D10.pdf": "05b38d3505e20a975f854b22e3b05d7178880fd16f88fa60b5db6f0bbaad3562" }, "planned_question_ids": [ "V19-Q07", "V19-Q33", "V19-Q05", "V19-Q39", "V19-Q36", "V19-Q29", "V19-Q16", "V19-Q13", "V19-Q28", "V19-Q22", "V19-Q09", "V19-Q35", "V19-Q26", "V19-Q14", "V19-Q25", "V19-Q21", "V19-Q04", "V19-Q02", "V19-Q24", "V19-Q38", "V19-Q01", "V19-Q40", "V19-Q37", "V19-Q34", "V19-Q20", "V19-Q31", "V19-Q10", "V19-Q30", "V19-Q08", "V19-Q11", "V19-Q17", "V19-Q23", "V19-Q18", "V19-Q32", "V19-Q15", "V19-Q03", "V19-Q12", "V19-Q19", "V19-Q27", "V19-Q06" ], "session_order": [ "A1", "B1", "B2", "A2", "A3", "B3" ], "artifact_verification_seconds": 2.5469693340128288, "runtime": { "name": "llama.cpp", "binary_sha256": "51afbf4ea6485b9ada19e99af97181f12baca42ef704b1e291c26a9ac0644eae", "version": { "command": [ "/opt/homebrew/Cellar/llama.cpp/0.6.0/bin/llama-server", "--version" ], "exit_code": 0, "stdout": "", "stderr": "version: 0.6.0 (build 11429, commit d81235049)\nbuilt with AppleClang 21.0.0.21000334 for Darwin arm64" } }, "before": { "os": { "command": [ "sw_vers" ], "exit_code": 0, "stdout": "ProductName:\t\tmacOS\nProductVersion:\t\t26.3\nBuildVersion:\t\t25D125", "stderr": "" }, "hardware": { "command": [ "sysctl", "hw.model", "hw.memsize", "hw.physicalcpu", "hw.logicalcpu" ], "exit_code": 0, "stdout": "hw.model: Mac16,7\nhw.memsize: 25769803776\nhw.physicalcpu: 14\nhw.logicalcpu: 14", "stderr": "" }, "memory_pressure": { "command": [ "memory_pressure", "-Q" ], "exit_code": 0, "stdout": "The system has 25769803776 (1572864 pages with a page size of 16384).\nSystem-wide memory free percentage: 39%", "stderr": "" }, "swap": { "command": [ "sysctl", "-n", "vm.swapusage" ], "exit_code": 0, "stdout": "total = 10240.00M used = 9358.75M free = 881.25M (encrypted)", "stderr": "" }, "disk": { "command": [ "df", "-h", "/" ], "exit_code": 0, "stdout": "Filesystem Size Used Avail Capacity iused ifree %iused Mounted on\n/dev/disk3s1s1 460Gi 11Gi 35Gi 25% 453k 365M 0% /", "stderr": "" }, "ac_power": { "command": [ "pmset", "-g", "batt" ], "exit_code": 0, "stdout": "Now drawing from 'Battery Power'\n -InternalBattery-0 (id=22216803)\t73%; discharging; 1:31 remaining present: true", "stderr": "" }, "power_mode": { "command": [ "pmset", "-g", "custom" ], "exit_code": 0, "stdout": "Battery Power:\n Sleep On Power Button 1\n powermode 0\n standby 0\n ttyskeepawake 1\n hibernatemode 3\n powernap 0\n hibernatefile /var/vm/sleepimage\n displaysleep 0\n womp 0\n networkoversleep 0\n sleep 0\n lessbright 1\n tcpkeepalive 1\n disksleep 0\n SleepServices 0\nAC Power:\n Sleep On Power Button 1\n powermode 0\n standby 0\n ttyskeepawake 1\n hibernatemode 3\n powernap 0\n hibernatefile /var/vm/sleepimage\n displaysleep 0\n womp 1\n networkoversleep 0\n sleep 0\n tcpkeepalive 1\n disksleep 0\n SleepServices 0", "stderr": "" }, "thermal": { "command": [ "pmset", "-g", "therm" ], "exit_code": 0, "stdout": "Note: No thermal warning level has been recorded\nNote: No performance warning level has been recorded\nNote: No CPU power status has been recorded", "stderr": "" }, "chip_inventory": [ { "chip_type": "Apple M4 Pro", "physical_memory": "24 GB", "number_processors": "proc 14:10:4", "machine_model": "Mac16,7", "machine_name": "MacBook Pro" } ] }, "task_owned_server_pid": 94929, "cold_process_to_health_seconds": 2.293061000003945, "ready_model_identity": { "models": [ { "name": "kingy-qwen3-4b", "model": "kingy-qwen3-4b", "modified_at": "", "size": "", "digest": "", "type": "model", "description": "", "tags": [ "" ], "capabilities": [ "completion" ], "parameters": "", "details": { "parent_model": "", "format": "gguf", "family": "", "families": [ "" ], "parameter_size": "", "quantization_level": "" } } ], "object": "list", "data": [ { "id": "kingy-qwen3-4b", "aliases": [ "kingy-qwen3-4b" ], "tags": [], "object": "model", "architecture": { "input_modalities": [ "text" ], "output_modalities": [ "text" ] }, "created": 1792096606, "owned_by": "llamacpp", "meta": { "vocab_type": 2, "n_vocab": 151936, "n_ctx": 8192, "n_ctx_train": 40960, "n_embd": 2560, "n_params": 4022468096, "size": 2491323904, "ftype": "Q4_K - Medium" } } ] }, "warmup_model_request_count": 1, "warmup_seconds": 0.09339508303673938, "parse_seconds": 0.03941287496127188, "index_seconds": 0.0026023330283351243, "loaded_pdf_pages": 30, "retained_attempts": 40, "planned_attempts": 40, "unexecuted_question_ids": [], "unexecuted_attempt_count": 0, "model_request_count": 90, "session_total_model_request_count": 91, "attempt_errors": 0, "timeouts": 0, "after": { "os": { "command": [ "sw_vers" ], "exit_code": 0, "stdout": "ProductName:\t\tmacOS\nProductVersion:\t\t26.3\nBuildVersion:\t\t25D125", "stderr": "" }, "hardware": { "command": [ "sysctl", "hw.model", "hw.memsize", "hw.physicalcpu", "hw.logicalcpu" ], "exit_code": 0, "stdout": "hw.model: Mac16,7\nhw.memsize: 25769803776\nhw.physicalcpu: 14\nhw.logicalcpu: 14", "stderr": "" }, "memory_pressure": { "command": [ "memory_pressure", "-Q" ], "exit_code": 0, "stdout": "The system has 25769803776 (1572864 pages with a page size of 16384).\nSystem-wide memory free percentage: 44%", "stderr": "" }, "swap": { "command": [ "sysctl", "-n", "vm.swapusage" ], "exit_code": 0, "stdout": "total = 13312.00M used = 11993.00M free = 1319.00M (encrypted)", "stderr": "" }, "disk": { "command": [ "df", "-h", "/" ], "exit_code": 0, "stdout": "Filesystem Size Used Avail Capacity iused ifree %iused Mounted on\n/dev/disk3s1s1 460Gi 11Gi 32Gi 27% 453k 330M 0% /", "stderr": "" }, "ac_power": { "command": [ "pmset", "-g", "batt" ], "exit_code": 0, "stdout": "Now drawing from 'AC Power'\n -InternalBattery-0 (id=22216803)\t67%; charging; (no estimate) present: true", "stderr": "" }, "power_mode": { "command": [ "pmset", "-g", "custom" ], "exit_code": 0, "stdout": "Battery Power:\n Sleep On Power Button 1\n powermode 0\n standby 0\n ttyskeepawake 1\n hibernatemode 3\n powernap 0\n hibernatefile /var/vm/sleepimage\n displaysleep 0\n womp 0\n networkoversleep 0\n sleep 0\n lessbright 1\n tcpkeepalive 1\n disksleep 0\n SleepServices 0\nAC Power:\n Sleep On Power Button 1\n powermode 0\n standby 0\n ttyskeepawake 1\n hibernatemode 3\n powernap 0\n hibernatefile /var/vm/sleepimage\n displaysleep 0\n womp 1\n networkoversleep 0\n sleep 0\n tcpkeepalive 1\n disksleep 0\n SleepServices 0", "stderr": "" }, "thermal": { "command": [ "pmset", "-g", "therm" ], "exit_code": 0, "stdout": "Note: No thermal warning level has been recorded\nNote: No performance warning level has been recorded\nNote: No CPU power status has been recorded", "stderr": "" }, "chip_inventory": [ { "chip_type": "Apple M4 Pro", "physical_memory": "24 GB", "number_processors": "proc 14:10:4", "machine_model": "Mac16,7", "machine_name": "MacBook Pro" } ] }, "session_wall_seconds": 262.583097041992, "frozen_inputs_unchanged": true }, "integrity_issues": [], "source_sha256": { "session-plan.json": "aadf624ad1e838fdbe76da950728df1f5c9ade8d9697ad403bca3b557292d31f", "session-summary.json": "c8ee19b52893571715d7c4b3a0c2197cb3dd352aa1aa2e3addd4726b1394896d", "attempts.jsonl": "288b5439384436ef6f9affd58739866a549569d05c8155e408e722f50c397adc" }, "complete_40_original_rows": true }, "provenance_verified": true }, "A2": { "attempts": 40, "answer_correctness": { "passed": 24, "total": 30, "unresolved": 0, "lower": 0.8, "upper": 0.8 }, "supported_answer_success": { "passed": 23, "total": 30, "unresolved": 1, "lower": 0.7666666666666667, "upper": 0.8 }, "abstention_accuracy": { "passed": 10, "total": 10, "unresolved": 0, "lower": 1, "upper": 1 }, "citation_precision": { "passed": 27, "total": 28, "unresolved": 1, "lower": 0.9642857142857143, "upper": 1 }, "unresolved_questions": 1, "explicit_page_citations": 27, "uniquely_quote_mapped_citations": 0, "all_semantic_judgments_resolved": false, "qualified": false, "complete_planned_ids": true, "thresholds": { "supported_answer_success": 0.9, "citation_precision": 0.95, "abstention_accuracy": 0.9 }, "thresholds_passed_on_lower_bounds": { "supported_answer_success": false, "citation_precision": true, "abstention_accuracy": true }, "latency": { "n": 40, "unavailable": 0, "p50": 3.6546819164996123, "p95": 7.422845583001617, "max": 7.787626666999131, "errors_retained": 0, "timeouts_retained": 0, "boundary": null }, "answer_model_request_count": 90, "original_session_receipt": { "original_summary": { "executor_version": "kingy-comparison-session/1.0.0", "system": "A", "repetition": 2, "status": "complete", "qualificationGranted": false, "complete_query_deadline_seconds": 90, "generation_seed": 271828, "client_offline_profile": "offline.sb", "client_offline_profile_sha256": "e5462ef1daae3785012cb8e4fa05b89105bf4ac4d98be9580a8c357365a4f54f", "timing_boundary": "runner.ask entry through its complete returned answer, source resolution and sufficiency check; excludes prior PDF import/index, cold start, warmup and later grading.", "cache_policy": "cache_prompt false; no claim that OS file caches were flushed.", "freeze_sha256": "d1abac923f15a5b6c4a10e51b3f17938789cf99747a218e91ca5d622d22a8f6f", "schedule_sha256": "a568a9bbf9a68dcb23a8bc0191cc151be64ab27e932a62873ba616eed2a1215c", "questions_sha256": "677e88becba7b22d057e5f172421686c61cecf79d170eb0b56d1c9232d51be90", "executor_sha256": "9e8e702e3967ab53f501de1145f26c98ff9582ae7867ab8853cd17f91babf1e2", "runner_sha256": "911298ba21797fe0704874ed85c8f306193de2a69d130ea7ebb32b18bb9064f7", "model_sha256": "7485fe6f11af29433bc51cab58009521f205840f5b4ae3a32fa7f92e8534fdf5", "corpus_pdf_sha256": { "V19-D01.pdf": "c6e49b6cd7067bfac30a956e09d00aaee438b2d021aae474d33a3a23a05694c7", "V19-D02.pdf": "e7bf34f61d9835bcac343fa16fd1bd5754ef97a071fdbea6562171bfaa569cdf", "V19-D03.pdf": "6a04c61ec3b02ef5a08dd3d648cc369f08da86366becc786d44c0abb1a5afe72", "V19-D04.pdf": "7c2dd8a8f3696639941362cdfab9bdac0f35735296168059f95ff4a869a87bc9", "V19-D05.pdf": "009256748075d06133d973fb17dcb28479e81f0325d73978903c160fd4feadb6", "V19-D06.pdf": "f9a23da40c1781721167177c61ebce0129f0fabca7a5bc346ae8af388db800cf", "V19-D07.pdf": "170213e1018f0e0593aec90f515c7563a45f7ae786c4832ca325c682e4507c8b", "V19-D08.pdf": "139ff8485db0f1ae53bf676522fe83656c305d1312ea4657ac119742a143a768", "V19-D09.pdf": "91d65f821a4edf1fb8c9689db251b82edfa32c08cb7445e8b23f15466001fbd0", "V19-D10.pdf": "05b38d3505e20a975f854b22e3b05d7178880fd16f88fa60b5db6f0bbaad3562" }, "planned_question_ids": [ "V19-Q23", "V19-Q30", "V19-Q02", "V19-Q07", "V19-Q40", "V19-Q20", "V19-Q28", "V19-Q27", "V19-Q01", "V19-Q12", "V19-Q24", "V19-Q26", "V19-Q22", "V19-Q31", "V19-Q19", "V19-Q32", "V19-Q06", "V19-Q36", "V19-Q03", "V19-Q04", "V19-Q35", "V19-Q15", "V19-Q16", "V19-Q11", "V19-Q29", "V19-Q18", "V19-Q09", "V19-Q17", "V19-Q14", "V19-Q21", "V19-Q05", "V19-Q10", "V19-Q39", "V19-Q08", "V19-Q25", "V19-Q34", "V19-Q38", "V19-Q13", "V19-Q37", "V19-Q33" ], "session_order": [ "A1", "B1", "B2", "A2", "A3", "B3" ], "artifact_verification_seconds": 1.9707937499988475, "runtime": { "name": "llama.cpp", "binary_sha256": "51afbf4ea6485b9ada19e99af97181f12baca42ef704b1e291c26a9ac0644eae", "version": { "command": [ "/opt/homebrew/Cellar/llama.cpp/0.6.0/bin/llama-server", "--version" ], "exit_code": 0, "stdout": "", "stderr": "version: 0.6.0 (build 11429, commit d81235049)\nbuilt with AppleClang 21.0.0.21000334 for Darwin arm64" } }, "before": { "os": { "command": [ "sw_vers" ], "exit_code": 0, "stdout": "ProductName:\t\tmacOS\nProductVersion:\t\t26.3\nBuildVersion:\t\t25D125", "stderr": "" }, "hardware": { "command": [ "sysctl", "hw.model", "hw.memsize", "hw.physicalcpu", "hw.logicalcpu" ], "exit_code": 0, "stdout": "hw.model: Mac16,7\nhw.memsize: 25769803776\nhw.physicalcpu: 14\nhw.logicalcpu: 14", "stderr": "" }, "memory_pressure": { "command": [ "memory_pressure", "-Q" ], "exit_code": 0, "stdout": "The system has 25769803776 (1572864 pages with a page size of 16384).\nSystem-wide memory free percentage: 73%", "stderr": "" }, "swap": { "command": [ "sysctl", "-n", "vm.swapusage" ], "exit_code": 0, "stdout": "total = 15360.00M used = 13884.56M free = 1475.44M (encrypted)", "stderr": "" }, "disk": { "command": [ "df", "-h", "/" ], "exit_code": 0, "stdout": "Filesystem Size Used Avail Capacity iused ifree %iused Mounted on\n/dev/disk3s1s1 460Gi 11Gi 23Gi 34% 453k 238M 0% /", "stderr": "" }, "ac_power": { "command": [ "pmset", "-g", "batt" ], "exit_code": 0, "stdout": "Now drawing from 'Battery Power'\n -InternalBattery-0 (id=21692515)\t22%; discharging; 0:48 remaining present: true", "stderr": "" }, "power_mode": { "command": [ "pmset", "-g", "custom" ], "exit_code": 0, "stdout": "Battery Power:\n Sleep On Power Button 1\n powermode 0\n standby 0\n ttyskeepawake 1\n hibernatemode 3\n powernap 0\n hibernatefile /var/vm/sleepimage\n displaysleep 0\n womp 0\n networkoversleep 0\n sleep 0\n lessbright 1\n tcpkeepalive 1\n disksleep 0\n SleepServices 0\nAC Power:\n Sleep On Power Button 1\n powermode 0\n standby 0\n ttyskeepawake 1\n hibernatemode 3\n powernap 0\n hibernatefile /var/vm/sleepimage\n displaysleep 0\n womp 1\n networkoversleep 0\n sleep 0\n tcpkeepalive 1\n disksleep 0\n SleepServices 0", "stderr": "" }, "thermal": { "command": [ "pmset", "-g", "therm" ], "exit_code": 0, "stdout": "Note: No thermal warning level has been recorded\nNote: No performance warning level has been recorded\nNote: No CPU power status has been recorded", "stderr": "" }, "chip_inventory": [ { "chip_type": "Apple M4 Pro", "physical_memory": "24 GB", "number_processors": "proc 14:10:4", "machine_model": "Mac16,7", "machine_name": "MacBook Pro" } ] }, "task_owned_server_pid": 96620, "cold_process_to_health_seconds": 2.0431850000022678, "ready_model_identity": { "models": [ { "name": "kingy-qwen3-4b", "model": "kingy-qwen3-4b", "modified_at": "", "size": "", "digest": "", "type": "model", "description": "", "tags": [ "" ], "capabilities": [ "completion" ], "parameters": "", "details": { "parent_model": "", "format": "gguf", "family": "", "families": [ "" ], "parameter_size": "", "quantization_level": "" } } ], "object": "list", "data": [ { "id": "kingy-qwen3-4b", "aliases": [ "kingy-qwen3-4b" ], "tags": [], "object": "model", "architecture": { "input_modalities": [ "text" ], "output_modalities": [ "text" ] }, "created": 1792225281, "owned_by": "llamacpp", "meta": { "vocab_type": 2, "n_vocab": 151936, "n_ctx": 8192, "n_ctx_train": 40960, "n_embd": 2560, "n_params": 4022468096, "size": 2491323904, "ftype": "Q4_K - Medium" } } ] }, "warmup_model_request_count": 1, "warmup_seconds": 0.0655304169995361, "parse_seconds": 0.036168124999676365, "index_seconds": 0.0016832500004966278, "loaded_pdf_pages": 30, "retained_attempts": 40, "planned_attempts": 40, "unexecuted_question_ids": [], "unexecuted_attempt_count": 0, "model_request_count": 90, "session_total_model_request_count": 91, "attempt_errors": 0, "timeouts": 0, "after": { "os": { "command": [ "sw_vers" ], "exit_code": 0, "stdout": "ProductName:\t\tmacOS\nProductVersion:\t\t26.3\nBuildVersion:\t\t25D125", "stderr": "" }, "hardware": { "command": [ "sysctl", "hw.model", "hw.memsize", "hw.physicalcpu", "hw.logicalcpu" ], "exit_code": 0, "stdout": "hw.model: Mac16,7\nhw.memsize: 25769803776\nhw.physicalcpu: 14\nhw.logicalcpu: 14", "stderr": "" }, "memory_pressure": { "command": [ "memory_pressure", "-Q" ], "exit_code": 0, "stdout": "The system has 25769803776 (1572864 pages with a page size of 16384).\nSystem-wide memory free percentage: 70%", "stderr": "" }, "swap": { "command": [ "sysctl", "-n", "vm.swapusage" ], "exit_code": 0, "stdout": "total = 16384.00M used = 15183.62M free = 1200.38M (encrypted)", "stderr": "" }, "disk": { "command": [ "df", "-h", "/" ], "exit_code": 0, "stdout": "Filesystem Size Used Avail Capacity iused ifree %iused Mounted on\n/dev/disk3s1s1 460Gi 11Gi 22Gi 35% 453k 227M 0% /", "stderr": "" }, "ac_power": { "command": [ "pmset", "-g", "batt" ], "exit_code": 0, "stdout": "Now drawing from 'Battery Power'\n -InternalBattery-0 (id=21692515)\t12%; discharging; 0:23 remaining present: true", "stderr": "" }, "power_mode": { "command": [ "pmset", "-g", "custom" ], "exit_code": 0, "stdout": "Battery Power:\n Sleep On Power Button 1\n powermode 0\n standby 0\n ttyskeepawake 1\n hibernatemode 3\n powernap 0\n hibernatefile /var/vm/sleepimage\n displaysleep 0\n womp 0\n networkoversleep 0\n sleep 0\n lessbright 1\n tcpkeepalive 1\n disksleep 0\n SleepServices 0\nAC Power:\n Sleep On Power Button 1\n powermode 0\n standby 0\n ttyskeepawake 1\n hibernatemode 3\n powernap 0\n hibernatefile /var/vm/sleepimage\n displaysleep 0\n womp 1\n networkoversleep 0\n sleep 0\n tcpkeepalive 1\n disksleep 0\n SleepServices 0", "stderr": "" }, "thermal": { "command": [ "pmset", "-g", "therm" ], "exit_code": 0, "stdout": "Note: No thermal warning level has been recorded\nNote: No performance warning level has been recorded\nNote: No CPU power status has been recorded", "stderr": "" }, "chip_inventory": [ { "chip_type": "Apple M4 Pro", "physical_memory": "24 GB", "number_processors": "proc 14:10:4", "machine_model": "Mac16,7", "machine_name": "MacBook Pro" } ] }, "session_wall_seconds": 170.39064462500028, "frozen_inputs_unchanged": true }, "integrity_issues": [], "source_sha256": { "session-plan.json": "a09b1e768e3820e10b2bf7096b49054ec424ab960bda7a4dcec8402666a6403b", "session-summary.json": "1d1c7d4841af89313a1d35b7b8f49addb5d5bffba60a27acedbaf55c93793f12", "attempts.jsonl": "01342616a9dfb753ed58aca815c3c532c921bb2879d0c1208512b1d986bdf269" }, "complete_40_original_rows": true }, "provenance_verified": true }, "A3": { "attempts": 40, "answer_correctness": { "passed": 24, "total": 30, "unresolved": 0, "lower": 0.8, "upper": 0.8 }, "supported_answer_success": { "passed": 23, "total": 30, "unresolved": 1, "lower": 0.7666666666666667, "upper": 0.8 }, "abstention_accuracy": { "passed": 10, "total": 10, "unresolved": 0, "lower": 1, "upper": 1 }, "citation_precision": { "passed": 27, "total": 28, "unresolved": 1, "lower": 0.9642857142857143, "upper": 1 }, "unresolved_questions": 1, "explicit_page_citations": 27, "uniquely_quote_mapped_citations": 0, "all_semantic_judgments_resolved": false, "qualified": false, "complete_planned_ids": true, "thresholds": { "supported_answer_success": 0.9, "citation_precision": 0.95, "abstention_accuracy": 0.9 }, "thresholds_passed_on_lower_bounds": { "supported_answer_success": false, "citation_precision": true, "abstention_accuracy": true }, "latency": { "n": 40, "unavailable": 0, "p50": 3.6692478959994332, "p95": 7.471811084000365, "max": 7.790317709001101, "errors_retained": 0, "timeouts_retained": 0, "boundary": null }, "answer_model_request_count": 90, "original_session_receipt": { "original_summary": { "executor_version": "kingy-comparison-session/1.0.0", "system": "A", "repetition": 3, "status": "complete", "qualificationGranted": false, "complete_query_deadline_seconds": 90, "generation_seed": 271828, "client_offline_profile": "offline.sb", "client_offline_profile_sha256": "e5462ef1daae3785012cb8e4fa05b89105bf4ac4d98be9580a8c357365a4f54f", "timing_boundary": "runner.ask entry through its complete returned answer, source resolution and sufficiency check; excludes prior PDF import/index, cold start, warmup and later grading.", "cache_policy": "cache_prompt false; no claim that OS file caches were flushed.", "freeze_sha256": "d1abac923f15a5b6c4a10e51b3f17938789cf99747a218e91ca5d622d22a8f6f", "schedule_sha256": "a568a9bbf9a68dcb23a8bc0191cc151be64ab27e932a62873ba616eed2a1215c", "questions_sha256": "677e88becba7b22d057e5f172421686c61cecf79d170eb0b56d1c9232d51be90", "executor_sha256": "9e8e702e3967ab53f501de1145f26c98ff9582ae7867ab8853cd17f91babf1e2", "runner_sha256": "911298ba21797fe0704874ed85c8f306193de2a69d130ea7ebb32b18bb9064f7", "model_sha256": "7485fe6f11af29433bc51cab58009521f205840f5b4ae3a32fa7f92e8534fdf5", "corpus_pdf_sha256": { "V19-D01.pdf": "c6e49b6cd7067bfac30a956e09d00aaee438b2d021aae474d33a3a23a05694c7", "V19-D02.pdf": "e7bf34f61d9835bcac343fa16fd1bd5754ef97a071fdbea6562171bfaa569cdf", "V19-D03.pdf": "6a04c61ec3b02ef5a08dd3d648cc369f08da86366becc786d44c0abb1a5afe72", "V19-D04.pdf": "7c2dd8a8f3696639941362cdfab9bdac0f35735296168059f95ff4a869a87bc9", "V19-D05.pdf": "009256748075d06133d973fb17dcb28479e81f0325d73978903c160fd4feadb6", "V19-D06.pdf": "f9a23da40c1781721167177c61ebce0129f0fabca7a5bc346ae8af388db800cf", "V19-D07.pdf": "170213e1018f0e0593aec90f515c7563a45f7ae786c4832ca325c682e4507c8b", "V19-D08.pdf": "139ff8485db0f1ae53bf676522fe83656c305d1312ea4657ac119742a143a768", "V19-D09.pdf": "91d65f821a4edf1fb8c9689db251b82edfa32c08cb7445e8b23f15466001fbd0", "V19-D10.pdf": "05b38d3505e20a975f854b22e3b05d7178880fd16f88fa60b5db6f0bbaad3562" }, "planned_question_ids": [ "V19-Q18", "V19-Q16", "V19-Q06", "V19-Q05", "V19-Q37", "V19-Q23", "V19-Q28", "V19-Q04", "V19-Q26", "V19-Q22", "V19-Q34", "V19-Q32", "V19-Q13", "V19-Q35", "V19-Q07", "V19-Q20", "V19-Q24", "V19-Q29", "V19-Q02", "V19-Q14", "V19-Q39", "V19-Q38", "V19-Q09", "V19-Q36", "V19-Q17", "V19-Q01", "V19-Q25", "V19-Q11", "V19-Q19", "V19-Q30", "V19-Q03", "V19-Q27", "V19-Q10", "V19-Q12", "V19-Q15", "V19-Q33", "V19-Q08", "V19-Q21", "V19-Q40", "V19-Q31" ], "session_order": [ "A1", "B1", "B2", "A2", "A3", "B3" ], "artifact_verification_seconds": 1.9626684160029981, "runtime": { "name": "llama.cpp", "binary_sha256": "51afbf4ea6485b9ada19e99af97181f12baca42ef704b1e291c26a9ac0644eae", "version": { "command": [ "/opt/homebrew/Cellar/llama.cpp/0.6.0/bin/llama-server", "--version" ], "exit_code": 0, "stdout": "", "stderr": "version: 0.6.0 (build 11429, commit d81235049)\nbuilt with AppleClang 21.0.0.21000334 for Darwin arm64" } }, "before": { "os": { "command": [ "sw_vers" ], "exit_code": 0, "stdout": "ProductName:\t\tmacOS\nProductVersion:\t\t26.3\nBuildVersion:\t\t25D125", "stderr": "" }, "hardware": { "command": [ "sysctl", "hw.model", "hw.memsize", "hw.physicalcpu", "hw.logicalcpu" ], "exit_code": 0, "stdout": "hw.model: Mac16,7\nhw.memsize: 25769803776\nhw.physicalcpu: 14\nhw.logicalcpu: 14", "stderr": "" }, "memory_pressure": { "command": [ "memory_pressure", "-Q" ], "exit_code": 0, "stdout": "The system has 25769803776 (1572864 pages with a page size of 16384).\nSystem-wide memory free percentage: 73%", "stderr": "" }, "swap": { "command": [ "sysctl", "-n", "vm.swapusage" ], "exit_code": 0, "stdout": "total = 16384.00M used = 15111.62M free = 1272.38M (encrypted)", "stderr": "" }, "disk": { "command": [ "df", "-h", "/" ], "exit_code": 0, "stdout": "Filesystem Size Used Avail Capacity iused ifree %iused Mounted on\n/dev/disk3s1s1 460Gi 11Gi 22Gi 35% 453k 227M 0% /", "stderr": "" }, "ac_power": { "command": [ "pmset", "-g", "batt" ], "exit_code": 0, "stdout": "Now drawing from 'Battery Power'\n -InternalBattery-0 (id=21692515)\t12%; discharging; 0:23 remaining present: true", "stderr": "" }, "power_mode": { "command": [ "pmset", "-g", "custom" ], "exit_code": 0, "stdout": "Battery Power:\n Sleep On Power Button 1\n powermode 0\n standby 0\n ttyskeepawake 1\n hibernatemode 3\n powernap 0\n hibernatefile /var/vm/sleepimage\n displaysleep 0\n womp 0\n networkoversleep 0\n sleep 0\n lessbright 1\n tcpkeepalive 1\n disksleep 0\n SleepServices 0\nAC Power:\n Sleep On Power Button 1\n powermode 0\n standby 0\n ttyskeepawake 1\n hibernatemode 3\n powernap 0\n hibernatefile /var/vm/sleepimage\n displaysleep 0\n womp 1\n networkoversleep 0\n sleep 0\n tcpkeepalive 1\n disksleep 0\n SleepServices 0", "stderr": "" }, "thermal": { "command": [ "pmset", "-g", "therm" ], "exit_code": 0, "stdout": "Note: No thermal warning level has been recorded\nNote: No performance warning level has been recorded\nNote: No CPU power status has been recorded", "stderr": "" }, "chip_inventory": [ { "chip_type": "Apple M4 Pro", "physical_memory": "24 GB", "number_processors": "proc 14:10:4", "machine_model": "Mac16,7", "machine_name": "MacBook Pro" } ] }, "task_owned_server_pid": 98305, "cold_process_to_health_seconds": 2.3159886249995907, "ready_model_identity": { "models": [ { "name": "kingy-qwen3-4b", "model": "kingy-qwen3-4b", "modified_at": "", "size": "", "digest": "", "type": "model", "description": "", "tags": [ "" ], "capabilities": [ "completion" ], "parameters": "", "details": { "parent_model": "", "format": "gguf", "family": "", "families": [ "" ], "parameter_size": "", "quantization_level": "" } } ], "object": "list", "data": [ { "id": "kingy-qwen3-4b", "aliases": [ "kingy-qwen3-4b" ], "tags": [], "object": "model", "architecture": { "input_modalities": [ "text" ], "output_modalities": [ "text" ] }, "created": 1792225470, "owned_by": "llamacpp", "meta": { "vocab_type": 2, "n_vocab": 151936, "n_ctx": 8192, "n_ctx_train": 40960, "n_embd": 2560, "n_params": 4022468096, "size": 2491323904, "ftype": "Q4_K - Medium" } } ] }, "warmup_model_request_count": 1, "warmup_seconds": 0.0843222080002306, "parse_seconds": 0.04124624999894877, "index_seconds": 0.0019298330007586628, "loaded_pdf_pages": 30, "retained_attempts": 40, "planned_attempts": 40, "unexecuted_question_ids": [], "unexecuted_attempt_count": 0, "model_request_count": 90, "session_total_model_request_count": 91, "attempt_errors": 0, "timeouts": 0, "after": { "os": { "command": [ "sw_vers" ], "exit_code": 0, "stdout": "ProductName:\t\tmacOS\nProductVersion:\t\t26.3\nBuildVersion:\t\t25D125", "stderr": "" }, "hardware": { "command": [ "sysctl", "hw.model", "hw.memsize", "hw.physicalcpu", "hw.logicalcpu" ], "exit_code": 0, "stdout": "hw.model: Mac16,7\nhw.memsize: 25769803776\nhw.physicalcpu: 14\nhw.logicalcpu: 14", "stderr": "" }, "memory_pressure": { "command": [ "memory_pressure", "-Q" ], "exit_code": 0, "stdout": "The system has 25769803776 (1572864 pages with a page size of 16384).\nSystem-wide memory free percentage: 62%", "stderr": "" }, "swap": { "command": [ "sysctl", "-n", "vm.swapusage" ], "exit_code": 0, "stdout": "total = 16384.00M used = 15086.69M free = 1297.31M (encrypted)", "stderr": "" }, "disk": { "command": [ "df", "-h", "/" ], "exit_code": 0, "stdout": "Filesystem Size Used Avail Capacity iused ifree %iused Mounted on\n/dev/disk3s1s1 460Gi 11Gi 22Gi 35% 453k 225M 0% /", "stderr": "" }, "ac_power": { "command": [ "pmset", "-g", "batt" ], "exit_code": 0, "stdout": "Now drawing from 'Battery Power'\n -InternalBattery-0 (id=21692515)\t5%; discharging; 0:09 remaining present: true\n\tBattery Warning: Early", "stderr": "" }, "power_mode": { "command": [ "pmset", "-g", "custom" ], "exit_code": 0, "stdout": "Battery Power:\n Sleep On Power Button 1\n powermode 0\n standby 0\n ttyskeepawake 1\n hibernatemode 3\n powernap 0\n hibernatefile /var/vm/sleepimage\n displaysleep 0\n womp 0\n networkoversleep 0\n sleep 0\n lessbright 1\n tcpkeepalive 1\n disksleep 0\n SleepServices 0\nAC Power:\n Sleep On Power Button 1\n powermode 0\n standby 0\n ttyskeepawake 1\n hibernatemode 3\n powernap 0\n hibernatefile /var/vm/sleepimage\n displaysleep 0\n womp 1\n networkoversleep 0\n sleep 0\n tcpkeepalive 1\n disksleep 0\n SleepServices 0", "stderr": "" }, "thermal": { "command": [ "pmset", "-g", "therm" ], "exit_code": 0, "stdout": "Note: No thermal warning level has been recorded\nNote: No performance warning level has been recorded\nNote: No CPU power status has been recorded", "stderr": "" }, "chip_inventory": [ { "chip_type": "Apple M4 Pro", "physical_memory": "24 GB", "number_processors": "proc 14:10:4", "machine_model": "Mac16,7", "machine_name": "MacBook Pro" } ] }, "session_wall_seconds": 170.76444929100035, "frozen_inputs_unchanged": true }, "integrity_issues": [], "source_sha256": { "session-plan.json": "79ef91e8b79601368f0ede6eb60ceae0e1775e385b4bd456a6ab06e97b856e12", "session-summary.json": "e08ee7f7f2c7d875f8ca3ac06ad61c3337e0ba6a0dab418cb2b85a68f6952518", "attempts.jsonl": "84c90623997a4ec0f4f13d61196a08c005107bdc2063216704c89a41496b38f8" }, "complete_40_original_rows": true }, "provenance_verified": true }, "B1": { "attempts": 40, "answer_correctness": { "passed": 7, "total": 30, "unresolved": 18, "lower": 0.23333333333333334, "upper": 0.8333333333333334 }, "supported_answer_success": { "passed": 7, "total": 30, "unresolved": 15, "lower": 0.23333333333333334, "upper": 0.7333333333333333 }, "abstention_accuracy": { "passed": 10, "total": 10, "unresolved": 0, "lower": 1, "upper": 1 }, "citation_precision": { "passed": 31, "total": 35, "unresolved": 0, "lower": 0.8857142857142857, "upper": 0.8857142857142857 }, "unresolved_questions": 18, "explicit_page_citations": 0, "uniquely_quote_mapped_citations": 31, "all_semantic_judgments_resolved": false, "qualified": false, "complete_planned_ids": true, "thresholds": { "supported_answer_success": 0.9, "citation_precision": 0.95, "abstention_accuracy": 0.9 }, "thresholds_passed_on_lower_bounds": { "supported_answer_success": false, "citation_precision": false, "abstention_accuracy": true }, "latency": { "n": 40, "unavailable": 0, "p50": 12.865, "p95": 20.873, "max": 40.455, "errors_retained": 0, "timeouts_retained": 0, "boundary": "Native app submission to documented logger final-answer event; includes native preprocessing/retrieval/embedding/inference; excludes later title generation and automation/export." }, "answer_model_request_count": 40, "original_session_receipt": { "session_path": "/Users/curtispyke/Documents/ChatGPT/kingy Wiki/implementation/2026-10-13/local-ai-guide/evidence/comparison-v19/sessions/B1", "logger": { "path": "/Users/curtispyke/Documents/ChatGPT/kingy Wiki/implementation/2026-10-13/local-ai-guide/evidence/comparison-v19/sessions/B1/model-io.jsonl", "sha256": "a3bad66c5c885ba5a10329d6f587465901975d8c8091a50077f7dec31d9a4382", "parsed_events": 90, "malformed_lines": [] }, "integrity_issues": [], "source_sha256": { "session-plan.json": "7bc4bdee162b7cf1a48e471152c871097ecbc76a0f5f767e439d4cb999507a93", "native-manifest.jsonl": "461b656308c67f420c640faa47d8a1293b31ffdf1f83137f1efec2756a2894cb", "pending.json": "38e0b9de817f645c4bec37c0d4a3e58baecccb040f5718dc069a72c7385a0bed" }, "retained_manifest_rows": 40, "unmanifested_pending_attempt": false, "complete_40_original_manifest_rows": true, "original_files_modified_by_analysis": false }, "provenance_verified": true }, "B2": { "attempts": 40, "answer_correctness": { "passed": 7, "total": 30, "unresolved": 18, "lower": 0.23333333333333334, "upper": 0.8333333333333334 }, "supported_answer_success": { "passed": 7, "total": 30, "unresolved": 15, "lower": 0.23333333333333334, "upper": 0.7333333333333333 }, "abstention_accuracy": { "passed": 10, "total": 10, "unresolved": 0, "lower": 1, "upper": 1 }, "citation_precision": { "passed": 31, "total": 35, "unresolved": 0, "lower": 0.8857142857142857, "upper": 0.8857142857142857 }, "unresolved_questions": 18, "explicit_page_citations": 0, "uniquely_quote_mapped_citations": 31, "all_semantic_judgments_resolved": false, "qualified": false, "complete_planned_ids": true, "thresholds": { "supported_answer_success": 0.9, "citation_precision": 0.95, "abstention_accuracy": 0.9 }, "thresholds_passed_on_lower_bounds": { "supported_answer_success": false, "citation_precision": false, "abstention_accuracy": true }, "latency": { "n": 40, "unavailable": 0, "p50": 6.6775, "p95": 8.468, "max": 8.855, "errors_retained": 0, "timeouts_retained": 0, "boundary": "Native app submission to documented logger final-answer event; includes native preprocessing/retrieval/embedding/inference; excludes later title generation and automation/export." }, "answer_model_request_count": 40, "original_session_receipt": { "session_path": "/Users/curtispyke/Documents/ChatGPT/kingy Wiki/implementation/2026-10-13/local-ai-guide/evidence/comparison-v19/sessions/B2", "logger": { "path": "/Users/curtispyke/Documents/ChatGPT/kingy Wiki/implementation/2026-10-13/local-ai-guide/evidence/comparison-v19/sessions/B2/model-io.jsonl", "sha256": "9583262ef0bb77ac0c654f1f2fb54da3769388bd7134ce078c9e1684fec80d59", "parsed_events": 82, "malformed_lines": [] }, "integrity_issues": [], "source_sha256": { "session-plan.json": "c49b4f3f27769872601ab0ed2ccb98cf9536c05514ceac0a7fa45d9c0a29f757", "native-manifest.jsonl": "6afc5995ad3936e86f2cd59ba00f920b6c93068a97a971203852ba4f354cec3e", "pending.json": "38e0b9de817f645c4bec37c0d4a3e58baecccb040f5718dc069a72c7385a0bed" }, "retained_manifest_rows": 40, "unmanifested_pending_attempt": false, "complete_40_original_manifest_rows": true, "original_files_modified_by_analysis": false }, "provenance_verified": true }, "B3": { "attempts": 40, "answer_correctness": { "passed": 6, "total": 30, "unresolved": 18, "lower": 0.2, "upper": 0.8 }, "supported_answer_success": { "passed": 6, "total": 30, "unresolved": 15, "lower": 0.2, "upper": 0.7 }, "abstention_accuracy": { "passed": 9, "total": 10, "unresolved": 0, "lower": 0.9, "upper": 0.9 }, "citation_precision": { "passed": 30, "total": 34, "unresolved": 0, "lower": 0.8823529411764706, "upper": 0.8823529411764706 }, "unresolved_questions": 18, "explicit_page_citations": 0, "uniquely_quote_mapped_citations": 30, "all_semantic_judgments_resolved": false, "qualified": false, "complete_planned_ids": true, "thresholds": { "supported_answer_success": 0.9, "citation_precision": 0.95, "abstention_accuracy": 0.9 }, "thresholds_passed_on_lower_bounds": { "supported_answer_success": false, "citation_precision": false, "abstention_accuracy": true }, "latency": { "n": 39, "unavailable": 1, "p50": 7.579, "p95": 18.601, "max": 92.536, "errors_retained": 2, "timeouts_retained": 1, "boundary": "Native app submission to documented logger final-answer event; includes native preprocessing/retrieval/embedding/inference; excludes later title generation and automation/export." }, "answer_model_request_count": 39, "original_session_receipt": { "session_path": "/Users/curtispyke/Documents/ChatGPT/kingy Wiki/implementation/2026-10-13/local-ai-guide/evidence/comparison-v19/sessions/B3", "logger": { "path": "/Users/curtispyke/Documents/ChatGPT/kingy Wiki/implementation/2026-10-13/local-ai-guide/evidence/comparison-v19/sessions/B3/model-io.jsonl", "sha256": "7dce9e12ccb1a86d88f04e739a0d6341bb47a5624bbcaaa88da8b0cb6cfaa84f", "parsed_events": 82, "malformed_lines": [] }, "integrity_issues": [], "source_sha256": { "session-plan.json": "490beffb5efee0a7a7de02509c14f790384b009a67da907edd45da0c638ddd3e", "native-manifest.jsonl": "16e47035d065567f965d1ee18a324dfa9c12082e34ae70a16e9fe2864f0ecfe2", "pending.json": "38e0b9de817f645c4bec37c0d4a3e58baecccb040f5718dc069a72c7385a0bed" }, "retained_manifest_rows": 40, "unmanifested_pending_attempt": false, "complete_40_original_manifest_rows": true, "original_files_modified_by_analysis": false }, "provenance_verified": true } }, "pairedEffects": { "complete_question_success": { "unique_questions": 40, "effect_bounds_A_minus_B": [ 0.04166666666666667, 0.4416666666666667 ], "cluster_bootstrap_95pct_sensitivity_envelope": [ -0.10833333333333332, 0.6083333333333333 ], "bootstrap_draws": 10000, "usable_bootstrap_draws": 10000, "interpretation": "Paired whole-question case-resampling sensitivity on this fixed corpus; not a p-value, population confidence claim or independent-repeat sample." }, "supported_answer_success": { "unique_questions": 30, "effect_bounds_A_minus_B": [ 0.044444444444444446, 0.5777777777777777 ], "cluster_bootstrap_95pct_sensitivity_envelope": [ -0.15555555555555556, 0.7666666666666667 ], "bootstrap_draws": 10000, "usable_bootstrap_draws": 10000, "interpretation": "Paired whole-question case-resampling sensitivity on this fixed corpus; not a p-value, population confidence claim or independent-repeat sample." }, "canonical_fact_correctness": { "unique_questions": 30, "effect_bounds_A_minus_B": [ -0.02222222222222222, 0.5777777777777777 ], "cluster_bootstrap_95pct_sensitivity_envelope": [ -0.2, 0.7666666666666667 ], "bootstrap_draws": 10000, "usable_bootstrap_draws": 10000, "interpretation": "Paired whole-question case-resampling sensitivity on this fixed corpus; not a p-value, population confidence claim or independent-repeat sample." }, "correct_abstention": { "unique_questions": 10, "effect_bounds_A_minus_B": [ 0.03333333333333334, 0.03333333333333334 ], "cluster_bootstrap_95pct_sensitivity_envelope": [ 0, 0.1 ], "bootstrap_draws": 10000, "usable_bootstrap_draws": 10000, "interpretation": "Paired whole-question case-resampling sensitivity on this fixed corpus; not a p-value, population confidence claim or independent-repeat sample." }, "emitted_citation_precision": { "unique_questions": 40, "effect_bounds_A_minus_B": [ 0.07967032967032972, 0.11538461538461542 ], "cluster_bootstrap_95pct_sensitivity_envelope": [ -0.048720553089791674, 0.23764907740774038 ], "bootstrap_draws": 10000, "usable_bootstrap_draws": 10000, "interpretation": "Paired whole-question case-resampling sensitivity on this fixed corpus; not a p-value, population confidence claim or independent-repeat sample." } }, "automaticWinner": null }