# Common private-PDF comparison method

Prepared without reading the new v1.4 bank or candidate outputs. Source review and consumed/synthetic unit controls are not a benchmark result.

## Systems and frozen inputs

Compare the complete Kingy document workflow with stock LM Studio document chat, including its official `lmstudio/rag-v1` preprocessor. A plain LM Studio inference backend using Kingy retrieval is a backend test and cannot represent native document chat. An SDK native-retrieval adapter needs a separate label unless its processed prompt, retrieval/configuration and citation mapping match stock GUI behavior on consumed controls.

Use the same M4 Pro Mac (14 CPU cores / 20 GPU cores, 24 GiB), OS/build and verified Qwen3-4B Q4_K_M bytes. Record both runtime binaries/builds, chat templates and exact settings. Target 8,192 context tokens, temperature zero, reasoning off, ten CPU threads, full Metal offload, flash attention and F16 K/V cache where supported. Record actual resolved values and unsupported settings. Do not imply identical runtimes: LM Studio's engine may differ from Kingy's llama.cpp build. Record native embedding artifact, parser, chunker, plugin revision, limits, thresholds and cached-index state. Default native document handling can choose full context or retrieval; preserve and report that choice.

Both systems see all ten identical PDFs and the identical requested question/field labels. Neither receives references, support spans, accepted values or other answers. Each native question uses a genuinely new chat with all attachments and no prior user/assistant exchange. Equal development opportunity uses only consumed data. Freeze code, system/user prompts, generation limits, timeouts, adapters, rubric and settings before accessing the new bank. The independently authored reference lists complete accepted fact phrases, including all requested qualifiers, and legitimate alternative exact source spans. New semantic equivalents require a new preregistered test or an explicitly reported blind adjudication; they cannot silently change this frozen result.

## Shared answer and evidence rules

`common_grade.py` is a dependency-free scorer. It accepts normalized final fields/citations or native prose through a format-only adapter. The adapter receives requested field names and retained native metadata, never reference values. JSON syntax or schema conformance is not an accuracy metric. Single-field prose can use its full answer; multi-field prose requires field labels. Unclear field alignment remains unresolved rather than being inferred from gold.

An answerable question succeeds only when every requested essential fact and qualifier matches a frozen complete accepted value, every claim has an explicitly linked supporting citation, and every emitted citation is an exact source-grounded quote that contains a declared legitimate evidence span from the correct document/page. Adjacent true source context can extend the minimal evidence span. Cosmetic whitespace and Unicode compatibility normalization are allowed in source matching; fuzzy matching is not. Cardinal words zero through twenty and valid thousands separators normalize in fact values; decimals, signs, identifiers, units, years and ambiguous numeric commas remain significant. Numeric mathematical/heavy minus and heavy plus map to their ASCII signs after Unicode compatibility normalization; superscript/subscript/fullwidth/small sign forms follow the same normalization. Ambiguous plus-or-minus and en/em dash tokens remain present and cannot silently become an unsigned positive value. This symmetric sign-preservation repair was made before fresh-bank access; the prior source is retained as common_grade.pre-unicode-minus.py.

Native numbered citations are linked only when the answer explicitly claims their IDs and retained metadata supplies an unambiguous ID-to-source/exact-quote mapping. Automatically displayed retrieval cards are not emitted answer citations. The scorer never supplies a missing citation, selects a reference source for the model, or adds another inference-based sufficiency check. Native free-form factual explanation is allowed when its vocabulary and required facts are grounded in the linked quotes. Unregistered semantic paraphrases and unreviewed extra assertions remain unresolved. Unsupported numeric facts, wrong sources/pages, fabricated quotes and missing citations fail. A clearly abbreviated required value with missing qualifiers fails. Do not use an LLM judge.

A consumed native preflight revealed that its parser glued the exact document filename ID to `Page` in the synthetic fixture footer, while pypdf inserted whitespace. Source matching symmetrically inserts one space only between the exact named document ID and a complete `Page N of M | Fixture edition VERSION` marker. Existing/extra whitespace already follows the ordinary whitespace rule. No arbitrary word/number joins, amounts, signs, units, page numbers or fixture versions are altered. Reference character spans remain checked against the original unmodified extracted text; normalized prefix lengths only resolve their matching location. The pre-repair source is retained as common_grade.pre-fixture-footer.py. This normalization does not establish compatibility with arbitrary real-world parser differences.

Explicit page-number citations and unique quote-to-page mappings are separate capability counts. A quote uniquely located in the original document can supply a common support score even if LM Studio did not emit a page number; this does not award explicit-page capability. A contiguous native quote spanning multiple pages can be grounded as a page range. Repeated/ambiguous locations require review; a stated wrong page fails. Character offsets supplied by the reference must still match its exact PDF text.

For unanswerable questions, a clean explicit factual decline such as `UNKNOWN` passes. A native explanation of missing information can pass when it is an exact grounded documentary-absence statement, covers the requested topic/fields and named-source scope, and contains no guessed requested value. A factual zero/no-charge policy is not missing information. Unsupported or ambiguous absence explanations remain unresolved; speculative requested answers fail. A correct decline on an answerable question still fails.

A consumed native preflight returned an exact requested field label followed by `UNKNOWN`. Before fresh evaluation, the format adapter was extended to recognize one line per requested field, with every value exactly `UNKNOWN` and optional final punctuation. Labels may use the adapter's existing colon/equal sign and bullet syntax. All requested fields must appear once; citations, missing or duplicate fields, extra fields, mixed factual values and extra prose cannot enter this clean-decline branch. The original native prose remains retained. Complete normalized `UNKNOWN` field values receive the same treatment. This recognizes explicit abstention formatting only; it does not interpret missing-information explanations or infer answers. The prior source is retained as common_grade.pre-labelled-unknown.py.

The scorer emits definite `pass`, definite `fail`, or `unresolved`. Unresolved cases stay in the denominator and produce conservative lower/upper bounds. Do not claim a winner if its advantage depends on unreviewed ambiguity. Any adjudication must be blind to system identity, retain raw evidence and rationale, use the same criterion for both systems, and be reported separately from deterministic bounds. A second independent reviewer should resolve disputed semantic cases before a comparative claim.

## Metrics and qualification

Report canonical answer correctness separately from complete supported-answer success, emitted-citation precision and clean abstention. The primary answer qualification is complete supported-answer success: at least 90% of answerable questions, at least 95% citation precision, and at least 90% correct abstention in every complete repetition. Use conservative lower bounds while any case is unresolved. Missing and duplicate planned IDs, crashes and timeouts remain attempts; an incomplete planned evaluation cannot qualify. A zero denominator is unavailable and cannot pass a threshold. Syntax success, explicit-page capability and manual-review workload are descriptive measures.

Record all planned IDs exactly once per system/repetition. Three fresh repetitions measure repeatability; they do not triple the number of independent questions. Use paired question-level comparisons and uncertainty clustered by unique question, rather than treating repeated attempts as independent samples. This fixed synthetic task on one Mac does not establish a general product/model winner or performance on arbitrary private PDFs.

## Execution order, latency and privacy

Declare `A = Kingy`, `B = LM Studio`; use session order `A1, B1, B2, A2, A3, B3` (three sessions each). Generate and retain a seeded question permutation for each paired repetition; both systems use its same order. Restart the relevant model/app/server before each session and keep only one tested system's models loaded. Each question is independent. Record AC/power mode, thermal state where available, concurrent workloads, memory pressure and swap. Use the same neutral warmup outside the bank, before document attachment. No golden setup exchange or repeated question is a warmup. Retain cache reuse and first-query index work explicitly.

Complete-answer latency starts at query submission against ready imported documents and ends when the final usable answer and its cited evidence are available. It includes retrieval and embedding/model work incurred by that query, all model phases (both Kingy requests), local resolution and the adapter. Report parsing/import/index preparation, cold process/model/embedding startup and first-query indexing separately. If native indexing occurs inside the first timed request, keep it in that latency as well as reporting its phase. Do not silently subtract it. Declare output token limits that permit the shared task's answer/evidence format on consumed controls; equal token caps are not a substitute for recording each pipeline's actual generation and total waiting time. Use the same 90-second complete query deadline and retain timeout outcomes.

Download/import preparation can use the Internet. Kingy measured sessions block outbound networking for the complete server/client process trees. The native isolation preflight adjustment in COMPARISON-PLAN.md applies to LM Studio: keep its Chromium protections enabled, use local inference on public synthetic documents, and report native egress enforcement as unverified. Disable external tools/MCP/LM Link in both workflows. This difference excludes a comparative privacy claim. Local native parsing, retrieval and inference are part of the baseline. Record embedding/runtime downloads and disk requirements as installation costs. Never include private keys, private documents or existing user chat records in the public evidence bundle.

## Native capture prerequisites

Before evaluation, a consumed-data preflight must prove all ten files are attached and processed. Capture final raw native prose, actual claimed citation IDs, citation IDs mapped to source filenames/exact passages, model processed-prompt/output logs, resolved settings, timing boundaries, native retrieval/full-context choice and plugin logs. Native chat exports alone may omit preprocessor metadata. If support linkage cannot be recovered, retain the attempt as unresolved; do not infer links from the reference or award all retrieved cards.

Official references checked October 13, 2026:

- Document workflow: https://lmstudio.ai/docs/app/basics/rag
- Official stock preprocessor: https://lmstudio.ai/lmstudio/rag-v1/files/src/promptPreprocessor.ts
- Stock configuration: https://lmstudio.ai/lmstudio/rag-v1/files/src/config.ts
- SDK native files/retrieval interfaces: https://github.com/lmstudio-ai/lmstudio-js/blob/main/packages/lms-client/src/files/FilesNamespace.ts
- Model IO logging: https://lmstudio.ai/docs/cli/serve/log-stream
- Local model import: https://lmstudio.ai/docs/cli/local-models/import
- Official release: https://lmstudio.ai/changelog/lmstudio/lmstudio-v0.4.26

The official Mac-arm64 latest URL redirected to the 0.4.26 Build 4 DMG while the changelog advertised Build 5. Record the actual installed build and hashes.

Run only the method controls with `python -B -m unittest -v test_common_grade.py` from this folder. No method control reads a fresh bank or calls a model.