← Kingy Local AI

FULL THREE-PAIR DOCUMENT STUDY · WORKFLOW 1.5.6

Kingy and native LM Studio: three paired PDF repetitions

document-comparison-v24-20261018.1 · analysis completed 2026-10-18T17:50:30.012401+00:00

40 unique question clusters × 3 paired repetitions = 120 attempts per workflow.

One shared Apple M4 Pro Mac, one separately frozen synthetic five-PDF/30-page English corpus,40 question clusters and three paired repetitions per workflow. Complete original answer deadlines are90seconds. Results describe these configured workflows only.

Each pair contains the same 30 answerable and 10 unknown questions. Repetitions measure repeatability on the declared system; these are 40 unique cases, not 120 independent questions or an independent physical reproduction.

Reading the frozen results

The independent source review found the requested factual values and roles correct for both workflows on all thirty answerable tasks. Both also cleanly declined all ten absent requests. The supported-answer score additionally requires the claimed source passages to support the requested facts; it is not a factual-answer-accuracy score alone.

Kingy provided supporting passages on Q11 and Q29 where native LM Studio supplied correct values but claimed chunks that omitted the requested facts. This is a demonstrated source-support advantage on two unique cross-page tasks, repeated three times. Native gave shorter complete answers on several other tasks. No overall semantic winner follows.

Native Q23 is complete and supported after semantic review, but its event-based deadline paraphrase remains unresolved under the frozen finite matcher. The original scores were not changed: each native repetition has 27 confirmed supported answers, one unresolved and two citation-support failures. All paired case-resampling envelopes include zero; forty unique tasks repeated three times are not 120 independent questions.

Independent current candidate engineering approval (public derivative; original hash retained) · Independent every-item corpus approval (public derivative; original hash retained) · Actual independent final source and scoring audit (public derivative; original hash retained) · Paired calculations and unresolved bounds (public derivative; original hash retained)

Repairs completed before the fresh corpus

Candidate 1.5.6 binds each requested field to its full question context, keeps sibling facts separate, and checks that a selected complete-token source passage supplies that field. The candidate and scientific scorer were fixed before a separate agent authored this fresh five-PDF, thirty-page corpus.

Kingy retrieves local PDF pages and uses the local model to select and check source passages, with up to four model requests per question. The observed result tests that whole configured workflow. Development replays and public smoke tests are documented separately from the 240 fresh study originals.

A new Python environment and an exact fresh extraction of the final ZIP passed all four public installation selftests. The unauthenticated model API returned 401, permitted loopback returned 200, and the declared external IPv4, external IPv6 and wrong-loopback-port probes were denied. This is a same-Mac installation check with existing prerequisites, not novice-user or second-machine validation.

Actual r7 engineering changes and retained development evidence (public derivative; original hash retained) · Independent current candidate engineering approval (public derivative; original hash retained) · Complete approved study freeze (public derivative; original hash retained) · Actual fresh package/selftest/security outcome (public derivative; original hash retained) · All four actual public selftest originals (public derivative; original hash retained) · Actual authentication and declared egress checks (public derivative; original hash retained)

Original completion and prospective method adherence

The independent eligibility review records original completion separately from strict prospective-method adherence.

Retained inference evidence and input equivalence grant no automatic exemption from the prospective method or launch gates. Actual original completion and strict-method eligibility (public derivative; original hash retained)

Fresh package installation check

The actual fresh package check passed. 4 of 4 public selftests passed. The separate release decision records every other launch gate.

Authentication and declared client/server egress checks: passed. This separate result does not turn a failed answer selftest into a passed installation. Fresh ZIP extraction on the recorded Mac with declared prerequisites; not novice-user research or separate-machine reproduction.

Actual fresh package/selftest/security outcome (public derivative; original hash retained) · All four actual public selftest originals (public derivative; original hash retained) · Actual four-selftest summary (public derivative; original hash retained) · Actual authentication and declared egress checks (public derivative; original hash retained)

Final independent source check and release decision

All local qualification, approval, completion, audit, package and security gates passed. Actual immutable publication and app readback are still required before claiming launch.

All 240 actual outputs were covered by review of 80 exact semantic/source bundles, each mapped to its three original attempts. No unsupported frozen pass or wrong factual value/role was found. Kingy supplied complete cited support on all answerable tasks. Native supplied correct requested factual values throughout; its Q11/Q29 claimed chunks omitted the requested support, and its complete, supported Q23 paraphrase remains frozen unresolved. Both systems cleanly declined all ten wholly absent requests. The frozen conservative bounds remain valid within their stated finite-recognition scope; these results do not establish an overall semantic winner.

The bounded final scoring audit passed within its declared scope. No overall comparative quality winner is claimed. Passed only for the unchanged prospective scorer results interpreted as conservative finite-recognition lower/upper bounds for complete supported answers, citation precision and clean abstention in this five-PDF/forty-task study. A frozen unresolved outcome is not a demonstrated factual error. Native Q23 is semantically complete and sourced despite remaining unresolved; no alias or result was changed. Native Q11/Q29 failures concern emitted citation support, not their correct answer values. This same-Mac independent-context AI audit is not human adjudication, second-hardware reproduction, timing/privacy validation, release qualification or a broad workflow/model superiority claim. The separate agent reviewed 80 distinct response bundles covering all 240 originals. This is an unblinded AI-assisted review, not human adjudication. The original scores and responses are preserved.

Actual independent final source and scoring audit (public derivative; original hash retained) · Actual independent every-case review notes (public derivative; original hash retained) · All actual failed or unresolved Kingy originals (public derivative; original hash retained) · Actual release decision and all gates (public derivative; original hash retained)

Observed differences in both directions

These are observations on this fixed corpus and configuration, not a general product ranking.

Complete paired accounting

All A1/B1, A2/B2 and A3/B3 attempts are retained. Confirmed passes form the conservative rate. The upper bound additionally includes unresolved judgments. These bounds are not confidence intervals. Canonical facts and complete supported answers are reported separately. Displayed native retrieval cards count only when claimed in the answer; uniquely quote-mapped citations can receive credit, with page capability retained separately.

Frozen common rubric; all six complete sessions
Workflow / repetitionCanonical factsSupported answersCitation source supportCorrect abstentionAttempt accounting
Kingy workflow 1.5.6A130/30 (100.0%)0 unresolved · bounds 100.0%–100.0%30/30 (100.0%)0 unresolved · bounds 100.0%–100.0%35/35 (100.0%)0 unresolved · bounds 100.0%–100.0%10/10 (100.0%)0 unresolved · bounds 100.0%–100.0%40/40 retained0 errors · 0 timeouts
Native LM Studio PDF chatB129/30 (96.7%)1 unresolved · bounds 96.7%–100.0%27/30 (90.0%)1 unresolved · bounds 90.0%–93.3%31/35 (88.6%)2 unresolved · bounds 88.6%–94.3%10/10 (100.0%)0 unresolved · bounds 100.0%–100.0%40/40 retained0 errors · 0 timeouts
Kingy workflow 1.5.6A230/30 (100.0%)0 unresolved · bounds 100.0%–100.0%30/30 (100.0%)0 unresolved · bounds 100.0%–100.0%35/35 (100.0%)0 unresolved · bounds 100.0%–100.0%10/10 (100.0%)0 unresolved · bounds 100.0%–100.0%40/40 retained0 errors · 0 timeouts
Native LM Studio PDF chatB229/30 (96.7%)1 unresolved · bounds 96.7%–100.0%27/30 (90.0%)1 unresolved · bounds 90.0%–93.3%31/35 (88.6%)2 unresolved · bounds 88.6%–94.3%10/10 (100.0%)0 unresolved · bounds 100.0%–100.0%40/40 retained0 errors · 0 timeouts
Kingy workflow 1.5.6A330/30 (100.0%)0 unresolved · bounds 100.0%–100.0%30/30 (100.0%)0 unresolved · bounds 100.0%–100.0%35/35 (100.0%)0 unresolved · bounds 100.0%–100.0%10/10 (100.0%)0 unresolved · bounds 100.0%–100.0%40/40 retained0 errors · 0 timeouts
Native LM Studio PDF chatB329/30 (96.7%)1 unresolved · bounds 96.7%–100.0%27/30 (90.0%)1 unresolved · bounds 90.0%–93.3%31/35 (88.6%)2 unresolved · bounds 88.6%–94.3%10/10 (100.0%)0 unresolved · bounds 100.0%–100.0%40/40 retained0 errors · 0 timeouts

Errors and timeouts can overlap and are not added together. Both remain failed attempts. The frozen citation metric checks for a registered source anchor in the emitted excerpt. It does not independently guarantee that the answer states the right fact: a citation can contain the expected source anchor while the answer still names the wrong event or value. Whole-answer support requires factual correctness and source checks together. Citation denominators include emitted partial or unresolved citations. Unchanged common atomic rubric1.5.0 (public derivative; original hash retained) · Frozen prospective comparison protocol (public derivative; original hash retained)

Qualification and unresolved judgments

The declared thresholds are at least 90% supported-answer success, 95% emitted-citation precision and 90% correct abstention in every repetition, using conservative lower bounds.

Unresolved semantic or citation judgments remain. Their possible outcomes remain in the displayed bounds; an unreviewed ambiguity does not establish a comparative winner. Qualification is an absolute threshold for this workflow and test scope. A relative score does not automatically qualify a buying recommendation.

Paired effects and uncertainty

Effects below are Kingy minus native LM Studio. The 10,000-draw sensitivity analysis resamples whole unique-question clusters and keeps the three repetitions together. The envelope describes case-resampling sensitivity on this fixed corpus. It is not a population confidence interval, p-value or market-wide ranking.

Arithmetic unresolved bounds and whole-question sensitivity
Outcome / clustersEffect bounds95% case-resampling sensitivity envelope
complete question success40 unique clusters5.0 pp to 7.5 pp0.0 pp to 17.5 pp
supported answer success30 unique clusters6.7 pp to 10.0 pp0.0 pp to 23.3 pp
canonical fact correctness30 unique clusters0.0 pp to 3.3 pp0.0 pp to 10.0 pp
correct abstention10 unique clusters0.0 pp to 0.0 pp0.0 pp to 0.0 pp
emitted citation precision40 unique clusters5.7 pp to 11.4 pp0.0 pp to 27.3 pp

No automatic winner is declared. Paired calculations and unresolved bounds (public derivative; original hash retained)

Complete-answer latency

Descriptive time includes each workflow's full answer pipeline. Native timing includes native preprocessing, embedding/retrieval and inference. Kingy timing includes retrieval, selection, source resolution and sufficiency checks. Import, cold start and neutral warmup are recorded separately. Median and nearest-rank p95 include retained measured failures; unavailable observations remain visible. These measurements do not establish model throughput, novice setup speed or a population speed advantage.

Seconds; descriptive observations only
SessionMeasured / unavailableMedianp95Maximum
A140 / 05.10410.89311.252
A240 / 04.95710.33910.660
A340 / 04.90510.30010.595
B140 / 07.5639.1299.472
B240 / 07.3049.2329.390
B340 / 07.3759.0159.447
Actual timing boundaries

Control provenance

Prospective native controls and separate timing boundaries

The actual study retained all 240 planned originals in the frozen A1, B1, B2, A2, A3, B3 order. Each native original used one journaled external System Events AXPress, same-conversation acceptance, the exact five PDFs, matching settings, and original completion events. Previous incomplete studies and repairs remain preserved.

Preparation, document import, cold load, the single document-free warmup and inter-original evidence-review pauses are separate from per-query timing. Native results retain both the original dispatch watchdog and submission-to-final-event interval. No clock was reset to make a late observation pass. Latency does not measure automation throughput, novice setup time or a general speed advantage.

These results cover one shared M4 Pro Mac, one synthetic English selectable-text corpus, one Qwen artifact and the declared runtime versions. LM Studio also documents local offline document chat. This study does not prove an exclusive privacy feature, cloud-cost saving, GPU-upgrade benefit or a market-wide product ranking.

All thirty answerable v24 questions require facts from one PDF each; five span adjacent pages within that PDF. None requires combining facts across different PDFs. Cross-document answer synthesis is unmeasured. The generic multi-document corpus description in the immutable tested ZIP README does not describe v24; this study-specific correction takes precedence. The installed package, original answers and frozen scores are unchanged.

Actual current-study control provenance (public derivative; original hash retained) · Actual original completion and strict-method eligibility (public derivative; original hash retained) · Frozen prospective comparison protocol (public derivative; original hash retained) · Complete approved study freeze (public derivative; original hash retained) · All six original-session provenance checks (public derivative; original hash retained) · Actual independent final source and scoring audit (public derivative; original hash retained)

V24 used the prospectively frozen preparation, bounded input-persistence and external one-shot System Events AXPress Send routes. Each native original used a distinct fresh conversation, the exact five PDF bytes and unchanged nine configuration groups. Source-bound fresh control runtimes, durable official logging, same-conversation first-user acceptance, original EOS/final-event evidence and both immutable90-second bounds were checked. Historical transport failures and repairs were preserved before this method/corpus freeze; no within-v24 transport or scientific-source amendment is asserted. Operator preparation and inter-original evidence-review pauses are outside per-query latency and do not establish setup speed, uninterrupted wall throughput or competitor reliability.

Frozen prospective comparison protocol (public derivative; original hash retained) · Complete approved study freeze (public derivative; original hash retained) · Independent exact method/source approval (public derivative; original hash retained) · Actual original completion and strict-method eligibility (public derivative; original hash retained) · All six original-session provenance checks (public derivative; original hash retained) · Actual independent final source and scoring audit (public derivative; original hash retained)

Exact configurations and failures

Kingy settings
{
  "studyId": "document-comparison-v24-20261018.1",
  "createdAt": "2026-10-18T17:58:16.379954+00:00",
  "comparisonFreezeSha256": "f968a685190b843d121167b1031130e79a8daaf5a86da13effef37cb7af45936",
  "methodFreezeSha256": "c3cfa47c1e7a30832d0fdad78774ff5543a1851f5d679389b9ce097540d51709",
  "publicMetadataOnly": true,
  "schemaVersion": "kingy-document-public-settings/1.0.0",
  "workflow": "Kingy",
  "candidateVersion": "1.5.6",
  "runnerVersion": "kingy-document-runner/1.5.6",
  "runnerSha256": "3bcc03ffef818fa7fb6062ae17f20aa09d2fcc25fc9d57572c77ec35bc1ef02b",
  "model": {
    "name": "Qwen3-4B-Q4_K_M",
    "bytes": 2497280256,
    "sha256": "7485fe6f11af29433bc51cab58009521f205840f5b4ae3a32fa7f92e8534fdf5"
  },
  "runtime": {
    "name": "llama.cpp",
    "build": "b11429",
    "commit": "d81235049",
    "binarySha256": "51afbf4ea6485b9ada19e99af97181f12baca42ef704b1e291c26a9ac0644eae"
  },
  "generation": {
    "temperature": 0,
    "seed": 271828,
    "contextTokens": 8192,
    "cpuThreads": 10,
    "parallelSequences": 1,
    "reasoning": false,
    "topP": 1,
    "selectionMaximumTokens": 256,
    "sufficiencyMaximumTokens": 192,
    "cachePrompt": false,
    "contextShift": false,
    "flashAttention": true,
    "kvCacheTypes": {
      "K": "f16",
      "V": "f16"
    }
  },
  "retrieval": {
    "maximumPagesPerField": 8,
    "maximumCandidatePassagesPerField": 12,
    "answerMode": "source-passage-ID selection, selected-quote grounding and semantic sufficiency checks"
  },
  "completeQueryDeadlineSeconds": 90,
  "timingBoundary": "Full runner.ask return, including retrieval, field selection, source resolution and sufficiency; cold startup, PDF parse/indexing and neutral warmup are separate.",
  "cacheLimit": "Request prompt caching disabled; OS file caches were not claimed flushed.",
  "sourceEvidenceIds": [
    "protocol",
    "freeze",
    "scorer-validation",
    "session-provenance"
  ],
  "sourcePins": {
    "startServerSha256": "7f54900199a61c495bf5f6753a5caa13b529f86061d56ec2fca5a63be3844b2a",
    "executorSha256": "0313919e7c3f11d8238c5afad191d2bd8b92de63c13cea691c56f360096d2527",
    "offlineProfileSha256": "e5462ef1daae3785012cb8e4fa05b89105bf4ac4d98be9580a8c357365a4f54f"
  }
}

Actual current-study kingy settings (public derivative; original hash retained)

Native LM Studio settings
{
  "studyId": "document-comparison-v24-20261018.1",
  "createdAt": "2026-10-18T17:58:16.379954+00:00",
  "comparisonFreezeSha256": "f968a685190b843d121167b1031130e79a8daaf5a86da13effef37cb7af45936",
  "methodFreezeSha256": "c3cfa47c1e7a30832d0fdad78774ff5543a1851f5d679389b9ce097540d51709",
  "publicMetadataOnly": true,
  "schemaVersion": "kingy-document-public-settings/1.0.0",
  "workflow": "Native LM Studio",
  "model": {
    "name": "Qwen3-4B-Q4_K_M",
    "bytes": 2497280256,
    "sha256": "7485fe6f11af29433bc51cab58009521f205840f5b4ae3a32fa7f92e8534fdf5"
  },
  "generation": {
    "temperature": 0,
    "seed": 271828,
    "contextTokens": 8192,
    "loadCpuThreads": 10,
    "predictionCpuThreads": 10,
    "parallelSessions": 1,
    "offloadRatio": 1,
    "flashAttention": true,
    "reasoning": false,
    "maximumPredictedTokens": 512,
    "topP": 1,
    "minP": 0,
    "repeatPenalty": 1,
    "contextOverflowPolicy": "stopAtLimit",
    "toolsType": "none"
  },
  "systemPromptEmpty": true,
  "effectiveGroupsVerified": [
    "load_config",
    "prediction_config",
    "plugins",
    "plugin_configs",
    "system_prompt"
  ],
  "effectiveSettingsSourceSha256": "d28305b7df7214eb6e4797591b90991ab6add825980bbc38a91ae10958a7668d",
  "effectiveSettingsProvenanceSha256": "01b3bbc26eafebb93f1f2b979c422ba655a99c7b7ffb3053b297dbbd7e886ea7",
  "promptTemplateFingerprints": {
    "loadTemplateCanonicalSha256": "0f2a3cc9c3acea3d78e406dc5d32686f0b2906dc75ca44cfdb7f1f28d4373cc1",
    "predictionTemplateCanonicalSha256": "1e69c61005de39c59ac787a5ec1ba4284fd353353f1fbc935bc29a4c3e8b15f3",
    "fingerprintSerialization": "UTF-8 JSON, sorted keys, compact separators, trailing newline"
  },
  "factualUserTemplateSha256": "6445968866a12a80297f52f1c97afe07ce1b425d349544f648a2faf294943b5e",
  "configurationGroupCount": 9,
  "configurationTemplateSha256": "a8409314a38f454a7aff4caabcf5c5d8b57cf7bb511e55200471df659e97e12e",
  "completeQueryDeadlineSeconds": 90,
  "cacheBehavior": "Actual native cache behavior remains unknown; no OS-cache flush or cold-cache claim.",
  "measuredOriginalsVerified": 120,
  "sessions": {
    "B1": {
      "originals": 40,
      "effectiveSettingsVerified": 40,
      "exactFivePdfInputMappings": 200,
      "oneFreshLoad": true,
      "oneDocumentFreeWarmup": true,
      "rawFirstUiCompleteCount": 40,
      "summarySha256": "67d16a55e6cdb69010c21155fa919ac3c7b8744d35fc6a776c668faebf60c961",
      "artifactBeforeReceiptSha256": "bf5a1ba0869e51d17cf8408871cde08a5af0db0a0413f1512b81a17b21f6b516",
      "artifactAfterReceiptSha256": "c11822b99cc999403d7b7e5b776d4573ea952ec4e2987564bd2cdce55e27388f",
      "hostBeforeReceiptSha256": "380a76b07a2bba55de5103843645ccc401fde5f2e37d9a43d436ff81c46eff3a",
      "hostAfterReceiptSha256": "ab1b498043f733915d9cde23668b9610d17a94cd791b78c25ae4614ae8e1dcdf",
      "sourceCacheReceiptSha256": "a105c40fd85ad0c6048adad9567455f4f3a9cb19f019f39ef8451b6efdd5a735",
      "cleanupReceiptSha256": "fffd1b965835218736b9ad7cc9a0fad4aaedee43721f9815216920e18701017c",
      "runtimeRestored": "2.60.0",
      "embeddingObservedLoadedAfter": true
    },
    "B2": {
      "originals": 40,
      "effectiveSettingsVerified": 40,
      "exactFivePdfInputMappings": 200,
      "oneFreshLoad": true,
      "oneDocumentFreeWarmup": true,
      "rawFirstUiCompleteCount": 40,
      "summarySha256": "4b2db9ad6ace277113cf952b4378b1e74858b98e29443a0dd9447a3b0c3f4c9b",
      "artifactBeforeReceiptSha256": "19cb48863f586ff2d400a9753169a226b6340904621eaae0a52ae461ae53d945",
      "artifactAfterReceiptSha256": "67bb99bbd3987ccb1beeb1b930bcc46852b984ccd5249c3a692d66002cc18da8",
      "hostBeforeReceiptSha256": "e0957bf85682a71f6a54396ace9b7ffb367cbc50326f1afd5d725f67f3f8337c",
      "hostAfterReceiptSha256": "1470f47da1df69deedf4bd0a44ba324534bcb1684ba9a4695690fda79dc9985f",
      "sourceCacheReceiptSha256": "3688d52132aeda81694b9c5a99dbfae8a5a6ae921f6c8daaa7cf762c0ee7cf78",
      "cleanupReceiptSha256": "4ec7e54664e980c908dea0414abb50d26745a0f01a074779497f512b48a8fb3d",
      "runtimeRestored": "2.60.0",
      "embeddingObservedLoadedAfter": true
    },
    "B3": {
      "originals": 40,
      "effectiveSettingsVerified": 40,
      "exactFivePdfInputMappings": 200,
      "oneFreshLoad": true,
      "oneDocumentFreeWarmup": true,
      "rawFirstUiCompleteCount": 40,
      "summarySha256": "cbbee494db25682bc0a344d6588ef50e213ea46f729462d99bf8fa0fb3c520e6",
      "artifactBeforeReceiptSha256": "85180eadf0dff72ff876be4fa48f8ad303c387522be0e776b3734528ad2b7340",
      "artifactAfterReceiptSha256": "806a21d1b04a676cbeeac25d4bd9faba72b1d9acd093b351f270c74b2cf3e80c",
      "hostBeforeReceiptSha256": "886ce08b761841ba31266c0bca01b76bc33433d41ecdd15c79157bfd281f7693",
      "hostAfterReceiptSha256": "aec142eb4a977d0182f3a46e0fe2c09e141e0bc03236491d27b486ef7918d1b7",
      "sourceCacheReceiptSha256": "c75fcc767ed2c7bef79f330ee932ae650b766b628dc9440aacd5f2bd6230ad09",
      "cleanupReceiptSha256": "09bf6b827a1b6437065aabed6f07abd1d94e74fe352513e078b3a3e919a25583",
      "runtimeRestored": "2.60.0",
      "embeddingObservedLoadedAfter": true
    }
  },
  "sourceEvidenceIds": [
    "protocol",
    "freeze",
    "session-provenance",
    "method-approval",
    "completion-eligibility"
  ]
}

Actual current-study native settings (public derivative; original hash retained)

Actual current-study native runtime (public derivative; original hash retained) · Actual current-study native rag (public derivative; original hash retained) · Complete approved study freeze (public derivative; original hash retained) · All six original-session provenance checks (public derivative; original hash retained)

The retained attempt records contain no runtime errors or timeouts. Quality failures and unresolved judgments remain in the score denominators and full raw evidence.

All120 Kingy originals and frozen grades (public derivative; original hash retained) · All120 normalized native originals and frozen grades (public derivative; original hash retained)

Limits and history

The original immutable v1.4 paired pilot remains separate historical evidence, including its incomplete native repetitions and post-output source review. Its scores and files are preserved. The original failed 1.5.0 study and its scorer limitation also remain unchanged. The original failed 1.5.1 study also remains unchanged. The preserved v19 study remains unchanged, including its scoring defects and incomplete native original. The preserved v20 study remains separate and unchanged, including its held-release decision and disclosed method limitations. The preserved v21 incomplete, ungraded study and preserved v22 incomplete, ungraded study remain unchanged. The preserved v23 incomplete, ungraded study remains separate and unchanged. This candidate1.5.6 report uses its own frozen rubric and fresh study; it does not rewrite earlier comparisons. Original immutable1.4 manifest (public derivative; original hash retained) · Preserved1.5.0 manifest (public derivative; original hash retained)

Inspect and reproduce

Report data · File hashes and sizes · Actual completed analysis and output hashes (public derivative; original hash retained)