Kingy and native LM Studio: three paired PDF repetitions

Frozen automatic scores, preserved

All A1/B1, A2/B2 and A3/B3 attempts are retained. Confirmed passes form the conservative rate. The upper bound additionally includes unresolved judgments. These bounds are not confidence intervals. Canonical facts and complete supported answers are reported separately. Displayed native retrieval cards count only when claimed in the answer; uniquely quote-mapped citations can receive credit, with page capability retained separately.

Errors and timeouts can overlap and are not added together. Both remain failed attempts. Citation denominators include emitted partial or unresolved citations. Original frozen common scoring rules · Original frozen paired protocol

Qualification and unresolved judgments

The declared thresholds are at least 90% supported-answer success, 95% emitted-citation precision and 90% correct abstention in every repetition, using conservative lower bounds.

• Kingy: A1 does not meet all thresholds; A2 does not meet all thresholds; A3 does not meet all thresholds.
• Native LM Studio: B1 does not meet all thresholds; B2 does not meet all thresholds; B3 does not meet all thresholds.

Unresolved semantic or citation judgments remain. Their possible outcomes remain in the displayed bounds; an unreviewed ambiguity does not establish a comparative winner. Qualification is an absolute threshold for this workflow and test scope. A relative score does not automatically qualify a buying recommendation.

Paired effects and uncertainty

The effects below retain the original automatic calculation. The citation-mapping limitation invalidates comparative source-support, citation and cited-abstention interpretations. These apparent effects cannot be cited as product advantages; use the separate source review with its limits.

Effects below are Kingy minus native LM Studio. The 10,000-draw sensitivity analysis resamples whole unique-question clusters and keeps the three repetitions together. The envelope describes case-resampling sensitivity on this fixed corpus. It is not a population confidence interval, p-value or market-wide ranking.

No automatic winner is declared. Original frozen paired calculations; method limitation disclosed

• A1: runner.ask entry through its complete returned answer, source resolution and sufficiency check; excludes prior PDF import/index, cold start, warmup and later grading.
• A2: runner.ask entry through its complete returned answer, source resolution and sufficiency check; excludes prior PDF import/index, cold start, warmup and later grading.
• A3: runner.ask entry through its complete returned answer, source resolution and sufficiency check; excludes prior PDF import/index, cold start, warmup and later grading.
• B1: Native app submission to documented logger final-answer event; includes native preprocessing/retrieval/embedding/inference; excludes later title generation and automation/export.
• B2: Native app submission to documented logger final-answer event; includes native preprocessing/retrieval/embedding/inference; excludes later title generation and automation/export.
• B3: Native app submission to documented logger final-answer event; includes native preprocessing/retrieval/embedding/inference; excludes later title generation and automation/export.

{
  "configuration": {
    "machine": {
      "platform": "apple-silicon",
      "chip": "M4 Pro",
      "model": "Mac16,7",
      "cpuCores": 14,
      "gpuCores": 20,
      "memoryGiB": 24,
      "osFamily": "macOS",
      "osVersion": "26.3",
      "osBuild": "25D125"
    },
    "runtime": {
      "name": "llama.cpp",
      "version": "b11429",
      "commit": "d81235049",
      "binarySha256": "51afbf4ea6485b9ada19e99af97181f12baca42ef704b1e291c26a9ac0644eae"
    },
    "python": "3.12.14",
    "pypdf": "6.10.0",
    "model": {
      "id": "Qwen3-4B",
      "quantization": "Q4_K_M",
      "repository": "Qwen/Qwen3-4B-GGUF",
      "revision": "bc640142c66e1fdd12af0bd68f40445458f3869b",
      "file": "Qwen3-4B-Q4_K_M.gguf",
      "bytes": 2497280256,
      "sha256": "7485fe6f11af29433bc51cab58009521f205840f5b4ae3a32fa7f92e8534fdf5",
      "license": "Apache-2.0"
    }
  },
  "workflowVersion": "1.5.0",
  "runnerSha256": "d5e3c68878af3926c8844e7406a5f9f2666c133eabe624bc34836a85161f17b2",
  "generationSeed": 271828,
  "context": 8192,
  "temperature": 0,
  "maxOutputTokens": 512,
  "thinking": false,
  "cpuThreads": 10,
  "gpuOffload": "full Metal",
  "parallel": 1,
  "flashAttention": true,
  "kv": "F16",
  "retrieval": "Local BM25 pages and generic field/clause selection; source-ID selection then local sufficiency inference",
  "answerModelRequests": [
    79,
    79,
    79
  ],
  "warmupRequests": 3,
  "cachePolicy": "cache_prompt false; no claim that OS file caches were flushed.",
  "timingBoundary": "runner.ask entry through its complete returned answer, source resolution and sufficiency check; excludes prior PDF import/index, cold start, warmup and later grading.",
  "nativeEgressComparison": "unverified; no comparative privacy claim"
}

Actual pinned Kingy configuration and full-answer timing boundary

{
  "load_config": {
    "fields": [
      {
        "key": "llm.load.llama.autoFit",
        "value": false
      },
      {
        "key": "llm.load.mlx.autoFit",
        "value": false
      },
      {
        "key": "llm.load.llama.cpuThreadPoolSize",
        "value": 10
      },
      {
        "key": "llm.load.contextLength",
        "value": 8192
      },
      {
        "key": "llm.load.llama.flashAttention",
        "value": true
      },
      {
        "key": "llm.load.promptTemplate",
        "value": {
          "type": "jinja",
          "jinjaPromptTemplate": {
            "template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0].role == 'system' %}\n        {{- messages[0].content + '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0].content + '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for index in range(ns.last_query_index, -1, -1) %}\n    {%- set message = messages[index] %}\n    {%- if ns.multi_step_tool and message.role == \"user\" and not('<tool_response>' in message.content and '</tool_response>' in message.content) %}\n        {%- set ns.multi_step_tool = false %}\n        {%- set ns.last_query_index = index %}\n    {%- endif %}\n{%- endfor %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {%- set content = message.content %}\n        {%- set reasoning_content = '' %}\n        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}\n            {%- set reasoning_content = message.reasoning_content %}\n        {%- else %}\n            {%- if '</think>' in message.content %}\n                {%- set content = message.content.split('</think>')[-1].lstrip('\\n') %}\n                {%- set reasoning_content = message.content.split('</think>')[0].rstrip('\\n').split('<think>')[-1].lstrip('\\n') %}\n            {%- endif %}\n        {%- endif %}\n        {%- if loop.index0 > ns.last_query_index %}\n            {%- if loop.last or (not loop.last and reasoning_content) %}\n                {{- '<|im_start|>' + message.role + '\\n<think>\\n' + reasoning_content.strip('\\n') + '\\n</think>\\n\\n' + content.lstrip('\\n') }}\n            {%- else %}\n                {{- '<|im_start|>' + message.role + '\\n' + content }}\n            {%- endif %}\n        {%- else %}\n            {{- '<|im_start|>' + message.role + '\\n' + content }}\n        {%- endif %}\n        {%- if message.tool_calls %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if (loop.first and content) or (not loop.first) %}\n                    {{- '\\n' }}\n                {%- endif %}\n                {%- if tool_call.function %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {{- '<tool_call>\\n{\"name\": \"' }}\n                {{- tool_call.name }}\n                {{- '\", \"arguments\": ' }}\n                {%- if tool_call.arguments is string %}\n                    {{- tool_call.arguments }}\n                {%- else %}\n                    {{- tool_call.arguments | tojson }}\n                {%- endif %}\n                {{- '}\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n    {%- if enable_thinking is defined and enable_thinking is false %}\n        {{- '<think>\\n\\n</think>\\n\\n' }}\n    {%- endif %}\n{%- endif %}"
          }
        }
      },
      {
        "key": "llm.load.llama.tryDirectIO",
        "value": false
      },
      {
        "key": "llm.load.llama.acceleration.offloadRatio",
        "value": 1
      },
      {
        "key": "llm.load.numParallelSessions",
        "value": 1
      },
      {
        "key": "llm.load.seed",
        "value": {
          "checked": true,
          "value": 271828
        }
      }
    ]
  },
  "prediction_config": {
    "fields": [
      {
        "key": "llm.prediction.promptTemplate",
        "value": {
          "type": "jinja",
          "jinjaPromptTemplate": {
            "template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0].role == 'system' %}\n        {{- messages[0].content + '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {{- '<|im_start|>system\\n' + messages[0].content + '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for index in range(ns.last_query_index, -1, -1) %}\n    {%- set message = messages[index] %}\n    {%- if ns.multi_step_tool and message.role == \"user\" and not('<tool_response>' in message.content and '</tool_response>' in message.content) %}\n        {%- set ns.multi_step_tool = false %}\n        {%- set ns.last_query_index = index %}\n    {%- endif %}\n{%- endfor %}\n{%- for message in messages %}\n    {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) %}\n        {{- '<|im_start|>' + message.role + '\\n' + message.content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {%- set content = message.content %}\n        {%- set reasoning_content = '' %}\n        {%- if message.reasoning_content is defined and message.reasoning_content is not none %}\n            {%- set reasoning_content = message.reasoning_content %}\n        {%- else %}\n            {%- if '</think>' in message.content %}\n                {%- set content = message.content.split('</think>')[-1].lstrip('\\n') %}\n                {%- set reasoning_content = message.content.split('</think>')[0].rstrip('\\n').split('<think>')[-1].lstrip('\\n') %}\n            {%- endif %}\n        {%- endif %}\n        {%- if loop.index0 > ns.last_query_index %}\n            {%- if loop.last or (not loop.last and reasoning_content) %}\n                {{- '<|im_start|>' + message.role + '\\n<think>\\n' + reasoning_content.strip('\\n') + '\\n</think>\\n\\n' + content.lstrip('\\n') }}\n            {%- else %}\n                {{- '<|im_start|>' + message.role + '\\n' + content }}\n            {%- endif %}\n        {%- else %}\n            {{- '<|im_start|>' + message.role + '\\n' + content }}\n        {%- endif %}\n        {%- if message.tool_calls %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if (loop.first and content) or (not loop.first) %}\n                    {{- '\\n' }}\n                {%- endif %}\n                {%- if tool_call.function %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {{- '<tool_call>\\n{\"name\": \"' }}\n                {{- tool_call.name }}\n                {{- '\", \"arguments\": ' }}\n                {%- if tool_call.arguments is string %}\n                    {{- tool_call.arguments }}\n                {%- else %}\n                    {{- tool_call.arguments | tojson }}\n                {%- endif %}\n                {{- '}\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- message.content }}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n    {%- if enable_thinking is defined and enable_thinking is false %}\n        {{- '<think>\\n\\n</think>\\n\\n' }}\n    {%- endif %}\n{%- endif %}"
          },
          "stopStrings": []
        }
      },
      {
        "key": "llm.prediction.llama.cpuThreads",
        "value": 10
      },
      {
        "key": "llm.prediction.repeatPenalty",
        "value": {
          "checked": true,
          "value": 1
        }
      },
      {
        "key": "llm.prediction.topPSampling",
        "value": {
          "checked": true,
          "value": 1
        }
      },
      {
        "key": "llm.prediction.minPSampling",
        "value": {
          "checked": true,
          "value": 0
        }
      },
      {
        "key": "llm.prediction.temperature",
        "value": 0
      },
      {
        "key": "llm.prediction.maxPredictedTokens",
        "value": {
          "checked": true,
          "value": 512
        }
      },
      {
        "key": "llm.prediction.reasoning.enableThinking",
        "value": false
      },
      {
        "key": "llm.prediction.contextOverflowPolicy",
        "value": "stopAtLimit"
      },
      {
        "key": "llm.prediction.tools",
        "value": {
          "type": "none"
        }
      }
    ]
  },
  "plugins": [
    "lmstudio/rag-v1"
  ],
  "plugin_configs": {
    "lmstudio/rag-v1": {
      "preset": "",
      "config": {
        "fields": [
          {
            "key": "retrievalLimit",
            "value": 8
          }
        ]
      }
    }
  },
  "system_prompt": ""
}

Frozen native effective settings verified on every measured answer