# KACI Wave One limitations

These findings concern the completed designed panel. Verification and pilot responses are excluded from the study results.

- These are official API responses. Consumer ChatGPT, Claude and Gemini interfaces can use different prompts, models, tools and routing.
- The English panel is designed, not a census or probability sample of real questions. The four strata have predetermined weights.
- Two observations per question measure limited run-to-run variation. They do not capture all randomness, personalization, geography or changes over time.
- Current-development questions depend on the actual collection date. A single collection window cannot describe lasting citation patterns.
- Native automatic search choice can yield no search or no citations. Citation annotations describe the provider's attribution, not an independent check that a source supports every claim.
- Output and tool limits can truncate demanding answers. Incomplete and failed responses require separate reporting.
- Citation occurrences can repeat a URL or domain within an answer. Occurrence share and response-presence share use different denominators and must not be interchanged.
- Redirect links may conceal an original domain until resolved. Unresolved destinations must be shown rather than assigned an invented origin.
- Registrable-domain aggregation merges subdomains and uses the ICANN suffix convention; it can conceal differences among hosted publications.
- YouTube metadata is time-sensitive. Captions available does not establish caption accuracy, content quality or whether the engine watched a video.
- Rule-based domain categories are imperfect; uncertain labels and review coverage must accompany category comparisons.
- The ten-question operational pilot provides a limited cost estimate. Usage-rated accounting uses documented prices, excludes tax, and can differ from invoices due to free allowances, discounts or provider billing rules.
- July comparability is limited to 220 reused questions and to changes in model and search surfaces since then.
- Kingy is both the researcher and a potentially cited publisher. Its sites and channel must be identified transparently and ranked by the same mechanical rules as other publishers.

- OpenAI Flex and Claude Batches change scheduling. Native tools and models are preserved, but Batch can perform more server-loop iterations before pause_turn. Queue latency and asynchronously changing search indexes can affect comparisons even inside a common 72-hour window. Request and archive timestamps include this uncertainty.

- Collection includes a process interruption, OpenAI Flex capacity backoff and a Google project spending-cap pause. These operational delays remain inside the original collection window, and different providers can finish the same question group at different times. A common window does not eliminate temporal confounding from changing search indexes.

### Provider response identifiers

In the collected Gemini Interactions archives, the provider returned neither an interaction response ID nor an HTTP request ID. Claude batch results include native message IDs, but no individual HTTP request IDs. The study retains immutable local call IDs, request hashes, raw archives and exact provider/question/run slots; available OpenAI response/request IDs and Claude message IDs remain intact. Missing Google identifiers limit provider-side reconciliation and independent server-side duplicate detection. The local ledger and archives show no duplicate completed slots; no missing provider ID is invented.

### Observed timing and missing metadata

The queue-inclusive collection span was 13.814 hours. Matched provider answers could be archived as much as 348.7 minutes apart following the Google cap pause; the median gaps were 4.3 minutes for OpenAI/Claude, 1.1 for OpenAI/Gemini and 5.7 for Claude/Gemini. These are local archive-time differences, not exact provider execution-time differences. Changing search indexes and the interruption therefore remain possible temporal confounders. Detailed pairs are in [timing-comparability.json](timing-comparability.json).

Public metadata was retrieved after collection for all 239 unique cited YouTube videos on October 5, 2026 UTC (October 4 Pacific). No video metadata was unavailable at retrieval. Channel attribution uses the returned channel IDs; the Kingy AI channel appears in one Gemini answer and no OpenAI or Claude answers in this collection. Views and other mutable fields are retrieval-time snapshots, not collection-time values. Caption availability does not establish transcript quality, factual support or whether a provider watched a video. Curtis reviewed the selected top 100 of 4,631 collection domains; the other 4,531 remain provisional. Category comparisons describe this selected subset, with all unreviewed citations pooled separately. Labels describe source type, not accuracy or quality. Native annotation styles differ by provider, so repeated-link occurrence counts can reflect annotation formatting as well as source choice.

## Top-100 human category review

Curtis reviewed the selected top 100 of 4,631 collection domains; the other 4,531 remain provisional. Category comparisons describe this selected subset, with all unreviewed citations pooled separately. Labels describe source type, not accuracy or quality.

The reviewed subset accounts for OpenAI 5,998 / 11,738, Claude 4,294 / 18,327 and Gemini 7,511 / 15,734 native citation occurrences. Selection used pooled citation frequency, not random sampling. The same domain-level conventions apply across providers. Whole-company labels include company blogs; shared hosting can contain many authors; academic does not guarantee peer review. Review approval does not validate factual support.
