Kingy AI testing methodology · Planned pilot
A cheap generation is not cheap if the product changes shape, the label falls apart, the interface becomes unreadable, or three more attempts are needed before a clip can be used. This protocol measures the result a product team actually needs: an acceptable product-demo clip, its observed generation cost, and the human time required to obtain it.
Current status: this is the pre-registered method for a planned Kingy AI pilot. Four tools, three product-demo jobs and four attempts per job would produce 48 primary attempted renders. No tool has been selected and no result, ranking or winner is claimed on this page.
The question
For a creator or product marketer who needs a short vertical product-demo clip, which eligible AI video tool produces the highest rate of acceptable outputs, at what observed generation cost and human effort?
The test is intentionally narrower than “Which AI video generator is best?” It does not establish a universal model ranking. It evaluates selected public product surfaces, under dated conditions, on three frozen assets and briefs.
Three real product-demo jobs
1. Packshot integrity
A studio product image must become a restrained landing-page hero clip. The product has to retain its colour, shape, cap, wordmark and identifying mark throughout the usable portion of the video. Extra products, people, invented labels and scene changes are prohibited.
2. Lifestyle product ad
A contextual still must become a subtle paid-social product moment. The product remains the clear focal point while light, camera and environmental movement stay controlled. The test penalizes identity drift, distracting additions and product geometry that changes between frames.
3. App-interface motion
A fictional mobile analytics screen must become a short launch-page demonstration. The app name, required values and layout have to remain stable and legible while a restrained interface animation occurs. Invented controls, altered numbers and unstable text are failures.
The source assets will be fictional, non-confidential and free of real people. Each file and exact prompt will be frozen with a cryptographic checksum before the first scored attempt. The source images must first pass a baseline review confirming that every required element is legible at the intended viewing size.
How the comparison stays fair
- One declared product route per tool. The cohort will use the cheapest public paid mode that satisfies the common output requirements. That selection rule is locked before results are seen.
- One common output specification. Duration, aspect ratio and minimum native resolution are set at the highest specification all four admitted tools can produce comparably.
- One semantic brief. The requested outcome and restrictions remain the same. Syntax-only adaptations required by a tool are frozen and recorded before scored generation begins.
- No repaired entries. Reviewers score the complete native generated clip. The scored version is not trimmed, upscaled, interpolated, colour-corrected, composited or repaired.
- Every submitted job remains in the record. Completed clips, policy blocks, queue failures and technical failures are retained. If a tool returns several variants from one job, every variant and its share of the charge are recorded.
- Order is randomized. Tool and brief order is distributed across blocks so one service is not tested only at a particular time or after the operator has learned from every other tool.
- Stability is checked separately. Five tool-and-brief combinations are selected in advance with a recorded random seed for one additional run. These retests do not overwrite the primary cohort.
What counts as an acceptable clip
Two reviewers independently assess anonymized renders on a 0–4 anchored scale. The review is identity-blind rather than perfectly tool-blind: generator signatures may occasionally make a product recognizable, so reviewers also record whether they believe they can identify the tool.
| Dimension | What it asks |
|---|---|
| Identity fidelity | Did the product or interface remain the same object? |
| Required information | Did labels, values and required visual elements remain present and legible? |
| Brief adherence | Did the requested movement and composition occur without prohibited additions? |
| Technical integrity | Are flicker, deformation, corruption and other artifacts within an acceptable range? |
| Product-demo usability | Could an editor use the clip for the defined landing-page or paid-social job without material repair? |
A render passes only when both reviewers score every dimension at least 3, each reviewer’s five-dimension average reaches 3.4, and neither reviewer identifies a critical failure. Critical failures include a materially changed product, illegible required identity, altered required values, an invented third-party brand, a prohibited person, a watermark or corrupt output.
Reviewer disagreement is preserved. An adjudicator can issue a separate final decision with a written reason, but cannot rewrite the original scores.
Reliability and quality are reported separately
A visually strong clip does not erase repeated service failures. A reliable queue does not make a weak result useful. The pilot therefore reports three different measures:
- Completion rate: completed outputs divided by valid submitted attempts.
- Acceptance among completed outputs: accepted clips divided by completed outputs.
- End-to-end acceptance: accepted clips divided by all valid submitted attempts.
Operator mistakes and duplicate submissions are recorded as protocol deviations rather than tool failures. Queue limits, recovery rules and valid exclusions are locked before the cohort opens.
How Kingy calculates real cost
Pricing pages explain what a vendor charges. They do not reveal how many attempts a particular product job may require. Kingy keeps four cost concepts separate:
- Direct generation cost: actual API charges or the value of credits consumed by valid attempts.
- Required access cost: a plan or export fee purchased solely to conduct the test.
- Existing-plan burden: a pre-existing subscription used during the pilot, disclosed separately rather than silently called free.
- Human time: active operating and independent-review minutes, reported separately from tool charges.
The primary observed calculation is:
Generation cost per acceptable clip = attributable generation charges ÷ accepted outputs
If a tool produces no acceptable output, its result is “not calculable: no accepted output,” accompanied by attempted spend and attempt count. It is never shown as zero cost. Required access charges and consumed-credit values are kept in non-overlapping cost views so a subscription is not counted twice.
A client can optionally apply an hourly labour assumption to the measured human minutes. That scenario is labelled as a planning estimate, not an observed tool price.
Which tools can enter the cohort
A candidate must support the same image-to-video job, meet the common native output specification, expose a usable public price basis, allow per-attempt accounting, permit local retention of the output, be accessible in the test region and have terms suitable for the fictional product assets and prospective editorial display.
Private models, vendor-managed prompts and undisclosed credits unavailable to ordinary buyers create an unequal advantage and are excluded from this cohort. Supplied access or another commercial relationship must be disclosed and cannot change the prompts, scoring or verdict.
What the eventual evidence record will show
- Test date, region, product surface, model label and access route
- Frozen input and prompt versions
- All valid attempts, returned variants and retained failures
- Native duration, resolution and relevant generation settings
- Independent scores, timestamped defect notes and adjudications
- Completion, acceptance and end-to-end acceptance rates
- Observed charges, access-cost treatment and measured human minutes
- Relevant pricing, terms and product documentation reviewed at the cutoff date
- Commercial relationships, exclusions, limitations and corrections
Limits of the pilot
Four attempts per job are enough to expose useful failures and estimate production friction, but not enough to establish permanent statistical superiority. Results remain descriptive and specific to the selected assets, tools, modes, region and dates. The pilot does not measure conversion, ad-platform approval, legal clearance for a client’s own assets, or every type of AI video work.
Any future headline and verdict must preserve those boundaries. A product update or material protocol change creates a new dated test version rather than silently rewriting the original record.
For AI companies and product teams
A credible product demo makes its input, action, output and limits visible. Kingy’s testing and distribution work is built around that evidence boundary. Product access, sponsorship or a supplied briefing can enable evaluation; it does not purchase a favourable result.
Protocol status: public methodology for the planned product-demo video pilot. Source and eligibility checks occur before any cohort is selected. See also the Kingy AI research methodology and source policy.