# Kingy.ai independent test: Meta Muse Voice Transcribe 1.0

**Test date:** September 1, 2026  
**Scope:** 30 unique minutes of public audio; 55 billable Muse audio-minutes across transcription, diarization, endpointing, and one language-bias repeat  
**Comparator:** OpenAI Whisper large-v3-turbo Q5_0 via whisper.cpp, CPU-only with Silero VAD 6.2.0  
**Successful Muse spend:** **US$0.165 calculated list price; CA$0.22 posted by Meta**  
**Paid-API hard cap:** **US$10.00**

## Bottom line

Muse beat the local Whisper baseline on every English slice in this small test. Across 25 minutes of clean read speech and a four-speaker mixed-headset meeting, Muse scored **11.93% WER** versus **15.34%** for Whisper: a **3.41-point absolute** and **22.2% relative** reduction in word errors.

The result reversed on one five-minute Hindi-English technical tutorial. Muse's output rendered most English software terms phonetically in Devanagari, while the reference retained them in Latin script. The resulting mixed-script WER-style score was **54.21%** for Muse versus **30.84%** for Whisper. Adding `languageBias: ["Hindi", "English"]` did not help: Muse scored **54.39%**. This is evidence of an output-convention mismatch on one clip, not proof that Muse is generally worse at Hindi or code-switching.

Muse's separate diarization and endpointing modes worked, but our proxy scores were not strong enough to treat their output as authoritative. The 15-minute AMI diarization pass produced **37.31% collar-free DER** under our disclosed, overlap-included scorer. A paced five-minute endpointing pass produced a first valid partial **573 ms after the relevant speech-start event**, but its simple activity matcher counted **9 false endpoint turns and 8 missed reference regions**. Both are small, protocol-sensitive measurements and are not comparable to Meta's provider chart or Artificial Analysis.

## Headline results

| Test slice | Duration | Muse | Whisper baseline | Lower is better |
|---|---:|---:|---:|---|
| LibriSpeech test-clean | 10 min | **1.09% WER** | 1.87% WER | Muse |
| AMI ES2004a, minutes 0–5 | 5 min | **15.17% WER** | 22.01% WER | Muse |
| AMI ES2004a, minutes 5–10 | 5 min | **24.90% WER** | 29.12% WER | Muse |
| AMI ES2004a, minutes 10–15 | 5 min | **17.09% WER** | 22.29% WER | Muse |
| **English aggregate** | **25 min** | **11.93% WER** | **15.34% WER** | **Muse** |
| MUCS Hindi-English | 5 min | 54.21% WER-style | **30.84% WER-style** | Whisper |
| MUCS, Muse with Hindi + English bias | 5 min extra | 54.39% WER-style | — | Bias was 0.19 points worse |

The English aggregate contains 3,840 reference words. Muse made 458 edit errors: 87 substitutions, 353 deletions, and 18 insertions. Whisper made 589: 131 substitutions, 437 deletions, and 21 insertions.

## Corpus and licensing

The corpus was frozen before the first billable request. It contains exactly 1,800 seconds of unique public audio:

| Source | Fixture | Audio | License/use |
|---|---|---:|---|
| [LibriSpeech test-clean](https://www.openslr.org/12) | Concatenated read-English excerpts | 10 min | CC BY 4.0 |
| [AMI Meeting Corpus](https://groups.inf.ed.ac.uk/ami/corpus/) | ES2004a Mix-Headset, three consecutive segments | 15 min | CC BY 4.0 |
| [MUCS 2021](https://www.openslr.org/104/) | One Hindi-English LibreOffice tutorial | 5 min | CC BY-SA 4.0 |

AMI **Mix-Headset** is a mixed close-microphone channel. This independent test did not test AMI's distant-microphone channel and must not be described as a far-field evaluation.

All fixture hashes and source provenance are in the artifact manifest. Because these are public datasets, they may have appeared in model training. That possibility weakens any claim about novel generalization.

## Systems and configurations

### Meta Muse

- Model: `muse-voice-transcribe-1.0`
- Batch transcription: `PUSH_TO_TALK` on all 30 unique minutes
- Diarization: separate `DIARIZATION` pass on the 15 AMI minutes
- Endpointing: separate realtime `ENDPOINTING` pass on AMI minutes 0–5, sent at the documented 80 ms cadence
- Language-bias check: a second MUCS batch pass with Hindi and English bias
- No automatic retries

### Local Whisper comparator

- Model: OpenAI Whisper large-v3-turbo, Q5_0 quantization
- Runtime: whisper.cpp
- Execution: CPU-only
- VAD: Silero 6.2.0
- Paid cost: $0

The Whisper model SHA-256 was `394221709cd5ad1f40c46e6031ca61bce88931e6e088c188294c6d5a55ffa7e2`; the VAD model SHA-256 was `2aa269b785eeb53a82983a20501ddf7c1d9c48e33ab63a41391ac6c9f7fb6987`.

This is not a like-for-like cloud comparison. Whisper was quantized, local, offline, CPU-bound, and had no native endpointing or diarization pass. It is an available zero-cost baseline, not a claim about the best competing commercial API.

## A post-hoc comparator correction

The installed whisper.cpp Metal path crashed with signal 11 during the first full-corpus attempt. CPU mode passed a smoke test. A CPU run without VAD then repeated the phrase “I'll leave space for everyone else” through a long quiet AMI region. We preserved those raw files but excluded that configuration from the headline comparison.

The scored Whisper configuration therefore uses Silero VAD. That decision was made after observing a failure and may favor Whisper. On the clean LibriSpeech fixture, WER moved from 1.80% without VAD to 1.87% with VAD, showing that even an apparently small configuration change can move a score.

## Scoring method

For the transcript tests, the scorer applies Unicode NFKC normalization, lowercases text, removes punctuation, collapses whitespace, retains fillers, then computes standard token edit distance:

`WER = (substitutions + deletions + insertions) / reference words`

The Hindi-English score uses the same token operation and is deliberately labelled **WER-style**. Tokenization and script choice make it a poor language-complete measure. We did not freeze a named-entity list before running the providers, so we did not add a post-hoc entity score.

## Diarization proxy

| AMI segment | Muse proxy DER | Muse proxy JER | Reference speakers | Hypothesis speakers |
|---|---:|---:|---:|---:|
| Minutes 0–5 | 46.48% | 58.76% | 4 | 4 |
| Minutes 5–10 | 44.55% | 49.12% | 4 | 4 |
| Minutes 10–15 | 25.25% | 30.07% | 4 | 4 |
| **Aggregate/macro** | **37.31% DER** | **45.98% macro JER** | — | — |

The custom scorer uses 10 ms frames, a 0 ms collar, overlap included, and an optimal one-to-one anonymous-speaker mapping. It derives reference activity from AMI annotations. Meta's launch chart reported 17.5% average DER over AMI-IHM, AMI-SDM, and VoxConverse, but Meta did not publish enough protocol detail to reproduce that number. Our 37.31% result is not a reproduction and should not be compared numerically as if it were.

## Realtime endpointing proxy

The five-minute realtime run emitted 4,190 events, including 349 partial transcripts, 30 predicted turns, and 30 speech-complete events.

| Measure | Result |
|---|---:|
| First valid partial after its speech-start event | 573 ms |
| Cumulative partial churn | 3.49 edits per 100 prior tokens |
| Predicted turns | 30 |
| Reference activity regions | 29 |
| False endpoint matches | 9 |
| Missed reference regions | 8 |
| Matched start-boundary MAE | 1,015 ms |
| Matched end-boundary MAE | 3,306 ms |
| Last completion after final audio byte | 20 ms |

There were about 20 seconds of initial silence, so the first partial arrived 21.98 seconds after socket opening; that is not model latency. The meaningful within-turn observation is 573 ms after the relevant speech-start event. Muse also emitted an empty false turn during initial silence.

Endpoint scoring uses AMI segment-derived activity and a simple greedy, non-reuse overlap matcher. It is a diagnostic proxy, not a standard endpointing benchmark. Network geography, one connection, and one meeting segment make these numbers high-uncertainty.

## Runtime and cost

Muse's five primary batch calls processed 1,800 seconds of audio in about 265 seconds of wall time: batch real-time factor **0.147**, or about **6.8× faster than realtime**. The full local Whisper transcription took 731 seconds, for RTF **0.406**. These figures are not a fair speed race: Muse ran on unknown Meta cloud hardware over a network; Whisper ran CPU-only on one local Apple Silicon machine.

Successful Muse processing totaled 55 audio-minutes. At $0.003 per minute, calculated list-price cost is **US$0.165**. After the run, Meta's account usage page recorded **3.3k audio-transcription seconds and CA$0.22 spend** for the period—consistent with 55 minutes after currency conversion and display rounding. The billing overview still showed CA$0.00 currently due, so that page had not yet rolled the usage into a payable balance.

The spend ledger conservatively reserved **US$0.195** because it includes the first 10-minute request, which failed with `billing_not_configured`. Meta's documentation says requests that fail before a transcript are not billed. The raw HTTP 402 response is retained.

The harness refuses any request whose conservative reservation would push the ledger above **US$10.00**. No subscription was purchased and no second paid competitor was used.

## What the test supports

- Muse was materially more accurate than this Whisper configuration on all four English slices.
- Muse handled easy read English exceptionally well in this sample.
- The AMI meeting remained much harder than read speech for both systems.
- Muse's script/transliteration convention was a bad fit for exact mixed-script fidelity on this MUCS clip.
- The documented language-bias hint did not fix that mismatch.
- Native speaker and turn events are useful product primitives, but our small proxy tests do not justify treating them as verified identity or authoritative boundaries.

## What the test does not support

- It does not prove Muse is the world's best ASR system.
- It does not reproduce Artificial Analysis or Meta's diarization chart.
- It does not test accents, telephony, distant microphones, heavy noise, music, clinical speech, legal speech, children, or broad multilingual coverage.
- It does not establish statistical significance or day-to-day service variance.
- It does not compare current paid streaming competitors.
- It does not measure confidence scores, word timestamps, punctuation, or production availability.
- It does not establish that Meta did not train on these public datasets.

## Reproducibility

The downloadable audit package contains the frozen protocol, corpus manifest and hashes, reference text/RTTM, runners, scorer, attempt log, model ledger, raw Meta responses and realtime events, raw Whisper outputs, failure response, spend ledger, and computed metrics. It excludes API credentials, downloaded model binaries, and the source audio files.

This report documents exactly what Kingy.ai ran. It makes no claim of broader hands-on use outside the preserved test.