TTS Leaderboard

How To Run It

Run each compatible TTS wrapper on the same case file and write each run under a distinct output directory. Source GEMINI_API_KEY at runtime; do not commit it to the repository.

oaj autojudge-local-tts \
  --model mlx-community/chatterbox-turbo-6bit \
  --cases examples/tts_multiturn_cases.jsonl \
  --judge-provider gemini \
  --judge-samples 3 \
  --limit 440 \
  --out runs/leaderboard/chatterbox

oaj autojudge-local-tts \
  --model mlx-community/Kokoro-82M-4bit \
  --synthesis-provider mlx_community_kokoro \
  --cases examples/tts_multiturn_cases.jsonl \
  --judge-provider gemini \
  --judge-samples 3 \
  --limit 440 \
  --out runs/leaderboard/kokoro

oaj autojudge-local-tts \
  --model mlx-community/Qwen3-TTS-12Hz-0.6B-Base-4bit \
  --synthesis-provider mlx_community_qwen3_tts \
  --cases examples/tts_multiturn_cases.jsonl \
  --judge-provider gemini \
  --judge-samples 3 \
  --limit 440 \
  --out runs/leaderboard/qwen3-tts

If the local console script is stale while developing from source, run the same commands through the module entry point:

PYTHONPATH=src:. .venv/bin/python -m open_audio_judge.cli autojudge-local-tts \
  --model mlx-community/chatterbox-turbo-6bit \
  --cases examples/tts_multiturn_cases.jsonl \
  --judge-provider gemini \
  --judge-samples 3 \
  --limit 440 \
  --out runs/leaderboard/chatterbox

Once each model run has written judge-report/results.jsonl, combine those result files into one side-by-side report:

oaj report \
  --results runs/leaderboard/chatterbox/judge-report/results.jsonl \
  --results runs/leaderboard/kokoro/judge-report/results.jsonl \
  --results runs/leaderboard/qwen3-tts/judge-report/results.jsonl \
  --baseline-model mlx-community/chatterbox-turbo-6bit \
  --out runs/leaderboard/comparison-report

Expected Output Files

Illustrative Report Preview

The preview samples 300 cases per model from the 440-case evaluation library: 900 model-case evaluations, with three judge attempts per evaluation producing 2,700 illustrative judgments.

Eval Set Samples

The public demo eval set contains 440 synthetic, public-safe cases grouped into research-backed TTS dimensions. Each category below contributes five shared cases that every model synthesizes before Gemini judges the generated audio.

Illustrative Model Ranking

Top Diagnostics

Selected Category Comparison

Baseline Model Deltas

Weakest Segments

Model-Category Action Matrix

Sample-By-Sample Breakdown

The real combined report contains one row per synthesized model/case pair. This preview shows representative rows across models, categories, slices, and score bands from 900 illustrative model-case evaluations.

Representative Result JSON

{
  "case_id": "tts-storytelling-bedtime-001-local-tts",
  "task": "tts_naturalness",
  "provider": "gemini",
  "overall_score": 95,
  "label": "accurate",
  "semantic_error_summary": "The audio is highly natural and fully faithful to the target text.",
  "error_categories": ["no_error"],
  "researcher_notes": [
    "The voice sounds highly natural with clear pronunciation and no artifacts."
  ],
  "metadata": {
    "synthesis_provider": "local_chatterbox",
    "synthesis_model": "mlx-community/chatterbox-turbo-6bit",
    "synthesis_voice": "af_heart",
    "synthesis_lang_code": "en",
    "eval_category": "storytelling_dialogue",
    "sample_kind": "local_synthetic_tts",
    "judge_sample_count": 3,
    "judge_sample_scores": [94, 96, 95],
    "judge_sample_average": 95.0
  }
}

How To Read The Report

Start with average score and label counts, then inspect score breakdowns by the model leaderboard, then inspect category-level rankings to find where one generator loses to another on the same eval set. Use the Weakest Segments cards to find low-scoring model/category/slice groups, then use the Model-Category Action Matrix to route the lowest model/category pairs to likely solution areas, taxonomy focus areas, source-basis signals, and representative samples. Use the top diagnostics to confirm repeated issue clusters such as pronunciation errors, artifact classes, or weak slices. Finally, filter the per-sample rows by label, status, model, category, or TTS slice, search for an issue or case id, sort by score, and inspect the averaged score, the three underlying Gemini scores, target text hash, generated-audio provenance, and any researcher notes.

This page is a committed illustrative artifact, not verified benchmark results. Real run outputs should stay under ignored runs/ directories unless the audio, text, and metadata are explicitly safe to publish.