Open Audio Judge TTS Leaderboard

How To Run It

Run each compatible TTS wrapper on the same case file and write each run under a distinct output directory. Source GEMINI_API_KEY at runtime; do not commit it to the repository.

oaj autojudge-local-tts \
  --model mlx-community/chatterbox-turbo-6bit \
  --cases examples/tts_multiturn_cases.jsonl \
  --judge-provider gemini \
  --judge-samples 3 \
  --limit 265 \
  --out runs/leaderboard/chatterbox

oaj autojudge-local-tts \
  --model mlx-community/Kokoro-82M-4bit \
  --synthesis-provider mlx_community_kokoro \
  --cases examples/tts_multiturn_cases.jsonl \
  --judge-provider gemini \
  --judge-samples 3 \
  --limit 265 \
  --out runs/leaderboard/kokoro

oaj autojudge-local-tts \
  --model mlx-community/Qwen3-TTS-12Hz-0.6B-Base-4bit \
  --synthesis-provider mlx_community_qwen3_tts \
  --cases examples/tts_multiturn_cases.jsonl \
  --judge-provider gemini \
  --judge-samples 3 \
  --limit 265 \
  --out runs/leaderboard/qwen3-tts

If the local console script is stale while developing from source, run the same commands through the module entry point:

PYTHONPATH=src:. .venv/bin/python -m open_audio_judge.cli autojudge-local-tts \
  --model mlx-community/chatterbox-turbo-6bit \
  --cases examples/tts_multiturn_cases.jsonl \
  --judge-provider gemini \
  --judge-samples 3 \
  --limit 265 \
  --out runs/leaderboard/chatterbox

Once each model run has written judge-report/results.jsonl, combine those result files into one side-by-side report:

oaj report \
  --results runs/leaderboard/chatterbox/judge-report/results.jsonl \
  --results runs/leaderboard/kokoro/judge-report/results.jsonl \
  --results runs/leaderboard/qwen3-tts/judge-report/results.jsonl \
  --baseline-model mlx-community/chatterbox-turbo-6bit \
  --out runs/leaderboard/comparison-report

Expected Output Files

Sample Report Preview

Eval Set Samples

The public demo eval set contains 265 synthetic, public-safe cases grouped into research-backed TTS dimensions. Each category below contributes five shared cases that every model synthesizes before Gemini judges the generated audio.

Model Leaderboard

Top Diagnostics

Category Leaderboard

Baseline Model Deltas

Weakest Segments

Model-Category Action Matrix

Sample-By-Sample Breakdown

The real combined report contains one row per synthesized model/case pair. This preview shows representative rows across models, categories, slices, and score bands from the 795 generated samples.

Representative Result JSON

{
  "case_id": "tts-storytelling-bedtime-001-local-tts",
  "task": "tts_naturalness",
  "provider": "gemini",
  "overall_score": 95,
  "label": "accurate",
  "semantic_error_summary": "The audio is highly natural and fully faithful to the target text.",
  "error_categories": ["no_error"],
  "researcher_notes": [
    "The voice sounds highly natural with clear pronunciation and no artifacts."
  ],
  "metadata": {
    "synthesis_provider": "local_chatterbox",
    "synthesis_model": "mlx-community/chatterbox-turbo-6bit",
    "synthesis_voice": "af_heart",
    "synthesis_lang_code": "en",
    "eval_category": "storytelling_dialogue",
    "sample_kind": "local_synthetic_tts",
    "judge_sample_count": 3,
    "judge_sample_scores": [94, 96, 95],
    "judge_sample_average": 95.0
  }
}

How To Read The Report

Start with average score and label counts, then inspect score breakdowns by the model leaderboard, then inspect category-level rankings to find where one generator loses to another on the same eval set. Use the Weakest Segments cards to find low-scoring model/category/slice groups, then use the Model-Category Action Matrix to route the lowest model/category pairs to likely solution areas, taxonomy focus areas, source-basis signals, and representative samples. Use the top diagnostics to confirm repeated issue clusters such as pronunciation errors, artifact classes, or weak slices. Finally, filter the per-sample rows by label, status, model, category, or TTS slice, search for an issue or case id, sort by score, and inspect the averaged score, the three underlying Gemini scores, target text hash, generated-audio provenance, and any researcher notes.

This page is a committed sample artifact. Real run outputs should stay under ignored runs/ directories unless the audio, text, and metadata are explicitly safe to publish.