Open Audio Judge ASR Leaderboard

Demo Run

First Model Set

Verified Leaderboard Results

Generated from runs/asr-leaderboard/full-35-combined/results.jsonl. The verified matrix covers 105 judged transcripts across 3 MLX ASR models and 7 research categories: acoustic_noise_robustness, entity_factual_integrity, negation_modality_scope, numeric_unit_integrity, semantic_paraphrase_preservation, temporal_scheduling_accuracy, transcription_accuracy_wer.

Category Breakdown

Total Gemini judge samples: 315. Refresh this block with .venv/bin/python scripts/refresh_asr_leaderboard_artifacts.py after rerunning the verified ASR model jobs. The combined local report is runs/asr-leaderboard/full-35-combined/report.html and the committed summary artifact is docs/asr-leaderboard-summary.json. The generated refresh report is docs/asr-leaderboard-refresh-report.md, and the generated shell playbook is docs/asr-leaderboard-refresh-commands.sh. The committed run manifest is docs/asr-leaderboard-run-manifest.json, with coverage validation in docs/asr-leaderboard-manifest-validation.json and seed-manifest validation in docs/asr-seed-manifest-validation.json. The next-refresh plan is docs/asr-leaderboard-next-runs.json, and the hosted artifact manifest is docs/asr-leaderboard-hosted-manifest.json. The artifact bundle index is docs/asr-leaderboard-artifacts.json. Runtime readiness is tracked in docs/asr-leaderboard-runtime-status.json; together they include the source result files, complete model/category matrix, missing-cell guidance, hosted copy map, and reproducible refresh workflow. Pass ASR_LEADERBOARD_HOSTED_DIR with --hosted-dir-from-env to copy the same verified artifacts into the hosted Pages checkout. Use docs/asr-leaderboard-report-index.md as the generated map from the demo page to the combined full-35 report and per-source run reports; use docs/asr-leaderboard-report-links.json for the same map in machine-readable form.

Generated Refresh Workflow

These commands are generated from the same workflow metadata written to docs/asr-leaderboard-summary.json and docs/asr-leaderboard-refresh-report.md.

Generated Model Refresh Commands

Load the Gemini secret only in the local shell before running live judge calls: source /Users/wangyauli/.openclaw/secrets/open-audio-judge-gemini.env.

If a primary MLX ASR model is unsupported locally, record that blocked state in the run notes before trying the documented fallbacks: mlx-community/whisper-small.en-asr-4bit, mlx-community/parakeet-rnnt-0.6b, mlx-community/GLM-ASR-Nano-2512-4bit.

Generated Artifacts

Source Run Reports

Each source run keeps its own local report alongside the JSONL file that feeds the combined 35-case leaderboard.

Eval Categories

Command Flow

source /Users/wangyauli/.openclaw/secrets/open-audio-judge-gemini.env

oaj autojudge-mlx-asr \
  --python-bin .venv/bin/python \
  --cases runs/asr-research-audio/tts_audio_cases.jsonl \
  --model mlx-community/whisper-large-v3-turbo-asr-fp16 \
  --judge-provider gemini \
  --judge-samples 3 \
  --out runs/asr-leaderboard/whisper-large-v3-turbo-full-gap

oaj autojudge-mlx-asr \
  --python-bin .venv/bin/python \
  --cases runs/asr-research-audio/tts_audio_cases.jsonl \
  --model mlx-community/Qwen3-ASR-1.7B-8bit \
  --judge-provider gemini \
  --judge-samples 3 \
  --out runs/asr-leaderboard/qwen3-asr-1.7b-full-gap

oaj autojudge-mlx-asr \
  --python-bin .venv/bin/python \
  --cases runs/asr-research-audio/tts_audio_cases.jsonl \
  --model mlx-community/VibeVoice-ASR-4bit \
  --judge-provider gemini \
  --judge-samples 3 \
  --out runs/asr-leaderboard/vibevoice-asr-full-gap

.venv/bin/python scripts/refresh_asr_leaderboard_artifacts.py

Expected Output Files

Representative Result JSON

{
  "case_id": "asr-numeric-transfer-001",
  "provider": "gemini",
  "overall_score": 42,
  "meaning_preservation": "major_loss",
  "error_categories": ["number_error", "contrast_error"],
  "metadata": {
    "candidate_model": "mlx-community/whisper-large-v3-turbo-asr-fp16",
    "eval_category": "numeric_unit_integrity",
    "asr_slice": "amount_contrast",
    "judge_sample_count": 3
  }
}

Research Notes

See docs/asr-eval-taxonomy.md for the category rationale and examples/asr_research_cases.jsonl for the seed manifest. These leaderboard results are verified local artifacts from all seven categories, with private/generated audio kept out of the repository.