Contents

ADR-0009: Voices and lip sync as provider chains

  • Status: Accepted (2026-09-29)
  • Date: 2026-09-29

Context

  • Rhubarb Lip Sync. v1.14.0 (April 2025) is the latest release and the project is mostly dormant. Its macOS release is Intel-only, and it crashes under Rosetta on this Mac. There is no Linux arm64 build. It builds and runs natively from source on Apple Silicon: CMake, Boost headers only, target rhubarb only. It accepts WAV (8–32-bit, 16–96 kHz) and Ogg, and takes the transcript with -d, which improves accuracy.
  • macOS say. System voices are licensed for personal, non-commercial use only, so they cannot voice shareable demo content.
  • Local TTS. Kokoro-82M (code, weights and G2P all Apache-2.0) is the best permissive local option and returns word timings. Caveat: its training data includes synthetic audio from closed models.
  • Cloud TTS. Several cloud voices return alignment data: phoneme timings from Cartesia and Hume, visemes from Azure, per-character timings from ElevenLabs.

Decision

  • TTS chain, per show: recorded (approved takes, pinned by hash) → configured cloud providers → kokoro (local, via a uv subprocess). Takes are cached by hash(text, voice binding, delivery, provider+version) and stored as FLAC.
  • Lip-sync chain: rhubarb (built from source by tools/rhubarb/install.sh, pinned to v1.14.0, run with -d <transcript> --extendedShapes GHX -f json) → later an alignment provider that maps TTS or forced-alignment phoneme timings onto the same A–H/X shapes. The engine sees only the resolved viseme track, never the provider.
  • Demo dialogue for M1 is generated once with Kokoro, committed as FLAC under CC BY 4.0, with provenance recording the model, voice and settings.

Consequences

  • No step in the M1 demo depends on a network service or a paid key.
  • Rhubarb is replaceable without touching the Timeline format.
  • The toolchain grows by one optional Python environment. It is only needed to regenerate demo audio or voice new lines locally.